MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Authors: Dongzhi Jiang
Renrui Zhang
Ziyu Guo
Yanwei Li
Yu Qi
Xinyan Chen
Liuhui Wang
Jianhan Jin
Claire Guo
Shen Yan
Bo Zhang
Chaoyou Fu
Peng Gao
Hongsheng Li
Venue: ICML 2025 poster (2025)
Keywords: Evaluation and Benchmark
MLLM
Reasoning
Primary area: deep_learning->everything_else
TL;DR: A Comprehensive Evaluation Suite for Multimodal Reasoning

Abstract: Answering questions with Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), yet its impact on Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth investigation. In this paper, we introduce **MME-CoT**, a specialized benchmark evaluating the CoT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. 
As the first comprehensive study in this area, we propose a thorough evaluation suite incorporating three novel metrics that assess the reasoning quality, robustness, and efficiency at a fine-grained level.
Leveraging curated high-quality data and a unique evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs, uncovering several key insights: *1)* Models with reflection mechanism demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and demonstrating the highest quality results; *2)* CoT prompting often degrades LMM performance on perception-heavy tasks, suggesting a potentially harmful overthinking behavior; and *3)* Although the CoT quality is high, LMMs with reflection exhibit significant inefficiency in both normal response and self-correction phases.
We hope MME-CoT serves as a foundation for advancing multimodal reasoning in LMMs.

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Dongzhi Jiang" !
Renrui Zhang I1 Ziyu Guo Yanwei Li Yu Qi
Xinyan Chen 31
Liuhui Wang"
Jianhan Jin 16
Claire Guo :7
Shen Yan Bo Zhang
Chaoyou Fu
Peng Gao 8 Hongsheng Li 189
Abstract
Precision
Answering questions with Chain-of-Thought Thought (CoT) has significantly enhanced the reasoning ca- pabilities of Large Language Models (LLMs), yet its impact on Large Multimodal Models (LMMs) still lacks a systematic assessment and in de-tepth investigation. In this paper, we introduce MMEE CoT, specialized benchmark evaluating the CoTT reasoning performance of LMMs, spanning six domains: math, science, OCR, logic, space-time, and general scenes. As the first comprehensive study in this area, we propose a thorough evalu- ation suite incorporating three novel metrics that assess the reasoning quality, robustness, and effi- ciency fine-grained level. Leveraging curated high-quality data and unique evaluation strategy, we conduct in-depth analysis of state-of-the- art LMMs, uncovering several key insights: Z) Models with reflection mechanism demonstrate a superior CoT quality, with Kimi k1.5 outperform- ing GPT-4o and demonstrating the highest quality results; 2) CoT prompting often degrades LMM performance perception-heayy tasks, suggest- ing potentially harmful overthinking behavior; and 3) Although the CoT quality is high, LMMs with reflection exhibit significant inefficiency in both normal response and self-correction phases. We hope MME-CoT serves as a foundation for advancing multimodal reasoning in LMMs.
92.0
Quality
85.4
Reflection Quality
100.0
80.2
505
512
Recall
49.3
79.5
49.2
77.3
722
61.7
41.
46.5
83.7
-31
2.0
90.6
-10
922
2.9
Relevance
92.9
Stability
Rate
Robustness
Efficacy
Qwen2 V1,72B
GPT-4o
Vago-72B
InternV1225-78-MMPO
Kimi k1.5
QVQ-72B
Figure I. Chain-of-
Chain-of-Thought Thought Performance of Leading LMMs in MME-CoT, Our evaluation assesses LMMs using three novel metrics that yield six distinct scores. Results reveal that current open-source models, including those reflection capabilities, still lag behind closed-source models like GPT-4o and Kimi k1.5 in key aspects of chain-of of-thutght reasoning.
promising advances in reasoning capabilities, exemplified by the recent OpenAl ol (OpenAI, 2024a) and DeepSeek- R1 (Guo et al., 2025a). By engaging in a more deliberate, stepwise reasoning process before reaching a final answer, this methodology presents an effective solution in tackling complex scenarios (Lu al., 2023; Guo al., 2025c; Jiang et al., 2025; Chen al., 2025).
1. Introduction
The emergence of Chain-of- Chain-of-Thought Thought (CoT) (Wei et al., 2022) in Large Language Models (L.MMs) has demonstrated
"Core Contribution 'rroject Lead 'Equal Contribution !CUHK MMLab *CUHK MiuLar Lab ByteDance 'NEU 'Ueen 'NJU "CUHKSZ 'hhanhhii Al Laboratory "CPII under InnoHK. Corre- spondence to: Hongsheng wwwli@ec.cuhke........ edu.............................
In parallel, the multimodal extensions of LLMss termed Large Multimodal Models (LMMs), have demonstrated re- markable proficiency across diverse visual domains, e.g., general image recognition (Zhang et al., 2023; Zhu et al., 2023; OpenAI, 2023; Zhang al., 2024a), temporal video
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MMB-CoT: Benchmarking Chain-of-Thouup in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
CoT Evaluation Suite
Quality
Robustness
Efficiency
Precision
Perception Task No Nerd for Rensrwang
Reasoning Task
many are presented
Nead Rosening
Model Output
Judgment
Relevance Rate
Step Judgment
ABCD 1s square (ABCD)
Relevant
4. [Shadow] I-BCD)-
Irrelevant
[ABII-18E5]- 314
Quortton Vhass aravect dhoddyy
Recall
Reflection Quality
Reflection Judgment
Key Step Annotation
Judgment
(Poud Azseect
(Dhen Asswe)
Alternatively masbe
Stability
Efficacy
cant.. But sime that helpe dscble-check
Ievalid
[ABCD] 625
Match
V
Unaiion Plase tirk wey step.
[Shanina] 455
Prediction
WEEh new perspect
(LMI Acaver)
JOOT Airme)
Dataset Overview
Math
General Scenes
Space-
Key Step Annotation
Wast enetes repmoordin the piemne? At maa zwe
Question:
Key Step
PR
Key Caption:
Caption: T.eeellely
Key Step Annotation
Question:
brstge boke
water surfiee
Key Conclasion: rdita
Aluudd A is
The disance
Answer: B
2. The scaweris
Key
Nore provided
Bi thme five
Key Captiun:
tmage
The vidia A/ti
Answer:
toral
L Logic
OCR
Science
Question:
Question:
Your tukk
Key Step
does
Key Stop Amnotation
to snleet the correc
Captions
Answer:
Question: contacttng suane boldiego singeddurgol Which shons the dlecro....
Key Capcion:
Key Step Annotation
[labdedd Ao
Conclusion: The faidd colamn tee contiistioon..
Key Conchasion: dlettic ticd... loxa outiile..
Key Cuncuuson:
sy empry brs...
Option
Oanoca 5. and conatten mokerute...
Answer: C
Answer: D
Figure 2. An Overview of MME-CoT, Our benchmark contains comprehensive CoT evaluation suite with three novel aspects and meticulously curated dataset encompassing six categories.
understanding (Li et al., 2023; Chen al., 2023), and 3D geometry perception (Guo al., 2024; Xu et al., 2023; Guo al., 2023; Jia et al., 2024). However, to what tent and how much CoT reasoning can benefit multimodal challenges still remains an open question. Although some previous efforts (Zhang et al., 2024c; Yu et al., 2023; Zhang et al., 2024d: Guo et al., 2025d) have been made to evaluate the CoT capabilities of LMMs, their examination insuffi ciently systematic and thorough, limiting our understanding of multimodal reasoning and its further development.
soning skills within LMMs (Figure Our benchmark spans six fundamental domains: math, science, OCR, logic, space-time, and general scenes, encompassing broad range of CoT-relevant scenarios. Unlike the simplistic metrics used in previous studies, MME-CoT introduces a rigorous evaluation framework that delves into the fine-grain mnn-grrieed CoT process of LMMs, assessing reasoning quality, robustness, and efficiency. Specifically, we address three critical re- search questions follows:
To bridge this gap, propose MME-CoT, comprehen- sive and specialized benchmark for evaluating the CoT rea-
each intermediate CoT step logically valid and faithful without hallucination? The outcome-oriented
2

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
evaluation paradigm, where most current benchmark adapts, omits the scenario where the model reaches the correct answer through flawed logic or random guess. This causes an illusion of inflated reasoning capabili- ties in the model. To delve into the reasoning process, we introduce two interpretable metrics to evaluate the Quality of CoT: 1) Recall, which quantifies reason- ing informativeness by measuring the proportion of tround-truth solution steps appearing in the response; 2) Precision, which measures faithfulness by evaluat- ing how many of the generated steps are accurate.
we notice that not steps within the long CoT are related to answering the question, and the model could be distracted by the image content, especially when handling general scenes, space-time, and OCR tasks. Around 30% to 40% of reflection steps fail to help answer questions, pointing out critical issues of current models' reflection capabilities.
The contributions of this paper are summarized as follows:
The MME-CoT benchmark is curated, covering a com- prehensive scope of six multimodal reasoning scenar- ios. The data collection and annotation process under- goes rigorous human verification, aiming to provide the community with a high-quality evaluation dataset for multimodal reasoning.
2. Does CoT interfere with perception tasks, and to what extent does it enhance reasoning tasks? While exist- ing studies primarily focus on the performance im- provements CoT brings to reasoning tasks, they often overlook whether CoT could inadvertently disrupt the model's ability to solve perception tasks that require minimal reasoning. To this end, we present the first in- vestigation into the Robustness of CoT in LMMs. Our benchmark incorporates two task categories (percep- tion and reasoning), and employs two distinct prompt- ing strategies ('direct answer' and 'step-by-step') to as- sess two metrics: /) Stability, which examines whether CoT negatively impacts the model's performance on direct perception tasks; 2) Efficacy, which measures the extent which CoT enhances the model's perfor- mance on complex reasoning tasks.
We identify critical issues in existing benchmarks, and introduce a thorough evaluation suite specialized for multimodal CoT reasoning, which meticulously exam- ines the reasoning quality, robustness, and efficiency.
We conduct extensive experiments and analysis on state-of-the-art LMMs with reasoning capabilities. We summarize our observations and insights, hoping to inspire future advancements of reasoning performance.
Our project page is at https://mmecot: wwr.ooottiiiii git.... io/
3. How can we assess the efficiency of CoT in a long reasoning process? Recent ol-like models have dis- tinguished themselves by employing excessively long CoT and reflection steps. This raises a critical trade-off question: does this approach strike an optimal balance between accuracy and computational cost? To investi- gate this, we present the first study on the Efficiency of CoT in LMMs. We evaluate efficiency using two key metrics: 1) Relevance Rate, which assesses the proportion of generated content that contributes to an- swering the question. 2) Reflection Quality, which analyzes whether each reflection step drives the ques- tion towards correctness.
2. Dataset Curation
2.1. Data Composition and Categorization.
MME-CoT composes 6 major domains with 17 subcate- gories, as visualized in Fig. 3. Different from textual reason- ing questions, the extra visual input significantly enriches the scope of the visual reasoning questions. With the image input, the model needs to frequently visit the image for rel- evant information according to current reasoning progress. Describing the image area of interest becomes a crucial part of the CoT process. Thus, in addition to complex prob- lems demanding rigorous logic, commonsense scenarios also pose challenging reasoning problem, as shown in the general scenes in Fig. 2. To maintain focus on the rea- soning process. we exclude questions that require complex domain-specific theorems or specialized knowledge.
Through our systematic evaluation and analysis, we discover that the fine-grained reflection capability greatly enhances the CoT quality, e.g., QVQ achieves F1 Score of 62.0%, largely surpassing Uwwe22-VL-2BB by 6.8%. Kimi k1.5 beats GPT-4o and achieves the best quality. As for the robustness, we surprisingly find that most models are in- terfered with by CoT on the perception tasks, implying harmful overthinking behavior. The worst case happens in Intern VL2.5-8B, where we witness a 6.8% degradation when applying CoT on the perception tasks. This signifi- cantly impedes the applicability of models using CoT rea- soning as a default practice. Moreover, for CoT efficiency,
In addition, to evaluate CoT robustness detailed in Sec- tion 3.2, we incorporate a variety of perception tasks along with the reasoning tasks in the benchmark. The reasoning tasks contain questions that demand multi-step logical in- ference, while the perception tasks consist of questions that primarily test visual recognition abilities or require very minimal reasoning. Existing benchmarks often conflate these two types of tasks, with perception and reasoning questions frequently appearing within the same categories.
3

---

MME-CoT: Benchmarking Chain-of-Thone in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Statistic
Number
Dosi
Chemmi
5.
Toral questions
1,130
Diggnes 1.e
6.%%
Physics
16.4%
Details 21.8%
Spatial
26.3%
Reasoning questions
Multiple-choicc questions
Pree-fomm questions
(74.18)
431
106
Perception questions
293
(25.99% (9))
Multiplo-choice questions
Doc 11.2%
Scricnie 25.2% 29
General Scenas 41.0%
Space- Time 33.5%
Free-fomm questions
OCR 16.4%
Walkogs 5.7%
Toral key step annocation
3,865
Total inference conclusions
Tropene
Spanial Sap
Math 29.9%
Canatove 30%
Alabrs 4.9%
Common
Sense 19, 69,
OCR 24.9%
Temporai 7.2%
Total image captions
Average inference conclusions
Average captions
1,198
1.4
more Geontey
Doe 7
Reference image caption ieem
SettCrevee Sne
Average reference caption
18.9%
Diagram 17.1%
Number of unique images
2,380
Number unique questions
unique nnswers
Masimum question length
477
Maximum anrwer length
Reasoning Task
Perception Task
Average question length
Average answer length
1.2
Figure 3. Category and Subcategory Distribution of MME-CoT.
Table Key Statistics of MME-COT.
To address this, we implement two-stage classification ap- proach combining both model-based and human assessment. Initially, we leverage LMMs to guide the preliminary cate- gorization by comparing their performance with and without CoT prompting. We employ GPT-4o (OpenAI, 2024b) and Qwen2-VL VL (Wang et al., 2024b) to answer questions us- ing both direct and COT' approaches. Superior performance with CoT indicates reasoning-dominet subcategory, while comparable inferior CoT performance suggests either perception-focused content or insufficient model reasoning capabilities. The results are shown in Appendix C.2. Sub- sequently, expert annotators review individual questions to finalize their classification. In total, MME-CoT contains 1,130 questions with 3,865 key step annotation. The detailed statisties of data compositions shown in Table 1. Please refer to Appendix C for more details about the distribution of data sources.
multiple solution paths, annotators required to provide all possible methods. Reference image captions capture visual information not covered by the image captions in key steps, mainly for precision calculation. We use the same two-phase method to obtain these annotations, with annotators verifying and correcting the details.
3. CoT Evaluation Strategy
Existing benchmarks only focus on evaluating the final an- swer of the questions, leaving the whole chain of thoughts unvisited. We argue that the CoT process reflects reason- ing capability from multiple aspects, serving as a crucial medium to understand LMM's thinking pattern and defi- ciency. Here, present the first holistic CoT evaluation suite to facilitate comprehensive understanding of the LMMs' reasoning abilities. We detail the evaluation of cor- rectness in Section 3.1, stability and efficacy in Section 3.2, and reflection quality in Section 3.3.
2.2. Data Annotation and Review
To facilitate CoT evaluation, we provide key steps anno- tation and reference image captions for all the reasoning questions. Key steps are defined as necessary steps for an- swering the question correctly, falling into two categories: (1) inference conclusions necessary conclusions reached through logical inference steps (including the final answer), and (2) image captions identifications of critical visual information. For efficient annotation, we implement two- phase process. First, GPT-4o generates initial versions of key steps annotations with questions, images, and ground truth answers as inputs, which yields more accurate ratio- nales compared to question-only prompting. Second, human annotators review these initial versions, correcting any er- rors or developing key steps independently when GPT-4o fails to provide reasonable output. All steps reduced to the simplest form, retaining only core conclusions and relevant visual element descriptions. For problems with
3.1. CoT Quality Evaluation
Existing methods typically rely on state-of-the-arr LLMs or LMMs to directly evaluate Shain-of-Thoughee reasoning based on self-defined criteria, using only the final answer as a reference (Hao al., 2024; Zhang et al., 2024c). We identify two primary issues with the strategy. First, the scor- ing process only attends to the logical validity of each step, omitting the helpfulness evaluation. Second, there is large number of complex visual reasoning questions that even the scoring model cannot solve. It is unreasonable for the scor- ing model to judge another model's 's reasoning process on these questions without knowing the ground truth solution process. Therefore, building upon our annotated key steps and reference image captions, we leverage interpretable metrics to evaluate the CoT correctness: recall and preci- sion (Figure 4). The two metrics respectively attend to the

---

MMB-CoT: Benchmarking Chain-of-Thoughe in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Question
Key Step Annotation
Square C has side length 22. Points and H on Lovertine Rj so that H G Points and lie outside square A B C so that E.F Hisa square. Compute area hexagon
1. GH is equal to 12.
2. The area of ARCI 484,
The area ERGH
BCD.
The area AEH and 30.
5. The area of AEPB equals the sum arens
ABCD, EFGH, AEH, and BFG.
Model Response
The area 688.
Eval Prompt
Step Partition
Judge the correctness
Determine if each key step appear in the prediction.
Precision Prompt
Recall Prompt
Eval Result
X
Step1 [logical inference]
hexagon AEFBCD be a square ABCD, a square EFGHt, and four triangles les AHE, 86f, CGF, and DHE,
1. equal to 12, [Match]
X area ABCD 484, [Missing]
Step2 [logical, inference] The side length of EFGH is 12.
3. The area of EFGH 144. [Match]
Step3 [logical inference] The area of square EFGH is 144 (12 x 12).
X 4. The area of AEH and BEG 30. [Conflict]
Step4 [background mation] Area triangle (base height) +
X 5. The area of AEFB equals the sum of areas of ABCD,
x
Step5 [logical inference] triangle (ME, CGF, DHE)
of
EFGH, AEH, BFG. [Conflict]
Step6 [logical inference)
X 6. The of AEFB 688. [Conflict]
x
hexagon AEFBCD 216.
Precision 2/5
Recall 2/6
Figure 4. Illustration of CoT Quality Evaluation. For recall, prompt GPT-44 match each key step annotation in the prediction. For precision, GPT-4o is instructed to split the prediction into steps and determine the correctness of all the image caption and logical inference steps.
two aspects of the CoT correctness: informativeness and accuracy. We denote the key steps as = UUZ I, where C {(I,,..,,.. } includes M key inference conclusions and I = (i1,..., includes N key image captions.
by the model. From another perspective, this metric also strictly examines the process's rigorousness toward reaching the correct answer, eliminating the probability of random guessing. For questions with multiple methods, we compute the recall on the most matched method.
Recall. We prompt GPT-4o (Hurst al., 2024) to deter- mine whether each key step occurs in the model's CoT response. Then we calculate the ratio of the matched key steps Smatched against all the annotated key steps:
We first instruct GPT-4o to partition the predic- tion into sequence of steps P, as shown in Fig. 7 in the Appendix. Each step is categorized into one of three classes: logical inference, image caption, and background informa- tion. The logical inference step draws an intermediate or final conclusion based on previously obtained informa- tion. The image caption step depicts elements of interest in (2) the image. The background information step states exter- nal knowledge question information. Visual reasoning can be primarily characterized as interleaved sequence of image captions and logical inferences, so we focus on measuring precision for these two key step types. We assess the correctness of logical inference steps (CP) and image caption steps (IP) using two criteria: If the step exists
Precision.
Sk matched Sk1
kn arg max
pko Chatched |Cho|
Ikk Inanchhed
Recalle
Recally
Iho
skk
mutched
Recall
Sko
where & denotes the th method of the problem. Intuitively, recall measures how many informative steps are reached
5

---

MMB-CoT: Benchmarking Chain-of-Though in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Task Type
Response
Perception Task
Reasoning Task
So cretre can sort
glass as that sealls lo aed are degrets alde, thene aeglas Me and rtrst, wed to deseppee the diagran basd the tapel sears low occaide the circlte
was
dives th 06, tid deprens, That the noher
way vinen centery monnt the cxretey they ore opuat leephh.
Question Sphins type of environmenn?
The changes
Relevance Rate
point particle
GT Aaswer. Rainforest
GT Answer Futh
Eval
Step Partition
Direct Answer
(Egiical
answer directly.
Judge the relevance.
Reievance
X
(Srage Captiot
Rainforest
Rate Prompt
Definition:
deurgatio beight
Marp Droprcat intereee
Stability
Efficacy Reasoning Tasks?
Compore
Compare
Doss CoT interfore Perception
Reflection Quality
livl losk
Reflection Step
Refracuue staps dacees triangle
Please answer step by step
Judge the validity.
Col Fratge
Reflection Quality
angles formed Fly liee it is tao mont teattieeee
Prtlactue stopd
The Sphinx is...
...
Prompi
Tip:
answer
will nove
Reflection Indication::
long Path
Alumratiods
dealle-hhed
Fartactuse Stup3
Extract Final Answer
@istrret Final Anewer
Validday Difinittinn:
sectionnt tangert
iss nerfin
Desert
Path
Figure 5. Illustration of CoT Robustness Evaluation, We pare the performance of applying CoT prompt and direct prompt on two types of tasks: perception and reasoning. The stability score measures whether CoT interferes with perception, while the efficacy score assesses the performance gain CoT on reasoning tasks.
Figure 6. Illustration of CoT Efficiency Evaluation. For rele- vance rate, we partition the prediction into steps and determine if is relevant by GPT-4o. For reflection quality, we prompt GPT--4o to identify the reflection steps by common indicators and judge the validity the reflection. The definitions relevance and validity are included.
in S. the step is correct. 2. If the step is logically correct or faithfully depicts the image based the annotations, the step is also correct. Thus, we compute precision as:
great importance. In real-world applications, what task is given is unknown in advance. Whether the model should perform CoT to solve the task is difficult to determine. In fact, there exists no golden standard to determine which question can benefit from CoT so far (Sprague et al., 2024). Instead of trying to define this criterion, we examine the performance of CoT across all kinds of tasks, both reason- ing and perception. We argue that ideal CoT process should assist in reasoning and not interfere with pure per- ception. Therefore, it can be applied for any tasks. Based on this, we propose to evaluate two metrics of CoT: stability and efficacy (Figure 5). We leverage two kinds of prompts: the direct prompt (DIR) and the CoT prompt (COT). The direct prompt asks the model directly provide the final an- swer, while the CoT prompt instructs the model to perform step-by-stey reasoning and finally give the answer. To di- rectly compare the performance difference caused by these two prompts, we conduct the direct evaluation, which only judges the correctness of the final answer, i.e., accuracy. We instruct GPT-4o mini (Hurst et al., 2024) to extract the final answer, and then compare it with the ground truth answer,
C ICPl
I PTP
Precisionc
Precision
(4)
Comeaa UIP
Precision
5
CC IIP
Intuitively, precision evaluates the faithfulness of each step, considering all the possible reasoning output. Finally, we calculate the F1 score as the metric of CoT quality.
3.2. CoT Robustness Evaluation
Here, perform the first investigation the robustness of CoT in visual reasoning- The effectiveness of CoT on reasoning tasks has been verified in many works (Wei et al., 2022; OpenAI, 2024a). However, how CoT impacts visual perception tasks or tasks requiring minimal reasoning still remains unknown. Despite the neglect, this question bears
6

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
following the two-step procedure introduced in (Zhang et al.,, 2024c).
where 0 corresponds to the overall relevance rate, and take 0.8.
Stability. We define the performance difference of the two prompts on the perception tasks P as the stability score:
Reflection Quality. The superior reasoning ability could be largely attributed to the reflection and verification pro- cess. However, our analysis reveals that not all reflective steps contribute meaningfully to finding correct answers. We identify distinet failure patterns in the reflection process. Some reflective steps mislead the reasoning by introduc- ing new errors or incorrect assumptions, while others are redundant, simply echoing previous conclusions without contributing new insights. To account for failure reflection scenarios, we propose to measure the validity of the reflec- tion. We define a valid reflection as either correctly pointing out the previous mistakes or verifying the previous conclu- sion with a new insight. Otherwise, the reflection only slows down the reasoning. To instruct GPT-4o determine all the valid reflection steps R, we list set of common indicators of the start of the reflection, such as "Wait" and "Alterna- tively", and illustrate the definition of valid reflection. For all the valid reflection steps Realid. the reflection quality is computed as:
Stability = Acceor ACc
Intuitively, applying the CoT prompt to perception tasks should not degrade performance compared with the direct prompt. Thus, model with stable CoT should be not less than O. Otherwise, the model's thinking process demon- strates inconsistency and harm. The overthinking process pushes over the original correct judgment.
Efficacy. Similarly, the performance difference of the two prompts on the reasoning tasks R is defined as the score:
Efficacy = Acceor Acc
(7)
Intuitively, CoT facilitates stepwise thinking and therefore benefits answering reasoning tasks. The difference reflects how much CoT can enhance reasoning.
3.3. CoT Efficiency Evaluation
|Rvatia [
Reflection Quality
(1I)
Models like ol generate extremely long thinking processes with reflection and verification of current steps and out- comes. We perform the first exhaustive analysis of the CoT efficiency of visual reasoning with two carefully designed metrics (Figure 6):
4. Experiments
In this section, conduct systematic evaluation of state- of-the-art models on MME-COT. We first detail the experi- ment setup in Section 4.1. Then in Section 4.2, we report the quantitative results and provide valuable insights derived from our analysis.
Relevance Rate. Although the long reasoning content allows for deeper thinking, it may also introduce a large amount of irrelevant information. As shown in the bottom left of Fig. 6, the model has identified the critical element in the image for answering the question, but it still gener- ates detailed description of other objects. This irrelevant information provides no helpful information to work out the answer. In the meantime, this extra content slows down the generation speed. Similar to the calculation of precision, we employ the same method to partition the prediction into steps. Then, we instruct GPT-4o to determine all the rel- evant steps Pretevant- The step is considered relevant only when the majority of its content works towards solving the question. We first compute the raw relevance rate and then apply scaling factor to amplify the differences between models. Let rx denote the raw relevance rate:
4.1. Experiment Setup
Evaluation Models. We select top-performing LMMs for comprehensive CoT evaluation. We test earlier mod- els such as VLaVV-OneeVision (7B, 72B) (Li et al., 2024a), Qwen2-VL (7B, 72B) (Qwen Team, 2024), MiniCPM-V- 2.6 (Yao et al., 2024b), and Intern VL2.5 (8B) (Chen et al., 2024c), which not trained for the reasoning capabil- ity. We also include GPT-4o (OpenAI, 2024b) as strong baseline model. Besides, we test recent models targeting reasoning, including LLaVA-CoT (1IB) (Xu et al., 2024), Mulberry (8B) (Yao et al., 2024a), I1tem (8B, 78B) (Wang et al., 2024c). Finally, we evaluate LMMs with reflection capabilities, including both closed-source models like Kimi k1.5 (Team et al., 2025) and open-source implementations such as QVQ-72B (Team, 2024) and Virgo- 72B (Du et al., 2025).
c Geleeant
Irelevan
re- =
ICPl
PI
IPI
(8)
Prosevant
(9)
PPI
Then, the final relevance rate Relevance Rate, is defined as:
Implementation Details. We define the CoT prompt as: Please generate a step-by-step answer, include all your in- termediate reasoning process, and provide the final answer
Relevance Rate,- = Ta a 1- a
C.I.0 (10)
7

---

MME-CoT: Benchmarking Chain-of- in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Table Evaluation Results of Three Aspects of CoT MME-COT. Models with receive detailed image captions instead of images.
We mark the highest score of each metric in red denotes unreliable results due the refusal answer directly most of the time.
CoT Quality
Col Robustness
Col Efficincyy
Model
Precision Image Concusion Recall lonage Concluiiu
Suiiiitt
CUT Porepption
Durect Proception
Eilieacy
CUT Reseeninng
Direct Recwering
Avg. Savee
Relevaice Raic
Iaage Conclasion
Reflection Quality
Opeisonnnee Lafhy
Mulleny
27.4
99.1
53.8
17.
26.5
17.3
2.0*
35.7
29.94
8.5*
10.4
89.5
79.0
50.8
95.4
100
30.9
50.9
47.3
43.5
22.2
24.4
23.2
-3.4
+3.8
46.1
49.8
-3.0
1644
19.4
96.7
93.4
85.3
93.4
100
LLaVA-CoT
34.9
53.9
75.6
46.2
25.8
35.8
24.4
51.5
24.4
99.4
98.8
85.2
95.4
100
DeepSeek VL2
$4.4
46.3
25.7
33.4
26.0
5.1
65.9
1.6
29.0
96.7
93.4
90.8
99.0
100
57.3
50.5
26.6
27.4
61.1
60.8
0066
28.2
99.1
97.7
100
MMniCMM-V-2..
39.8
57.3
63.4
45.4
30.5
47.5
26.7
-3.3
-4.5
59.6
64.0
2.3
26.1
28,3
97.8
85.7
74.6
97.6
100
InteraVI.2 2,5-88
60.0
52.4
50.8
31.3
40.4
10.6
63.8
0.9
90
29.5
98.4
96.8
93.0
9
100
Qwes2-VL--B
61.6
49.3
46.6
30.5
62.6
4.6
30.4
94.9
98.8
100
IntcmaVI.2.5888MM0
43.0
60.4
60.5
49.9
35.4
44.9
31.8
066
0.3
62.5
62.1
0.8
30.1
29.3
94.7
89.3
84.0
96.4
100
2iteraVV22.575805 MPO
$2.7
73.6
68.4
63.0
41.1
53.6
29.1
0.2
2.0
64.3
70.3
2.4
38.0
25.6
99.6
99.2
95.1
98.0
100
Qwen2-V0 VIL-220
77.3
90.3
42.2
68.9
175.2
2.3
36.6
96.5
92.9
98.7
100
Virgo
79.5
71.6
12.7
49.2
60.5
47.7
2.6"
2.2
73.7*
75.9*
3.1"
49.5
75.0
90.6
79.8
95.6
60.6
QVQ-720 -288
62.0
80.
73.9
77.5
80.5
60.1
9
70.0
41.0
74.0
83.7
63.9
95.1
61.7
Closed source LMM:
Clasde-3.5
59.4
77.2
65.1
31.6
46.2
59.7
47.6
10.5
11.0
74.1
63.1
9.9
41.5
31.6
95.4
90.9
79.5
99.0
100
Gemini- F.
63.8
80.3
60.2
74.5
.2.9
56.7
53.6
6.3
59
78.4
72.5
6.6
47.5
40.9
97.7
95.5
91.4
98.4
100
85.4
$1.4
512
49.9
1.0
72.0
35.5
96.0
92.0
82.4
99.
100
Kimi k1.5
64.2
92.0
[8.1
89.8
49.3
62.9
47.9
14
29
65.7
62.9
0.0
40.0
40.0
82.2
92.2
82.2
97.2
72.2
at the end. and the direct prompt as: Please directly provide the final answer without any other output. We only calculate recall of image observation and logical inference on ques- tions where key inference conclusion or image observation exists. We employ GPT-4o mini for the direct evaluation and GPT-4o for all other criteria. For hyperparameters, we follow the settings in VLMEvalKit (Duan et al., 2024).
demonstrates the highest scores in precision evaluations. For CoT robustness. Mulberry obtains the highest average score. However, when we look into its output, we find it still generates lengthy rationales despite receiving a direct prompt. Even worse, the direct prompt seems to be an out-o--distribution input for Mulberry, frequently leading to nonsensical outputs. Further analysis of other models' predictions reveals that LLaVA-CoT, Virgo, QVQ, and Kimi k1.5 similarly neglect the direct prompt, instead generating extended rationales before answering. Consequently, their robustness scores may be misleading. Once again, GPT-4o achieves the highest robustness score. Among open-source models, only Intern -PO in both its 8B and 78B variants, attains a positive robustness score. Finally, for CoT efficiency, VL2.5-VL2..888 obtains the maximum relevance of 98.4%, 4%, suggesting its consistent focus on questions.
4.2. Quantitative Results
We conduct extensive experiments on various LMMs with our proposed CoT evaluation suite. The main results are presented in Table 2 and Table 3. We begin by analyzing the overall performance and then highlight key findings.
Overall Results. In Table 2, we present the overall perfor- mance of three CoT evaluation perspectives with specific metrics. To provide a comprehensive understanding, we report precision, recall, and relevance for both logical infer- ence and image caption steps. For robustness, we provide the direct evaluation result on the perception and reasoning tasks, with either CoT or direct prompt. We employ the average value of the stability and efficacy as the final robust- ness metric. Notably, we define the reflection quality as 100 on models incapable of reflection.
Now, we summarize our key observations as follows:
Models with reflection largely benefit CoT quality. As shown in Table 2, the Fl scores of the two models with reflection capability most closely approach GPT-4o. After specifically fine-tuning for the reasoning capabilities from Qwen2-VL-72B, QVQ surpasses its base model by 5.8%. Notably, although QVQ generates longer CoT sequences than Qwen2-VL-72BB QVQ's precision still exceeds Qwen2- VL-72B by 2.9%, indicating superior accuracy in each rea- soning step. Kimi kl.5 also surpasses the previous state-of- the-art model GPT-4o, obtaining the highest CoT quality.
For CoT quality, Kimi k1.5 achieves the highest Fl score. Open-source models with larger sizes consistently demon- strate better performance, highlighting the scalability of LMMs. Notably, Qwen2-VI--2BB outperforms all other open-source models without reflection, even surpassing Ww...LL..-777--MPO which is specifically enhanced for reasoning. Analysis reveals that GPT-4o achieves supe- rior performance across all recall metrics, while Kimi k1.5
Long CoT does not necessarily cover key steps. Despite high precision in long CoT models, the informativeness of each step is not guaranteed. We observe that the recall trend
8

---

MME-CoT: Benchmarking Chain-of- in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Table Evaluation Results of Three Aspects of CoT across Categories in MME-COT. CoT. Models with receive detailed image captions
instead of images. Best performance is marked in red. denotes unreliable results due to the refusal answer directly most of the time.
General Scenes
Space-Time
OCR
Math
Science
Logic
Model
Quality Robustness Efficinccy Quality Robustness Efficiency Quality Robustness Efficiency Quality Elficiency Quality Efficiency Quality Efficiency
Open-source LMMs
Mulberry
33.9
4.3
88.0
18.2
1,0*
69.2
26.7
6.6
63.2
29.1
93.9
29.1
96.1
13.9
99.5
LLaVA-OV-7B
41.8
-6.2
90.9
23.8
-6.7
62.4
44.1
-0.2
71.4
27.4
98.7
28.5
97.9
12.2
99.0
LLaVA-CoT
38.2
94.9
33.6
84.5
37.4
88.9
35.3
95.5
36.4
96.7
14.9
98.6
DeepSeek-VLL VL2
43.9
-11.5
99.8
98.1
LLaVA-OV-72B
41.8
2.3
99.5
29.0
-0.9
71.8
40.8
-1.7
92.0
38.4
99.4
35.4
97.8
18.4
91.1
Vi2.CPM-V-2.6
47.1
3.2
93.9
49.3
14.4
85.5
63.7
+.9
81.0
32.9
97.6
29.5
95.2
16.9
96.8
I22.mVL2...8B
43.8
-6.4
93.6
50.7
-8.9
99.5
44.7
99.4
40.9
95.0
40.8
98.6
19.5
98.4
Qwea2-VL-78
46.7
-3.4
89.6
$1.7
-11.8
86.5
93.1
18.4
88.4
IMS--VL2.588-MMP
47.2
2.9
97.1
51.8
-0.2
87.3
59.6
-1.0
90.7
37.4
96.7
39.0
97.8
20.9
89.9
Intem 11577LL25-788MMP
47,9
0.0
94.6
55,5
-2.3
95.9
72.2
2.2
86.6
50.6
97.6
48.5
98.9
24.2
93.6
Qwan2-VL-72B
51,9
-2.9
94.4
59.7
-5.3
93,4
77.6
2,5
90,9
49.6
98.9
53.6
99.5
40.0
94,0
Virgo-72B
60.5
0.5%
-3.8*
65.1
80.2
81.0
59.3
QVQ-72B
62.6
70.1
58.2
51.4
76.9
60.7
61.4
81.1
57.7
83.2
44.6
61.1
Closed source LMMs
C,5
63.8
7.1
95.4
51.7
11.4
83,4
65.8
10.1
91.1
53.0
98.9
59.6
98.6
41.7
88.4
(emmiai-2.0-Flan
55.5
7,2
56.1
7.3
95.7
96.6
65.1
97.0
99.1
99.8
GPT4o
62.3
-1.7
98.1
66.3
5.5
82.3
83.3
-1.0
91.0
60.8
98.4
64.1
99.7
27.2
96.0
Kini k1.5
73.8
14.0
84.1
69.3
-3.3
58.9
83,3
7,5
83.6
58.3
90.8
48.8
85.9
41.3
60.3
among GPT-4o, QVQ, and Virgo does not align with their CoT Rea. performance (i.e., their final answer accuracy the reasoning tasks under the CoT prompt). Specifically, while both Virgo and QVQ outperform GPT-4o in direct evaluation, they lag behind in recall. This suggests that long CoT models sometimes reach correct answers while skipping intermediate steps, which contradicts the principle of stepwise reasoning and warrants further investigation.
Reflection often fails to help. While reflection is key feature of long CoT models for answer verification, both QVQ and Virgo achieve reflection quality scores of only about 60%, indicating that approximately 40% of reflection attempts fail to contribute meaningfully to answer accuracy. Even for the dlosed-soure model Kimi k1.5, over 25% re- flection steps are also invalid. This substantial failure rate compromises efficiency by potentially introducing unneces- sary or distracting steps before reaching correct solutions. Future research should explore methods to reduce these in- effective reflections to improve both efficiency and quality.
COT impairs perception task performance in most mod- els. Surprisingly, most models exhibit negative stability scores, indicating that CoT interferes with perception tasks. The most significant degradation occurs in InterVVL..5 8B, where performance drops by 6.8%. This reveals in- consistency and potential overthinking in current models, presenting a significant barrier to adopting CoT as the de- fault answering strategy. Among models that provide direct answers, only LLaVA-OV-72B and Intern achieve modest positive score of 0.3%.
5. Conclusion
In this paper, we have introduced MME-CoT, comprehen- sive benchmark designed to evaluate Chain-of- Thought rea- soning in Large Multimodal Models. Our dataset comprises six categories to cover most scenarios of visual reasoning tasks. To gain a thorough understanding of the reasoning process, we design a novel CoT evaluation suite with three metries. Our systematic evaluation obtains useful insights into the issues within the current ofate-of-the-art Large Mul- timodal Models. We identify critical flaws in all the tested open-source models. As the field continues to evolve, MME- CoT stands as a valuable tool for measuring progress and identifying areas for improvement in the development of more sophisticated multimodal AI systems.
More parameters enable models to grasp reasoning bet- ter. We find that models with larger parameter counts tend to achieve higher efficacy scores. This pattern is evident across VA-OV, InternVL2.5-MPP and Qwen2-VL. For instance, while Qwen2- VL-7B shows 4.8% decrease in performance when applying CoT to reasoning tasks, its larger counterpart, Qwen2-VL-72B, demonstrates a 2.4% improvement. This discrepancy suggests that models with more parameters could better grasp the reasoning ability under the same training paradigm.
9

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Acknowledgements
multi-modality models. In Proceedings of the 32nd ACM International Conference Multimedia, pp- 11198 11201, 2024.
This project is supported in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, and in part by NSFC-RGC Project N. .CUHK498/24. Hongsheng Li is Pl of CPII under the InnoHK.
Gao, P., Zhang, R., Liu, C., Qiu, L., Huang, S., Lin, W. Zhao, S., Geng, S., Lin, Z., Jin, P., et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. ICML 2024, 2024.
Impact Statement
Golovneva, O., Chen, M., Poff, S., Corredor, M., Zettle- moyer, L., Fazel-Zarandi, M., and Celikyilmaz, A. Roscoe: A suite of metrics for scoring step-by-step rea- 2022.
This paper presents work whose goal advance the field of Computer Vision and Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
soning, arXiv preprint arXiv:
Guo, D., Yang, D., Zhang, H.,, Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501, 12948, 2025a.
References
Chen, D., Chen, R., Zhang, S., Wang, Y., Liu, Y., Zhou, H., Zhang, Q. Wan, Y., Zhou, P., and Sun, L. Mllm- as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning. 2024a.
Guo, Z., Zhang, R., Zhu, X., Tang, Y., Ma. X., Han, J., Chen, K., Gao, P., Li, X., Li, H., et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.0061 10611,, 2023.
Chen, G., Zheng. Y.-D., Wang, J., Xu, J., Huang. Y., Pan, J.. Wang, Y., Wang, Y., Qiao, Y., Lu, T., et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292 2023.
Guo, Z., Zhang, R., Zhu, X., Tong, C., Gao, P., Li, C., and Heng, P.-A. Sam2point: Segment any 3d as videos in zero-shot and promptable manners. arXiv preprint arXiv:2408. 2024.
Chen, Q., Qin, I., Zhang, J., Chen, 7., Xu, X., and Che, W. co A novel benchmark for multi- multi-step muuli--odall ohain-of of ofotthtth In Proc. of ACL, 2024b.
Guo, Z., Lin, H., Yuan, 7., Zheng, C., Qiu, P., Jiang, D., Zhang, R., Feng, C.-M., and Li, Z. Pisa: A self-augmented data engine and training strategy for 3d understanding with large models. arXiv preprint ar arXiv:2503. 2025b.
Chen, X., Zhang, R., Jiang, D., Zhou, A., Yan, S., Lin, W., and Li, H. Mint-cot: Enabling interleaved visual tokens in mathematical chain-of-thought reasoning. arXiv preprint MxXiv:2506.05311, 2025.
Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang. J., and Heng, P.-A. Sciverse: Unveiling the knowledge com- prehension and visual reasoning of Imms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627, 2025c.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao. Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05277 2024c.
Guo, Z., Zhang, R., Tong, C., Zhao, Z., Gao, P., Li, H., and Heng, P.-A. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., al. How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024d.
2025d.
Hao, S., Gu, Y., Luo, H., Liu, T., Shao, X., Wang, X., Xie, S., Ma. H., Samavedhi, A., Gao, Q. et al. Llm reasoners: New evaluation, library, and analysis of step- by-step reasoning with large language models. arXiv preprint arXiv:2404.05221, 2024.
Du, Y., Liu, Z., Li, Y., Zhao, W. X., Huo, Y., Wang, B., Chen, W., Liu, 7.,, Wang, Z., and Wen, J.-R. Virgo: A preliminary exploration on reproducing ol-like mllm. arXiv preprint arXiv:2501.00
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, Liu, Z., and Sun, M. Olympiadbench: A challenging benchmark for promoting agi with olympiad- -evel bilingual multimodal scientific problems, 2024.
2025.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vimevalkit: An open-source toolkit for evaluating large
10

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-40 system card. arXiv preprint #xXi:2410.21226, 2024.
Lin, Z., Liu, C.,, Zhang, R., Gao, P., Qiu, L.,, Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi- modal large language models, ECCV 2024, 2023.
Jia, Y.. Liu, J., Chen, S., Gu, C., Wang. Z., Luo. L., Lee, L.. Wang, P.. Wang, Z., Zhang, R., et al. Lift3d foun- dation policy: Lifting 2d large-scale pretrained mod- els for robust 3d robotic manipulation. arXiv preprint arXiv:2411. 2024.
Liu, H.,, Li, C., Wu, Q., and Lee, J. Visual instruction tuning. In NeurlPS, 2023.
Lu, P., Bansal, H., Xia, T., Liu, J., yue Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. ArXiv, 1bs/223 2023.
Jiang, D., Song. G., Wu, X., Zhang, R., Shen, D., Zong, Z., Liu, Y., and Li, H. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint XiXv:2404.00655, 2024a.
OpenAL. URL GPT-4V(ision) system card, 2023. D0c-4v-systen:
Jiang, D., Zhang, R., Guo, Z., Wu, Y., Lei, J., Qiu, P., Lu, P., Chen, Z., Song, G., Gao, P., et al. Mmsearch: Benchmarking the potential of large models as multi- modal search engines. arXiv preprint arXiv:2409.1295) 2024b.
OpenAL. Introducing openai ol, 2024., 2024a. URL
/ttps://openall com/o1/
OpenAl. Hello gpt-4o. https: / wwwp/ppeaaa.oooom com/ madex/hello-e 20004.
Jiang, D., Guo, 7., Zhang, R., Zong, 7., Li, H., Zhuo, I.., Yan, S., Heng, P.-A., and Li, H. T2i-r1: Reinforcing image generation with collaborative semantic-leve and token-level cot. arXiv preprint 25Kx:25005000003 2025.
Peng. T., Li, M., Zhou, H., Xia, R., Zhang, R., Bai, L., Mao, S., Wang, B., He, C., Zhou, A., et al. Chimera: Improving generalist model with domain-specific experts. arXiv preprint ar arXiv:2412.05983, 2024.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint ar MXXv:2408.03326 2024a.
Prasad, A., Saha, S., Zhou, X., and Bansal, M. Receval: Evaluating reasoning chains via correctness and informa- tiveness. arXiv preprint arXiv:2304.10703, 2023.
Qwen Team. Qwen2-vl. 2024.
Li, F. Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next-interles image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning trans- ferable visual models from natural language supervi- sion. In International Conference on Machine Learning, 2021. URL https: //api./pppppp semanticscholar. org /tsc/ooopuID::2359910
Tackling multi-
Li, J. Li, D., Xiong, C., and Hoi, S. Blip: Boot- strapping singuage-image pre- pre-training for unified vision- language understanding and generation. In International Conference on Machine Learning, pp. -2880-12900. PMLR, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, 2022.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L.., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023.
pp.
Shen, D., Song, G., Zhang. Y., Ma, B., Li, L., Jiang, D., Zong, Z., and Liu, Y. Adt: Tuning diffusion models with adversarial supervision. arXiv preprint AXiv:2504.11422 2025.
Lin, X., Zare, A., Huang, S., Yang, M.-H., Chang, S.-F., and Zhang, Personalized video comment generation. In Al- Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 16806-16820, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 00.18653/112222.mmdinn....m 979. URL attpps:/ /ttaa/aaaattttoo...oooo wwtp:///...gg/220.....// fi findi iasiing--ae nmn--ennlpp 979/.
Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., and Durrett, G. To cot or not to cot? thain-of-thougeee helps mainly on math and symbolic reasoning. arXiv preprint MxXiv:2409.12188 2024.

---

MME-CoT: Benchmarking Chain-of-Thougee in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1, 5: Scaling reinforcement learning with Ilss, arXiv preprint arXiv:2501.12591 2025.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang. J., Wang, J., Yang. J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai. J., He, J.. Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang. P., Peng. R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y. Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang. Z., and Fan, Z. Qwen2 technical report. arXiv preprint arx 10671, 2024.
Team, Q. Qvq: To see the world with wisdom, December 2024. URL, https: //ts::///wwwa ///eeweeln 1m. m. g thub. i lo/blog/ qvq-72b 1ee-772b-reeee poe-pprevieww.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971,, 2023.
Yao, H., Huang, J., Wu, W., Zhang, J., Wang, Y., Liu, S., Wang, Y., Song, Y., Feng, H., Shen, L.., et al. Mulberry: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18314 2024a.
Wang, F., Fu, X., Huang, Y., Li, Z., Liu, Q., Liu, X., Ma, M. D., Xu, N., Zhou, W., Zhang. K., et al. Muirbench: A comprehensive benehmark for robust multi-image image under- standing. arXiv preprint ar arXiv:2406.09411 2024a.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H.. Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint vrXiv:2408.01800 2024b.
Wang, P., Bai, S., Tan, S., Wang. S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Wang, W., Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Zhu, J., Zhu, X., Lu, L., Qiao, Y., and Dai, J. Enhancing the reasoning ability of multimodal large language mod- els via mixed preference optimization. arXiv preprint arXiv:2411. 104444 2024c.
Ying, K., Meng, F. Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., et al. Mmt- bench: comprehensive multimodal benchmark for eval- uating large vision- languae models towards multitask agi. arXiv preprint arXiv:2404. 100224444.0006666 16006, 2024.
Wang, W., Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Zhu, J., Zhu, X., Lu, L.., Qiao, Y., et al. Enhancing the reasoning ability of multimodal large language mod- els via mixed preference optimization. arXiv preprint arXiv:2411. 10444, 2024d.
Yu, W., Yang, Z., Li, L.. Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint aiXiv:2308.m 0.24999 2023.
Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S. Sun, Y., Yu, B., Zhang, G., Sun, H., Su, Y., Chen, W., and Neubig, G. Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark, 2024. URL https:ts///........ arxiv. org aass/////////// 2409. 02813.
Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H.. Malladi, S., Chevalier, A., Arora, S., and Chen, D. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint 24Xiv:2406.1 18521, 2024e.
Zhang, H., Li, H., Li, F., Ren, T., Zou, X., Liu, S., Huang, S., Gao, J., Zhang, L.. Li, C., et al. Llava grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02949, 2023.
Wei, J.. Wang, X.. Schuurmans, D.,, Bosma, M., Xia. F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zhang, R., Han, J., Liu, C., Zhou, A., Lu, P., Qiao, Y., Li, H., and Gao, P. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR 2024, 2024a.
Xu, G., Jin, P., Li, H., Song, Y., Sun, L., and Yuan, L. Llava-cot: Let vision language models reason step-by- step, 2024. htlphtts:///rre org/ abs/2411. 10440.
Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., and Qiao, Y. LLLMA--daaters Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024b. URL https:// openreview net/ forum? i id 401 XAHN2W.
Xu, R., Wang, X., Wang, T., Chen, Y., Pang, J., and Lin, D. Pointllm: Empowering large language models to un- derstand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
12

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P. Zhou, A., Lu, P., Chang, K.-W., Gao, P., al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? ECCV 2024, 2024c.
Zhang, R., Wei, X., Jiang, D., Zhang, Y., Guo, Z., Tong, C., Liu, J., Zhou, A., Wei, B., Zhang, S., al. Mavis: Mathematical visual instruction tuning. arXiv preprint mxXiv:247.088739 2024d.
Zhang, Y., Bai, H., Zhang, R., Gu, J., Zhai, S., Susskind, J., and Jaitly, N. How far from intelligent visual deductive reasoning? In COLM, 2024c.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592,, 2023.
Zong, Z., Jiang, D., Ma, B., Song, G., Shao, H., Shen, D., Liu, Y., and Li, H. Easyref: generaledr group image reference for diffusion models via multimodal Ilm. arXiv preprint arXiv:2412.0961 2024a.
Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., and Liu, Y. Mova: Adapting mixture of vision experts to multimodal context, arXiv preprint ,,44, 2024b.
13

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Appendix Overview
Section A: Related Work.
Section B: More Evaluation Details.
Section C:
More Dataset Details.
Section D: More Experiment Details
Section E: Error Analysis.
Section F: More Qualitative Examples.
Section G: Evaluation Prompts.
A. Related Work
A.1. Large Multimodal Models
The field of multimodal (Radford et al., 2021; Li et al., 2022; OpenAI, 2023; Rombach et al., 2022: Shen et al., 2025; Jiang et al., 2024a; Zong et al., 2024a) AI has experienced extraordinary growth, particularly through the development of Large Multimodal Models (LMMs) (Liu et al., 2023; Zhu al., 2023; Lin et al., 2023; Zong et al., 2024b; Qwen Team, 2024). These models build upon the achievements of Large Language Models (LLMs) (Touvron et al., 2023; Yang et al., 2024) and advanced vision models (Radford al., 2021), expanding their capabilities to process multiple kinds of visual input (Li et al., 2024b; Guo et al., 2023; Li et al., 2023; Guo et al., 2025b).
Closed-soures models, such as OpenAI's GPT-4o (OpenAI, 2024b), have demonstrated exceptional capabilities in visual understanding and reasoning. However, their closed-source nature creates barriers to widespread adoption and further development by the broader research community. In response, significant progress has been made in developing open-source alternatives. Early approaches like LLaVA (Liu et al., 2023). --Aateerr (Zhang et al., 2024b), and MiniGPT-4 (Zhu al., 2023) established foundation by combining frozen CLIP models for image encoding with LLMs, enabling multimodal instruction tuning. Subsequent developments through projects such Intern V1.2 (Chen et al., 2024d), Qwen2-Vi VL. (Qwen Team, 2024), SPHINX (Gao et al., 2024; Lin et al., 2023), and MiniCPM-V (Yao et al., 2024b) have expanded these capabilities by incorporating more diverse visual instruction datasets and broadening application scenarios.
Recently, with the introduction of ol (OpenAI, 2024a), the field of LMMs has also focused on enhancing the reasoning capability. (Wang et al., 2024d) introduces mixed preference optimization with automatically constructed data. (Yao et al., 2024a) proposes leverage collective knowledge from multiple models to identify effective reasoning paths. Besides, several works (Team, 2024; Du al., 2025) have demonstrated the ability replicate behaviors similar to ol models, particularly regarding multi-step CoT reasoning with iterative self-reflection and verification processes.
A.2. Reasoning Evaluation
Several methods have been developed evaluate reasoning in natural language processing, including ROSCOE (Golovneva al., 2022) and ReCEval (Prasad et al., 2023), which assess reasoning chains across multiple dimensions such correctness and informativeness. However, these approaches are limited to text-only scenarios and do not address the unique challenges present in visual reasoning tasks. Furthermore, the emergence of long bhain-of-though (CoT) reasoning has introduced additional considerations, such as output efficiency and reflection quality, which existing evaluation methods do not adequately address.
On the other hand, various multimodal benchmarks have been developed to assess reasoning abilities across specific domains. Current exploration of visual reasoning predominantly focuses on the mathematics (Zhang et al., 2024d: Peng al., 2024) domains. Vista (Lu et al., 2023) provides a comprehensive collection of mathematical problems that assess mathematical and logical reasoning abilities. Building on this, Math (Zhang et al., 2024c) introduces a new benchmark by climinating redundant textual information to evaluate whether LMMs can accurately interpret graphical representations. OlympiadBench (He et al., 2024) further raises the complexity bar by incorporating challenging Olympiad- mathematics and physics problems. Despite these advances in specialized domains, broader applications such as general-scene reasoning remain relatively unexplored. Recent developments have begun to expand beyond purely scientific reasoning. For instance,
14

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
M-CoT (Chen al., 2024b) and SciVerse (Guo al., 2025c) incorporate commonsense tasks alongside scientific reasoning and knowledge-based assessment in the multimodal benchmark. However, most existing benchmarks focus solely on evaluating final answers while overlooking the intermediate steps, thus providing limited insights into the process through which models arrive at their conclusions.
B. More Evaluation Details
We provide the illustration of the step partition in Figure 7.
Question
Donna a laser She points at the point beam reflects and then as shown, arriving at point
DE=EF=1 length BD, in
Background Information
Image Caption
Logical Inference
D-B-1
Model Response
Step Partition
To find the length of 80, ve can use the properties of right triangles and the aiven inforastion.
Stept [background informstion]
Tn find the length of RD, we can use the propert of right triangles and the given information.
#<Identify Right Trianglesies
Trsangle le LDE is a right trisngle with angle CUE 45, Triangle ErG
right triangte
Triangle
right
Triangle CDE
Step2 [image description]
right triangle with ang CE 45%;
triangle with angle DGB a. 451
triangle le LTG a right triangle le with ang BTG 457; triangle EGB is a right triangle with Le E68 45'
2. +HUse the 45-45-90 Trianqle Theorenzee
45-15-20 triangle, sides the ratie 1:1//2; Since DE EF 1 D. the hypstenuse EG is 192 a IG
Step3 (hackground inforeation]
a<calculate
45-15-98 Triangle Theoren: 45-15-20 triangle,
DE EF
4. +-Use the Pythagoreon Theoren in Triangle EG3:.4
BGT.
Stepa Hlogical inference]
-2" =E3 B61,
Since DE
1 n, the hypotenuse ES is sarti2) n.
44= LB DGF.
e<<alculate
Stepo [logical inference]
-Since 30 EB 86, we need to find EB and 86,
Pythegorean
tD 4,
Since EB 1both are the hypotonuses 45-45-30 triangtes), can write 203 = 4,
Steph [logica] inference] 6812 8612
Sotving EB, get 2.
[logicl inference] sqrt(2)
Therefore, IB = /2.
6. e<Ca Lculate 00:80
B5
2/2.
Step8 [logical inferiecce
B0 sgrti2) + sort(2) = Zegrt(2)
Figure 7. Illustration of Step Partition. We instruct GPT-4o to divide each step into three categories: image caption, background information, logical inference. The step partition result is later used to perform step-wise reasoning evaluation. We focus on evaluating the image caption and logical inference steps, which are the keys to visual reasoning.
15

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
C. More Dataset Details
C.1. Data Source Distribution
We visualize the data source distributions in our benchmark, which consists of 15 sets, including MathVerse (Zhang et al., 2024c), MMMUPro (Yue al., 2024), OlympiadBench (He al., 2024), MMT-Bench (Ying et et.,, 2024), MuirBench (Wang al., 2024a), ml-rpm-bench (Zhang al., 2024e), MMSearch (Jiang et al., 2024b), CharXiv (Wang et al., 2024e), and Sci Verse (Guo et al., 2025c).
olympiad
mathrerse
sciverse
6.9%
10. 10.1%
commgen 19%
6.7%
mmmmppoo 3,0%
olympiad
Science 18.7%
10.2%
charate 2.8%
Math 22.1%
OCR 18.6%
Space- Time
mint 6.5%
muir 12.7%
General
muir 8. 8.3%
Scenes 19.2%
6.0%
mett 2,2%
muir 5,3%
13.9%
mlrp 6.6%
mmt
Figure 8. Data Source Distribution of MME-CoT.
16

---

MMB-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
C.2. Prelininary Categorization Result
Table Accuracy of MMT-Bench for different subcategories. ACT: Action Understanding: AUT: Attribute Similarity; CNT: Cartoon
Understanding; CIM: Counting: DOC: Diagram Understanding; EMO: Difference Spotting: HAL: Geographic Understanding: IIT:
Image-Test Text Matching; IRT: Ordering: IQT: Scene Understanding: MEM: Visual Grounding; MIA: Visual Retrieval: OCR: Object
Recognition: PLP: Physical Layout Prediction; RRE: Relationship Extraction; TMP: Temporal Reasoning; VCP: Visual Comprehension;
VCR: Visual Coherence Reasoning; VGR: Visual Generation; VII.: Visual Identification; VPU: Visual Prediction Understanding: VRE:
Visual Reasoning Evaluation.
ACT
AUT
CNT
CIM
DOC
EMO
HAL
IIT
IRT
1QT
MEM
MIA
OCR
PLP
RRE
TMP
VCP
VCR
VGR
VII.
VPU
VRE
File Name
GPT40-cot
0.60 0.60 0.67 0.79 0.30 0.71 0.50 0.63 0.10 0.60 0.77 0.36 0.76 0.48 0.86 0.80 0.49 0.48 0.82 0.85
GPT4-direct
0.53 0.60 0.44 0.81
0.23 0.69 0.33 0.66 0.25
0.80
0.42
0.78
0.36 0.89 0.85
0.41
0.37
0.85 0.85
Qwen2-VL-7B--ot
0.53 0.61 0.34 0.65 0.77 0.53 0.74 0.40
0.31 0.20
0.78
0.58
0.60
0.43
0.69
0.43
0.85
0.90
0.54
0.35
0.79 0.81
Swen2-VL-7B-diin
0.49 0,67 0.40 0.78 0.75 0.52 0.73 0.43 0.31
0.10
0.78 0.55 0.60
0.54
0.69
0.40
0.85
0.85
0.67
0.38
0.85 0.82
Table 5. Accuracy of MUIRBench for different subcategories. AU: Action Understanding: AS: Attributc Similarity; CU: Cartoon
Understanding: CO: Counting: DU: Diagram Understanding: DS: Difference Spotting; GU: Geographic Understanding: ITM: Image- Text
Matching: OR: Ordering: SU: Scene Understanding: VG: Visual Grounding: VR: Visual Retrieval.
File Name
AU
AS
CU
CO
DU
DS
GU
ITM
OR
SU
VG
VR
GPT40-cot 0.48
0.57
0.55
0.75
0.82
0.64
0.82
0.38
0.88
0.56
0.70
GPT4o-direct
0.45
0.62
0.59
0,50
0.88
0.62
0.55
0.86
0.33
0.74
0.38
0.77
Qwen2-VL-7B-cot
0.38
0.51
0.42
0.43
0.43
0.27
0.21
0.55
0.13
0.69
0.37
0.28
Qwen2-V1--7B-die
0.39
0.47
0.44
0.41
0.40
0.33
0.25
0.51
0.13
0.67
0.31
0.20
Table 6. Accuracy of OlympiadBench for the mathematics and physies subcategories.
File Name
Mathematics
Physics
GPT4o-cot
0.25
0.04
GPT4o-direct
0.07
0.03
Qwen2- VL-7B-cot
0.05
0.01
Qwen2-VL VL-7BB-drett
0.07
0.01
17

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
D. More Experiment Details
D.I. More Findings
Long CoT models may be more susceptible to distraction. Long CoT models may demonstrate lower relevance scores compared other models. They frequently generate content unrelated to solving the given question, corresponding to their relatively low recall scores compared to direct evaluation, like QVQ. Although few models with short CoT, like Mulberry and L.LaVA-V-7B, also obtain low relevance rate, find that it is because these models may keep repeating words when dealing with specific type of questions, resulting in irrelevant judgment. The fine-grained metric reveals that models tend to lose focus when describing images, often producing exhaustive captions regardless of their relevance to the question. From Table 3, we find that this phenomenon prevails in general scenes, space-time, and OCR tasks. This behavior can significantly slow inference by generating substantial irrelevant content. Teaching long CoT models to focus on question-critical elements represents a promising direction for future research.
D.2. Human Agreement
we conduct additional human evaluations to verify the validity the GPT-4o assessment from two perspectives (Chen et al., 2024a; Lin et al., 2024):
1. Human Agreement Rate: A binary (yes/no) human evaluation assess agreement with the model's per-step judgments.
2. Hallucination Detection: We assess whether any steps identified by GPT-4o are hallucinated contain hallucinations.
Our human agreement study covers four key metries: Recall, Precision, Relevance rate, and Reflection quality. We randomly sample 54 predictions (9 questions from each subject) from QwenVL2-72B and QvQ-72B, totaling 216 predictions and 2,368 steps. The results are shown in Table 7.
Table 7. Human Evaluation Results.
Metric
Agreement
Hallucination
Recall
98.5%
0%
Precision
94.1% %
2.1% 1
Relevance rate
90.8%
0%
Reflection quality
86.1% %
0%
These results demonstrate a high correlation between GPT-4o evaluations and human judgment, indicating that GPT-4o is reliable tool for CoT evaluation. This result also indicates that all of our analysis and conclusions are valid.
E. Error Analysis
In this section, we analyze error patterns the LMMs' CoT and reflection process.
We categorize the CoT errors into four types and provide the examples in Figure and distribution in Figure 11:
Visual Perception.
The model mistakenly perceives the images and causes the error.
Visual Reasoning. The model fails to grasp the key information in the images and therefore gets a wrong answer.
Logical Reasoning. The model exhibits flawed logical reasoning process such wrong planning of the solution.
Calculation. The model makes a mistake in the calculation.
As for reflection, we define that an effective reflection should either correct previous mistakes or validate correct conclusions through new insights. We examined 200 model predictions from QVQ and identified four distinct error types that hinder productive reflection. These patterns are illustrated in Fig. 10 and their distribution is shown in Fig. 12.
The four major error types are:
18

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Ineffective Reflection. The model arrives an incorrect conclusion and, upon reflecting, continues make incorrect adjustments. This is the most common type and is also witnessed most frequently.
Incompleteness. The model proposes new analytical approaches but does not execute them, only stopping at the initial thought. The reflection slows down the inference process without bringing any gain.
Repetition. The model restates previous content or methods without introducing new insights, leading to inefficient reasoning.
Interference. The model initially reaches correct conclusion but, through reflection, introduces errors.
Understanding and mitigating these errors is crucial for improving the reliability of LMM reflection mechanisms. The analysis provides the opportunity to focus on solving specific error types to enhance the overall reflection quality.
Visual Perception Error
Visual Reasoning Error
Quistics
Quartics
Question: the
the
diflerer onr. Preaer idertity the outhhrr A the thind image B. the funth image C. Nonz of choicos provriee
many
image of the row?
Answer:
D. the accond image
Answer:
Modd Rorperic
Madel Reqpone
Ersor
unturat Riscausey aspiene Nekker informpo
onterninc mayy mare lane, psaka are provent the wcord image of Use Usird
nood
nater
Dass Precessy Unctuting
1. vediccond Inage of the First Reuce
to exbecint Isugees Ssacavee Smcole tika Procosss, ascoading Usir repalarity
There other distinct peoks scssate In this inages
3. wethurd Inquoee Zaacasocs houure reluted sdsptive contrel of sparnaty, wnto orth bncceess biseasses The setas hoctire is decervattess ee
1925 umage alse shows castrul poak with LIAA rat region disttact
in thie
approdinatngg
hawe stcrinct
probusillyy Founstr, introuiee ane mole about The sntpp of farrrion recuer: (oonee))))
The there elddee
cana anesentactor,
prosessss. (ounth sidde;
diecuse
Therefors, the ruaber additional clntinct pusla the socand congared zera. Aeuerr: (aooeel))
Logical Reasoning Error
Calculation Error
Quartion
Question
Queetion:
Five distinet imoqers are be chouce fromt the act A.S.A.A...AAAA.... and placed in some order the
bores
diugmm Each
nuti top now then containe the product
of imeges is the two boses connected to it cow dicesly ahove. integess
ways ploced in the now to that the integer hottom box is 905 0050000000 Auswez:
Questio:: A small soy car rolls dowe theee rampo with the same beight and boriznctal leuath, but difieent shapes, sooting from net. The car anye contect with the ramp ar timee and rmnzs lost Order nempe from Swarel to skownst time takes faseet and zamp slowet. ctter
my car drop
imothras] For cuanppe, if d'aamp as your snower choias.
Medi Respeaac
Amwar: 213
Tive esscanct ontegers tram the set NIVI, 2. 3. by BVF) with that the soes eccerding to the gance duggia.
Ercor
let factorizs 1082239900 Gecerstadd resuired productus 9883299900
froe the ast styll, 2. 3 5, 8130 whass product is 1125 vitees 20d Vecnes 114 310 U1 201 50 201 104 2121) VIS DV) 1G Vetine mutches the factoriantion aed 2353200000. Let a try to fhe consunation eetterse 100d letach strtuuieer:
angte of the reep, as stssper angle vitt resutt In faster descens. zue the grovicatiin
When ingial
2. asiaap Deaz The vegle to SiGersesrest.
viltap dest The
authenee yoR lubiic 1O grass (IP14).
the order Tros feaceat to stoest lac
4, sechezes USU lubtce de MSSUlee: This goees VI5U.
1991
and Uhe faitol.
eethezee 1IT lelac 1770te:: This 1ITV New, latta versity 2n2 Stanes Atumss
Visses THis mitches the
kang (39) to leact steep and thas The cloweet.
requred product.
stues Villes 322 Aitass
Acover: (20066(213)
Bou, read to arrange these five namers 25 top There 31535 uoss chanss arragge intepers 238. Anownts 121
Figure 9, Examples of CoT Error Types.
19

---

MMB-CoT: Benchmarking Chain-of-Thoughl in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Ineffective Reflection
Repetition
Quorkin
Question: The bls moving Srwwaed What is the connec order images according
Peoptel picture nieen Day. And the right picture takes today. I've pained lox weight and I'm annoyod, How many kilogannd people! A 21.4
Quoction: Cunvenatioo:
A. the secoad imsge, the firs imsae. the thind imsage
overage?
Noue provided
10.7
C. Nore ofe choicen provided
D. 1:07
Answen 17
image. image, the socund imag
the thind imags the second fire image
Reflection
Reqpess
Reffection
menlhly total wouhh namber
Reaciing Tie
tonal cove Llnoerr) 28 (Pebnars) (Narrt) 30 (errl1 31 HWy1 New, to fird the average acectlyy woight gadr, permipu reed to condder the
Locking the oaages, The creast appeain horizertal aer3st
Tiieer laage (lettyy canood tnoer (noatin) lmage (rApt"), siice roce
match
and, unting
aitt creels cifficieet.
Alternativety, pernass the sucstaas exeets airferent perspectives Mist, rayao the horzzentat sevdis Free top bottaa, thee the seguence unald lee firee lrage thepl, shird lmage
Kiternatavelo
the probiir
ascrage satly weagst pain, Khenl
apeenty
aays
Repeired haite
Ne Srall Whon Anoer Ofter Befkerie
buaabe sgrnoy daysument aparoctncte 3.22 kgrmEEt::
That's roahlly 1:27 og per serts, shuch clone to curtier citalle.ie af
Incompleteness
Interference
Quatke
Quaetion
figre,, straigh panilel
than the degres ofange i B:105
Quostion: Suppose these ideniical pit on each of's plans, that two neightrriing planes have gits. Let Rtn k) reperient the sumber of ways which he place the gifts For cuampie, 85,3)=2, as shown below.
squally spaced anound circaler sibe: Bos wishss to place
Medi Roppes:
Dutemine the smollze poodile valae of'n arnnng possiti onderel imagere (n. kj which fin, postive mulippl of 2009
Reflection
Anower:
Requosse
de tab,, Let en cees it haen trtanele with acgles ehy 3y, and angle Tlee'
Reflection
shoild 1965
Cennett Anover
an Whing Asaver
Einencal confrictears, pertage
dxteralne fi
acconding sentier cateatation, L. 1485.
S. a. aigits rast
complicated circutar
theme's deconstatancy here, becweee cornegenning to rua selle choild alce 199".
Uhen asplc being
diffecertly.
Oie Withos Tryin
Figure 10. Examples of Reflection Error Types.
Interference
Vienal Perception 11.9%
1.8% Repetition
Calendation 18.5%
Ineffective Reflection 76.0%
Visual Reasoning
Incompletenees 17.3%
Logical Reasoning 9%
Figure II. CoT Error Distribution of MME-CoT.
Figure 12. Reflection Error Distribution of MME- CoT.
20

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
F. More Qualitative Examples
Precision & Recall
Queation:
Key Steps Annotation:
Find the lnngh oTDB
PM
The ingg Shows two rffnnge 9TR and CAB,
FE-3 DE-s, CBai5, AB-45 Coacision:
Choicess A. 2.4 C.o D..
minegl
The kmnth of 8x9, which dooct D
Amswer:
The ooneet unswe
GPT4o:
Prediction:
trinnless traesele anc dr Usssppl (ib,, the probice uss liss asin the cantort par trianlice since the correcpezzing
carefally
Shes confire the sinclerity of the Lruelll from the clagrom: dath craungl the ase and traerpls ter lure 5e daynee are tee argics in craengis lef, Since the aspios troangic and is 190ogrees, the thud angles tse property
stellars are prpprtioon the isi or ke mas find ot, to we sill use the sice ratue:
trinngles cintiar trisngles,
Vfrac (eC) (r
is In triangle out: proper Laee (raclesiiee (treetasedll U.
sumplall Uhe ratas as the comttiand suder Viras (ie) te 5. 1l ase hanal Aeavert the length 38 aat a.
Step
Judgment:
Information
catun the lmage provtded, reed carefullyy aealize tue plwer stace triang thare cae anglet, which makes their corresperetin sinee propertioeal.
concllltion:
sten typa: Iaag Bese iption
daeclasian
trlangle IFF de angle (right teisnglee).
destription
casclusion The pine arcs Indicatee that one the other angles trisegle ARC equal to cee the
prentee Bulli Io uaglle Ask und trunglic the have o ser angite drashs, traeegles), the park are intieate wosles triannte aod up 18ll trineglee cistlar).
argles duer with eount. Tats
Rarigraand Information
conclasion: for simttar triangees, the sidea are proportieral, That la: 4B/DE SC/EF 4C/27.
ster typer teage description
conclasion: in tranglic D87:
stea, typu: Logical Inferance
troangle Abc: 10 = 45, Dc 15, trtangls ter: ET 5. EP b.
pranive: Horte auzen,
1y, to = proportion:
pranise: 40/08 1573,
praniser suitl 3,
praaton: or 4525
pranis: be 9.
length
Precision 11/11
Key Steps Ansotation Matching:
stows Hbdd, teea, Lb-1,, 12445
Key Concluston:
Traergic p 48C is sintler to traengls 2tt
4B/DE SU/PE
The Teeest whic correspoats to
Recall 5/5
Figure 13, Examples of Precision and Recall Evaluation.
21

---

MMB-CoT: Benchmarking Chain-of-Thoughh in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Qwen2-7b:
Prediction:
To find the length of DE, similar, corresponding Atrac(EF)(BC
the property sinilar triangles. Since triangles DEF and ABC proportional.
the following proportion: AI \frac(DE)(a)) Given that =,, 80 15, and AB 45, we can substitute these values into the proportion: ML (E)))) = MMrac(3315)) V Simplifying the right side of the equation: Al (T)) (frac(1)(5) VI To find DE, can cross-multiply; VI Acdot DE 45 Acdot 1 Therefore, length
\frac(45H5)
The correct
answer 15 D:
Step Partition & Judgment:
step step_type: background information conclusion: To Find the length of DE, we can use the property of sinilar triangles.
step type: logical inference
premise: triangles DEF and ABC are similar.
conclusion: The corresponding sides propertional.
step type: logical inference
presise: the cor resnenigg sides are proportional. conclusion: We can
the following proportion: DE/AB EF/BC.
step_type: background information
conclusion: Given that EF 3, BC 15, and AB 45.
step_type: logical inference
prasise: DE/AB EF/BC and EF = 3, BC 15, AB 45.
conclusion: Substitute these va lues into the proportion: DE/45 3/15.
step_typen type: logical inference
step,
premise: DE/45
3/15.
conclusion: Simplifying the right side of the equation: DE/45 1/5.
step
step_typer type: logical inference
pramise: DE/45 1/5.
conclusion: To find DE, can cross-multiply: 5 x DE 45 1.
step step_type: logical inference
premise: 5 DE 45.
conclusion: DE 45/5.
step_ type: logical inference 45/5. conclusion: DE = 9.
presise:
stop step_typer type: logical
premise; DE = 9.
conclusion: The length of DE is 9. The correct answer is D: 9.
Precision= 8/8
Key Steps Annotation Matching:
Key Caption:
The inage shows two triangle FDE and CAB, FE=3, DEsx, CB=15, AB=45
Key Conclusion:
Triangle ABC sinilar triangle DEF
AB/DE
BC/FE
The length of is 9, which carresponds to choice
The correct answer is
Recall= 5/5
Figure 14, Examples of Precision and Recall Evaluation.
22

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
0V0-228-Previes
Prodiction:
letts
trasngle they are other than diagrs, Speciiically, in trierate porraseeds wais crasple atgle trungis tamooptdds angle triangle ssc
glvee.
are orsendee:
castter
coasing Aoc. angle coreamecias argle paoms carrasaading ssgle and sigle corrasaneiig to angle lat the cisgrim a
Are
Abc.
trunple da
to
Locling sol's lock lhe conritettes the lake again. In trueg ti sas, cuttls and 3. to asd in cosered. of Mally ro. In trise aee, sates te, te eh with boin caa traanlle AoC. tual's cogron, speci ccally aggle dogress, sanee share's seaare part C. witch dowtts
bating
avste
to eath other. diven teat angte also be the right segla is trtangle stc. inst, tet tronngl dee, aeplas b, and are all some In trianple are is san tur to treenll soc, wath atpte at A beay commn to lett. fet is the duairen it paist atared. wats, pertazs pain diagra cencription. trueegit
sures ioport jasat First,
jastt
Lot try ratrass the prsbiin basod aeather rolae tetwvon the audey trteppic iain rese is laad"" Ef uhit in tranngt carraaann nd to stach sides in traangle 48t1 If angie cornapeeds arpte as a. angle as comrespands and correppnds at. steretore, the ratte corresssstin saaes dn 21 at H-- dut csing tringts
camon with
the hhre's a.
tine weats and
Vast, mate next
caall let denvic the tergttr Let 22 a
dante
aotter we E2 air
safe anyial tranngi wath right argis
was since trissyle ase io can lar to rrine eec, wits be cerresponting ze du to conresvviiigg to M., 13b0eett1211 a / is Fiuslaty 2 as: that's a 5 So, a Tharefore, a 45 11 / S1
Step
Judgrents
step teaer tus teckgroae antormo
Gor
ice, zther ABC.
inage deceraatio tritagle
trimpgee because corresonntie angles
ogpll.
step. teae: inags doseratton
deserigtion
Lasion: there's sods labeled votn largth J.
laage deseriation
conclasio:: trurgii ARC s3855 As, UC, and MG, with BE teang seats as betng units.
stog. tope: Lougs dosrastion
trinngis
segroo, apecitionlly
degress.
inderence
preniser trisogle Ae 15 siptt-ongtee triongte with right welte ae is 45 wetts, and ac is 15
concti secclasoon sance traanpic atnd traanpte ABc, ther corropention sants
strg. ryan: legtisl Inderence
triangts
trisngte craanpte ARC, with anglc astng commn to tath,
the rit regie.
legiiaa
prensse: tranngic AEt
form
strg ryan tegiial iederence
teisnpte
uxiss, aegle
Is 18 soyrces.
legtiaa Inderwace
praddse: triargle clatlar to trinegle MMC the neti their correppaatngg cides an casi.
preciee: in trivigle ABC, B6 15 unire, ND 45 untte, and aegta C degreet.
tegtial indeowace
predise: ssa't the leeath dC-
togicat Interesce
"ighe-ingted extangte right
45.
Pyrtageness
Infevence
prenise: trassale A8C) de" MCI REr, os W9,
sqp2c18901 31-agra12).
<teg, ryan: tegtost Inderence
cornequeztin to
correcawviing to Ar, and pereeooeetin to RC.
(3ae0rr1211 precdes: csnality tharts
legtost indeowcee conclasionn: 25
steg, topm: tegrcst Inference coeclasionn
cettes
tyan, legttss inderwace
preniss: sonclasion: the correct choice cs D:5.
Precision= 18/19
Figure 15. Examples of Precision and Recall Evaluation.
23

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
QVQ-72B-Preview:
Annotation Steps Matching:
Key Caption:
V
The inage shows two triangle FDE and CAB, FE=3, DE=x, CB=15, AB=45
Key Conclusion:
Triangle ABC similar to triangle DEF
V AB/DE BC/FE
The length of DE 9, which corresponds to choice D
The correct answer is D
Recall= 4/5
Figure 16. Examples of Precision and Recall Eyaluation.
24

---

MME-CoT: Benchmarking Chain-of-Thhughee in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Precision & Recall
Key Strps Annotation:
Bewas Gljes the ddjet wili
Key Copiaa:
milkk the Man inciogle whick can
socee liv lave
Cenlicn comprrelunglo
Choues Objic 7. J.Cljjec
Buck auruuua
Key
soceallle the atiled soutd clicie
Corparol with Di Objec eut Orchees has onz and chiee
Anmwer Objec
6PT
Pradiction:
uettactee battr lake portar
qurtian as asmess cpccitially base sparre mase aootooas salass
ang bortoe monno wellow techatee seetaii weree deseralie
The
crear batter sost radk its teaiitorat,, the whtan caject 13l pectapiatts Cajact separrs
and.
snsiile sopr copttation is buils lils mors cherty ony brtmeed loom cattonti
uedindan wilte bink waen bootin nits lals the food thiy scaratie tol
statile
to that Anseittty bettortwe bodt basee asvertice th lnstrmoon. draiing aethots kitcen
visst ancmume sttemativety
anapiity qqueeran proanitaeed inpanitile desemnie beatenchens
6 Step Partition & Judgmest
Cociiiion: The feod qartian is to aatess henlttirus spcctiiallyy in This
fectrr
natrittennl whee that
bas. bage sapcopties
year reatect fish, poststy cestatetng shettfis
uniratly bealthy.
eopetalles ctamtow this sidi to six eatritianl valas, prositiag
11tarte and Flsen
snclarden repetables adds nutritional
steg, bypn logucat asterencs
ca with 11 at's caaeed wuth excauser ool,
stea hyan: lostcal Intermee
that the feed Onject ds fried cocket with
certeiter feal Sugst
ompore Boxt
autraant coppaned oaject
bype: Infermee
Fredie: Frest Tocdi peserstly coecddrd lass healthy when canaunes is Bose des to their hoge
1998, 139, Logioal Infermnce
ca cenioeee:: Obtect lo rore W0 Worsett-dees optiins
preercattes fecis
vegrratis of aslance groaps
Bacigread Diioomstion
ceettiies relping votille act daotee, and thet, tare sore aoeetaiity suace sar'
Sype Bacigruad Drionestion
deterstring beateshen satets les
Losteal Intereeee
certaiienn Exjec bad
Precision= 9/11
say Steps Amnttation Matching:
Caption:
Aapect fuitian the arange rectaggar caets tiee truct nut unak,
Cojact Ele rectangic som 1ike Frilisd rearse cutlan
Bay Conclusion:
Eiject
Tun tass cale cap wich Bicet
Recalla 3/4
Figure 17. Examples of Precision and Recall Evaluation.
25

---

MME-CoT: Benchmarking Chain-of-Thougle in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Owen2-7b:
Prediction:
deternine object healchier Object object within orange
Object health crsteria: protein yegurt
*e0bject within blue rectangle), analyze
possibly restangle)ee:
various
Yogurt good
granola healthy probiotics.
Fruits provide
fiber.
Overall,
mutritious real, xedject tte object withan Fried foods typically appears appear healthy choice corpared yegurt nutrients.
rectangto)
possibly pancales
tiber
analysis, wefbject yogurt
granola)se healthier than wedbfect
fried food) *fiinal
Objoct
Step Partition Judgnent:
type: background informatiion
determine object healthier Obfect (the object within rectanglel Object (the object within blue rectangla), let's health criteria.
object based
type: description
cone conclusion: Object (the object within the urange rectangle) to be boel of yogurt topped possibly granela.
background information
conclusion: Yogurt good
prabiotics.
stop_type:
background information
conclusion:
provide vitanins, ninerals, fiber.
type:
background information
conclusion: Nuts and granola add healthy fats and fiber.
type: logical inference
of protein probistics. Fruits provide vitasins, minerals,, and add hes ity fats fiber, conclusion: Dverall, this looks Like balanced and nutritious neal.
prenise:
fiber.
description
X conclusion: Object (the object within the blue rectangle) to be of fried food, possibly pancakes wafflles.
background information
conclusion: Fried foods typically high in calories, saturated fats, and sodiun.
background informstion
conclusion: They any also de lou in fiber and other essential nutrients.
type: logical inference
prenise: fried foods are typically high in calories, saturated fats, and sodius. They also be Tow
fiber and other essential nutrients.
conclusion: This does not appear to be healthy choice conpared to the yogurt bowl.
type: logical inference
prenise:
overalll this Tooks tike balanced and nutritious meal. This does not appear to be healthy bowl.
choice compared the Lusiliiion: Based this andlysis, Object (the yogurt boul with fruits and granola) is healthier than Object 2 Ithe fried food).
type:
Togical inference
prenise: based this analysis, Object (the yogurt howl with fruits and granola) is healthier than Object 2 It fried food).
conclusion: Final Answert A. object
Precision 5/6
Key Steps Annotation Matching:
Caption:
Object
twithin the orange rectangte) seens like a fruit and nut salad.
Object 2 (within the blue rectangle) seers Like frilled or roasted chicken
Key Conclusion:
Compared with Object Object less ca loriess fat compared with Object
healthier because the fruit and Object
The correct answer A.
Recall= 3/4
Figure 18. Examples Precision and Recall Evaluation.
26

---

MMB-CoT:
Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
DVQ-728-Preview:
Prediction:
Alright, I've get this question rectangle), Object (blue restangle), on appesrance alone. First, orange reclangle, Fron the description, sounds Like a breakfes setup. There's plate with what appeers to be yogurt topped with fruits like bananas, straberrries and poss possibly fueberries atong tea, aug of coffee, and some roasted poratoes compartson is between
are txo objects,
colored rectangles orange equally healthy, or
opt impessable
Object lorange rectangle ic. object
to undorstand what's
some
granola, There's also glass Object is in blue rectangle. It's plate roasted poratoes. So, and granola yogurt bool, and plate of roasted potatoes, Now, I need to assess which of these is hea healthier. Let is consider the nutrients each provides. The yogurt bowl with truits and granola: source of protein and calciun. are rich in antinxidants and vitarin C. GGanolaa la can vary, but it often contains eats, which are high in riber, and possibly nuts or seeds for healthy fats and additional protein. The roasted potatocs: Potatoes are source of carbohydrales, vilamin C, and polassiun.
background.
Yogurt 15 a good Bananas provide potassiun and vitarin C. Strawberries and blueberries
How they' 're roasted nallers: if it's they deeply tried or cooked with
with niniral oll and seasoning, they can be quite healthy. Hovever, unhealthy fats, Given that, the yogurt bawl seers offer more
lot of fat, they night be higher ba salanced neal with combination of protein, carbohydrates, fiber, and vitamins froz the fruits and granola. The roasted potatoes, wiile they can be part of a healthy diet, especia bly if prepared hes stellylly, don't provide as diverse range or nutrients
the yogurt boul. Porcover, desending boal. But, to be fair, if the they could still be quite healthy, Hovever, nutrient density But shoule consider there's specific
the preparation, the
potatoes night be higher in calories from fat compared potatoes roasted with
the
little
yogurt bool potatoes Therefore,
fruits granola is Likely be healthier choice due
would be A. Object example, the option
that the
potatoes night
besed on visual, information provided, yogurt bowl well-balanced and nutritious, So, Oofect
Step Partition & Judgnent:
step type:
background informeLion
conclusion: There are tuo cojects, each within a colored rectongles ane in arange and one in blue. The task is to detersine which one is thtr. The opt ions are A. dbject lorange rectanglel, B. Object 2 Iblue rectanglel, c. Both equally healthy, D. It's inpossible to deternine based on agpearance alanc.
step, type: image description
conclusion: object is in the orange rectangle, There's a plate with what appears to be yogurt topped with fruits like bananas, strawberries, and possibly blucberries, along with some granola. There's also rossted potatoes the background.
glass of iced
of coffee,
step_type: image description
conclusion: Object is in blue rectangle, plate roasted potatoes.
step_type: lopical
preeise: comparison between
and grano yogurt granola.
plate roasted potatoes, of protein,
yegurt carbohydrates, fiber, vitanins the
stop type: logical inference
premise: roasted potatoes, while they can part a healthy diet, especially if prepared hea ithily, provide as conclusion: The yogurt bowl is likely to healthier choice due its nutrient densiry and diversity.
range nutrients yogurt boxt.
step, type: logical inference promise: based on the visual information provided, the yogurt boul looks well balanced and nutritious. conclusion: A. Object is the correct answer.
precision: 5/5
Steps Annotation Natching:
Caption:
Object 1 (within Uhe orange rectangle) seess Like fruit and nut salad.
Object Iwithín the blue rectangte) seens 1ike fritled roasted chicken
Key Conclusion:
Compared object Object seens more healthserr because trust and Object has and fat corpared with Ubject
answer is
recall: 3/4
Figure 19, Examples Precision and Recall Evaluation.
27

---

MMB-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Relevance Rate
Key Steps Annotation:
Question:
How many puppies in the images are sticking out their tongues?
Key Caption:
the first image, there one puppy, while tongue not sticking out. the second image, there one puppy, while tongue not sticking out.
Choices: A. Zero B. Two C. Three
D. None of the choices provided
E. Four
Key Conclusion:
1. no puppies are sticking out their tongues.
Answer:
A. Zero
2. The answer is A.
GPT4o:
irrelevant information
Prediction:
Let's carefully analyze the images step by step: ### **Inage 1 Analysis:** The puppy sitting on an orange blanket. The position of the mouth and tongue of the puppy is visible. Upon claser inspection, **Inage Analysis:** The position of the mouth and tonguc is also visible here. Upon careful observation, **the puppy is again not sticking out its tongue.** ##е Intermediate Observations: In both images, the puppies are not sticking out their tongues. and Final Answer: **λ. Zeront
**the puppy is sticking out purple background.
tongue.**
puppy is sitting
Judgments:
step_type: inage description
conclusion: The puppy is sitting on an orange blanket.
step_types image description
conclusion: The position the mouth and tongue the puppy visible.
step type: logical inference
conclusion: The puppy is not sticking out its tongue.
X step_type: image description
conclusion: The puppy is sitting on purple background.
step_type: image description
conclusion: The position of the mouth and tongue is also visible here.
step type: logical inference
conclusion: The puppy is again not sticking out its tangue.
step_type: logical inference
conclusion: In both images, the puppies sticking their tongues.
step_type: logical inference
conclusion: Final answer is A. Zero.
Raw Relevance Rate 6/8
Figure 20. Examples of Relevance Rate Evaluation.
28

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Qwen2-7b:
inrelevant information
Prediction:
To determine how many puppies are sticking out their tongues each image, let's analyze each inage step by step: 1. #*first Imagess: There is one puppy in the image. - The puppy is not sticking out its
one puppy tongue. Since neither in either image is sticking out its tongue, the correct answer is: A. Zero
tongue. 2.
Images4:
image.
puppy not sticking
Judgments:
step_ background information
conclusion: to determine how many puppies are sticking out their tongues in each image, let's analyze each image step by step.
step_type: image description puppy the first image.
conclusion: there
type: image description
conclusion: the puppy in the first image is not sticking out its tongue.
step_type: image description
conclusion: there
puppy the second image.
step_type: image description conclusion: the puppy in the second image is not sticking out its tongue.
step_type: logical inference
conclusion: since neither puppy either image sticking tongue, is: A. Zero.
Raw Relevance Rate 6/6
Figure 21, Examples of Relevance Rate Evaluation.
29

---

MMB-CoT: Benchmarking Chain-of-Thougle in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
QVD-72B-Preview:
irrelevant information
Prediction:
Alright, I've got this question here, It says, "How nany puppies are sticking out their tongues in each two inages provided. Okay, puppies are sticking out their tongues in each one, First, let's look at the first image. There's corgi against wooden background
image?" And there
look both images and count many
puppy sitting on an orange has tri-color coat-white, black, and tan-and it's wearing black collar. Its tongue not sticking So, in this inage, the number of puppies with their tongues out zero. Now, moving on the second image. This one features another corgi puppy, this time against purple fabric background. This puppy also has tri-color coat and is looking directly from what can see, the puppy's mouth closed, and there's s tongue visible. So, again, with their tongues out in this image. Wait a minute, the question says "each inage, and there are two images. for
puppy
the camera, need to check its tongue sticking out, Hrm, puppies
inage, need images, there's only one puppy, and neither of them is sticking out their tongue. So, for both inages, the count
number puppies sticking out their tongues.
But let's double-check. Maybe there's something I'm missing. In the first image, the puppy is clearly not showing its tongue. In the second image, similarly, no tongue is visible. So, think it's safe say
their tongues. Looking back options provided: A Zero, В. Two, C. Three, D. None of the choices provided, E, Four. Since in each image, the number is zero, and there should be A. Zero. Wait, but maybe there's a trick here. Maybe "each image" refers to all images combined. no, the question specifically says "in inage," so think it's asking for the number per image, not in total across all images. So, final answer is A. Zero. Final Answer: A.
image,
zero puppies sticking
inages, but
question specifies "in each image," " the
Judgments:
/ step type: background information
conclusion: question many puppies sticking their tongues in each inage.
X step_type: image description conclusion: The first image shows cargi puppy orange blanket with tri-color coat and black collar.
step_type: logical inference conclusion: In the first image, the puppy's tongue is sticking out, the count is
step_type: inage description
conclusion: The second image shows a corgi puppy against a purple fabric background with tri- color coat.
step_type: logical inference conclusion: In the second image, the puppy's 's mouth is closed, so the count is zero.
step_type: logical inference conclusion: for each image, the number of puppies sticking out their tongues is zero.
step_type: logical inference
conclusion: Double-checking confirms zero puppies sticking out their tongues in both images.
step_type:
background information
conclusion: Reviewing the answer choices: A. Zero, B. Two, C. Three, None of the choices provided, Four.
step_type: logical inference conclusion: The answer 1s A. Zero, as the count is zero in each image.
step_type: logical inference
conclusion: Considering if "each image" refers images conbined, but specifies per image.
step_type: logical inference
conclusion: Final answer is A. Zero.
Raw Relevance Rate = 9/11
Figure 22, Examples of Relevance Rate Evaluation.
30

---

MMB-CoT: Benchmarking Chain-of-Thouge in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Reflection Quality
B
Question:
Key Steps Annotation:
In the diagram, ACDF is rectangle with AC=200 and CD=50. Also, triangle FBD and triangle AEC congruent triangles which are right- angled and E, respectively. What is the area of the shaded region?
Key Conclusion:
1. join BE
2. BAFE is rectangle
3. BCDE is rectangle
4. 1/4 of rectangle ABEF is shaded and
1/4 of rectangle BCDE is shaded
The total shaded area 1/4(10000)
2500
Answer: 2500
6. The answer 2500
QVQ-72B-Preview:
redundant reflection
Prediction:
Now, triangle F80 has a right angle at B, so is sonewhere on FD, think. Wait, no, FD is the botton side FBD triangle must be one of the endpoints, but that doesn't make sense. Wait, maybe B is not FD. Similarly, triangle AEC is right-angled at E, and is on FD, perhaps. Wait, naybe I need to think differently. Since congruent right-angled triangles, perhaps the legs of these triangles right-ang triangles. Let's say, for triangle F80, the legs, and for triangle AEC, AE and EC the legs. Given that the triangles are congruent,
rectangle.
vertices F, B,
it's right-angled
B,
200
and CD 50, and the triangles are equal in length. Let's denote the legs and BD FB EC and BD AE. But since both triangles are right- and congruent, their corresponding sides of
equal. Wait, these two triangles. Maybe should find the area of one triangle and then double it, and then subtract the the intersection directly. me try
need find the
of shaded region, which
overlapping
non-overlapping parts. Alternatively, perhaps to find the equations of the lines that form the triangles to find their intersection points.
find
Judgments:
conclusion: is not FD, reconsidering the position of
conclusion:
The legs the triangles equal, reconsidering length the sides
conclusion:
The goal to find the area the shaded region, reconsidering the method calculating it
Figure 23. Examples of Reflection Quality Evaluation.
31

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
G. Detailed Evaluation Setup
G.1. CoT Quality Evaluation Prompts
Recall Evaluation Prompt
You are an expert system to verify solutions to image-based problems. Your task is to match the ground truth middle steps with the provided solution.
INPUT FORMAT:
1. Problem: The original question/task
2. A Solution of a model
3.
Ground Truth: Essential steps required for a correct answer
MATCHING PROCESS:
You need match each ground truth middle step with the solution:
Match Criteria:
The middle step should exactly match in the content or is directly entailed by certain content in the solution
All the details must be matched, including the specific value and content
You should judge all the middle steps for whether there is match in the solution
OUTPUT FORMAT:
"step index": \textless integer\ textreate,, "judgment": "Matched" "Unmatched"
1
ADDITIONAL RULES:
Only output the ISON array with no additional information.
Judge each ground truth middle step in order without omitting any step.
Here the problem, answer, solution, and ground truth middle steps:
[Problem]
{question]
[Answer]
{answer)
[Solution]
{solution}
[Ground Truth Information]
{gt annotation]
32

---

MMB-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Precision Evaluation Prompt
Task Overview
Given a solution with multiple reasoning steps for an image-based problem, reformat it into well-structured steps and evaluate their correctness.
# Step 1: Reformatting the Solution
Convert the unstructured solution into distinct reasoning steps while:
Preserving all original content and order
Not adding new interpretations
Not omitting any steps
## Step Types
1. Logical Inference Steps
Contains exactly one logical deduction
Must produce a new derived conclusion
CCannot be just a summary or observation
Image Observation Steps
Pure visual observations
Only includes directly visible elements
No inferences or assumptions
3. Background Information Steps
External knowledge or question context
No inference process involved
## Step Requirements
Each step must be atomic (one conclusion per step)
No content duplication across steps
Initial analysis counts as background information
Final answer determination counts as logical inference
Step Evaluating Correctness
Evaluate each step against:
## Ground Truth Matching
For image observations:
Key elements must match ground truth observations
For logical inferences:
CConclusion must EXACTLY match be DIRECTLY entailed by ground truth
## Reasonableness Check (if no direct match)
Step must:
Premises must not contradict any ground truth or correct answer
Logic is valid
Conclusion must not contradict any ground truth
Connclusion must support or be neutral to correct answer
## Judgement Categories
"Match": Aligns with ground truth
"Reasonable": Valid but not in ground truth
33

---

MME-CoT: Benchmarking Chaun-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
":rong": Invalid or contradictory
"N/AA: For background information steps
# Output Requirements
1. The output format must be in valid JSON format without any other content.
2. For highly repetitive patterns, output it as single step. Output maximum 40 steps. Always include the final step that contains the answer.
Here the json output format: ## Output Format
Sstep_type": "image observation (oovvatioonlogiaal inference lbackground information", "premise": "Evidence (only for logical inference) ", "conclusion": "Step result",
"judgment": "Match Reasonable Wrong IN/A"
Here the problem, and the solution that needs to reformatted to steps:
[Problem]
{question}
[Solution]
{solution]
[Correct Answer]
(answer)
[Ground Truth Information]
{gt annotation]
G.2. CoT Efficiency Prompt
Relevance Rate Evaluation Prompt
# Task Overview Given a solution with multiple reasoning steps for an image- to get a solution (ignore correct or wrong) of each step.
image-based based problem, evaluate the relevance
Step 1: Reformatting the Solution Convert the unstructured solution into distinct reasoning steps while: Preserving all original content and order
Not adding new interpretations
Not omitting any steps
## Step Types
Logical Inference Steps
Contains exactly one logical deduction
34

---

MME-CoT: Benchmarking Chain-of-Thoughe in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Must produce a new derived conclusion
Cannot be just a summary observation
2. Image Description Steps
Pure visual observations
Onll includes directly visible elements
No inferences or assumptions
Background Information Steps
External knowledge or question context
No inference process involved
## Step Requirements Each step must be atomic (one conclusion per step)
No content duplication across steps
IIitial analysis counts as background information
Final answer determination counts as logical inference
Step Evaluating Relevancy relevant step is considered as: 75% content of the step must be related to trying get a solution (ignore correct or wrong) to the question.
IMPORTANT NOTE:
Evaluate relevancy independent correctness. As long as the step trying to get to a solution, is considered relevant. Logical fallacy, knowledge mistake, inconsistent with previous steps, or other mistakes do not affect relevance. logically wrong step can relevant if the reasoning attempts address the question.
The following behaviour is considered as relevant:
i. The step is planning, summarizing, thinking, verifying, calculating, or confirming intermediate/final conclusion helpful to get solution.
ii The step is summarizing or reflecting on previously reached conclusion relevant to get solution.
ii.. Repeating the information in the question or give the final answer.
iv. relevant image depiction should be one of following situation:
help obtain a conclusion helpful solve the question later;
2. help identify certain patterns in the image later;
3. directly contributes to the answer
Depicting analyzing the options of the question is also relevant.
vi. Repeating previous relevant steps are also considered relevant.
The following behaviour is considered as irrelevant: i. Depicting image information that does related what asking in the question. Example: The question asks how many are present in all the images. If the step focuses other visual elements like the road building, the step is considered as irrelevant.
ii Self-thought not related to what the question is asking.
ii.. Other information that is tangential for answering the question.
Output Format
"step_type": "image observation logical inference bbackround information", "conclusion": "A brief summary of step result", "relevant"; "Yes INo"
Output Rules
35

---

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Direct JSON output without any other output Output most 40 steps
Here is the problem, and the solution that needs to be reformatted to steps: [Problem]
{question]
[Solution]
{solution}
Reflection Quality Evaluation Prompt
Hereś refined prompt that improves clarity and structure:
Task
Evaluate reflection steps image-based problem solutions, where reflections are self-corrections or reconsideration of previous statements.
# Reflection Step Identification
Reflections typically begin with phrases like:
"But xxx"
"Altemnttiely, xxx"
"Maybe should"
"Let double-check"
"Wait xxx"
"Perhaps xxx"
It will throw a doubt of its previously reached conclusion or raise a new thought.
Evaluation Criteria
Correct reflections must:
Reach accurate conclusions aligned with ground truth
2. Use new insights to find the mistake of the previous conclusion verify its correctness.
Invalid reflections include:
1. Repetition Restating previous content or method without new insights
Wrong Conclusion Reaching incorrect conclusions vs ground truth
Incompleteness Proposing not executing new analysis methods
Other Additional error types
# Input Format
[Problem]
{question]
[Solution]
{solution]
[Ground Truth]
36

---

MMB-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
{gt annotation]
# Output Requirements
The output format must be in valid JSON format without any other content.
Output maximum 30 reflection steps.
Here the json output format:
## Output Format
"conclusion"; "One-sentence sunmary of reflection outcome", "judgment"; "Correct WWong"", "error_type"; IN/A/Repetition Wrong Conclusion Incompleteness Other" |Other
# Rules
Preserve original content and order
No new interpretations
3. Include ALL reflection steps
4. Empty list if no reflections found
5. Direct JSON output without any other output
G.3. Direct Evaluation Prompt
Answer Extraction Prompt
You are an Al assistant who will help me to extract an answer of a question. You are provided with question and a response, and you need to find the final answer of the question.
Extract Rule:
[Multiple choice question]
1. The answer could be answering the option letter or the value. You should directly output the choice letter of the
answer.
2. You should output a single uppercase character in A, B, C, D, I (if they are valid options), and Z.
3. If the meaning of all options are significantly different from the final answer, output Z.
[Non Multiple choice question]
Output the final value of the answer. could be hidden inside the last step of calculation or inference. Pay attention what the question asking for extract the value of the answer.
2. The final answer could also be short phrase or sentence.
3. If the response doesn't I give final answer, output Z.
Output Format: Directly output the extracted answer of the response.
[In Context Examples]
Question: {question}
Answer: {response)
Your output:
37

---

MMB-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
Answer Scoring Prompt
You are an Al assistant who will help me to judge whether two answers are consistent.
Input Illustration:
[Standard Answer] is the standard answer to the question. [Model Answer] is the answer extracted from model's output to this question.
Task Illustration: Determine whether [Standard Answer] and [Model Answer] consistent.
Consistent Criteria:
[Multiple-Choice questions]
If the [Model Answer] option letter, then must completely matches the [Standard Answer]. If the [Model Answer] not an option letter, then the [Model Answer] must completely match the option content of [Standard Answer].
Haan-Muliiple-Chotee questions]
1. The [Model Answer] and [Standard Answer] should exactly match.
2. If the meaning is expressed in the same way, it is also considered consistent, for example, 0.5m and 50cm.
Output Format: If they are consistent, output 1; if they are different, output O.
DIRECTLY output or 0 without any other content.
[In Context Examples]
Question: {question}
[Model Answer]: fextract
[Standard Answer]: {gtanswer)
Your output:
38


Reviews:
[{"review_id": "NkgFYjPqbq", "reviewer": "Reviewer_NFBU", "summary": "The paper introduces MME-CoT, a specialized benchmark evaluating the CoT reasoning performance of LMMs. \n\nIt is the first comprehensive study in LMM CoT evaluation: it spans six domains: math, science, OCR, logic, space-time, and general scenes; and it proposes a thorough evaluation suite incorporating novel metrics about reasoning quality, robustness, and efficiency.\n\nIn-depth analysis of state-of-the-art LMMs are shown, e.g., CoT performance on perception-heavy tasks, inefficiency of reflection mechanism.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.\n\n> **Q1: Difficulty in understanding Sec 2.2 and difference with existing metrics** \n\nThanks for your valuable advice. **We improve the clarity of Sec 2.2 below:**\n\n  For CoT evaluation, we provide *key steps* and *reference image captions* for all questions. \n\n  + *Key steps* are defined as **necessary steps** for answering the question correcly.\n\n    All the key steps fall into two categories: \n\n    1. *Inference conclusions*: Necessary conclusions reached through logical inference steps (including the final answer)\n    2. *Image captions*: Identifications of critical visual information\n\n    To efficiently annotate key steps for all questions, we implement a two-phase process. First, GPT-4o generates initial versions of key steps annotations (containing both inference conclusions and image captions) with the questions, images, and final answers as inputs. Including the final answers significantly improves the quality of the initial versions compared to using only questions and images. Second, human annotators review these initial versions, correcting any errors or developing key steps independently when GPT-4o failed to provide reasonable output.\n\n    We reduce all steps to the simplest form, preserving only core conclusions and relevant visual element description. For problems with multiple solution paths, annotators are required to provide all possible methods.\n\n  + *Reference image captions* are visual information not covered by the image caption in key steps.\n\n    These reference captions are mainly for the calculation of precision. We use the same method as key steps to obtain the annotation: GPT-4o first generates initial version and then annotators review and correct the errors.\n\n\n**Difference with existing metrics:**\n\nFew works evaluate the multimodal CoT process. MathVerse [1] represents one such effort. MME-CoT differs in several key aspects:\n\n1. **MME-CoT evaluates the CoT quality in two ways: precision and recall**, corresponding to CoT faithfulness and informativeness. While [1] only instructs GPT-4 to judge the correctness of step, which could be viewed as only evaluating precision.\n2. **MME-CoT contains ground truth key steps annotation**, enabling more reliable evaluation, especially for tasks beyond GPT-4o ability. [1] contain no such GT reference.\n3. **MME-CoT considers both inference conclusions and image captions in CoT on all visual reasoning tasks**. [1] only targets on math tasks without identifying different types of CoT steps.\n\n> **Q2: Need to consider tasks involving both perception and reasoning**\n\nSorry for the confusion caused. We'd like to clarify our task definitions:\n\n+ The perception tasks are the tasks primarily test visual recognition abilities or require very minimal reasoning. \n\n+ The reasoning tasks additionally require logical inference steps along with visual perception tasks. Therefore, **tasks requiring both perception and reasoning are exactly the reasoning tasks in MME-CoT**. \n\n  Examples in Fig. 2 (bottom section) of our paper all showcase this two requirements: first perceiving visual cues, and then reasoning based on the perception.\n\n+ **For your examples**: Counting red balls belongs to perception task since it requires minimal reasoning. Determining relationship between people belongs to reasoning task. Similar reasoning task also occurs in MME-CoT, as shown in Fig. 1 in https://anonymous.4open.science/r/reb-3538/README.md\n\n> **Q3: The reflection evaluation seems to be beyond the model's capability**\n\nThanks for your valuable advice. We identify GPT-4o as an well-qualified evaluator for assessing the reflection quality:\n\n1. **GPT-4o shows competitive results of reflection quality evaluation in the human agreement experiments.** \n\n   We conduct additional human evaluations to verify the validity of GPT-4o assessment from two perspectives:\n\n   1. **Human Agreement**: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.\n   2. **Hallucination Detection**: We assess whether any reflection steps identified by GPT-4o are hallucinated or contain hallucinations.\n\n   We randomly sample 54 predictions (9 questions from each subject) from QvQ-72B. The results are below:\n\nAgreeement|Hallucination\n-|-\n86.1%|0%\n\n2. **Additional instruction for reflection quality evaluation prompt.**\n\n   Acknowledging the challenges in identifying valid reflections, we incorporate specialized prompt design. We identify and list common reflection errors to better guide validity assessment (detailed in Lines 906-910).\n\n> **Q4&5: Comparison with existing metrics and more examples**\n\nWe provide the comparison in Fig. 7 and more examples in Fig.8-18 in the link above.\n\n[1] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?", "claims_and_evidence": "Main claim: existing evaluation for LMM CoT is insufficiently systematic and thorough, while the proposed MME-CoT is a comprehensive and specialized benchmark for evaluating the CoT reasoning skills within LMMs.\n\nThe claim is clear and convincing. MME-CoT spans six fundamental domains and introduces comprehensive metrics. Also, diverse typical LMMs are evaluated on MME-CoT and analyzed.", "methods_and_evaluation_criteria": "The proposed metrics are sound for CoT evaluation. \nThe curated dataset spans six important domains and contains large-scale test data (1,130 questions with 3,865 key reasoning steps).\nExperiments have been conducted for several typical LMMs.\n\n**Questions:**\n\n1. For CoT quality evaluation, it is difficult to understand L186-204 and identify the difference compared with existing metrics.\n\n2. For CoT robustness evaluation, the paper proposes stability score based on perception tasks and efficacy score based on reasoning tasks. What about tasks involving both perception and reasoning? For example, counting red balls in a picture of balls with multiple colors, reasoning the relationship of two people from their activity, expression and scene. Such visual reasoning tasks are typical and important. \n\n3. For CoT efficiency evaluation, a reflection should be \"valid\", either correctly pointing out the previous mistakes or verifying the previous conclusion with a new method. How to judge its validity? It needs comprehensive judgement and seems beyond capabilities of automatic evaluation.", "theoretical_claims": "No Theoretical Claims.", "experimental_designs_or_analyses": "I have checked experimental designs and analyses.", "relation_to_broader_scientific_literature": "The key contributions of the paper is related to LMM CoT evaluation.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-organized. The figures and tables are clear and easy to understand.", "other_comments_or_suggestions": "1. Comparison with existing metrics can be shown in figures to better illustarte the novelty of MME-CoT benchmark.\n\n2. More visualization and examples can be shown in the supplementary to support the analysis and conclusion.", "questions_for_authors": "My questions are mainly about metric design, which are listed in the \"Methods And Evaluation Criteria\" part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KFbOzdU7mK", "reviewer": "Reviewer_yz6o", "summary": "This paper introduces MME-CoT, a new benchmark for evaluating Chain-of-Thought (CoT) reasoning in Large Multimodal Models (LMMs). The work addresses a timely and important gap in the evaluation of multimodal reasoning. The authors have identified key limitations in existing benchmarks and propose a more comprehensive evaluation suite.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.\n\n> **Q1: Need studies on human agreement**\n\nThank you for your advice. As you suggested, we conduct additional human evaluations to verify the validity of the GPT-4o assessment from two perspectives [1, 2]:\n\n1. **Human Agreement Rate**: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.\n2. **Hallucination Detection**: We assess whether any steps identified by GPT-4o are hallucinated or contain hallucinations.\n\nOur human agreement study covers four key metrics: Recall, Precision, Relevance rate, and Reflection quality. We randomly sample 54 predictions (9 questions from each subject) from QwenVL2-72B and QvQ-72B, totaling 216 predictions and 2,368 steps. The results are as follows:\n\nMetric|Agreeement|Hallucination\n-|-|-\nRecall|98.5%|0%\nPrecision|94.1%|2.1%\nRelevance rate|90.8%|0%\nReflection quality|86.1%|0%\n\nThese results demonstrate a high correlation between GPT-4o evaluations and human judgment, indicating that GPT-4o is a reliable tool for CoT evaluation. This result also indicates that all of our analysis and conclusion are valid.\n\nWe will incorporate references to the papers you suggested in our final version.\n\n> **Q2: Confidence interval is not provided**\n\nThanks for your advice. Our default experiment setting uses temperature 0 for GPT-4o to ensure reproducible results. To address your concern, we set the temperature to 1 and evaluate 50 predictions for 5 times. The resulting 95% confidence interval widths are below:\n\nPre.|Rec.|Eff.|Sta.|Rel.|Ref.\n-|-|-|-|-|-\n0.018|0.026|0.004|0|0.049|0.069\n\nThese narrow confidence intervals demonstrate the statistical reliability of our findings. The largest interval width is only 0.069 for Reliability, indicating that our results are stable and reproducible even with stochastic sampling.\n\n> **Q3:  Small separation between models of the current metric** \n\nWe respectfully disagree with the reviewer's assessment for the following reasons:\n\n1. **MME-CoT separation is comparable to other benchmarks:**\n\n   The difference is listed below:\n   Benchmark|Difference\n   -|-\n   F1-score (MME-CoT)|3.05\n   MMMU|0.83\n   MathVista|1.43\n   MathVerse|3.47\n\n2. **The robustness and efficiency scores differ from traditional metrics, so the scale of the model's separation is also different.**\n   \n   + The robustness score measures the performance difference within different prompts. This difference results from the intrinsic model attributes.\n   + The efficiency score specifically identifies models with excessively long CoT and reflection steps (e.g., QvQ and Virgo). As shown in Table 2, these two models score over 20 points lower in CoT efficiency compared to others, demonstrating significant separation.\n\n> **Q4: Experimental setup is too simple**\n\nWe respectfully disagree that our experimental setup is too simple:\n\n1. **Our experimental setup is sophisticated and comprehensive**\n\n   Compared with previous multimodal evaluation works [3, 4], our study explores multiple dimensions: different prompt strategies (CoT prompt vs. direct prompt), evaluation methods (CoT evaluation vs. direct evaluation), and diverse evaluation aspects (quality, robustness, and efficiency).\n\n2. **Our focus is assessing natural CoT behavior, not improving it**\n\n   The primary objective of our paper is to evaluate how current LMMs perform reasoning when confronted with problems and to assess the quality of their reasoning processes. We specifically examine how models behave with standard prompting approaches rather than engineering prompts to encourage specific behaviors. The latter, while valuable, is beyond our current scope and for future research.\n\n> **Q5: Results of Claude and Gemini**\n\nThanks for your advice. The results of Claude and Gemini are listed below:\n\nModel|Pre.|Rec.|Eff.|Sta.|Rel.|Ref.\n-|-|-|-|-|-|-\nGemini-2.0-Flash|80.3|52.9|6.6|5.9|95.5|100\nClaude-3.5|77.2|48.2|9.9|11.0|91.0|100\n\nPlease refer to the updated table in Table 1 in https://anonymous.4open.science/r/reb-3538/README.md\n\n[1] MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark \n\n[2] Personalized Video Comment Generation\n\n[3] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?\n\n[4] Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi", "claims_and_evidence": "All the claims are problematic because the metrics themselves are not verified by human at all. Moreover, without any confidence interval, it is too early to draw any conclusion in a statistically significant manner.", "methods_and_evaluation_criteria": "The reviewer is really concerned about the validness of the current evaluation suite.\n\n1. The evaluation relies heavily on GPT-4o(mini) for assessing various aspects of CoT (e.g., Recall, Precision, Relevance). While GPT-4o(mini) is a strong LLM, it's not perfect. Analysis of the agreement between LMM's judgments and human evaluations is necessary. For example, performing similar studies on human agreement in [a,b] could really make the work much more sound.\n\n2. Confidence interval is necessary to draw meaningful conclusions to illuminate future research.\n\n3. Current metrics have a rather small separation between models.\n\n4. The current experimental setup is too simple, lack of sufficient variants to perform in-depth analysis. For example, to understand the importance of reflection/length of CoT, it is expected to at least try different prompts to encourage or discourage the model to include certain behavior in the prompts and perform more controlled experiments. \n\n[a] MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark\n[b] Personalized Video Comment Generation\n\n[Update]: The reviewer appreciates the rebuttal and raised score accordingly based on the promised results. But given the current agreement value and the separation between models. The reviewer is still concerned on how this metric effectively reflect the actual rank between models and would like to call this out for ACs to consider.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Please check details in Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "As mentioned in Methods And Evaluation Criteria, references providing approaches to verify agreement with human shall be leveraged and cited.", "other_strengths_and_weaknesses": "Lack of more evaluation of other most capable models such as Gemini and Claude.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please check above for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d2Azz9bLTq", "reviewer": "Reviewer_nZUW", "summary": "This paper introduces MME-CoT, a novel benchmark for evaluating chain-of-thought (CoT) reasoning capabilities in Large Multimodal Models (LMMs). The authors present a comprehensive evaluation framework that assesses three critical aspects of multimodal reasoning: quality, robustness, and efficiency. The insights derived from the experiment results are valuable and helpful to develop better CoT model.\n\n## update after rebuttal\nThe authors's rebuttal has resolved most of my concerns. I hope the rebuttal contents can be added in the final version. I generally think this is a good paper and will left the AC to decide the acceptance", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.\n\n> **Q1: Potential GPT-4o bias in evaluation**\n\nThank you for your valuable advice. We want to address your concern from two aspects:\n\n1. **High Human-GPT-4o Alignment**\n\n   Our human alignment experiment shows strong correlation between GPT-4o evaluations and human judgment, confirming GPT-4o as a reliable tool for CoT evaluation. Specifically, we investigate two perspectives:\n\n   1. **Human Agreement Rate**: A binary (yes/no) human evaluation to assess agreement with the model's per-step judgments.\n   2. **Hallucination Detection**: We assess whether any reflection steps identified by GPT-4o are hallucinated or contain hallucinations.\n\n   We cover four key metrics: Recall, Precision, Relevance rate, and Reflection quality. We randomly sampled 54 predictions (9 questions from each subject) from QwenVL2-72B and QvQ-72B, totaling 216 predictions and 2,368 steps. The results are as follows:\n\n  Metric|Agreeement|Hallucination\n  -|-|-\n  Recall|98.5%|0%\n  Precision|94.1%|2.1%\n  Relevance rate|90.8%|0%\n  Reflection quality|86.1%|0%\n\n2. **Best Available Automated Evaluation Method**\n\n   While acknowledging the inherent limitations, GPT-4o represents the current state-of-the-art approach for automatic CoT evaluation. GPT-4o has been widely adopted and validated for evaluation across various multimodal [1] and reasoning tasks [2]. Given our reference solutions and human alignment results, we believe GPT-4o is the most reliable option currently available for this complex evaluation task.\n\n> **Q2: Issues of true direct answers**\n\nThanks for your valuable advice.  We report the actual direct answer (defined as less than 20 words) ratio in the table below: \n\nModel|Direct Ans Ratio\n-|-\nMulberry|0.2600\nLLaVA-OV-7B|0.9712\nLLaVA-CoT|0.0000\nLLaVA-OV-72B|0.9704\nMiniCPM-V-2.6|0.9944\nInternVL2.5-8B|0.9920\nQwen2-VL-7b|0.9776\nInternVL2.5-8B-MPO|0.9656\nInternVL2.5-78B-MPO|0.9992\nQwen2-VL-72b|0.9776\nVirgo-72B|0.6728\nQVQ-72B|0.0008\nGPT4o|0.9648\n\nFour models demonstrate reluctance to provide direct answers (we mark these four models with \\* in Table 2). Based on this, we recalculate stability and effectiveness using only questions that receive direct answers. The updated table is shown in Tab. 1 in https://anonymous.4open.science/r/reb-3538/README.md\n\n**We would like to emphasize that all conclusions in our current paper remain valid.** Our analysis and conclusion do not consider any results with \\* to ensure validity. Thanks for pointing this out, and we will stress this further in the final version.\n\n> **Q3: CoT error analysis**\n\nThanks for your advice. We provide the analysis of CoT error types and reflection error types.\n\n+ CoT Error Types: Visual Perception, Visual Reasoning, Logical Reasoning, Calculation.\n+ Reflection Error Types: Ineffective Reflection, Repetition, Incompleteness, Interference.\n\nPlease refer to the example and error ratios in Fig. 3-6 in the link above.\n\n> **Q4: Human verification of GPT-4o correctness**\n\nPlease refer to our response to Q1.\n\n> **Q5: The cost of evaluating GPT-4o**\n\n**The average cost of evaluating one model is 16 dollars**. We have also experimented with evaluating with GPT-4omini for all the tasks, but the human evaluation shows low agreement. With the recent development of LLM, the open-source models might be used for evaluation in the future.\n\n**We also plan to release a testmini set**, which composes 200 questions with all the subjects included. **Evaluation using testmini can reduce evaluation costs to around 3.5 dollars.**\n\n> **Q6: Most models score 100 in reflection quality**\n\nYes, as illustrated in Line 381-382, **for models not generating reflection, we define their reflection quality to be 100** since the absence of reflection can be viewed as the most efficient approach.\n\n> **Q7: How does GPT-4o judge the answer for OCR tasks?**\n\nThanks for your advice. We look into 20 predictions and their corresponding evaluation results. We observe that:\n\n1. The tested models typically only OCR the content directly relevant to the question rather than performing comprehensive OCR on all text. The models tend to summarize other visual information briefly. This results in concise OCR content in predictions.\n2. **We observe that GPT-4o demonstrates high accuracy in judging OCR content**, likely because the OCR content is usually highlighted in the key step annotations. Nevertheless, **we will still enhance our evaluation prompt to explicitly instruct GPT-4o to follow the OCR metric design you suggested**, ensuring that OCR responses are only considered correct when they exactly match the reference.\n\n[1] Tarsier: Recipes for training and evaluating large video description models\n\n[2] Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Yes. This paper applies an LLM-as-judge evaluation method, and also provided LLM with the reference solution and background knowledge required to make the judgment. Therefore, although there might be concern on the hallucinations of LLM-as-judge, it's overall acceptable.", "theoretical_claims": "No. There is no theoritical claims in this paper.", "experimental_designs_or_analyses": "Yes. I checked section 4 for the details and the setting makes sense. Also Apendix B for prompt template", "relation_to_broader_scientific_literature": "1. This paper presents the first benchmark that evaluates the quality of CoT in terms of multiple aspects including precision and recall, also the robustness, etc. Previous benchmarks only focus on the outcome of the final answer and thus makes the paper's contribution significant.\n2. This paper clearly distinguished the vision perception tasks from the reasoning tasks and performance analysis on them in different approaches. The insights that overthinking can do harm to the perception is a unique observation.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths \nSame as \"Relation To Broader Scientific Literature\"\n\n### Weaknesses\n1. Potential GPT-4o bias: The evaluation relies heavily on GPT-4o for judging various aspects of model outputs. This introduces potential biases, as GPT-4o itself may not be perfect at evaluating reasoning processes. This is a minor concern as the author already tried to give the users a reference solution and background knowledge.\n2. There can be potential issues in the evaluation setting, where the comparison of CoT reasoning and direct reasoning can contain problems. Although for direct reasoning, the prompt is \"Please directly provide the final answer without any other output\", there is still a great change that the model ignore this instruction and keeps output the CoT, which is the behavior the model is trained on. Therefore, this can makes the comparison between CoT reasoning and direct reasoning meaningless. The authors should analyze and report the percentage of the response from direct reasoning that is actually direct. Otherwise, part of the conclusions like stability of the perception and reasoning tasks can be wrong conclusions.\n3.", "other_comments_or_suggestions": "1. Are there any failure cases analysis in terms of the error type of the CoT, like calculation error, lack of background knowledge, etc? It's quite helpful for people to understand the behavior of the models on CoT generation. \n2. If there are still resources, I would suggest the authors to conduct a human verification of the correctness of the GPT-4o as a judge. Although it's provided with the solution and each step along with the background knowledge, there is still a possibility that the GPT-4o can hallucinate. A ablation can alleviates concerns for this.", "questions_for_authors": "1. What's the cost of evaluating a model since GPT-4o can be expensive.\n2. The reflection quality of most models are 100, only Virgo-72B and QVQ-72B have pretty low scores. Are there any insights behind this result? Is that because we rarely observe reflection in other models?\n3. For OCR tasks, how did GPT-4o judge the answer? The OCR response should only be regarded as correct when it's the same as the reference, otherwise 0. Will GPT-4o also give score if the response has pretty minor issue like missing a character or work, but the meaning is still the same?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qwDfNVqXBR", "reviewer": "Reviewer_3D9g", "summary": "This paper introduces MMECoT, a benchmark designed to evaluate the Chain-of-Thought (CoT) reasoning performance of Large Multimodal Models (LMMs) across six domains: math, science, OCR, logic, space-time, and general scenes. It proposes a comprehensive evaluation suite with three novel metrics to assess reasoning quality, robustness, and efficiency at a fine-grained level. The study analyzes state-of-the-art LMMs and uncovers key findings: 1) Models with a reflection mechanism, like QVQ, show superior CoT quality, approaching GPT-4o; 2) CoT prompting tends to degrade LMM performance on perception-heavy tasks, possibly due to overthinking; and 3) Despite high CoT quality, LMMs with reflection mechanisms are inefficient during both response and self-correction phases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments. We find them extremely helpful and will incorporate them in the final version. We address each comment in detail, hoping to address your concerns.\n\n> **Q1: Accuracy concern of questions of multiple correct solutions**\n\n**We believe this concern is addressed in our paper.** Our methodology explicitly accounts for questions with multiple correct solutions in both annotation and evaluation phases:\n\n1. **Annotation Phase**: As detailed in Lines 203-206 of the main paper, annotators are explicitly instructed to provide all key steps for all possible solution paths.\n2. **Evaluation Phase**: Our evaluation framework accommodates multiple solutions:\n   - For recall computation (Lines 245-246 and Equation 1), we compute recall scores for all possible solutions and select the maximum as the final recall score\n   - For precision, relevance rate, and reflection quality, we provide GPT-4o with key step annotations for all possible solutions in the evaluation prompt, enabling proper assessment of the CoT process considering all valid approaches\n\n> **Q2: How is CoT obtained in models that don't generate it autonomously?**\n\nThanks for your advice. **We have incorporated the illustration of how to obtain the CoT output in Line 257 - 261 in the right columns of the main paper.** Specifically, we employ a CoT prompt to instruct the model to first perform step-by-step reasoning and finally give the answer. As illustrated in Line 355-358, the CoT prompt is:\n\n> Please generate a step-by-step answer, include all your intermediate reasoning process, and provide the final answer at the end.\n\nWe empirically find that the models that don't generate CoT automatically like LLaVA-OV could give a detailed reasoning process before providing the final answer.\n\nGLM-4v cannot receive multiple images as input, so we only evaluate DeepSeek-vl2. The result is:\n\nModel|Pre.|Rec.|Eff.|Sta.|Rel.|Ref.\n-|-|-|-|-|-|-\nDeepSeek-VL2|81.2|43.0|-1.6|-5.1|93.3|100\n\nPlease refer to the detailed table in Table 1 in https://anonymous.4open.science/r/reb-3538/README.md\n\n> **Q3: Selection process for key step annotation and the number of key step annotation for each QA** \n\nThanks for your advice. We want to clarify the two points below:\n\n1. **We have illustrated how to obtain the key steps in Line 187 - 205 in the main paper.** We detail the process below:\n\n   Key steps are defined as **necessary steps** for answering the question correcly, such as identifying critical values in math diagrams or reaching intermediate conclusions essential to determining the final answer.\n\n   To efficiently annotate key steps for all questions, we implemented a two-phase process. First, GPT-4o generated draft annotations using the questions, images, and final answers as inputs. Including the final answers significantly improved draft quality compared to using only questions and images. Second, human annotators reviewed these drafts, correcting any errors or developing key steps independently when GPT-4o failed to provide reasonable output.\n\n   All the key steps fall into two categories: \n\n   1. *Inference conclusions*: Necessary conclusions reached through logical inference steps (including the final answer)\n   2. *Image captions*: Identifications of critical visual information\n\n   We reduce all steps to their simplest form, preserving only core conclusions and relevant visual element descriptions. For problems with multiple solution paths, annotators are required to provide all possible methods.\n\n2. **All QA corresponds to multiple key steps.** \n\n   1. **We provide a dataset visualisation in the bottom part of Fig. 2**. The key caption and key conclusion correspond to the inference conclusions and image captions of the key steps.  We will make this figure more clear in the final version.\n\n   2. **We also provide the statistics of the key step annotations in Table 1.** There are 837 reasoning questions with 3,865 key step annotations in total. On average, each question contains 4.6 key step annotations. This result can also be derived by the summation of the average inference conclusions and image captions listed in the table. We further look into each question and find out that all the questions have at least 3 key step annotations. We will make this more clear in the final version.\n\n\n   We will explicitly add this result in the table in the final version.", "claims_and_evidence": "Yes, the claims are well-supported and make sense in light of the experimental results.", "methods_and_evaluation_criteria": "Yes, the proposed CoT benchmark for LMMs is highly meaningful, as it addresses the need for systematic evaluation of reasoning across multiple domains.", "theoretical_claims": "This work does not include proofs for its theoretical claims. However, based on the results provided by the authors, the findings appear to be reasonable.", "experimental_designs_or_analyses": "Yes, the experimental designs and analyses appear to be sound based on the reported correlation studies.", "relation_to_broader_scientific_literature": "The key contributions of this paper build on prior research in Chain-of-Thought (CoT) reasoning for Large Language Models (LLMs) but extend the focus to Large Multimodal Models (LMMs). Previous work on CoT in LLMs (e.g., GPT-4) has shown improved reasoning capabilities. This paper expands that by evaluating how CoT affects LMMs in multiple domains, revealing novel insights.", "essential_references_not_discussed": "The references discussed in the paper are sufficient for understanding the context and key contributions. The study thoroughly covers the relevant advancements in Chain-of-Thought (CoT) reasoning and Large Multimodal Models (LMMs).", "other_strengths_and_weaknesses": "**Strengths.**\n\nThis paper's strengths lie in its comprehensive approach to evaluating Chain-of-Thought (CoT) reasoning in Large Multimodal Models (LMMs). By introducing the MMECoT benchmark, it provides a detailed and systematic assessment across six diverse domains, offering novel metrics to evaluate reasoning quality, robustness, and efficiency. The in-depth analysis of state-of-the-art LMMs uncovers valuable insights, such as the role of reflection mechanisms in enhancing CoT quality and the potential inefficiencies that arise in self-correction phases. This work fills a gap in multimodal reasoning research, offering both a practical evaluation suite and a foundation for future advancements in the field.\n\n**Weaknesses.**\n\n1. For math-related problems in the benchmark, there are often multiple correct solutions, meaning that there are several valid paths. In such cases, it becomes difficult to fully assess the accuracy of the CoT process.\n\n2.Some models tested in the paper, such as LLaVA-OV, Qwen2-VL, and InternVL2.5, do not generate CoT processes autonomously. The paper does not describe how the CoT outputs for these models were obtained. If these models are indeed capable, could other open-source models, like DeepSeekVL2 and GLM-4V, also be tested?\n\n3. The selection process for Key Step Annotation is not provided in the paper, and it is also unclear whether a QA corresponds to a single or multiple Key Step Annotations.", "other_comments_or_suggestions": "Please see weakness and suggestions above.", "questions_for_authors": "Please see weakness and suggestions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents MME-CoT, the first comprehensive benchmark designed to evaluate Chain-of-Thought (CoT) reasoning in Large Multimodal Models (LMMs) across six diverse domains using three novel metrics for reasoning quality, robustness, and efficiency. The study reveals that while reflection-based models like QVQ achieve high CoT quality, they often struggle with efficiency and may underperform on perception-heavy tasks due to overthinking prompted by CoT.


There was considerable discussion around this paper. The reviewers raised several concerns, including the heavy reliance on a single evaluation method (GPT-4o-mini), the absence of confidence intervals, and various aspects of the evaluation setup. The authors were highly active in the discussion and even questioned the expertise of the reviewers, something not uncommon in cases of disagreement. Overall, I align with the reviewers' consensus: this is an interesting paper with a potentially valuable and focused contribution. If there is space in the program, it would make sense to accept it.
Decision: Accept (poster)