MCoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought (2024)

Abstract: Multi-modal Chain-of-Thought (MCoT) requires models to leverage knowledge from both textual and visual modalities for step-bystep reasoning, which gains increasing attention. Nevertheless, the current MCoT benchmark still faces some challenges: (1) absence of visual modal reasoning, (2) single-step visual modal reasoning, and (3) Domain missing, thereby hindering the development of MCoT. Motivated by this, we introduce a novel benchmark (MCoT) to address the above challenges, advancing the multi-domain, multi-step, and multi-modal CoT. Additionally, we conduct a thorough evaluation involving abundant MCoT approaches on Vision Large Language Models (VLLMs). In addition, we highlight that the current VLLMs still struggle to correctly reason in MCoT and there remains a large gap between existing VLLMs and human performance in MCoT, despite their superior results on previous MCoT benchmarks. To our knowledge, we take the first meaningful step toward the multi-domain, multi-step, and multi-modal scenario in MCoT. We hope that MCoT can serve as a valuable resource, providing a pioneering foundation in multi-domain, multi-step, multi-modal chain-of-thought research.

heading: 1 Introduction
text: Recent advancements in Large Language Models (LLMs) have led to notable improvements in Chainof-Thought (CoT) in textual modality (Wei et al., 2022a,b; Wang et al., 2023b; Hu et al., 2024). In addition, some works begin to extend the textual CoT capabilities to multi-modal CoT reasoning (MCoT). Take Figure 1 (c) as an example, multi-modal CoT requires both the visual and textual features to generate a rationale and a final answer. To this end, Lu et al. (2022a) introduced ScienceQA benchmark and laid the foundation for MCoT. Subsequently, Zhang et al. (2023c) proposed a two-stage approach
* Corresponding Author
during multi-modal reasoning for MCoT. Additionally, Wang et al. (2023a) developed T-SciQA framework to distill high-quality rationales from ChatGPT, which attains an average accuracy of 96.2%, surpassing even the human accuracy of 88.4%.
Inspired by the recent remarkable advancements in the MCoT literature (surpassing human performance), we seek to explore an interesting question: Has MCoT task been solved perfectly? In our deep analysis, the conclusion is definitely “NO”. As shown in Figure 2 (a), we observe that current benchmarks are too simple, leading to an overestimation of current progress. Furthermore, we find that the existing benchmarks exhibit three major drawbacks (see Figure 2 (b)): (1) Absence of visual modal reasoning: As shown in Figure 1 (a), the model can successfully produce rationale and answer solely based on the textual modality context of “supports the plant”, which cannot truly reflect the
8199
ability of multi-modal CoT model. (2) Single-step visual modal reasoning: As illustrated in Figure 1 (b), the model only requires a single-step “feather” object to predict the correct rationale and answer, which cannot be satisfied in the complex multi-step CoT scenario. (3) Domain Missing: Commonsense and mathematics are important domains for evaluating multi-modal CoT (Wei et al., 2022b; Qin et al., 2023), but the current benchmarks lack these topics, hindering the comprehensive evaluation progress of multi-modal CoT. Nevertheless, in real-world scenarios, multi-step MCoT reasoning is frequently observed in diverse domains. For example, as illustrated in Figure 1 (c), vision large language models (VLLMs) are required to identify the correct options integrating at least two multimodal reasoning steps (as indicated by the orange and green lines.). Such multi-step MCoT tasks are required to effectively perform multi-step reasoning across multiple modalities, which cannot be achieved by previous single-step multi-modal CoT approaches.
Motivated by these observations and issues, we introduce a novel benchmark about multi-domain multi-step multi-modal chain-of-thought reasoning (M3CoT) based on ScienceQA (Lu et al., 2022b). Specifically, to address the first issue, we directly remove samples that could infer the final answer without the need for images. To tackle the second issue, we manually annotate and select multistep multi-modal samples. Specifically, we provide expert annotators with textual context and rationales without images. Experts are required to determine when multi-step reasoning cannot be resolved solely based on textual context. Subse-
quently, we present the images to experts to ascertain whether multi-step reasoning occurred across textual and visual modalities. To solve the third issue, we explore LLM-guided augmentation to synthesize the multi-step MCoT data for commonsense and mathematics domains. We evaluate abundant representative MCoT approaches on M3CoT in extensive scenarios, yielding several key takeaways: (1) VLLM shows CoT emergence phenomenon at the parameter level over 10 billion (≥ 13B); (2) Fine-tuning has better hope on multi-step MCoT, compared with the failures of vanilla in-contextlearning, tool usage, and prompting strategies. (3) M3CoT is tough enough and all methods still struggle compared with human performance.
In conclusion, the primary contributions of our work are summarized as follows:
• We identify the weaknesses of current multimodal CoT benchmarks that can not handle complex multi-step reasoning scenarios, which motivates researchers to rethink the current progress of multi-modal CoT.
• To the best of our knowledge, we are the first to consider the multi-domain, multistep, multi-modal CoT scenario and introduce M3CoT to this end.
• We evaluate abundant representative MCoT approaches on M3CoT and summarize some insightful takeaways, hoping to inspire more breakthroughs in this direction.
To facilitate further research, all data and code are available at https://github.com/LightChen233/ M3CoT.

heading: 2 Problem Formalization
text: This section describes the definition of multi-step multi-modal CoT. Specifically, unlike the traditional textual CoT, multi-step multi-modal CoT should consider a scenario involving an image I , a question Q, a context C and a set of n options O = {o1, ..., on}. First we construct a textual prompt T :
T = Prompt(Q,C,O), (1)
where Prompt(·) represents any method used to convert textual inputs into an instruction format.
Then, model should generate a step-wise rationale Rm = {S1, ..., Sm}, each step determined by1:
Si = argmax Si∈Rm
P (Si|I, T ), (2)
P (Si|I, T )= { P (Si|T ,Ri−1), Si /∈ S; P (Si|I, T ,Ri−1), Si ∈ S, (3)
where S indicates steps that require multi-modal reasoning. This reasoning is considered multi-step and multi-modal if |S| ≥ 2.
Finally, the model arrives at the final answer Y , which is denoted as:
Y = argmax o∈O P (o|Rm). (4)

heading: 3 Dataset Annotation
text: This section describes the annotation process of M3CoT, including: Absence of Visual Modal Reasoning Sample Removal (§ 3.1), Multi-step MCoT
1All step segmentation in this paper follows the ROSCOE (Golovneva et al., 2023).
Sample Construction (§ 3.2), MCoT Domain Augmentation (§ 3.3), and Quality Assurance (§ 3.4). The samples we generate and retain at each stage are detailed in Figure 11.

heading: 3.1 Absence of Visual Modal Reasoning Sample Removal
text: This section focuses on addressing the absence of visual modal reasoning challenge from ScienceQA. Automatic Sample Removal: First, we directly filter out samples without images, thereby refining the dataset to include only those samples that potentially require multi-modal reasoning. Manual Annotation: Despite the automatic process, some samples containing images are still irrelevant for multi-modal reasoning (see Figure 1 (a)). Therefore, we further employ manual annotation, requiring experts to verify whether each sample meets the criteria for MCoT. Specifically, our annotation process and instructions are shown in Appendix A.2.

heading: 3.2 Multi-step MCoT Sample Construction
text: This section aims to incorporate multi-step reasoning characteristics from the last processed data. Automatic Sample Removal: In this step, we first automatically filter out some simple samples with overly simplistic rationales, which comprise less than two steps. By doing this, we can reduce the manual annotation burden and increase the reliability of M3CoT. More details are illustrated in Appendix A.3. Multi-step Sample Manual Annotation: After automatic sample removal, we further utilize manual annotation to obtain the final multi-step multi-
modal reasoning dataset. Specifically, human experts are first provided with textual context and rationales without visual modality. They are focused on determining whether it is necessary to answer the samples multiple times based on the visual content. Once experts find that multi-step reasoning needs multiple times reasoning based on the image, we will provide them with corresponding images to let them finally confirm whether the sample needs to utilize multi-step image and text modalities reasoning to obtain the final multi-step reasoning paths.

heading: 3.3 MCoT Domain Augmentation
text: In order to make up for the missing data in previous work on mathematics and commonsense, we constructed M3CoT based on MATH (Hendrycks et al., 2021) and Sherlock (Hessel et al., 2022) dataset to enhance the dataset within respective domains. More details are illustrated in Appendix A.4.2. Mathematics Domain Augmentation: It is worth noting that MATH (Hendrycks et al., 2021) is a single modal dataset only with textual questions, rationales and answers, lacking corresponding options and images. To construct the options, we first prompt LLM to generate the related and similar options. Then, for lack of images, we further convert the geometry code and formula code into images, and use HTML framework to splice them together. Commonsense Domain Augmentation: In order to expand the field of commonsense, we use the Sherlock (Hessel et al., 2022), which only contains some visual clues and does not contain any specific questions, options, and answers. Therefore, following Zhang et al. (2023b), we require LLM to cautiously generate questions, options, and answers. Specifically, we incorporate the multiple visual clues in Sherlock to LLM and enforce LLM generates models based on multiple image clues instead of single ones to ensure multi-step multimodal reasoning.

heading: 3.4 Quality Assurance
text: This section aims to improve annotated data quality. More details are shown in Appendix A.5. Onboarding Test: Annotators are all required to undergo a preliminary test, annotating 100 samples. Their results are evaluated by three experts, and only those achieving at least 80% accuracy proceed to subsequent annotation tasks. Human Annotation: To address potential hallucinations or logical errors in generated samples,
annotators are first asked to review and refine the rationale, ensuring accuracy and coherence. Human Recheck: After that, these annotators are required to recheck all data twice to determine if the data meets multi-step multi-modal reasoning criteria and possesses a coherent logical rationale. All samples in M3CoT for which at least two annotators agree can be accepted. The kappa coefficient between annotators achieves 0.85, which indicates perfect agreement (Landis and Koch, 1977).

heading: 4 Data Analysis
text: This section provides some detailed data analysis to better understand M3CoT. Basic statistics M3CoT is partitioned randomly into three subsets: train, validation, and test splits, containing 7,863, 1,108, and 2,358 samples, respectively. Compared to ScienceQA, M3CoT demands more intricate reasoning, with an average length of 294, much higher than ScienceQA’s 48. Multi-modal diversity As shown in Figure 4 (a), M3CoT features diverse image types, which categorized by CLIP (Radford et al., 2021). Furthermore, Figure 4 (b) demonstrates that M3CoT spans

heading: Biology

heading: Adaptations Genes to traits

heading: Ecological interactions Ecosystems

heading: Classification Scientific names

heading: Physics

heading: Chemistry

heading: Chemical reactions

heading: Concentration Calculation Concentration Comparison
text: the data we selected from ScienceQA.
a broader semantic space, suggesting the enhanced semantic richness and coverage. Rationale diversity In comparison to existing benchmarks, the rationale process in M3CoT is characterized by an increased proportion of steps that are more uniformly distributed, as shown in Figure 5. Specifically, ScienceQA averages 2.5 steps, OKVQA 3.0, MMMU only 1.0, and VCR only 1.0. In M3CoT, the reasoning process involves a significantly higher average of 10.9 steps, highlighting the complexity and challenges presented by M3CoT. Domain Diversity In M3CoT, questions are categorized into three primary domains: science knowledge, mathematics and commonsense. As illustrated in Figure 6, the dataset encompasses 17 topics and 263 categories, highlighting the extensive variety of the questions. This variety is essential for assessing the generalization abilities of various models and for furthering multi-modal research.

heading: 5 Experiments

heading: 5.1 Experiments Setting
text: We evaluate various VLLMs in M3CoT, including Kosmos-2 (Peng et al., 2023), InstructBLIP (Dai et al., 2023), LLaVA-V1.5 (Liu et al., 2023), CogVLM (Wang et al., 2023c), Gemini (Google, 2023), GPT4V (OpenAI, 2023). In addition, we explore some prompting strategies. Specifically, we utilize Direct approach to submitting samples in the VLLMs required format; CoT (Ko-
jima et al., 2022) with “Let’s think step-by-step!”; Desp-CoT (Wu et al., 2023b) with an initial image description prompting; CCoT (Mitra et al., 2023) with better description in graph format. Following the settings of Kojima et al. (2022); Qin et al. (2023), we extract the final generated answer through regular expressions.
5.2 Results for M3CoT
Results are presented in Table 1. We have the following observations:
There remains a significant disparity between open source VLLMs and GPT4V. Open source VLLMs still lag behind GPT4V by at least 7.98% on the M3CoT benchmark. It highlights the limitations in the interaction and reasoning capabilities of existing open-source VLLMs, when compared to GPT-4V, especially in advanced tasks.
It exhibits a significant gap between GPT4V and human. Despite GPT4V’s impressive results, it substantially trails human performance, demonstrating GPT4V still struggles to M3CoT.
Zero-shot Multi-modal Chain-of-Thought only benefits larger VLLMs. As shown in Table 1 and Table 3, MCoT strategy fails to enhance reasoning abilities in VLLM with fewer than 13B parameters. Therefore, larger VLLMs (≥ 13B) can better observe emergent capabilities.

heading: 5.3 Analysis
text: To gain a deeper understanding of why VLLMs fail on M3CoT, we analyze various factors to explore what influence the performance on M3CoT. We provide more analysis details in Appendix B.3.1 to confirm our speculations.
Multi-step MCoT poses a greater challenge than single-step one. As shown in Figure 7 (a), VLLM has achieved amazing performance in single-step reasoning. However, compared with single-step MCoT data in ScienceQA, multi-step MCoT data in M3CoT maintains at least a 29.06% performance decrease (Figure 7 (a)). In order to further understand the difference in model reasoning with different numbers of steps, we calculated the accuracy of different steps. As illustrated in Figure 7 (b), an increase in the number of reasoning steps is associated with a significant decline in the model’s performance. In Figure 7 (c), minimal rationale semantic distribution overlap between datasets further proves that the multi-step MCoT is an Out-of-Distribution (OOD) problem compared with single-step MCoT. For all, we attribute the low performance to the multi-step complexities for M3CoT. Multi-step MCoT needs higher rationale quality for better performance. We comprehensively assess the predicted rationale quality of various VLLMs based on five dimension criteria. As shown in Figure 8, we observe that rationale quality incrementally improves M3CoT performance, while it markedly impacts the accuracy in CoT tasks. In the future, we believe that improving rational quality is one of the key challenges to solving M3CoT. Multi-step MCoT needs more multi-modal interaction. To assess the necessity for more complex multi-modal interaction reasoning in M3CoT, we examine how multi-modal interaction degrees impact performance. Specifically, we measure this by defining the similarity between images and reasoning steps to judge which steps are related to the image, identifying steps to sufficient similarity as
multi-modal interaction steps. Figure 9 illustrates a positive correlation between averaged multi-modal interaction steps and reasoning performance, indicating M3CoT benefits from more multi-modal reasoning steps for optimal performance.

heading: 5.4 Exploration
text: In addition to the zero-shot CoT evaluation, we further evaluate the models on M3CoT under three setups: (1) Multi-modal Tool Usage; (2) Multimodal In-Context-Learning; (3) Fine-tuning. To our knowledge, these are the first comprehensive exploration of the multi-modal CoT scenarios.

heading: 5.4.1 Tool Usage Exploration
text: Multi-modal tool usage on text-modal LLM fails on M3CoT. Several studies highlight that ChatGPT can well use external multi-modal tools to help multi-modal reasoning. However, Table 2 reveals that those tool-usage models in single modality are significantly worse than that of GPT4V by
28.21%, and some even are worse than random baseline. We attribute it to the fact that the current tool usage framework cannot observe the visual modal during planning, which caused incorrect tool planning and tool usage, like confusing description and captioning tools. (as shown in Appendix B.3.2) This indicates the necessity for enhanced multimodal information interaction within M3CoT. And we will show more implementation details in Appendix B.2.1.

heading: 5.4.2 In-Context-Learning Exploration
text: Performance can not be boosted by text-only examples. Contrasting with textual CoT (Wei et al., 2022b; Shi et al., 2022), we find that ICL, even with chosen in-domain examples, fails to significantly improve multi-modal reasoning, as shown in Figure 10 (a). This suggests a need for more diverse multi-modal examples for M3CoT. Performance may even be harmed by image and text interleaving example. In Figure 10 (b), it reveals that LLaVA-13B, untrained in interleaved image-text data, suffers performance degradation with more samples. Surprisingly, despite be-
ing trained on interleaved image-text data, OpenFlamingo (Awadalla et al., 2023) still exhibits a slight performance decline. In contrast, GPT4V, which is thoroughly trained on high-quality imagetext interleaving examples, improves the performance as the number of shots increases. However, its performance is still lower than direct CoT. These indicate the future direction of high-quality interleaved samples and multi-step cross-modal interaction to enhance performance. All implementation details are shown in Appendix B.2.2.

heading: 5.4.3 Finetuning Exploration
text: To further explore the improvement on M3CoT, we conduct finetuning experiments for more effective multi-modal reasoning. We will show more implementation details in Appendix B.2.3. Finetuning on M3CoT can result better performance. Table 2 reveals that our benchmark training set significantly enhances model performance. It enables traditional vision-language models (VLMs) to surpass the zero-shot VLLMs, which is the value of our dataset in boosting VLM effectiveness. Finetuned VLMs (the lowest is 44.85%) outperform most open-source VLLMs with zero-shot prompting (the highest is 38.86%). In addition, some finetuned VLMs have even surpassed Gemini’s overall
accuracy of 47.50%, which demonstrates that finetuning can effectively boost the performance. Finetuning on VLLMs tends to be more effective than on Traditional VLM. Further, we found that the performance of VLLMs generally improves as their number of parameters increases. This also proves the importance of utilizing models with sufficient parameters in our M3CoT to achieve the target performance.
Takeaways: (1) Visual and textual information should be both considered for tool planning. (2) We should consider better multimodal interleaving for better ICL in M3CoT. (3) Fine-tuning has better hope on multi-step MCoT, compared with the failures of vanilla in-context-learning, tool usage, and prompting strategies.

heading: 6 Related Work
text: Chain-of-Thought (Wei et al., 2022b) (CoT) is a highly effective step-by-step strategy for enhancing zero-shot and few-shot reasoning in Large Language Models (LLMs) (Kojima et al., 2022; Zhou et al., 2022; Zelikman et al., 2022; Qin et al., 2023, 2024a; Zhuang et al., 2023). In addition, some works begin to extend the textual CoT capabilities to multi-modal CoT reasoning (MCoT) (Wang et al., 2023d; Singh et al., 2023; He et al., 2023). To this end, Lu et al. (2022a) introduce the ScienceQA benchmark, laying the foundation for MCoT. Subsequently, Zhang et al. (2023c) formally formalize the MCoT concept and enhanced its performance using a two-stage approach during multi-modal reasoning. Additionally, Wang et al. (2023a) develop a novel framework to integrate more knowledge with high-quality CoT rationales from larger LLMs for better MCoT reasoning. Further, Mondal et al. (2024) integrate CoT, Knowledge Graphs, and multi-modalities together for better MCoT. Ge et al. (2023); Zheng et al. (2023); Yao et al. (2023) manually decouple the chain-of-thought reasoning steps, integrating better multi-modal interaction. Building upon these works, Tan et al. (2023) introduce the self-consistency mechanism (Wang et al., 2022) into the training process to enable more accurate/reliable reasoning. Wei et al. (2023) propose a novel approach to improve reasoning capabilities in image and text encoders through the integration of multi-hop cross-modal attention and sentence-level contrastive learning. Chen et al. (2023) further extend the MCoT benchmark to generation tasks for
better commonsense reasoning evaluation. Compared with previous works, we first propose M3CoT to explore the multi-step MCoT and extend their application across a broader range of domains. In addition, we conduct comprehensive experiments on M3CoT and highlight some takeaways, facilitating future research.

heading: 7 Conclusion
text: In this work, we introduce a novel benchmark (M3CoT), toward multi-domain, multi-step, and multi-modal chain-of-thought scenarios, which is developed through a detailed and comprehensive process. In addition, we conduct a comprehensive analysis involving abundant multi-modal CoT methodologies to understand the limitation of existing frameworks on M3CoT. We sincerely aspire that our work can reassess existing advancements and inspire future research by highlighting new challenges and opportunities.

heading: Limitations
text: We introduce a new benchmark for multi-domain, multi-step, and multi-modal Chain of Thought (M3CoT) reasoning, performing in-depth analysis and exploring various CoT methodologies to better understand existing frameworks. However, limited by unavoidable human subjectivity, manual annotation may introduce potential biases that may affect the reliability of the data. Furthermore, with
the advent of globalization, multilingualism has become increasingly important (Qin et al., 2024b). However, due to regional and cost restrictions, the data does not take into account multilingual backgrounds and was developed only in the single language of English. Moreover, due to the possibility of discontinued or retired models, we will pay more attention to the open-source models in the future.

heading: Ethical Considerations
text: Data Access We sourced our data from the ScienceQA (Lu et al., 2022b), MATH (Hendrycks et al., 2021), TabMWP (Lu et al., 2022c), KiloGram (Ji et al., 2022), and Sherlock (Hessel et al., 2022). These are open-source and freely available for academic research, aligning with the commitment to ethical data use.
Participant Recruitment We recruit participants from universities and require all participants to pass the CET-6 exam or IELTS score of 6 or above. In addition, all participants come from all over China and may have some national biases. We blurred national differences in the data set as much as possible, limiting it to common human commonsense. All annotators gave informed consent and were compensated in excess of the local minimum wage. In addition, the site does not require IRB review.
Dataset Collection Process Our annotation process begins with an onboarding test, introducing the
task through 100 example questions. Participants are compensated $20 for this initial phase, aimed at acquainting them with the task. Subsequently, annotators receive $15 per hour for their contributions, accumulating approximately 450 humanhours for manual annotations. Following this, a recheck process to ensure correct labeling is added an additional 60 hours of work. Overall, six experts and three students are engaged to fulfill the annotation and recheck tasks.

heading: Acknowledgments
text: This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62306342, 62236004 and 62441603. This work was also sponsored by CCF-Baidu Open Fund and Excellent Young Scientists Fund in Hunan Province (2024JJ4070). We are grateful for resources from the High Performance Computing Center of Central South University.

heading: Appendix

heading: A Dataset Annotation Details

heading: A.1 Statistical Analysis of Existing Datasets
text: In this study, we conduct a comprehensive analysis of the prevalence of multi-step reasoning within existing datasets. The analysis focuses on the "MMCoT" column in Figure 2(b), which represents the proportion of multi-step multi-modal CoT (Chain of Thought) data. This examination is critical as it reveals a significant deficiency in the current datasets regarding multi-step reasoning capabilities. Our findings indicate that at least 79% of the data across all the benchmarks we examined lack sufficient multi-step reasoning elements. This highlights a pervasive issue in the design and utilization of these datasets, where the absence of complex reasoning processes could impede the development of more sophisticated multi-modal models.
The term %MMCoT, as used in this context, is consistent with the representation in Figure 2(b). Both denote the proportion of multi-step multimodal CoT data. To ensure the accuracy of our analysis, we employed a rigorous sampling method. We selected a stratified random sample consisting of 20% of the dataset, evenly distributed, for manual inspection. This approach allowed us to precisely determine the proportion of multi-step multi-modal CoT data within the existing datasets.
The manual inspection involved a detailed review of each sampled data point to identify and categorize the presence of multi-step reasoning. This meticulous process ensured that our findings were not only statistically significant but also reflective of the true nature of the datasets. The results underscore the necessity for enhanced dataset designs that incorporate more multi-step reasoning tasks, thereby facilitating the development of advanced multi-modal models capable of handling complex reasoning scenarios.
In conclusion, our statistical analysis sheds light on a critical gap in existing datasets, emphasizing the need for more comprehensive data that can better support the advancement of multi-modal reasoning capabilities.

heading: A.2 The Details of Absence of Visual Modal Reasoning Sample Removal
text: We develop the annotation interface based on the open-source Gradio framework. We segment the dataset, distribute the scripts, and deploy them to lo-
cal computers. In addition, we have designed some manual guidelines for annotators. These guidelines need to be followed by annotation experts. Specifically, our method flow is as follows:
1. We will first mask the image so that the expert can only see the text modal questions, options, rationale, and answers.
2. The expert needs to directly judge whether the question can directly infer the rationale. If it cannot be inferred, it may be necessary for the visual modal information for the rationale generation.
3. And then we ask the experts to check the image to confirm this.
For each step, the guideline instructions are as follows:
[Instruction1] Firstly, we will conceal all visual content, allowing you to concentrate exclusively on the textual elements, including the questions, available options, correct answers, and rationales. You need to follow these instructions for annotation:
• You should determine whether the text alone can convey the full context and solution logic of the question.
• You should evaluate whether the textual information is sufficient for a comprehensive understanding and rationale formulation.
[Instruction2] In cases where the text-based evaluation indicates a need for visual information, please click the “Image Display” button. Then, the image will be reintroduced for further recheck, confirming the requirement for a multi-modal approach in certain scenarios.
We used three experts to conduct majority voting to judge this matter. This part removes at least 30% of ScienceQA’s multi-modal sample, which also illustrates the limitations of the existing data.

heading: A.3 The Details of Multi-step MCoT Sample Construction
text: Automatic Sample Removal: In this step, we automatically filter out samples with overly simplistic rationales comprising fewer than two steps. This
process reduces the manual annotation burden and enhances the reliability of M3CoT. Since the rationale in ScienceQA includes at least one conclusion and one step of reasoning, samples with fewer than two steps indicate that multiple visual cues were not used for MMCoT reasoning. Thus, this filtering step minimizes annotation workload and costs. Notably, samples with multi-step and single-step reasoning still require manual evaluation in our study.

heading: A.4 Domain Augmentation Details

heading: A.4.1 Mathematics Domain Augmentation Details
text: Firstly, the MATH (Hendrycks et al., 2021) dataset consists solely of textual mathematical questions, complete with detailed rationales and answers. However, it does not include multiple-choice options and illustrative images, limiting its utility in M3CoT.
To address this, we employ gpt-3.5-turbo to generate relevant and similar multiple-choice options for each question, enhancing the dataset’s versatility. Specifically, for the option generation phase, we prompt the LLM with specific questions from the dataset, followed by instructions to generate four plausible options, one correct and three distractors. The prompt included guidelines for ensuring the options are closely related to the question’s content, challenging yet not misleading. Specifically, the prompt is defined as follows:
[Input] Here is the mathematical question: <Context, Question, Answer, Option>
[Instruction] Generate four multiple-choice options: one correct answer and three plausible but incorrect distractors. Ensure the distractors are relevant and challenging without being misleading. The options should closely relate to the question’s subject matter and provide a meaningful test of the reader’s understanding.
To compensate for the absence of visual content, we developed a method to translate mathematical expressions and geometric figures described in the questions into visual representations. This process involved generating PNG images from the mathematical and geometric codes. Subsequently, we used a combination of HTML and CSS to integrate these images with the textual content, creating a cohesive multi-modal dataset.

heading: A.4.2 Commonsense Domain Augmentation Details
text: In our study, we aim to enhance commonsense reasoning domain for M3CoT by leveraging visual clues from the Sherlock (Hessel et al., 2022) dataset. Unlike traditional vision question answering datasets, Sherlock (Hessel et al., 2022) does not include predefined questions, options, or answers, focusing instead on visual information to stimulate inference and deduction.
Recent advancements in LLMs have showcased their potential in generating diverse and contextually relevant data (Zhang et al., 2023b). Building on this, our approach involves prompting LLMs with visual clues from Sherlock, tasking them to generate coherent and contextually appropriate questions, multiple-choice options, and corresponding
answers. This process demands careful design to ensure the prompts effectively communicate the visual information and desired output format to the LLM.
To ensure comprehensive multi-step, multimodal reasoning, we develop a prompting methodology to trigger LLMs to consider multiple visual clues simultaneously. Specifically, our experimental setup includes detailed prompting strategies that describe the visual clues from a structured manner to natural language description, allowing the LLM to understand and interpret the information accurately. The prompt is defined as follows:
[Instruction] Given the visual clues, generate a question that requires commonsense reasoning to answer. Then, provide four options (A, B, C, D), one of which is the correct answer, and the others are plausible distractors.
[Highlighting] Ensure that the question and options leverage insights from at least two visual clues for deeper multi-modal interactions.
<One-shot Example> [Visual Clue] <multiple detailed description of the visual clues>
where [One-shot Example] denotes we used oneshot in-context-learning to allow the model to better learn the generation of related samples.
In addition, for some topics in ScienceQA where the data is too sparse due to the last “absence of multi-modal reasoning sample removal” and “multi-step multi-modal sample construction”, we used a similar method to synthesize data in ScienceQA or used other open source data sets, like TabMWP (Lu et al., 2022c), KiloGram (Ji et al., 2022), for data augmentation. Moreover, we also manually synthesized some geographical images through Matplotlib, constructed some samples using rules, and polished and modified them using ChatGPT.

heading: A.5 Quality Assurance Details
text: Due to space limitations, we only describe the rationale rewriting (§A.5.3) section in the appendix. This part is mainly to improve the quality of the rationale data set.

heading: A.5.1 Human Annotation Details
text: In order to better mark the correctness of the logical chain of reasoning, first, we divide the steps according to the ROSCOE (Golovneva et al., 2023) settings to obtain a clearer step-by-step rationale visualization. Secondly, we provide the corresponding sample image, question, options, answer and step-segmented rationale for experts to annotate each time. During annotation, we allow experts to discard samples that are of poor quality and cannot be modified to ensure the quality of the data set.

heading: A.5.2 Human Recheck Details
text: In assessing the capability of a given sample to fulfill the criteria for multi-step multi-modal reasoning, this process employs a structured approach. Initially, we decompose the reasoning rationale into discrete steps by ROSCOE (Golovneva et al., 2023). Following this segmentation, experts are required to focus more on ascertaining which step needs image modality in rationale. After that, it requires further verification that the sample has considered integration of image modality for a minimum of two distinct steps. This methodological framework ensures a thorough recheck of the sample with the specified requirements of multi-step multi-modal reasoning. Specifically, the instructions given by our experts are as follows:
[Input] Here is an example: <EXAMPLE>
[Instruction] You need to judge whether a given sample meets the requirements of multi-step multimodal reasoning.
1. First, we have broken down the steps for you.
2. Secondly, please determine which steps in rationale require image modality.
3. Finally, please confirm whether the sample requires image modality for at least two steps.
where “[EXAMPLE]” represents an example containing an image, question, options, answer, stepsegmented rationale and annotation detail information.
Furthermore, the human recheck process actually has two rounds. In the second round, the sample discard rate is less than 5%.

heading: A.5.3 Rationale Rewriting
text: The rationale quality within the ScienceQA dataset has been found poor expression, with some explanations not adequately addressing the posed questions. To mitigate this issue, we have employed gpt-3.5-turbo to perform rationale rewriting, aiming to elevate the overall quality of M3CoT before human annotation. Specifically, to achieve this, we designed a specific prompting strategy for the LLM, which prompt is defined as follows:
[Instruction] Improve the quality of ScienceQA dataset rationales by rewriting them for enhanced relevance, accuracy, and clarity.
1. Read the question and the provided rationale from the ScienceQA dataset.
2. Evaluate the existing rationale for its relevance and accuracy in answering the question.
3. Rewrite the rationale to better answer the question, ensuring the new version is clear, concise, and directly related to the question’s core topic. Maintain scientific accuracy and use accessible language suitable for the intended audience.
[One-shot Example] Question: <Question> Original Rationale: <Rationale> Rewritten Rationale: <Rewritten Rationale>
[Input] Question: <Question> Original Rationale: <Rationale> Rewritten Rationale:
This approach ensures that the rewritten rationales are not only relevant but also adhere to a high standard of clarity and coherence. Each rationale is assessed both automatically and manually to confirm its relevance and quality improvement over the original version.
A.6 Image Redundancy Removal
Additionally, we observed a significant number of highly similar samples in the ScienceQA. To reduce redundancy and maintain diversity for image, we remove samples where the questions are identical, and the grayscale image similarity exceeded 99%.

heading: B Experiment Details

heading: B.1 Main Result Details
text: Due to space limitations, we only show some of the LLM test results in the main table. The specific experimental results are shown in Table 3.

heading: B.1.1 Heuristic baselines
text: This study employs two heuristic baselines. The first is a random selection method, where an answer is chosen randomly, with its accuracy determined by averaging three random seeds. The second baseline evaluates human performance through participants who must successfully complete preliminary qualification tasks. The “Human” accuracy is the average accuracy achieved by three participants.

heading: B.2 Exploration Details

heading: B.2.1 Tool Usage Details
text: Model Selection In this section, we introduce a suite of tool-augmented LLMs, including HuggingGPT (Shen et al., 2023), VisualChatGPT (Wu et al., 2023a), IdealGPT (You et al., 2023), and Chameleon (Lu et al., 2023b). Specifically, VisualChatGPT (Wu et al., 2023a) and IdealGPT (You et al., 2023) are engineered to tackle complex issues through iterative problem-solving processes. Conversely, HuggingGPT (Shen et al., 2023), and Chameleon (Lu et al., 2023b) employ LLMs to decompose complicated challenges into a series of manageable sub-problems, addressing them in a sequential manner. This array of approaches highlights the diverse capabilities and potential of LLMaided visual reasoning systems in executing sophisticated problem-solving strategies.
B.2.2 In-Context-Learning Details Model Selection In this section, we explore three notable models: LLaVA-V1.5-13B (Liu et al., 2023), OpenFlamingo-7B (Awadalla et al., 2023), and GPT4V (OpenAI, 2023), each demonstrating unique capabilities in the context of In-Context Learning (ICL). LLaVA-V1.5-13B (Liu et al., 2023) is built upon a foundation of instructionfollowing data of high quality without any specific image-text interleaving training. OpenFlamingo (Awadalla et al., 2023) is a VLLM, optimized for tasks involving complex image-text interleaving sequences. GPT4V (OpenAI, 2023) is the state-of-the-art VLLM that can learn efficiently from limited image-text interleaving demonstrations. Therefore, we assume it is well-trained in
image-text interleaving scenarios.
Exemplar Selection In order to complete indomain sample selection as much as possible, we only randomly select samples under the same categories from the development set.

heading: B.2.3 Finetuning Details
text: Model Selection Our fine-tuning section incorporates a carefully curated selection of models, which includes a series of traditional Vision-Language Models (VLMs) and Vision Large Language Models (VLLMs). Specifically, VLMs contain MMCoT (Zhang et al., 2023c), MC-CoT (Tan et al., 2023), and MMR (Wei et al., 2023). VLLM include LLaMA-Adapter (Zhang et al., 2023a), LLaVAV1.5 (Liu et al., 2023), CogVLM (Wang et al., 2023c). This selection was strategically made to encompass a wide range of parameter sizes, architectures, and functionalities. This diversity ensures a comprehensive evaluation of the state-of-the-art in multi-modal capability on finetuning settings.
Experiment Setting In the context of finetuning VLLMs, parameter efficient tuning always achieves better performance compared with fullparameter tuning and offers a training cost reduction (Hu et al., 2021; Liu et al., 2023). To leverage these benefits, we employ LoRA (Hu et al., 2021) for parameter-efficient tuning across our experiments. However, for specific cases like the LLaMA-Adapter, we integrate a compact adapter module for fine-tuning, adding minimal additional parameters to the model’s architecture.
Our training configurations include a selection of batch sizes from {2, 4, 8} and learning rates ranging from [1e− 6, 8e− 5]. We standardize the maximum sequence length to 512 tokens for uniformity across all model trainings. The experiments are conducted on NVIDIA A100 and A800 GPUs to ensure optimal performance and efficiency. For all experiments, model selection is based on the best performance on the development set, which is then validated on the test set for final evaluation.

heading: B.3 Error Analysis

heading: B.3.1 Zero-shot Chain-of-Thought Error Analysis
text: In order to further analyze the typical errors in the data set, we analyzed the cases of GPT4V on different domains (as shown in Figure 12, Figure 13, and Figure 14). We can find that all cases have logical errors and visual information interaction errors
or deficiencies. This view is also consistent with Section 5.3 mutual confirmation. Therefore, we believe that the lack of high-quality logical reasoning capabilities and complex multi-modal interactions of large models leads to the failure of multi-step multi-modal reasoning of the model.

heading: B.3.2 Tool Usage Error Analysis
text: In the context of tool-usage methodologies, an initial challenge emerges from the image information mistake and logical mistake, as highlighted in Appendix B.3.1. This limitation becomes particularly problematic in complex scenarios involving multiple tools and steps, as seen in multi-modal task planning. These scenarios demand precise tool selection and sequencing; however, the lack of visual interaction during tool planning leads to frequent errors in tool selection (as shown in Figure 16) and tool-chain redundancy(as shown in Figure 15). Incorrect tool planning or selection can cascade through the process, culminating in complete failure of the intended task (as shown in Figure 16). This issue underscores the need for enhanced model capabilities in processing and integrating visual modalities to accurately navigate multi-step, multi-tool workflows on M3CoT.
GPT4V CoT Alright, let's break this problem down. Step 1: From the figure: AB is the diameter of circle O. This means ∠AOB is 180° because it's a straight angle. Given: ∠CAB = 40° Step 2: Recall a property of a circle: The angle subtended by a semicircle (an angle subtended by the diameter) is always a right angle. This means: ∠CBA = 90° Step 3: From the above information: ∠CAB + ∠CBA + ∠CBO = 180° [Image Information Mistake] Given ∠CAB = 40°, and we know ∠CBA = 90° So, ∠CBO = 180° - 90° - 40° = 50° Step 4: [Image Information Mistake / Logical Mistake] Now, note that in circle O, the arc CD subtends both ∠CAB and ∠ADC at the circumference, making them angles subtended by the same arc. A property of angles subtended by the same arc is: Angle subtended by an arc at the center is double the angle subtended by it at any point on the remaining part of the circle. This means: 2 * ∠CAB = ∠CBO 2 * 40° = 80° But, ∠CBO = 50° (as we've calculated above) Thus, the given information is inconsistent. Given our calculations: (A) 25° is incorrect. (B) 30° is incorrect. (C) 45° is incorrect. (D) 50° is incorrect. Based on the provided information and the options, the correct answer is: (C) All of the answer choices given in the text are wrong.
Question:
Let's analyze the options given for the Geometry problem in the image.
Options:
(A) the answer is option A in the image
(B) the answer is option D in the image
(C) All of the answer choices given in the text are wrong.
(D) the correct choice is option C in the image.
(b) Response from GPT4V with CoT prompting on mathematics domain.Figure 13: Response from GPT4V with CoT prompting on mathematics domain.

heading: Future Work
text: Moreover, due to the possibility of discontinued or retired models, we will pay more attention to the open-source models in the future