Multimodal Chain-of-Thought Reasoning in Language Models
Authors: Zhuosheng Zhang
Aston Zhang
Mu Li
hai zhao
George Karypis
Alex Smola
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.

Published in Transactions on Machine Learning Research (05/2024)
Multimodal Chain-of-Thoughi Reasoning in Language Models
Zhuosheng Zhang
shongss@xftu.edu.cm
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University
Aston Zhang*
a=@ustonzhang, com
CenAI, Meta
Mu Li Amazon Web Services
muli@cc.. cmau. edu
Hai Zhao
Sauohai@@es.ejtu.ddu..o
Deportment of Computer Science and Engineering, Shanghai Jiao Tong University
George Karypis
Sltrrpps@@mmzoon..oo com
Amozon Web Services
Alcx Smola Amason Web Services
alen@smola.cm org
Reviewed on OpenReview: https:// openreview. net/ forum? id=yIpPWEVfuR
Abstract
Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, cxisting CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (lext) and vision (images) modalities into two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal- CoT, our model under 1 billion parameters achieves seate-of-the-ass performance on the ScienceQA benchmark. Our analysis indicates that Multimodal CoTo offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https: ttops://gtt. thub. com/ www.aaamsceencceeeeeaaaaaaaaaaee
1 Introduction
Imagine reading textbook with no figures or tables. Our ability to knowledge acquisition is greatly strengthened by jointly modeling diverse data modalities, such as vision, language, and audio. Recently, large language models (LLMs) (Brwwn et al., 2020 Thoppilan et al. 2022 Rae et al. 2021 Chowdhery et al.) have shown impressive performance in complex reasoning by generating intermediate reasoning steps before inferring the answer. The intriguing technique called chain-of-thought (CoT) reasoning (Wei et al.
2022
2022b
Kojima al. 2022 Zhang el. al. 2023d
* Work done Amazon Web Services. Correspondence to: Zhuosheng Zhang and Aston Zhang.

---

Published in Transactions on Machine Learning Research (05/2024)
However, existing studies related to CoT reasoning are largely isolated in the language modality (Wang
Input
Language
Vision
let al. 2022c; "hou et al.) 2022 Lu et al. 2022b Fu
Will these magnets attract or repel each ofher??
Question:
itt al.
2022). with little considcration of multimodal
Context:
Two magnets are placed as shown. Hint:
Magnets that attract pull logether. Magnels that repel push apert.
scenarios. To elicit. CoT reasoning in multimodalityy
we advocate a Multimodal-CoT paradigm. Given the inputs in different modalities, Multimodal-Cof decomposes multi-step problems into intermediate reasoning steps (rationale) and then infers the an- swer. Since vision and language are the most popular modalities, we focus on those two modalities in this work. An example is shown in Figure !I
Options: (A) attract
(B) repel
Output
Rationale: these megnites attract regel? To find out, look which poles closest to each other. The north pole of one magnett closest the south pole of the other megnet. Poles that are different, attract, So, these magnets will attract each other. Answer: The answer is (A).
Figure 1: Example of the multimodal CoT task.
In general, Multimodal CoT reasoning can be elicited through two primary paradigns: (i) prompting LLMs and (ii) fine-tuning smaller models challenges as follows.
We will delve into these paradigms and delineate their associated
The most immediate way to perform Multimodal-CoT is to transform the input of different modalities into a unified modality and prompt LLMs to perform CoT [Zhang et al. 2023a Lu et al. 2023 Liu et al. 2023
Alayrac et al. 2022 Hao et al. 2022 Yasunaga et al. 2022j. For example, it is possible to generate a caption for an image by captioning model and then concatenate the caption with the original language input to be fed into LLMs (Lu al. 2022a). The development of large multimodal models such as GPT-4V (OpenAI, 2023) and Gemini (Reid et al. 2024) has notably enhanced the quality of generated captions, resulting in finer-grained and more detailed descriptions. However, the captioning process still incurs significant information loss when transforming vision signals into textual descriptions. Consequently, using image captions rather than vision features may suffer from a lack of mutual synergy in the representation space of different modalitics. In addition, LLMs either have paywalls or resource-consuming to deploy locally.
To facilitate the interaction between modalities, another potential solution is to fine-tune smaller language models (LMs) by fusing multimodal features (Zhang et al. 2023c Zhao et al., 2023). As this approach allows the flexibility of adjusting model architectures to incorporate multimodal features, we study fine-tuning
models in this work instead of prompting LLMs. The key challenge is that language models under 100
billion parameters tend to generate hallucinated rationales that mislead the answer inference (Ho et al. 2022
Magister et al. 2022 Ji et al. 2022: Zhaagg et al. 2023b).
To mitigate the challenge of hallucination, we propose Multimodal-Cof that incorporates language (Lext) and vision (images) modalities into a two-stage framework that scparates rationale generation and answer inference P
In this way, answer inference can leverage better generated rationales that are based on multimodal information. Our experiments were conducted on the ScienceQA (Lu et al. 2022a) and A-OKVQA (Schwenk et al. 2022 datasots, which are the latest multimodal reasoning benchmarks with annotated reasoning chains.
Our method achieves state-of-the-art performance on the ScienceQA benchmark upon the release. We find that Multimodal-Cof is beneficial in mitigating hallucination and boosting convergence. Our contributions are summarized as follows:
(i) To the best of our knowledge, this work is the first to study CoT reasoning in different modalities in scientific peer-reviewed literature.
(ii) We propose two-stage framework by fine-tuning language models to fuse vision and language representa- tions to perform Multimodal-Coff. The model is able to generate informative rationales to facilitate inferring final answers.
(iii) We elicit the analysis of why the naive way of employing CoT fails in the context and how incorporating vision features alleviates the problem. The approach has been shown to be generally effective across tasks and backbone models.
We refer to small models models with less than billion parameters (hereinafter dubbed as IB-models).
2This work focuses on the language and vision modalities.
2

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Representative CoT techniques (FT: fine-tuning; KD: knowledge distillation). Segment in-context. learning techniques; Segment 2: fine-tuning techniques. To the best of our knowledge, our work is the first to study CoT reasoning in different modalities in scientific peer-reviewed literature. Besides, we focus on IB- IB-models, without relying the outputs of LLMs.
Models
Mutimodal Model / Engine Training CoT Role
CoT Source
Cero-Shot-Cof Kojima al. 2022
GPT-3.5 (175B)
ICL
Reasoning
Template
Few-Shot-CoT Wei ct al. 20226
X
PaLM (540B)
ICL
Reasoning
Hand-crafted
Self- Solf-Consistency-Coo Wang cl al.
2022b
X
Codex (175B)
ICL
Reasoning
Hand-crafted
Least-to-Most Prompling Zhou
2022
Codex (175B)
ICL
Reasoning
Hand-crafted
Retrieval-CoT (Zhang et a...
2023d
X
GPT-3.5 (175B)
ICL
Reasoning
Auto-generated
PromptPG-CoT" [Lu et al_
2022b
X
GPT-3.5 (175B)
ICL
Reasoning
Hand-crafted
Auto-CoT Zhang et al.
2023d
X
Codex (175B)
ICL
Reasoning
Auto-generated
Comploxity-CoT Fu ot 2022
GPT-3.5 (175B)
ICL
Reasoning
Hand-crafted
Fow-Shot-PoT Chen ot al. 2022
X
GPT-3.5 (175B)
ICL
Reasoning
Hand-crafted
UnifiedQA [L et al.
2022a
T5 (770M)
FT
Explanation Crawled
Fine- Tono-TTnedTn T5 XXL
(Magister et al.
2022
X
T5 (11B)
KD
Reasoning LLM-generated
Fine-TTuee-CoT (Ho et al.
2022
X
GPT-3 (6.7H)
KD
Reasoning L.
I.LM-gemerated
Multimodal-Col (our work)
T5 (770M)
FT
Reasoning
Crawled
2
Background
This section reviews studies eliciting CoT reasoning by prompting and fine-tuning language models.
2.1 CoT Reasoning with LLMs
Recently, CoT has been widely used to elicit the multi-step reasoning abilities of LLMss WWei et al. 2022b Concretely, CoTT' techniques encourage the LLM to generate intermediate reasoning chains for solving a problem. Studies have shown that LLMs can perform CoT reasoning with two major paradigms of techniques: Vero-Shot-CoT (Kojima et al.) 2022 and Few-Shot-CoT (Wei et al. 2022b Zhang et al. 2023d). For Zero-Shot-Cof. Kojima al. (2022) showed that LLMs are decent zcro-shot reasoners by adding a prompt like "Let's think step by step" after the test question to invoke CoT reasoning. For Few-Shot-CoT,, few step-by-step reasoning demonstrations are used as conditions for inference. Each demonstration has commonly obtained by hand-crafting or automatic generation. These two techniques, hand-crafting and automatic generation are thus referred to as Mamual-CTT (Wei et. al., 2022b) and Aulo-CoT (Zhang el. al. 2023d).
question and a reasoning chain that leads to the final answer. The demonstrations
With effective demonstrations, Few-Shot-CoT often achieves stronger performance than Zero-Shot- Shot-CoTTTTTT and has attracted more research interest. Therefore, most recent studics focused on how improve Few-Shot-CoT. Those studies categorized into two major research lines: (i) optimizing the demonstrations; (ii) optimizing the reasoning chains. Table compares typical CoT techniques.
The performance of Few-Shot-CoTT relies on the quality of demonstrations. As reported in Wei et al. (20226), using demonstrations written by different annotators results in dramatic accuracy disparity in reasoning tasks. Beyond hand-crafting the demonstrations, recent studies have investigated ways to optimize the demonstration selection process. Notably, Rubin et al. (002)) retrieved the semantically similar demonstrations with the test instance. However, this approach shows a degraded performance when there are mistakes in the reasoning chains (Zhang et al., 2023d). To address the limitation, Zhang et al. (2023d) found that the key is the diversity of demonstration questions and proposed Auto-CoT: (i) partition questions of given dataset into few clusters; (ii) sample representative question from cach cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics. In addition, reinforcement learning (RL) and complexity-based selection strategies were proposed to obtain effective demonstrations, Fu et al. (2022) chose examples with complex reasoning chains (i.e., with more reasoning steps) as the demonstrations. Lu et al. (2022b) trained an agent to find optimal in-context examples from candidate pool and maximize the prediction rewards on given training examples when interacting with GPT-3.5.
Optimizing Demonstrations
3

---

Published in Transactions on Machine Learning Research (05/2024)
Optimizing Reasoning Chains A notable way to optimize reasoning chains is problem decomposition. Zhou et al. (2022) proposed least-to-most prompting to decompose complex problems into sub-problems and then solve these sub-problems sequentially. As a result, solving a given sub-problem is facilitated by the answers to previously solved sub-problems. Similarly, [hot et al. (2022) used diverse decomposition structures and designed different prompis to answer each sub-question. In addition to prompting the reasoning (PoT), which modeled the reasoning process as program and prompted LLMss to derive the answer by executing the generated programs. Another trend is to vote multiple reasoning paths for test question. WWang et al. (2022b) introduced self-consistency decoding strategy to sample multiple outputs of LLMs and then took majority over the final answers. WWang et al. (002cc) and Li et al. (2022c) introduced randomness in the input space to produce more diverse outpuls for voling.
chains as natural language texts, Chen et al. (2022) proposed program-of-thongg
2.2 Eliciting CoT Reasoning by Fine- Tuning Models
A recent interest is eliciting CoT reasoning by fine-tuning language models. Lu el al. (2022a) fine-tuned the encoder-decoder T5 model on a large-scale dataset with CoT annotations. However, a dramatic performance decline is observed when using CoT to infer the answer, i.e., generating the reasoning chain before the answer (reasoning). Instead, CoT is only used as an explanation after the answer. Magister et al. (2022) and Ho et al. (2022) employed knowledge distillation by fine-tuning student model the chain-of-thought outputs generated by larger teacher model. Waan al. (2022a) proposed an iterative context-aware prompting approach to dynamically synthesize prompts conditioned on the current step's contexts.
There is a key challenge in training 1B-models to be CoT reasoners. As observed by Wei et al. 2002b), models under 100 billion parameters tend to produce illogical CoT that leads to wrong answers. In other words, it might be harder for 1B-models to generale effective CoT than directly generating the answer. It becomes even more challenging in a multimodal setting where answering the question also requires understanding the multimodal inputs. In the following part, will explore the challenge of Multimodal-CoT and investigate how to perform effective multi-step reasoning.
3 Challenge of Multimodal-CoT
Existing studies have suggested that the CoT reasoning ability may emerge in language models at certain scale, e.g., over 100 billion parameters (Wei et al. 2022a). However, it remains an unresolved challenge to elicit such reasoning abilities in 1B-models, let alone the multimodal scenario. This work focuses on iB-models as they can be fine-tuned and deployed with consumer- grade GPUs (c.g., 32G memory). In
this section, we will investigate why 1B-models fail CoT reasoning and study how design an effective approach to overcome the challenge.
3.1 Towards the Role of CoT
To begin with, we fine-tune text-only bascline for CoT reasoning on the ScienceQA benchmark (Lu et al. 2022a We adopt FLAN-Alpacagae as the backbone language model Our task is modeled a text generation problem, where the model takes the textual information as the input and generates the output sequence that consists of the rationale and the answer.
As an example shown in Figure | the model takes the concatenation of tokens of the question text (Q), the context text (C), and multiple options (M) as the input. To study the effect of CoT, we compare the performance with three variants: (i) No-CoT which predicts the answer directly (QCM-A); (ii) Reasoning
Table 2: Effects of COT in the one-stage setting.
Method
Format
Accuracy
No-CoT
QCM-A
81.63
Reasoning
QCM-RR
69.32
Explanation
QCM->AR
69.68
where answer inference is conditioned to the rationale
(QCM-RA):: (iii) Explanation where the rationale is used for explaining the answer inference (QCM-AR).
'httpe://github cow/declare-eeeeeeeeeeeeeeeoeeec lab/flan alpaca 200M model Raffel al. 2020 fine-tuned Stanford Alpaca
data (Taoii al. 2023). Implementation details are presented in Section 5.2
4

---

Published in Transactions on Machine Learning Research (05/2024)
Problem
Question: Will these magnets attract repel each other?
Vision
Context: Two magnets placed shown. Hint: Magnets that attract pull together.
N
N
Magnets that repel push apart.
Options:
(A) attract
(B) repel
Gold Rationale: Will these magnets attract or repel? To out, look at which poles are closest each other. The north pole of one magnet closest the south pole other magnet, Poles that are different attract. So, these magnets attract each other.
Answer: The answer (A).
Baseline
+ Vision Features
Will these magnets attract or repel? To find out, look at which poles are closest to each other. The south pole of one magnet is closest to the south pole of the other magnet. Poles that are the same repel. So, these magnets will repel each other. Answer: The answer is (B).
Generated Rationale:
Generated Rationale: Will these magnets attract or repel? To find out, look at which poles are closest to each other. The north pole of one magnet is closest to the south pole of the other magnet. Poles that are different attract. So, these magnets will attract each other. Answer: The answer is (A).
Figure Example of the two-stage framework without vision features (baseline) and with vision features (ours) for generating rationales and predicting answers. The upper part presents the problem details with gold rationale, and the lower part shows the outputs of the baseline and our method incorporated with vision features. We observe that the baseline fails to predict the right answer due to the mislcading by hallucinated rationales. More examples are shown in Appendix A.1
Surprisingly, as shown in Table 2 observe a 412331% accuracy decrease (81.63%-<6) -69.2%% if the model predicts rationales before answers (QCM-RR). The results imply that the rationales might not necessarily contribute to predicting the right answer. According to Lu et al. (002aa). the plausible reason might be that the model exceeds the maximum token limits before obtaining the required answer or stops generating the prediction early. However, we find that the maximum length of the generated outputs (RA) always less than 400 tokens, which is below the length limit of language models (i.e., 512 in T5 models). Therefore, it deserves a more in-depth investigation into why the rationales harm answer inference.
3.2 Misleading by Hallucinated Rationales
To dive into how the rationales affect the answer pre- diction, we separate the CoT problem into two stages, rationale generation and answer inference 3 We re- port the RougeL score and accuracy for the rationale generation and answer inference, respectively. Table shows the results based on the two-stage framework. Although the two-stage baseline model achieves 90.73 Rougel, score of the rationale generation, the answer inference accuracy is only 78.57%. Compared with the QCM-A variant (81.63%) in Table 2 the result shows that the generated rationale in the two-stage framework does not improve answer accuracy.
Table 3: Two-stage setting of (i) rationale generation (RougeL.) and (ii) answer inference (Accuracy). Method
(i) QCM + R (fi) QCMR- A
Two-Stage Framework
90.73
78.57
w/ Captions
90.88
79.37
w/ Vision Features
93.46
85.31
Then, we randomly sample 50 error cases and find that the model tends to generate hallucinated rationales that mislcad the answer inference. As an example shown Figure D the model (left part) hallucinates that, e The south pole of one magnet is closest to the south pole of the other magned"; due the lack of reference to the vision content. We find that such mistakes occur at a ratio of 56% among the error cases (Figure 3 (aa)
3.3 Multimodality Contributes to Effective Rationales
We speculate that such a phenomenon of hallucination is due to a lack of necessary vision contexts for performing effective Multimodal-Corl To inject vision information, a simple way is to transform the image into caption [Lu ct al. 2022a and then append the caption in the input of both stages.
4The details will he presented in Section
5

---

Published in Transactions on Machine Learning Research (05/2024)
However, shown in Table using captions only yields marginal performance gains (10.80%). Then, explore an advanced technique by incorporating vision features into the language model. Concretely, we feed the image to the VIT model Dosovitskiy et al. 2021b) extract vision features. Then we fuse the vision features with the encoded language representations be- fore feeding the decoder (more details will be presented in Section 4. Interestingly, with vision features, the RougeL score of the rationale generation has boosted to 93.46% (QCM-R), which correspondingly cont ributes to better answer accuracy of 85.31% (QCMR++)
Resolved (60.7%)
Others (44%)
Hallucination (56%)
Unresolved (29. (%))
(a) ratio
mistakes
w/ features
Figure The ratio of (a) hallucination mistakes and (b) correction rate w/ vision features.
With those effective rationales, the phenomenon hallucination mitigated 60.7% hallucination mistakes in Section 3.2 have been corrected (Figure b)),
example shown in Figure (right part) | The
analysis so far compellingly shows that vision features are indeed beneficial for generating effective rationales and contributing to accurate answer inference. As the two-stage method achieves better performance than one-stage methods, we choose the two-stage method in our Multimodal-CoT framework.
4 Multimodal-CoT
In light of the discussions in Section 3 that the answer inference leverage better generated rationales that are based on multimodal information. In this section, we will overview the procedure of the framework and elaborate on the technical design of the model architecture.
we propose Multimodal-CoT.. The key motivation is the anticipation
Rationale
Vision
S
N
N
Will these magnets attract or repel? To find out, look at which poles are closest to each other. The north pole of one magnet closest to south pole of the other magnet. Poles that different attraci. So, these magnets will attract each other.
Language
Ralionale Generation
Answer Inference
The (A).
Question: Will these magnets attract or repel each
Two magnets placed as shown. Hint: Magnets attract pull together. Magnets that repel push apart.
Context:
Options: (A) attract
(B) repel
Figure 4: Overview of our Multimodal-Coll framework. Multimodal-Col consists of two stages: (i) rationale generation and (ii) answer inference. Both stages share the same model structure but differ in the input and output. In the first stage, we feed the model with language and vision inputs to generate rationales. In the second stage, we append the original language input with the rationale generated from the first stage. Then, we feed the updated language input with the original vision input to the model to infer the answer.
4.1 Framework Overview
Mol consists of two operation stages: (i) rationale generation and (ii) answer inference. Both stages share the same model structure but differ in the input X and output Y. The overall procedure is illustrated in Figure We will take vision-language as an example to show how Multimodal-CoT works.
In the rationale generation stage, we feed the model with X [Xhangaage Xvision} where X1 the language input in the first stage and Xvision represents the vision input, i.e., the image. For example, X can be instantiated as concatenation of question, context, and options of multiple choice reasoning
language represents
TThe left mistakes mainly about map understanding, requiring commonsense signals (Section

---

Published in Transactions on Machine Learning Research (05/2024)
problem (Lu et al. 2022a] shown in Figure The goal is to learn rationale generation model R F(X) where R is the rationale.
In the answer inference stage, the rationale At is appended to the original language input X Language to construct the language input in the second stage, X langunge = X language R where denotes concatenation. Then, we feed the updated input X' (Xaagaagee X the answer inference model to infer the final answer A = F(X)..
In both stages, we train two models with the same architecture independently. They take the annotated elements (e.g,, X R XR + A, respectively) from the training set for supervised learning. During inference, given X, the rationales for the test sets are generated using the model trained in the first stage; they are used the second stage for answer inference.
4.2 Model Architecture
Given language input Xlanguage E fanguage X² Language) and vision input Xvision: we compute the probability of gencrating target text Y (cither the rationale or the answer in Figure of length N by
N
Manguage: X Xvision)
= II po (Y; X Xlanguage, Xvision: Yei), i=1
(1)
where po (Y; X Xanguage: Xvision: Yx) is implemented with Transformer-based network (Vaswani et al. 2017). The network has three major procedures: encoding, interaction, and decoding. Specifically, feed the language text into Transformer encoder to obtain textual represcntation, which is interacted and fused with the vision representation before being fed into the Transformer decoder.
Encoding The model F(X) takes both the language and vision inputs and obtains the text representation Hanguage and the image feature Hvision by the following functions:
Hhanguage
(2)
Hvision W₁ Visionl>xtractoor(xvtiii))
(3)
where LanguageEncoder() is implemented as a Transformer model. We use the hidden states of the last layer in the Transformer encoder as the language representation Hianguage E gnxdd where n denotes the length of the language input, and d is the hidden dimension. Mcanwhile, VisionExttactor(()) is used to vectorize the input image into vision features. Inspired by the recent success of Vision Transformers (Dosovitskiy al. 2021ah, we fetch the patch-level features by frozen vision extraction models, such as ViT (Dosovitskiy al, 2021b). After obtaining the patch-level vision features, apply learnable projection matrix Wn, convert the shape of MisionExtracttr(Xyysss into that of Hiangaggei thus we have Hviston E pmmd where is the number of patches.
Note that our approach is general to both scenarios with or without image context. For the questions without associated images, we use all-zero vectors as the "blank features" with the same shape as the normal image fcatures tell the model to ignore them.
Interaction After obtaining language and vision representations, we single-head attention network to correlate text tokens with image patches, where the query (Q), key (K) and value (V) are Hanguagee Hvision
and Hvision, respectively. The attention output Hattn E R is defined as: Hatin = Softmax OK V
Vdz
where dy is the same the dimension of Haaguage because single head is used.
Then, we apply the gated fusion mechanism (Zhang et al. 2020 Wu et al. 2021 Li et al. 2022a to fuse
Haaguage and Hvision. The fused output Huse E ERa is obtained by:
X Sig + W Hattn
(4)
Huse = (1 (1-A) A). Hanggaae
X Hattn vision?
(5)
where Wi and W. are learnable parameters.
7

---

Published in Transactions on Machine Learning Research (05/2024)
Decoding Finally, the fused output Huse is fed into the Transformer decoder to predict the target Y.
5
Experiments
This section will present the benchmark dataset, the implementation of our technique, and the basclines for comparisons. Then, we will report our main results and findings.
5.1 Dataset
Our method is evaluated on the ScienceQA (Lu et al., 2022a and A-OKVQA (Schwenk et al. 2022 benchmark datasets. We choose those datascts bocause they are latest multimodal reasoning benchmarks with annotatod reasoning chains. ScienceQA is a large-scale multimoda science question dataset with annotated lectures and explanations. Tt contains 2 multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. There are 12k, 4k, and 4k questions in the training, validation, and test splits, respectively. A-OKVQA is a knowledge-based visual question answering benchmark, which has 25k questions requiring a broad base of commonsense and world knowledge to answer. Tt has 17k/11/6k questions for settings, we use the multiple-choice setting to keep consistency with ScienceQA.
As A-OKVQA provides multiple-choice and direct answer evaluation
5.2 Implementation
The following part presents the experimental settings of Multimodal-Con and the baseline methods.
Experimental Settings We adopt the T5 encoder-decoder architecture Raffel et al. 2020g under Base (200M) and large (700M) settings in our framework. We apply ''LAN-Alpcce to initialize our model weights We will show that Multimodal CoT generally effective with other backbone LMs, such as UnifiedQA (Khashabi et al. 2020) and FLAN-T5 (Chung et al. 2022 (Section 6.3 The vision features the frozen ViT-large encoder (Dosovitskiy et al. 2021b). We fine-tune the models up to 20 epochs, with learning rate of 5e-5. The maximun input sequence length is 512. The batch size is 8. Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs. More details are presented in Appendix
obtained by
Baseline Models We utilized three categories of methods
baselines:
(i) Visual question answering (VQA) models, including MCAN (Yu et al. 2019 Top-Dowe Anderson et al. 2018), BAN (im et al. 2018). DFAF (Gao et al. 2019). ViLT (Kim et al. 2021). Patch-TRM (Lu et al. 2021 and VisualBERT (Li et al. 2019). These VQA baselines take the question, context, and choices as textual input, while utilizing the image as visual input. They employ a linear classifier to predict the score distribution over the choice candidates.
(ii) LMs, including the text-to-text UnifiedQA model (Khashabi et al.. 2020) and few-shot learning LLMs (GPT-3.5, ChatGPT, GPT-4, and Chameleon (u et al. 2023)). UnifiedQA (Khashabi et al. 2020) is adopted as it is the best fine-tuning model in Lu el al. 2022a). UnifiedQA takes the lextual information as the input and outputs the answer choice. The image is converted into a caption extracted by an image captioning model following Lu et al. (2022a). UnifiedQA treats our task as a text generation problem. In Lu et al. (202a)), it is trained to generate target answer text, i.e., one of the candidate options. Then, the most similar option is selected as the final prediction evaluate the question answering accuracy. For GPT-3.5 models (Chen et al.] 2020),, we use the text-davinci-O00 and text-davinci-003 engines due their strong performance. In addition, the few-shot prompting, where two in-context examples from the training set are concatenated before the test instance. The few-shot demonstrations are the same as those in Lu et al. 2022a).
also include the comparison with ChatGPT and GPT-4. The inference is based
(iii) Fine-tuned large dision-langual model. We select the recently released -LaMA-Adapaa (Zhang et al. 2023a ) LLaVA LLu al. 2023). and InstructBI.IP (Dai et al. 2023 as the competitive large vision-language

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Main results (%). Size = backbone model size from the ScienceQA leaderboard (*-" means unavailable or unknown). Question classes: NAT = natural science, SOC = social science, LAN language science, TXT = text context, IMG = image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. Segment 1: Human performance; Segment 2: VQA baselines; Segment 3: LM baselines, i.e., UnifiedQA and few-shot learning LLMs; Segment 4: Fine-tuned large vision-language models; Segment. 5: Our Multinodal-Co1 results. Prior published best results marked with underline. Our best average result is in bold face. f denotes concurrent studies, cither through citation comparison with Multimodal-CoT.
Model
Size
NAT
S0C
LAN
TXT
IMG
NO
G1-6
G7-12
Avg
Human
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
88.40
MCAN Yu et al.
2019
95M
56.08
46.23
58.09
59.43
51.17
55.40
51.65
59.72
54.54
Top-Down Anderson et al.
2018
70M
59.50
54.33
61.82
62.90
54.88
59.79
57.27
62.16
59.02
BAN Kim
2018
112M
60.88
46.57
66, 64
62.61
52.60
65.51
56.83
63.94
59.37
DEAF Gao
2019
74M
64.03
63.55
65.88
54.49
64.11
57.12
67.17
60.72
ViI.T (Km BE al. 2021
113M
60.48
63.89
60.27
63.20
61.38
57.00
60.72
61.90
61.14
Patch-
TRM
[Lu et al.
2021
90M
65. 19
46.79
65.55
66.96
55.28
64.95
58.04
67.50
61 .4.42
VisualBERT [Ti al.
2019
111M
59.33
69.18
61.18
62.71
62.
58.54
62.96
59.92
61.87
UnifiedQA Lu et al.
2022a
223M
71.00
76.04
78.91 91
66.42
66.53
81.81
77.06
68.82
74.11
GPT-3.5 (text-davinci-U0) (UU)
Lu et al.
2022a
173B
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
GPT-3.5 (text-davinci-OU))
173B
77.71
68.73
80.18
75.12
67.92
81.81
80.58
69.08
76.47
ChatGPT [Lu
2023
78.82
70.98
83.18
77.37
67.92
86.13
80.72
74.03
78.31
GPT-4 (Lu ct al. 2023
85.48
72.44
90.27
82.65
71.49
92.89
86.66
79.04
83.99
Chameloon [Chat GPT Lu el al.
2023
81.62
70.64
84,00
79.77
70.80
86.62
81.86
76.53
79.93
Chameloon (GPT-4) [L el al.
89.83
74.13
89.82
88 22
77.64
92.13
88.03
83.72
86.54
LLaMA-Adapter [Zhang et al.
2023a
6B
84.37
88.30
84.36
83.72
80.32
86.90
85.83
84.05
85.19
LLaVA Liu et al.
2023
13B
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
90.92
InstructBLIP (Dai
2023
11B
90.70
Mutimodal-Cos mom
223M
84,06
92.35
82.18
82.75
82.75
84.74
85.79
44
86.31
Mutimodal-CoT.ao.
738M
91.03
93.7U
86.64
90.13
88,25
89.48
91.12
89.26
90.45
baselines. For LdaMA-Adapter the backbone model is the 7B LLaMA model fine-tuned with 52k self-instruct demonstrations. To adapt to our tasks, the model further fine-tuned on the ScienceQAA dataset.
5.3 Main Results
Table shows the main results in the ScienceQA benchmark. We observe that Mutinodal-CoT,, achieves substantial performance gains over the prior best model in publications (86.54%-990.5) The efficacy of Multimodal-CoT is further supported by the results obtained from the A-OKVQA benchmark in Table 5
It is worth noting that Chameleon, I.JaMA-Adapter Table 5: Results A-OKVQA. Baseline results are LLaVA, and InstructBLIP are concurrent works re- leased several months after our work. In the subsequent- Section 6.2 we will show that our method is orthog- onal those multimodal models (e.g., InstructBI.IP) and be potentially used with them together to im- prove generality further, i.e., scaled to scenarios where human-annotaee rationales unavailable, thereby establishing the effectiveness across diverse tasks.
from (Cen et al. 2023 and Schwenk et al. 2022
Model
Accuracy
BERT
32.93
GPT-3 (Curie)
35.07
IPVR (OPT-66B)
48.6
ViLBERT
49.1
Language-only Baseline
47.86
Multimodal-CoT
50.57
Ablation study results in Table 0 show that both the integration of vision features and the two-stage framework design contribute to the overall performance.
Furthermore, we find that Multimodal CoT demonstrates the ability to mitigate hallucination (Section 3.3 and improve convergence (Section 6.1
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 6: Ablation results of Multimodal-Cole
Model
Base
Large
Multinodal-CoT
85.31
90.45
w/o Two-Stage Framework
82.62
84.56
w/o Vision Features
78.57
83.97
6 Analysis
The following analysis will first show that Multimodal-Cof helps enhance convergence speed and has the feasibility of adaptation to scenarios without effectiveness of Multimodal-CoT] with different backbone models and vision features. We will also conduct an error analysis to explore the limitations to inspire future studies. We use models under the base size for analysis unless otherwise stated.
rationales. Then, we investigate the general
6.1 Multimodality Boosts Convergence
Figure shows the validation accuracy curve of the baseline and Multimodal-CoT across different training epochs. "One-stage" is based on the QCM-A onput- format as it achieves the best performance in Table 2 and "Two-stage" is our two-stage framework. We find that the two- stage methods achieve relatively higher accuracy at the beginning than the one-stage baselines that generate the answer directly without CoT. However, without the vision features, the two- stage baseline could not yield better results as the training goes on due to low-quality rationales (as observed in Section 3). In contrast, using vision Teatures helps generate more effective rationales that contribute to better answer accuracy in our two-stage multinodal variant.
One-stage Baseline One-stage Multimodal
-
Two-Stage Baseline - Two-Stage Multimodal
90
80
Accuracy
70
60
5
10
Epoch
Figure 5: Accuracy curve of the No-CoT baseline and Multimodal-CoT variants.
6.2 When Multimodal-CoT Meets Large Models
A recent flame is to leverage large language models or large vision-language models to generate reasoning chains for multimodal question answering problems (Zhang et al. 2023a Lu et al., 2023 Liu et al. 2023 Alayrac al.] 2022 Hao et al.) 2022 Yasunaga et al.] 2022). We are interested in whether we can use large models to generate the rationales for CoTT:: thus breaking the need for datascts with human-annotated rationales. During the first-stage training of Multimodal-Cof, based on human annotation in the benchmark dat asets. Now, we replace the target rationales with those generated ones. As ScienceQA contains questions with images and without images, we leverage InstructBLIP and ChatGPT to generate the rationales for questions with paired images and questions without paired images, respectively Then, we combine both of the generaled pseudo-rationales as the target rationales for training (Multimodal-CoT w/ Generation) instead of relying on the human annotation of reasoning chains (Multimodal-Cof) w/ Annotation).
target rationales are
Table shows the comparison results. We see that using the generated rationales achieves comparable performance to using human-annotated rationales for training. In addition, the performance is also much better than directly prompting those baseline models to obtain the answer (in the QCM-A inference format).
FExample are provided in Appendix C.1
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 7: Result comparison with large models. We also present the results of InstructBLIP and ChatGPT basclines for reference. The inference format for the two baselines is QCM-A.
Model
IMG
TXT AVG
InstructBLIP
60.50
ChatGPT
56.52
67.16
65.95
Multimodal-Col w/ Annotation 88.25
90.13
90.45
Multimodal-CoT w/ Generation
83.54
85.73
87.76
We see that Multimodal-CoT can work effectively with large models. The findings above compellingly show the leasibility of adaptation to scenarios without human-annolated rationales, thereby establishing the effectiveness our approach across diverse tasks.
6.3 Effectiveness Across Backbones
To test the generality of the benefits of our approach other backbone models, alter the underlying LMs to other variants in different types. As shown in Table g our approach is generally effective for the widely used backbone models.
Table 8: Using different backbone LMs. Method
Table 9: Using different vision features.
Accuracy
Feature
Feature Shape
Accuracy
Prior Best (u et al. 2022a
75.17
VIT
(145, 1024)
85.31
CLIP
(49, 2048)
84.27
MM-CoT on UnifiedQA
82.55
DETR
(100, 256)
83.16
MM-CoT on FLAN-T5
83.19
ResNet
(512, 2048)
82.86
MM-CoT on FLAN-Alpaca
85.31
6.4 Using Different Vision Features
Different vision fcatures may affect the model performance. We compare three widely-used types vision features, ViT (Dosovitskiy al. b021b), CLIP (Radford et al. 2021), DETR (Carion al.) 2020), and ResNet (He et al. 2016). ViT, CLIP, and DETR patch-like features. For the ResNet features, repeat. the pooled features of ResNet-50 to the same length with the text sequence to imitate the patch-like features, where each patch is the same as the pooled image features. More details of the vision features are presented in Appendix B.1
Table shows the comparative results of vision features. We observe that ViT achieves relatively better performance. Therefore, we use ViT by default in Multimodal-Cof.
6.5 Alignment Strategies for Multimodal Interaction
We are interested in whether using different alignment strategies for multimodal interaction may contribute to different behaviors of multimodal-CoT. To this end, we tried another alignment strategy, i.c., image-grounded text encoder, in BLIP Li el. al. (2022b). This alignment approach injects visual information by inserting one additional cross-attention layer between the self-attention layer and the feed-forward network for each transformer block of the text encoder. Our current strategy in the paper is similar to the unimodal encoder as in BLIP, which is used for comparison. In Table 10 we see that using other alignment strategics also contributes to better performance than direct answering.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 10: Result comparison with different alignment strategies for multimodal interaction.
Model
Accuracy
Direct Answering
82.62
Unimodal encoder
85.31
text encoder
84.60
6.6 Generalization to Other Multimodal Reasoning Benchmarks
We are interested in evaluating the generalization capability of Multimodal-CoT to datasets outside its training domain. For this purpose, we utilize the widely-recognized multimodal reasoning benchmark, MMMU Yue et al. 2024 and conduct evaluation of Multimodal-CoT MMMU without further training.
Table 11: Generalization performance MMMU.
Model
Size
Accuracy
Kosmos-2 Peng et al. 2024
1.6B
24.4
Fuyu (Bavishi et al. 2024
8B
27.9
OpenFlamingo-22 (Aaadalla al.
2023
9B
28.7
MiniGPT4-Vin Vieuna (Zhu et al. 2023
13B
26.8
Multimodal-CoT
738M
28.7
GPT-4V(ision)
Ap
2023
56.8
Gemini Ultra (Reid al.
2024
59.4
As shown in Table 11 it is evident that Multimodal-Cof! demonstrates effective generalization to MMMU, achieving better performance than various larger models around 8B.
6.7 Error Analysis
To gain deeper insights into the behavior of MultimodaalCofT and facilitate future research, manually analyzed randomly selected examples generated by our approach. The categorization results are illustrated in Figure G We examined 50 samples that yielded incorrect answers and catogorized them accordingly. The examples from each calegory can be found in Appendix D
The most prevalent error type is commonsense mistakes, accounting for 80% of the errors. These mistakes occur when the model is faced with questions that require com- monsense knowledge, such as interpreting maps, counting objects in images, or utilizing the alphabet. The second error type is logical mistakes, constituting 14% the errors, which involve contradictions in the reasoning pro- coss. Additionally, we have observed cases where incorrect answers are provided despite the CoT being either emply correct, amounting to 6% of the errors. The CoT in these cases may not necossarily influence the final answer.
6%
14%
Commonsense
Logical
80%
Others
Figure 6: Categorization analysis.
The analysis reveals patential avenues for future research. Enhancements can be made to Multimodal-CoT by: (i) integrating more informative visual features and strengthening the interaction between language and vision enable comprchension of maps and numerical counting: (ii) incorporating commonsense knowledge; and (iii) implementing filtering mechanism, such as using only relevant CoTs to infer answers and disregarding irrelevant. ones.
12

---

Published in Transactions on Machine Learning Research (05/2024)
7 Conclusion
This paper formally studies the problem of multimodal CoT. We propose Multimodal-CoT that incorporates language and vision modalities into a two-stage framework that separates rationale generation and answer inference, so answer inference can leverage better generated rationales from multimodal information. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Analysis shows that Multimodal-CoT has the merits of mitigating hallucination and enhancing convergence speed. Our error analysis identifies the potential to leverage more effective vision features, inject commonsense knowlodge, and apply filtering mechanisms to improve CoT reasoning in future studies.
References
Jean-Baptiste Alayrac, Jeff Donahuc, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 05-23716-23736, 2022.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Tei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognittion, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 6077-6086. IEEE Computer Society, 2018.
doi: 10. 1119//0PP.20011
Anas Awadalla, Trena Gao, Josh Gardner, Jack Hessel, Yusuf Hanaly, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Opentlamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.0138 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşılar. Fuyu-8b: A multimodal architecture for ai agents, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (cds.), Advances in Neural Information Processing Systems 33: Amnual Conference on Nearal Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020: 16th Furopean Conference,
End-to-end object detection with transformers. Tn Computer Glasgow, UK, August 23 28, 2020, Proceedings, Part 1, pp. 213 229, 2020.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervisod models are strong semi-supervised learners. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Mlaria-Florina Balcan, and Elsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 6-12, 2020, virtual, 2020.
Wenhu Chen, Xucguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for mumerical reasoning tasks. ArXiv preprint, abs/ bbs//221..12568 2022.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-bard based visual reasoning. ArXiv preprinl, abs/2301.05222 00232,, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi,
13

---

Published in Transactions on Machine Learning Research (05/2024)
Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vodant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim; Barret. Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXio preprint, abs/ abs//224.003311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instuucioonfinetune language models. ArXiv preprint, abs/ abs/2210.11 1146,, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Authony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpost vision-language models with instruction tuning, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 9-7, 2021. OpenReview.net, 2021a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, TCI.R 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-stop reasoning. ArXiv preprint, abs/2210.00022, 2022.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra- and inter-modality attention flow for visual question answering. In IEFF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 0639-6648 Computer Vision Foundation IEEE, 2019. doi:
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. ArXiv preprint, has/2206.063366 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770 778. IEEE Computer Society, 2016. doi:
Namgyu Ho, Laura Schmid, and Se Young Yun. Large language models are reasoning teachers. ArXio preprint, abs/22 2bs/2212.10071, 2022.
Jie Huang and Kevin preprint, abs/2212. 10403, 2022.
Chang. Towards reasoning in large language models: A survey. ArXio
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2022.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, 2020. Association for Computational Linguistics. doi: 10. 100533//112222.ndinnggsss
14

---

Published in Transactions on Machine Learning Research (05/2024)
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. ArXio preprint, 0bss/2210002400 2022.
Jin-Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi. and Roman Garnett (cds.), Advances in Newral Information Processing Systems 31: Annuul Conference on Newral Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 1581, 2018.
Wonjae Kim, Bokyung Son, and Todoo Kim. Vili: Vision-and-language transformer without convolution region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5583-5548 5594 PMLR, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXio preprint, abs/2205.11916, 2022.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Auxiang Ma, and JingBo Zhu. On vision features multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 6327-6337, Dublin, Ireland, 2022a. Association for Computational Linguistics. doi: 10.18653/v1 ht/e///v/22222 acl..acllooog.......
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szopesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 12888 129000 PMLR, 2022b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. ArXie preprint,
2019.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. ArXiv preprint, abs/2206..2255 2022c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv preprint, 0////2334.088.. 2023.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Dalasels and Benchmarks, 2021.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Gong-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for seience question answering. Advances in Neural Information Processing Systems, 35:2507 2521, 2022a.
Pan Lu, Liang Qiu, Kai- Chang, Ying Nian Wu, Song- Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. ArXiu preprint, abs/2 /2/22229146100 2022b.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang. A survey of deep learning for mathematical reasoning. ArXiv preprint, Abss/2212.10555 2022c.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and--laa compositional reasoning with large language models. In The Thirty-seventh Conference on Neural Information Processing Systems (NewrIPS 2023), 2023.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. Ar preprint, abs/22 22212.08410 2022.
15

---

Published in Transactions on Machine Learning Research (05/2024)
OpenAL. Gpt-4v(ision) system card, 2023.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations, 2024. URL
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763, PMLR, 2021.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Tisa Anne Hemdricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna. Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hcchtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112.11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-tex transformer. J. Mach. Learn. Res., 21: 91:140:11140067, 2020.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridon, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXie preprinl or Siv:2403.005533 2024.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655 2671, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1 /ttp//2222aacllmma
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avio, Israel, October 23 27, 2022, Proceedings, Port VIII, pp. 146 162. Springer, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction -oriructtioo-oloowinnn model. Stanford Center for Research on Foundation Models. https://orfm. stanford. edu/2023/00 /00022300//// 13/ 3/alpac.. html, 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, YaGuang Li, Hongrac Lee, Huaixin Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yauping Huang, Maxin Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Shung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina,
16

---

Published in Transactions on Machine Learning Research (05/2024)
Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kusmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications. ArXiv preprint, abs/2201.08222 08239, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Mia Polosukhin. Attention is all you need. In Tsabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017.
Boshi Wang, Xiang Deng, and Huan Sun. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2714-2730, Abu Dhabi, United Arab Emirates, 2022a. Association for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Sell-consistency improves chain of thought reasoning in language models. ArXiv preprint, abs/2203.1112 2022b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale-angmented 2022c.
ensembles in language models. ArXiv preprint, abs//2077.0
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, hbss2220..11 2022b.
Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 0153-6166, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/22 1/20//1111221/l/////// lcl-long.4888
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-angmentee multimodal language modeling. Proceedings of the 40th International Conference Machine Learning, PMT.R, pp. 39755-39769, 2022.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6281-6290. Computer Vision Foundation IEEE, 2019. doi: 2019.00644.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init. attention. ArXio preprind, abs/ abs/2303.16619, 2023a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song the ai ocean: A survey on hallucination in large language models. arXiv preprint ar xrx::220090122 2023b.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. Neural machine translation with universal visual representation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview. net, 2020.
Zhuosheng Zhang, Kehai Chen, Rui Wang. Masao Utiyama, Ficchro Sumita, Zuchao Li, and Hai Zhao. Universal multimodal representation for language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1- 18, 2023c. doi:
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting large language models. In The Eleventh International Conference Learning Represenlations, 2023d.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering Asion-language model with multi-modal in-context learning. arXiv preprint arXiv:2300.0P 07915, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. ArXie preprint, hoss/2205.1066 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023.
18

---

Published in Transactions on Machine Learning Research (05/2024)
A Extended Analysis for the Challenge of Multimodal-CoT
A.1
Additional Examples of Misleading through Hallucinated Rationales
Based on case studies (Section 3.2), have observed tendency for the baseline model to generate hallucinated rationales. Here, we present additional examples to illustrate this phenomenon, as depicted in Figure
Problem
Question: Which solution has higher concentration yellow particles?
Vision
Context:
The diagram below model of solutions. yellow ball represents one particle solute.
Options: (A) Solution
(B) Solution
(C)
same
Gold Rationale: Solution A and Solution B, the yellow particles represent the solute. figure out which solution has higher concentration yellow particles, look both the number of yellow particles and volume of the salvent in each container. Use the concentration formula to find the number of yellow particles per millliter. Solution has more yellow particles per milliitee. So,
higher concentration yellow particles.
Sokene valume:
Sorstion
Sakator
The
(A).
Baseline
Vision Features
Rationale: In Solution and Solution the yellow particles represent the solute. To figure out which solution has a higher concentration of yellow particles the volume of the solvent in each container. Use the concentration formula to find the number of yellow particles per millliter. Solution A has more yellow particles milliiter, So, Solution of yellow particles. Answer: answer (B).
Rationale: In Solution A and Solution B. the yellow particles represent the solute. To figure out which solution has higher concentration yellow particles, look both the number of yellow particles and the volume of the solvent in each container. Use the concentration formula yellow particles per milliiter. yellow particles per milliliter, So, Solution B has higher concentration yellow particles.
yellow particles,
more
higher concentration
Answer: The answer is (A).
Problem
Question:
Think about the magnetic force belween the magnets in each pair. Which of the
Vision
following statements
Pair
Pair 2
Context:
The images below show two pairs of magnets. The magnets in different pairs do not affect each other. the magnets shown are made of the same material.
C
C
Options: (A) The magnitude the magnetic force smaller Pair
(B) The magnitude of the magnetic force smaller Pair
2-2 BIB
3BL Bim
(C) The magnitude of the magnetic force is same both pairs.
The magnets in Pair attract. The magnets in Pair 2 repel. But whether the magnets attract or repel affects only the direction of not affect magnitude the magnetic force. The sizes of the magnets Pair are the same as in Pair 2. The distance between the magnets also the same. So, the magnitude of the magnetic force is the same in both pairs.
Gold Rationale:
the magnelic force.
magnet sizes distance affect magnitude the magnetic
Answer: The
Baseline
Vision
Rationale: Distance affects the magnitude of the magnetic force. When there is a greater distance belween magnets, the magnitude of magnetic force between them is smaller. There is greater distance between the magnets Pair than Pair 1. So, the magnitude of the magnetic force smaller in Pair than in Pair (B).
magnets But whether the magnets attract or repel affects only the direction of the magnetic force, It does affect the magnitude the magnetic force. Both magnet sizes and distance affect the magnitude of the magnetic force. The sizes of the magnets in Pair are the same in Pair
Rationale: magnets
repel.
The
magnets is the same. magnitude of the magnetic force is the same in both pairs. The answer is (C).
between
Answer:
Examples of the two-stage framework without vision features (bascline) and with vision features (ours) for generat ing rationales and predicting answers. The upper part presents the problem details, and the lower part shows the outputs of the baseline and our method.
Figure 7:
A.2 Two-Stase Swo-Stage Training Performance with Different Sizes of LMs
In Section 3, we observed that the inclusion vision features has a positive impact on the generation of more effective rationales, consequently resulting in improved answer accuracy. In addition incorporating vision features, another approach to addressing the issue of incorrect rationales is to scale the size of the language the answer accuracy achieved by two-stage training framework, both with
model (LM).
19

---

Published in Transactions on Machine Learning Research (05/2024)
and without the integration of vision features. Notably, when employing larger LM, the baseline accuracy (without vision features) experiences significant enhancement. This finding suggests that scaling the LM size could potentially alleviate the problem of incorrect rationales. However, it is crucial to acknowledge that the performance still falls considerably short of utilizing vision features. This outcome further validates the effectiveness of our Multimodal-CoT methodology across varying LM sizes.
w/o Vision Modality
w/ Vision Modality
90.45
90
Accuracy (%)
85.31
83.97
80
78.57
70
base
large
Figure 8: Answer accuracy with different sizes of LMs.
A.3 Discussion of the Possible Paradigms to Achieve Multimodal-CoT
As discussed in Section 1, there are two primary approaches to facilitate Multimodal-CoT reasoning: (i) prompting LLMs and (ii) fine-tuning small models. The common approach in the first approach is to unify the imput from different modalities and prompt LLMs to perform reasoning (Zhang et al. 2023a Lu et al.) 2023 Liu et al. 2023 Alayrac et al., 2022; Hao et al. 2022 Yasunaga et al. 2022). For instance, one way achieve this is by extracting the caption of an image using captioning model and then concatenating the caption with the original language input to feod LLMs. By doing so, visual information is conveyed to LLMs as texl, effectively bridging the gap between modalities. This approach can be represented as the pup format <image- + caption, question + caption + answer>. We refer to this approach as Caption- based Reasoning (Figure Dh). It is worth noting that the effectiveness of this approach depends on the quality of the image caption, which may be susceptible to errors introduced during the transfer from image captioning to answer inference.
In contrast, intriguing aspect of CoT is the ability to docompose complex problems into series of simpler problems and solve them step by step. This transformation leads to modification of the standard format <question - answer> into <question - rationale > answer>. Rationales, being more likely to reflect the reasoning processes leading to the answer, play a crucial role in this paradigm. Consequently, we refer to approaches following this paradigm as CoT-based Reasoning. The nomenclature has been widely adopted in the literature (Hunng & Chang 2022 Zhang et al. 2023d Lau et al., 2022c).
Our work aligns with the paradigms of CoT-based Reasoning in the context of multimodal scenarios, specifically employing the <question + image - rationale answer> framework (Figure Øp). This approach confers advantages two fronts. Firstly, the Multimodal-CoT framework leverages feature-level interactions between vision and language inputs, enabling the model to gain a deeper understanding of the input information and facilitating more effective inference of answers by incorporating well-founded rationales. Our analysis has demonstrated that Multimodal-Cor offers notable benefits by mitigating hallucination and enhancing convergence, resulting in superior performance on our benchmark datascts. Secondly, the lightweight nature of Multimodal-Cor T renders it compatible with resource constraints and circumvents any potential paywalls.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Image
Question
Image
Question
Caption Model
Model
Caption
Prompted Text
Rationale
LLM
Model
Answer
Answer
(a) Caption-based Reasoning
(b) Co CoT-based Reasoning
Figure 9: Paradigms to achieve Mult
B Experimental Details
B.1 Details of Vision Features
In Scction 6.2, we compared four types of vision features, VIT (Dosovitskiy al. 2021b), CLIP (Radford
et al. 2021), DETR (Carion et al. 2020 and ResNet (He et al. 2016). The specific models are: (i) VIT: vit patnee_pacch__ 384 (īi) CLIP: RN101 (iii) DETR: detr resnet101 de5 (iv) ResNet: use the
averaged pooled features of a pre-trained ResNet50 CNN.
Table 12 presents the dimension of the vision features (after the function VisionExtractorr() in Eq. 3). For Res Net-50, we repeat the pooled features of ResNet-50 to the same length as the text sequence to imitate the patch-like features, where each patch is the same as the pooled image features.
Table 12: Feature shape of vision features
Method
Feature Shape
ViT
(145, 1024)
CLIP
(49, 2048)
DETR
(100, 256)
ResNet
(512, 2048)
B.2 Datasets
Our method is evaluated on the ScienceQA (Lu et al. 2022a and A-OKVQA Schwenk et al. 2022 benchmark datasets.
ScienceQA is large-scale multimodal science question dataset with annotated lectures and explanations. It contains 21k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. The dataset. is split into training, validation, and test splits with 12k, 4k; and 4k questions, respectively.
A-OKVQA is a knowledge-basse based visual question answering benchmark, which has 25k questions requiring a broad base of commonsense and world knowledge to answer. Each question annotated with rationales that
21

---

Published in Transactions on Machine Learning Research (05/2024)
explain why particular answer correct according to necessary facts knowledge. It has 17k/ /k/ questions for main/val//ess
For ScienceQA, our model is evaluated on the test set. For A-OKVQA, our model is evaluated on the validation set as the test set hidden.
B.3 Implementation Details of Multimodal-CoT
As the Multimodal-CoT task requires generating the reasoning chains and leveraging the vision features, we adopt the T5 encoder-decoder architecture (Raffel et al. 2020) under Base (200M) and large (700M) settings in our framework. We apply FLAN-Alpaca to initialize our model weights ITT We will show that Multinodal-ooT is generally effective with other backbone LMs, such as UnifiedQA (Khashabi et al. 2020) and FLAN-T5 (Chung et al. 2022 (Section 6.1). The vision features are obtained by the frozen large encoder (Dosovitskiy al. 2021b). Since using image captions can slightly improve model performance, shown in Section 3.3, we append the image captions to the context following [u et. al. (2022ap. The captions are generated by InstructBLIP DDai al. 2023). We fine-tune the models up to 20 epochs, with learning rate selected in (5e-5, Se-5). The maximum imput sequence lengths for rationale generation and answer inference are 512 and 64, respectively. The batch size is 8. Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs.
C Further Analysis
C.1 Examples of Rationale Generation with Large Models
A recent flame is leverage large language models large vision-language models to generate reasoning
chains for multimodal question answering problems (Zhang et al. 2023a Lu et al. 2023 Liu et al. 2023
Alayrac et al., 2022 Hao et al. 2022 Yasunaga et al. 2022). We are interested in whether we can use large models to generate the rationales for Mullimodal-Cof thus breaking the need for datasets with numan-annotater rationales. During the first-stage training of Multimodal-Col1 our target rationales are based on human annotation in the benchmark datasots. Now, we replace the target rationales with those generated by an LLM or vision-language model. Concretely, and the question without images (TXT) to Tnst (Dai al. 2023 (Figure 10a) and ChatGPT (Figure 10b) for zero-shot inference, respectively. Then, we use the generated pseudo-rationales as the target rationales for training instead of relying on the human annotation of reasoning chains.
feed the questions with images (IMG)
Input
Input
Language
name colony shown?
Question: How long is hiking trail?
Context: N/A
Context:
Select the best estimate.
(A) Maryland
Hampshire
(A) meters
(B) millimeters
Options:
Options:
(C) Rhode Island
(D) Vermont
(C) 4 kilometers
(D) 4 centimeters
Answer:
The answer is (B).
Answer:
The answer is (C).
Please explain
Please explain the rationale to choose the correct answer. Solution:
Solution:
Output
Output
of New Hampshire, which located in the northeastern region the United States. Thare are several other states visible on the map, including Maine, Vermont, Massachusetts, Rhode Island, Connecticut, New York, New Jersey, Pennsylvania, Maryland, Delaware, Virginia, North Carolina, South Carolina. Georgia, Florida, and Alabama. Additionally, there are several smaller states scattered tnroughout the map, such as West Virginia, Kentucky, and Tennessee.
The green represents
The best estimate for the length hiking trail (C) 4 kilometers. Hiking trails are typically longer than few meters or centimeters, and 4 millimeters is too small to reasonable estimate. Therefore, kilometers the most appropriate option.
(a) Rationale generaled by InstrucIBLIP
(b) Rationale generated by ChatGPT
Figure 10: Rationale generation examples.
22

---

Published in Transactions on Machine Learning Research (05/2024)
C.2 Detailed Results of Multimodal-CoT on Different Backbone Models
To lest the generality of the benefits of our approach to other backbone models, alter the underlying LMs to other variants of different types. As detailed results shown in Table 13 our approach is generally effective for the widely used backbonc modcls.
Table 13: Detailed results Multimodal-CoT on different backbone models.
Model
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Avg
MM-CoT on UnifiedQA
80.60
89.43
81.00
80.50
80.61
81.74
82.38
82.86
82.55
MM-CoT FLAN-15
81.39
90.89
80.64
80.79
80.47
82.58
83.48
82.66
83.19
MM-CoT FLAN-Alpace
84.06
92.35
82.18
82.75
82.75
84.74
85.79
84.44
85.31
D Examples of Case Studies
To gain deeper insights into the behavior of Multimodal-CoT and facilitate future research, manually analyzed randomly selected examples generated by our approach. The categorization results are illustrated in We examined 50 samples that yielded incorrect answers and categorized them accordingly.
Figure I1
6%
14%
Commonsense
Logical
80%
Others
Figure 11: Categorization analysis.
The most prevalent error type is commonsense mistakes, accounting for 80% of the errors. These mistakes occur when the model is faced with questions that require commonsonse knowlodge, such as interpreting maps (Figure 12,,, counting objects in images (Figure 12 p), or utilizing the alphabet (Figure 12
The second error type logical mistakes, constituting 14% of the errors, which involve comparison mistakes (Figure L3a) and contradictions in the reasoning process (Figure 13b).
Additionally, we have observed cases where incorrect answers are provided despite the CoT being either empty or correct, amounting to 6% of the errors. The CoT in these cases may not necessarily influence the final answer (Figure 14p.
The analysis reveals potential avenues for future research. Enhancements can be made Multimodal-CoT by: (i) integrating more informative visual features and strengthening the interaction between language and vision to enable comprehension of maps and numerical counting; (ii) incorporating commonsense knowledge; and (iii) implementing filtering mechanism, such using only relevant Colss to infer answers and disregarding irrelevant ones.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Problem
Vision
Question: Which of these cities marked on map?
Context:
N/A
Options:
(A) Las Vegas
(B) San Francisco
(C) Phoenix
(D) Lake City
Rationale: The city Vegas, Nevada, Salt Lake City, Phoenix, and Francisco marked with gray circles
the map below.
Answer: The answer is (A).
Prediction
Rationale:
The city San Francisco, California.. Lake City, Las Vegas, Phoenix The answer (B).
gray circles map
Answer:
(a) Mistakes about maps.
Problem
Complete the text to describe the diagram. Solute particless moved in both directions across the permeable membrane, more solute particles moved across the membrane (). When there was an equal concentration on both sides, the particles reached equilibrium.
Question:
Vision
The diagram below shows solution with one solute. Each solute particle is represented by green ball. The membrane, represented dotted permeable to the solute particles. The diagram shows how the solution can change over time during the process of diffusion.
Context:
solution
by membrane.
Options: (A) left than to the right
(B) to the right than the left
particles reached equilibrium, there were solute particles on the left side of the membrane and 5 solute parlidess on the right side the membrane. When solute particles reached equilbrium, there were 4 solute particles on each side of membrane. There
Rationale (extract) shows you
changed during process diffusion. Before the
solute particle the left side of the membrane than before. for the solute particles reach equilibrum, more solute particles must have moved across the membrane to the left than to the right. The answer (A).
Prediction
Rationale: (extract) shows you how the solution changed during the process diffusion. Before the solute particles reached equilibrium, there were 5 solute particles on the left side of the membrane and 4 solute particles on the right side of the membrane. When the solute particles reached equllibrium,
right side membrane before. So, for the solute particles to reach equilibrim, more solute particles must have moved across the membrane to the right than to the right. The answer is (B).
particles on
more
Answer:
(b) Mistakes about counting numbers in the image.
Problem
Vision
Question: Which word would you find dictionary page with following guide words? helping hunter
Context: The diagram below shows solution with one solute. Each solute particle is represented by a green ball. The solution
membrane, represented permeable to the solute particles, The diagram shows how the solution can change over time during the process of diffusion.
by
dotted
Options: (A) hospital
(B) hardship
Rationale Put the words in alphabetical order. Since hospital is between the guide words helping hunter, it would be found on that page.
Answer: The answer is (A).
Prediction
alphabetical order. Since hardship the guide helping it would found page. Answer: The answer (B).
Rationale: Put
(c) Mistakes about utilizing alphabet.
Figure 12: Examples of commonsense mistakes.
24

---

Published in Transactions on Machine Learning Research (05/2024)
Problem
Vision
Question: Compare the Context: NIA
cars. Which car was moving at lower speed?
N/A
Options: (A) car that moved 445miles in 10hours
(B) car that moved 170miles 10hours
Rationale: Look at the distance each car moved and the time it took to move that distance. One car moved 170 miles in 10 hours. The other car Natice each spent the same amount of moving.
moved miles 10
moved 170 miles moved distance
in that time. So, that car must have moved lower speed. The answer is (B).
Answer:
Prediction
Rationale: Look at the distance each car moved and the time took to move that distance. One car moved 445 miles in 10 hours. The other car moved
hours. Natice each spent same amount time moving. The car moved 445 miles moved a shorter distance that time. So, that car must have moved lower speed.
Answer:
The answer is (A).
(a) Logical mistake where the model fails at comparisons.
Problem
Question: Based this information, what is Sage's phenotype the coat color trait?
Vision
Context: In group of horses, some individuals have a black coat and others have a reddish-brown cost. In this group, the gene coat color has two alleles.
for reddish-tree brown coat is recessive to the Sage horse from this group. Sage has the homazygous genotype II for the coat color gene.
(L).
N/A
Options: (A) black coat
(B) reddish-broowd brown coat
Il. Sage's genotype of li has only alleles. The I allele for reddish-brown coat. So, Sage's phenotype for the coat color trait must be reddish-brown coat. To check this answer, consider whether Sage's alleles are dominant or recessive, dominant allele, recessive allele. Sage's genolype of has only recessive alleles. An organism with only recessive alleles for gene will have the recessive allele's version of the trail. So, Sage's phenotype for the coat color trait must be reddiah-brown coat.
Rationale:
Sage's genotype for the coat color
The allele reddiah-brown coat ()) recessive the allele black coat (L). This means
Answer: The answer
Prediction
Sage's genotype for the coat color gene Il. Sage's genolype of Il has only alleles. The allele is for a meddish-broww coat. So, Sage's phenotype for the coat color trait must be a reddlshh-bowwn coat. To check this answer, consider whether Sage's alleles are dominant or recessive. is recessive to the allele for black coat (L). This means dominant allele, and is recessive allele. Sage's genotype of ll has only recessive alleles. An organism with only recessive alleles for gene will have the recessive allele's version of the trait. So. Sage's phenotype Answer:
Rationale:
The allele for reddish-brown
coat color The answer is (A).
black coat.
(b) Logical mistake where the there contradiction the rationale.
Figure 13: Examples of logical mistakes.
Problem
Question: Which trait did Palaeopython have? Select trait you can observe on the fossil.
Vision
Context: This picture shows a fossil of an ancient animal called Palaeopython. Palaeopython lived in trees and could grow more than six feet long.
Options:
(A) black stripes on its skin
(B) large fins on Its body
(C) a long. thin body
more about traits of ancient organisms. Fossils can preserve the remains of body parts and activities, A fossil of a body part, such tail or a wing. can tell you what an organism looked like. A fossil of an organism's activities, such as a burrow or a foolprint, can tell you about the organism's behavior. Here are three examples fossills and the traits that you can observe from them: This fossil tells you that the plant had small leaves arranged in branched pattern. This is a fossil of an animal's footprint. This fossil tells you that the animal could walk on formation. When organism's body turns into fossil, only few body parts are usually preserved.
Rationale:
way organism looks or is called
Scientists use fossill
fossil of animal. This fossil tells you that the animal had spiral-shaped shell.
fossil plant.
organism's fossil may not show
organism's traits. This because most body parts are destroyed during fossil
Answer: The answer is (C).
Prediction
Rationale: The way organism looks or acts is called trait. Scientists use fossils to learn more about the traits of ancient organisms. Fossils can preserve the remains of body parts and activities A fossil a body part, such as a tail or a wing, can tell you what an organism looked like. fossil organism's activilies, such burrow that you can observe from them: This is a fossil of an animal. This fossil tells you that the animal had a spiral-shaped shell. This is a fossil of a plant. is a of animal's footprint. This
footprint, tell you about the organism's behaviar. Here
three examples fossils and the traits
This fossil tells you animal could walk on land. An organism's fossil may not show all of the organism's traits. This is because most body parts are destroyed during fossil formation. When an organism's i body turns into fossil, only a few body parts are usually preserved. Answer: The answer (B).
plant had small leaves arranged
branched pattern.
you that the
Figure 14: Examples of answers are incorrect while the CoT is correct.
25


Reviews:
[{"review_id": "4nxwOM5sd9", "reviewer": "Reviewer_Czj7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe evaluation results are decent considering the small model size.\n\nWeaknesses:\n\n1. The fundamental limitation of this work is about its novelty and significance. Despite that the approach is called multimodal CoT, the approach is not a new prompting technique. Instead, the major part of this work is on how to fuse vision features and small-scale pretrained language models and then finetune the resulted vision-language model to perform multimodal reasoning. However, this submission misses the recent rapid progress of large multimodal models. For example, Gemini and GPT-4V have demonstrated impressive performance across various multimodal reasoning benchmarks, while there are neither discussion nor empirical comparison to such SOTA models. Also, the authors need to evaluate those multimodal benchmarks that are used in the Gemini technical report, instead of only evaluating on 2 benchmarks.\n\n2. This work designs the two-stage prompting and demonstrates that it improves the performance. However, note that the two-stage prompting trains 2 separate models for rationale and answer generation, which increases the total model size. From this perspective, the effectiveness of two-stage prompting mainly comes from the insufficient model size, and it is unclear whether it is necessary with more powerful and larger multimodal models.\n\n3. Table 7 is confusing. Did the authors use the text-only chatGPT? If so, how did they feed images into the model? Also, I don't think the results with generated rationales and human annotations are \"comparable\" as described in the paper, the performance with generated rationales is clearly worse.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Multimodal CoT, which includes training a small-size vision-language model, and a two-stage prompting pipeline that separates rationale and answer generation. They first motivate their two-stage prompting design with a text-only model, and show that without vision features, CoT performs worse than direct prompting. Then they show that two-stage prompting improves the performance with vision features. Next, they demonstrate their finetuned vision-language models based on FLAN-Alpaca, and the evaluation on 2 benchmarks show that their models with <1B parameters perform on par with LLaVA, which is a much larger model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Justify the novelty and significance of the approach, especially two-stage prompting.\n\n2. Add missing related work discussion and empirical comparison to recent large multimodal models.\n\n3. Add evaluation on more recent multimodal reasoning benchmarks, such as those evaluated in the Gemini technical report.\n\n4. Explain how to compute the results with image input for chatGPT in Table 7.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FAcxg9FyZM", "reviewer": "Reviewer_omWv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n- The paper provides a nice analysis about the motivation of introducing multimodal inputs for question answering and demonstrated the bad effects of the hallucinated rationales. The explanation is clear and reasonable.\n- The proposed two stage pipeline sounds reasonable for me. It achieves good performance in various dataset\n\nWeakness: the proposed method seems straightforward and not novel. However, as I am not the expert in the field. I can only make an educational guess.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a two-stage pipeline for multi-modal question answering. The pipeline contains two stages. In the first stage it generates rationale and then it generates the final answer using a separate network, conditioning on the original questions as well as the generated rationales. In both stages it trains a multi-modal network that takes both text and images as input. The authors analyze how hallucinated rationales misled the performance of the network. In experiments, they demonstrate that their methods can outperform several large vision language models with relative small networks with limited sizes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No requested changes", "broader_impact_concerns": "no broader concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Orhj6iRlFX", "reviewer": "Reviewer_MP8C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The proposed method is quite general, and strong for its size (< 1B). It outperforms most other larger models (in the 6-13B scale) on ScienceQA. The method is also complementary with stronger VLMs, and the authors show (Sec. 6.2) that these models can be used to further improve results by training on their generated outputs rather than on human annotated reasoning chains.\n- The analysis on failure modes of CoT and introducing multimodal inputs in Sec. 3 are informative.\n\n\n## Weaknesses:\n- As all the models are benchmarked on finetuned performance, it’s unclear how well the proposed method generalizes outside of the finetuning domain. It would be helpful to see such experiments, e.g., testing the ScienceQA finetuned model on MMMU [1] (both on the science and non-science domains).\n- Similar to the above, does the finetuned model still do well on other vision-language tasks such as image captioning? Or is it highly specialized for VQA-type problems?\n- Ablations on the importance of the visual backbone and language backbone would be appreciated (e.g., model size, image resolution, etc.). It would be useful to have information on how using larger visual encoders and language models affect performance.\n\n\n**References**\n\n[1] Yue, Xiang, et al. \"Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.\" arXiv preprint arXiv:2311.16502 (2023).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method of leveraging Chain-of-Thought (CoT) prompting in a multimodal setting, to improve performance on vision-language tasks. They analyze both prompt based methods and finetuned models (of around 1B parameters). The paper also investigates why models of 1B-parameter scale fail at CoT reasoning, which they find is partly due to hallucinated rationales. The proposed method has two stages: (1) rationale generation and (2) answer inference. They show that the proposed method outperforms much larger models on the ScienceQA and A-OKVQA datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would strengthen the work:\n- Results on generalization or transfer to datasets that the model was not trained on.\n- Analysis on different language model and visual backbones.", "broader_impact_concerns": "There are no major issues with the ethical implications or broader impact of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers concur that this paper presents a valuable analysis for the application of CoT reasoning in multimodal models. One of the striking weaknesses of this paper is basically novelty of the architecture, given that it shares a lot of similarity with the LLAVA approach. Even if novelty is not a determining factor for acceptance at TMLR, the novelty of the proposal might affect the interestingness of the work for TMLR, which is a determining factor. Nevertheless, it caught to my attention that the authors state "It is worth noting that Chameleon, LLaMA-Adapter, LLaVA, and InstructBLIP are concurrent works released several months after our work", this prompted me to verify that the first version of this paper appeared on arXiV a while ago, in 2023. In this light, I will consider this work indeed concurrent to LLaVA and the preceeding works. I appreciated the effort that the authors made to rerun their experiments (Table 2) and include more recent baselines (i.e. LLaVA in Table 5) along with the experiments on MMMU, which were in response to a reviewer concern.
Decision: Accept (poster)