DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models
Authors: Ge Zheng
Bin Yang
Jiajin Tang
Hong-Yu Zhou
Sibei Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Chain-of-Thought Reasoning
Multimodal Science Question Answering
Vision and Langauge

Abstract: A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: “keeping critical thinking” and “letting everyone do their jobs” in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models
Ge Zheng' 1 Bin Yang!" Jiajin Tang
Hong-Yu Yu Zhou?
Sibei Yang
Shanghai Tech University
The University of Hong Kong
Project Page: https://toneyaya github. io/ddcot/
Abstract
A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in shot prompting and fine-tuning learning, significantly outperforming state-of- the-art methods but also exhibit impressive generalizability and explainability.
Introduction
One of the fundamental aspirations of Al systems is to address complex tasks with reliability and efficiency that are aligned with human capabilities [31, 48, 58, 11, 52, 53, 49, 50]. In tackling such complex tasks, humans rely on multi-step reasoning that integrates information from various modalities. Recently, language models (L.Ms) [3, 37, 55, 8, 7] have shown remarkable progress in range of multi-step reasoning tasks by prompting [60, 25, 70, 15, 54] or fine-tuning [35, 13, 59, 27] with the chain of thought (CoT) that mimics the human reasoning process.
However, most studies on CoT reasoning have focused solely on the language modality [44, 10, 32, 72, 22, 5, 68, 66), with minimal attention multimodal contexts. UnifiedQA [31] and MM-CoT [71] are the pioneering works of eliciting CoT reasoning in employ multimodal inputs, as illustrated in Figure (a), necessitating training for the generation of the intermediate reasoning steps (i.e., rationales) either as an explanation accompanied by an answer or as reasoning prior to inferring the answer. Despite the henefits of these multimodal CoT reasoning methods in enhancing the ability of LMs to answer multimodal science questions, several significant challenges have impeded their deployment: (1) Labor-intensive annotation: manual annotation for rationales is time-consuming, costly, and challenging ensure consistency and completeness in
multimodality. They
Equual contribution.
Corresponniig author. yangsboshanghaaitech.ce edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Inout distribution)
Ours
Which nutrients are mairly provited by the foods the image?
Split: Please think by question down
Rationals: The image made up of rumber nutrients. The names nutrients the image are not correct.
quections
Vitamins
Bub--uestion(LLMM Q
(1) What foods are in the image?
They nutrients, They mainty provided by foods image.
Proten
(2) What nutrinnt are commonty found the foods
C. Pats
Image?
They are not witam ns or minerals.n hhyy are not tats.
anoaers(VAA)) C
(1) Branges (2) Staani C
Fitrol a type of fot the image is made up fot is protein.
Zoro
Rationale(LLMss;
Oranges are in the image and they are commanly known to be a good source of Viaamn C Tharafore, the nutfient mairly provided by the foods in the image is Vnammsg
Firertue
Fetrol protein. It made protein plant.
Our
37.34%
The answer is (B).
X
MM-CoT
78.49%
54 54.1%%
(8)
shot
78.00%
The sensowr (A)
UnittedQA
GPP
70765
Finoturing
Oranges are commoniy known to be good fruit, The answer (8).
GPT
NM CoT
Outs
(6)
(C)
The answer (A)
(6)
Figure/ 1: Comparison of existing multimodal CoT methods and our DDCoT on (a), (c), (d) and (e) an out-of-distribution example to illustrate the generalizability and (b) performance of zero-shot and fine-tuning learning, demonstrating that we are the first to generate general multimodal rationales.
multi-step CoT reasoning, thereby restricting the utilization of these approaches. (2) Flexibility: the effectiveness of multimodal rationales generated by existing methods [31, 71] is limited either zero- shot prompting or fine-tuning learning with LMs, as demonstrated in Figure (b). Our observations indicate that the rationales generated by MM-CoT fail to provide any benefits for zero-shot prompting. while the rationales generated solely by GPT-3 do not enhance the reasoning ability of the fine-tuning model. (3) Generalizability: the current multimodal CoT reasoning approaches [31, 71] exhibit limited generalizability to problems that require novel and unseen reasoning paths. As shown in Figure (()) and (e), basic science questions that are out-of-distributioo pose a challenge for these models. Moreover, models trained on specific subset, which includes arbitrary two subjects of questions in natural science, social science, and language science, perform worse when applied to another subject of questions (see Section 4). (4) Explainability: the objective of multimodal CoT extends beyond inferring answers to include providing explanations, yet the interpretability of current generated rationales still requires further improvement.
This study aims to explore overcoming the aforementioned challenges and develop a zero-shot, generalizable and explainable approach for generating rationales to enhance the reasoning abilities of LMs both zero-shot and fine-tuning learning. To achieve this, we first probe the the incorporation of rationales into multimodal reasoning and determine a two-step rationale generation and utilization process. Then we explore (1) the role of rationales in the utilization phase, where they serve as authentic knowledge providers, and (2) the challenges of intensified hallucinations in rationale generation phase. Based on these factors, we formulate the following approach.
To generate general multimodal rationales, we propose novel multimodal CoT prompting approach called Duty-Distinet Chain-of- Chain-of-Thought Prompting (DDCoT), which generate multimodal rationales using language-only LLMs [3, 37], considering the explored factors above. For the first factor, our key insight is "critical thinking keeping a healthy dose of skepticism" ": explicitly indicating the uncertainty in the rationale generation process is critical since it helps improve the correctness of the generated rationales, which serve as key inputs guiding LMs in the thinking process of inferring answers, particularly in the zero-shot setting. For the second factor, our key insight is to "lel everyone do their jobs - divide labor and work together": ": it is necessary to prompt LLMs to explicitly identify the reasoning and recognition responsibility of LLMs and off-the-shelf visual models, in order to overcome the language hallucination issue that arises when directly generating rationales from interleaved multimodal inputs. Based on this, we propose a sequential process of negative-space prompting, visual recognition, and joint reasoning to perform interleaved reasoning and recognition. More detailed analysis of the observed factors and motivation behind the two insights are elaborated in Section 3.1 and Section 3.2.
To utilize the generated rationales to facilitate multimodal question answering of LMs, we leverage them as inputs to explicitly guide the LMs' chain of thought and the attention to multimodal inputs in both scro-shot prompting and fine-tuning learning. We combine generated rationales with problem

---

statements as inputs to the LLMs for zero-shot learning. For fine-tuning learning, we propose deep- layer prompting (DLP) and rational- comppeseee visual embedding (RCVE) to utilize the rationales better to filter, encode, and jointly infer over interleaved multimodal inputs.
To sum up, our contributions are three-fold: (1) This work is the first study the zero-shot multimodal rationale generation. We deeply analyze the challenges and insights in multimodal CoT for the rationale generation: rationale sensitive in zero-shot prompting, the knowledge-requiire in fine- tuning due to catastrophic forgetting, and hallucinations intensified due to interleaved multimodal inputs. We hope these insights could benefit future research work. (2) We propose novel DDCoT prompting to maintain a critical attitude and identify reasoning and recognition responsibilities through the combined effect of srraaive--pace design and deconstruction. The resulting rationales can directly be part of multimodal inputs to improve the reasoning abilities of LMs in both zero-shot prompting and fine-tuning learning. (3) With the rationales, our methods consistently outperform the state-of-the-an art LMs, improving both GPT-3 and UnifiedQA by +2.53% and +8.23% respectively on questions that have image context, while exhibiting impressive generalizability and explainability.
2 Related Work
CoT Reasoning of LLMs. LLMs have been demonstrated successful in natural language processing. Recently, zero-shot [25] and few-shot [60, 44] multi-step thinking prompts have been found to significantly improve LLMs' reasoning ability, thus such thain-of-thought (CoT) methods have attracted growing interest. Some works are interested in example selection in terms of similarity [44, 32], diversity [70], and complexity [10]. Other methods optimize the reasoning pipeline, exploring to introduce programming approach [5], explicit decomposition of problems [72, 22] or calibration coordination across multiple rationales [59, 27, 10]. Inspired by these works, focus extending CoT reasoning to multimodality, whilst tackling inherent complications that emerge from therefrom.
Transferring Specialized Reasoning Skills to Small Models. In addition to research on CoT reasoning in LLMs, some studies conduct CoT on models with a smaller number of parameters. The works [35, 13] distill the CoT capabilities of LLMs into smaller models, facilitating the performance of smaller models in specific tasks. Unlike them to generate rationales in inference, our work utilizes the generated rationales via CoT reasoning from LLMs to explicitly guide the understanding of the image for multimodal reasoning.
Cross-modal CoT Reasoning, The pioneering work [31] for multimodal CoT proposes ScienceQA, a dataset consisting of scientific questions involving multimodality with annotated rationales. They perform zero-shot prompting on GPT-3 and fine-tuning learning on UnifiedQA [20] to generate rationales and answers simultaneously.
The following work MM-CoT [71] devises a two-stage framework in which the model initially learns generate rationales based on the ground-truth annotations, and then utilizes all available information to generate the final answer. However, their generated rationales can only exclusively benefit either zero-shot or fine-tuning learning.
Integrate Visual Modality to Language Models. With the demonstrated capability of LLMs to incorporate open-world commonsense knowledge [40, 43], an increasing number of studies are focused on making visual modality available to well-established LLMs for solving complex visual and multimodal problems. Some approaches [16, 26, 67, 28, 69, 6, 12, 63] incorporate additional training data to align image features with linguistic space. Others [33, 62, 47J take advantage of the scheduling ability of LLMs dynamically integrate off-the-shelf vision models, extracting image information into text form explicitly. Unlike them directly integrating multimodal inputs once, we explicitly identify the uncertainty in rationale generation and complement visual information step by step.
3 Method
Our work focus on how to best put visual information in the text to generate generalizable and explainable rationales. In this section, we first introduce the concepts and motivation in exploration (Sec 3.1,, and then present our concrete method designs for rationale generation (Sec 3.2) and utilization (Sec 3.3).
3

---

Question
w/lo
(b) R with partal K
Think about the magnetic forza between the magnetts each pale Which of the following statemonts is true?
Zere-shot: C)
X
The stringth of magnatic foror is stranger when the
Ubtanie between the megnets is smallei
Context:
Fine tunc:
Finetunes
inages below two pairs magnets. taagnet in different pairs do not each other the magnets shown made the same meteriall
(c) Wronng
The stronigth of magnetic force deponds on the distinco betwein the magnets and the strength of thair magnetic fieid. The difference betweon Poir and Fair 2 not specited in the contest. Looking et the images, we can see that the distance between the megnets in Per is smalier than the distance beiwwen the mognets Per 2. The strength of megnetii lonce is stronger when the distance betivoon the magnets smalles
magnets are Captionn
magnets
magnets pair are foot sport.
thae
Zero-shot:
Fineuner
(d) Cornect
The strength of magnetic force depends on the distance between the maggatt and the strnntt of their magnettc fells The difference between Poir and Pair 2 not specified in the concest. Looking at the Images, we can see that the distance between the magnets in Pair 1 Binger than the distance betwoen the magnets in Fair 2. The strongth of magnetic force is strunger when the distence between the magnets smaller.
Options:
A. The magneticc force is stronger Pair 8. The magnetti force is stronger Pair The strength
magnetic force the both
Zero-shat A
Zt
Fistune:
Figure 2: An example shows the importance of input rationales in multimodal reasoning as well as the disparate roles of rationales in zero-shot and fine-tuning scenarios.
3.1 Motivation: Leverage Rationales for Multimodal Reasoning
The success of rationales in unimodal reasoning [25, 60, 44] motivates us leverage rationales to enhance both reasoning capabilities and interpretability in multimodal reasoning. In this section, we first investigate the incorporation of rationales into multimodal reasoning in Section 3.1.1, and then examine the roles of rationales in enhancing multimodal reasoning in Sections 3.1.2, while also illuminating the challenges generate general rationales with LLMs Section 3.1.3.
3.1.1 Two-step Reasoning Process: Multimodal Rationales Generation and Utilization
Following [31, 71, 69], we first prompt LLMs to generate answers accompanied by rationales and observe that it cannot improve multimodal reasoning capabilities, i.e., 74.04% accuracy of generating only answers and 75.17% accuracy of generating both rationales and answers [31].
Moreover, as Figure 2(a) illustrates, despite sufficient image information provided in the form of image caption, the LLM,, GPT-3 [3] in this case, fails to jointly reason the question and the caption to infer answer. The possible reason is that LLMs have difficulty understanding dense image information.
Motivated by that people usually extract key information from images in the context of questions. rather than indiscriminately focusing on all dense information, we explore to provide the rationale as a structured logical chain to explicitly guide the understanding of the image. As shown in Figure 2(a) and (d), incorporating the rationale as input facilitates the model's s multimodal reasoning capabilities. Therefore, we employ a two-step reasoning process: multimodal rationale generation, and their subsequent utilization.
3.1.2 Roles of Rationales Differ in Zero-shot and Fine-tuning Learning
In further exploration, we find that the effect of ratio- nale varies in zero-shot and fine-tuning models.
[alWnong R commonly good source Fass Therefore, nutrient mainty provided foods in the
[0l/o R
image
Caption:
table
Rationale-sensitive reasoning for zero-shot prompt- ing. Upon conducting zero-shot prompting on LLMs, such as ChatGPT [37], we find that the model tends to reason in line with the input rationales. For example, without referring to additional rationales, the common sense knowledge encoded in ChatGPT allows to an- swer the basic biological question "which nutrients are mainly provided by the foods in the image", as shown in Figure 3(c). However, the assertion "oranges are known to be good source of fats" in misleading rationale results in a failure of ChatGPT to answer the same question (see Figure 3(a)). Given language models' tendency to reason primarily based on input rationales, the accuracy of rationales becomes crucial for zero-shot prompting LLMs.
Zhro-shot: V Fine-tunc:
imaga
Cranges arw in the image and they are commonty known be good fruit.
[d18 with partial
Zero shot:
The food in image made up of a number of nutrients. They are mainty provided by the foods tha image They ane not vitmmns or mineralsnThey are not lass Fetrol is typo of fat. The fat in the mage made up of fat and is not protein niooroll is prossin. It made protein rich plant.
Zero
Fine
[e/Corrett R
Oranges they
known
Thersfor, Vaaami
mainty provided
image Vitamiins
X
Fine-tur:: B X
Zaro-shat A Finc-tunc:
Figure 3: An example for disparate roles of rationales in zero-shot and fine-tuning see- narios, with question depicted in Figure 1.

---

Question
Which of the followin organsms the socondary consumar thas food web?
The strongth of inggntiic forse depends on the di disiance bnsween the imgriids and the shonghh of OVER magrette Helth The di difterencn betwown Par and Per not spectied the co conteet. looking at the images, we can aos that the distance between the magretts in Pair l larper ihan the disanne betweon the magnets in Pair 2. The strerghh of magnetic force stronger whan the distince between the magnets is sttulie
Conteo:
Below is food web from an ocean ecoseter Monto ty medal fow the natior through ecceystem oegeniors
Bay, coest forma
by orga
Finetunee Answer:
Overll:
[D]GPT Caption
The cuoodion asking for the secondary consumer in Bhe gween Food wab. in food mndt, the sacondary consumer the orga hare that primary consattic Eocling the grean Food wedy can wee that peimary cansumers which the sardinss and the otaas. Thandlore, the answer kelp: The ple mideiip h tertary consumee, whech meern rots the secordary consumte
Caption:
A food web
zooplarkta kize''
Optione:
Fire Angwer:
Ovent:: 76 49%
A. lappplatiion diyyisplekiion
1c/Ours
oods Acconding the suppleennaay information, Uhe organisms that the producens (phyyioplankknn and lalo). The seconalay corsumars food web
qussiio
the wourddar connaree
C. help
D. plarfin mechlipman
the oxqariers that eut the primary connumerrs. Bead in the faod wil, the (lariin medalagmaan is the saill argeniom that eats the primary conamees (roosiendenn and smell fen), so secendary consumet Rise-tuse Ansonr:
Ovend: 85.5%%
Figure 4: An example of the hallucination challenge which is exacerbated by providing interleaved multimodal information at once in generating rationales with LLMs.
Knowledge-required reasoning for fine-tuning. In contrast to zero-shot prompting learning, the fine-tuning models not only require the correctness of rationales but also exhibit a strong reliance on comprehensive prior knowledge embedded in rationales. As shown in Figure 3d) and 2(b), lacking common sense knowledge, such as the nutrient mainly provided by oranges and factors affecting magnetic force, the fine-tuning model fails to answer these two questions. Compared to zero-shot prompting, the fine-tuning model showcases enhanced error tolerance (as shown in Figure 2(c)), while concurrently showing increased susceptibility to knowledge deficiencies. This stems from the catastrophic forgetting during fine-tuning in LMs [14, 4, 65].
3.1.3 Challenge Lies in LLMs with Multimodality: Hallucinations Intensified
To generate general rationales simultaneously fulfill the above two roles for assisting language models in understanding multimodal information, we first analyze the ofate-of-the-art method [71]. which is trained to generate rationales relying on manual annotations. However, it generates rationales without considering their correctness, thereby leading to the risk of misleading language models. In addition, based on training with manually annotated rationales, it lacks the ability to generate rationales that encompass sufficient relevant knowledge for out-of-distributioo questions, leading to a significant performance drop (see Figure 3(b) and the left table in Table 2). Unlike MM-CoTT [71], we resort to LLMM to generate rationales with zero-shot prompting, leveraging their intrinsic capacity for generalization. We further explore the strategies to generate rationales.
Uni-modal rationales have limited effect. We start by directly prompting "let's think step by step" without any image information, resulting in visually irrelevant rationale (see Figure 4(a)) and poor performance (see row "w/ naive R" in the right table in Table 2).
Interleaved information exacerbates hallucinations. To generate multimodal rationales involving image information, the challenge lies in mitigating the language hallucinations [24, 36, 17, 2}. which is exacerbated by providing interleaved multimodal information at once. A naive attempt is to extract the image caption and combine it with the question rationales. Despite the integration of image information, the generated rationales remain suboptimal for image-related questions, which is not aligned with our initial expectations. Upon analyzing various cases, we observe that interleaved information predisposes LLMs towards hallucination, generating fabricated visual information. As illustrated in Figure 4, we extracted the caption "a food web" from the image, which lacks the necessary information. Consequently. the language model resorts to imagining image-related information such as "the primary consumers are the zooplankton and the kelp" (see Figure 4(b)), leading to difficulty in distinguishing reliable knowledge from the language hallucinations.
joint prompt to generate
3.2 Zero-shot DDCoT Prompting for Multimodal Rationale Generation
We generate general multimodal rationales based on the following key insights, which are derived from Section 3:1:
(1) Utilize two-step reasoning process, considering that L.M has difficulty jointly reasoning multi- modal information directly (Section 3.).
5

---

Siotement
DDCoT
Question:
Which the main persusive spoeal used in this ad?
Split:: Please think steg-by-step and deconstruct the quest on down to sub-questions
Options:
pethos (emotion) ethos (cheracter) logos (reeson)
Sub-question:
(1) What is the image the ad?
(2) What emotions characters are depicted the image?
Quastion
Zero-shot
Are there words stogans the that can provide dues to the persuasive
Relionses)
egpeal
Sub-answers: (1) Uncertain (2) Uncortain (3) Uncortain
VQA: Understending (xb-questioos image]
Brand
LLMs
Sub-answers: (1) sica (2) catl33
the slogan. has bwice
meet of the leading brand)
2x the meat
Thought: not smoton and
Answear
Reorgnize: Rationale: image
cat,
ard offeri
thairters amottins depicted
imagel slogan used in ad "vist
Fine-cune
has twice the meat of the kaaiing brand," which appeats to the busionnels resson lsdie. Therefore, the main persuasive appeat used in this ac logos (reason). Output: enswer C.
Rationales
Embesdding
Question
LM Encoder Layer
Embeddiing
Quwation
JeSSwer
MMCsT
advertisment Rationale: train[GT-R] Solution: purpose
UM Encoder Layer
persuade people to do something An ad that appeals to patios (eihes do one of the following: ntrigger fear, such es the fear of embarassmentran a dasiro, such as the destrn to appear attratheniin the product to a pasitive feeling, such as adventure, love, or koury The ad appeals to pothos, ar amotiian arguesi that the product a postive feeling, such as adventure, love, or loury.
to
Emipddiing
LM Decodar
Prosee
Peekkre
Layer Phanol
Figure 5: An overview of our DDCoT and its utilization to improve the multimodal reasoning of LMs. Note that although errors encounter in the second sub-problem during visual recognition, the language model rectifies this error in the joint reasoning step with critical thought.
(2) Generate rationales that meet the requirements of both zero-shot and fine-tuning learning, filled with knowledge and requiring fidelity (Section 3.1.2).
(3) Alleviate the intensified hallucinations from interleaved information (Section 3.1.3).
Accordingly, we propose Duty-Distinct Chain-of- ofoughttt Prompting (DDCoT), which encompasses three steps: (1) We utilize LLMs intrinsic knowledge to generate multimodal rationales. (2) We explicitly cue the LLMs to differentiate the responsibilities of reasoning and recognition step by step. (3) We explicitly mark the negative space for uncertain parts, emphasizing critical thinking in rationale generation. Our DDCoT jointly exploits the reasoning ability in LLMs and the image understanding capability of visual question- guestion-answering models for general multimodal rationale generation.
Breaking Reasoning Down to Recognition Steps with Negative-Space Prompting.
First, with the given question, context, and options, we prompt the LLM [37] to deconstruct the input question into a sequence of basic sub-questions breaking the complex reasoning chain into simple steps, as shown in Figure 5. Diverging prior works NLP community [9, 19], we introduce the following prompting designs: (1) We utilize single-stage deconstruction to simplify the problem- solving process. Specifically, we employ the instruction "please think step-by-step and deconstruct the question down necessary sub rub-questions" to obtain the sub-question sequence at once. (2) Then we explicitly prompt the LLMs to determine whether each sub-question can be answered without visual information. In cases where sub-question involving visual recognition are unanswerable, the LLMs are instructed to answer "uncertainty" as a negative space. We provide the model with the following prompt: "Assume that you do not have any information about the picture, try to answer the sub- queb-uueno and formulate the corresponding sub-answed as 'Uncertain' if the sub-qu question cannot be determined". By adopting critical stance towards sub-questions and introducing the explicit assumption of invisibility, we successfully mitigate hallucination in LLMs when dealing with sub-questions involving images, reducing factual errors.
Visual Recognition to Obtain Visual Complements. However, the obstruction caused by negative space prevents LLMs from directly activating the chain-of-though! reasoning. In order to fill the negative space, we leverage the image understanding capability of the off-the-shelff models to acquire visual information as visual complements. Specifically, we employ the visual question answering (VQA) model [26] to individually answer the sub-questions with negative space, which correspond to simple visual recognition problems. Note that our approach can solely leverage the basic visual recognition capability of the VQA model and robust to its possible inference errors thanks to the following joint reasoning integration.
Integrate to Joint Reasoning. With a sequence complete sub-answers including visual recogni- tion results, we resort to LLMs again to integrate the information and engage in reasoning processes

---

to derive the rationales. Incorporating our obtained sub-question and corresponding sub- answers as supplementary information, we prompt LLMs "think step by step" to perform joint reasoning with linguistic and visual information and generate multimodal rationales. Furthermore, we prompt explicitly with "note that the supplementary information given may not always be valid" and "select valid information to form the rationale" " to encourage a critical attitude towards the supplementary information and thoughtful analysis. By explicitly highlighting uncertainty, LLMs are able to pay attention to the origin question information and intrinsic knowledge, effectively filtering and even rectifying supplementary knowledge to derive more reasonable answers, as shown in Figure 5.
3.3 Utilization of Multimodal Rationales in General Scenarios
In this section, we introduce the utilization of our rationales to achieve multimodal reasoning, both for zero-shot and fine-tuning learning. For zero-shot learning, we consider prompting the model with rationales, including established knowledge and reasoning chain of thought. Additionally, for fine-tuning, our proposed rationales, in combination with our proposed deep-layer prompting and rationale-compresseed visual embedding, improve deep multimodal understanding and reasoning.
Utilization for Zero-shot Prompting, As shown in Figure 5, for LLMs like ChatGPT, we utilize zero-shot prompts that combine rationales generated by Section 3.2 with our problem statements as inputs to the model. Our rationales generated by explicitly questioning the uncertainty with improving reliability can facilitate LLMs to make more precise assessments and exhibit fewer hallucinations.
Utilization for Fine-tuning Learning. The framework of the fine-tuning process is shown in Figure 5. During fine-tuning, we introduce learnable prompts in both shallow and deep layers to align cross-modal information and facilitate multimodal reasoning. In addition, instead of directly inputting the entire image into the LM, we use the multimodal rationales to guide the filtering of key image features as visual input embeddings for the LM.
Deep-Layer Prompting (DLP) is designed to assist in the alignment and joint reasoning of multimodal inputs multiple levels. It employs not only learnable prompts facilitate the alignment of visual and linguistic semantics at a shallow level [74, 73, 51, 34] but also utilizes explicit rationales to jointly encode multimodality by learning different prompts for each encoder layer [18, 21. 30, 29]. Specifically, we randomly initialize learnable prompts P E R NC for T. encoder layers. For the 1-th layer, we load the prompt P: N2Cx the begging and end of the visual input embeddings.
Camponal-Compressed Visual Embedding (RCVE). Instead of inputting visual features into the LM directly, we compress visual input embeddings according to our multimodal rationales. Specifically, we leverage the rationales to jointly comprehend the textual and visual contexts, serving as the prior knowledge filtering visual features. Given the text embeddings T R N. that contains context and rationale, the global visual input Vg E RC, and the local visual inputs Vg, E IR XC we first update the global visual feature based its similarity to text embeddings as follows.
V1 = Attention (V..))
(1)
where V E RC is the updated visual feature and Attention(-. -) is the standard multi-head cross- attention module [56]. Next, instead of directly using the updated visual feature Vú capture relevant local visual inputs Vi, we introduce low-rank intermediate vectors as crucial mediators for filtering local inputs as follows:
V, reshapeeMMLL (V)) V Attention( Vr, Vi),
where MLP(-) denotes a three-layer linear layer with activation function, V, N.C,C. represents N, low-rank vectors, and V is the final visual embeddings to input to the LM's encoder.
4 Experiment
ScienceQA benchmark [31] is the first multimodal science question-answer dataset comprising 21,000 questions with multiple choices and images. Following previous works [1], we divide ScienceQA into training, validation, and test sets, which contain 12,726, 4,241, and 4,241 examples, respectively. Additionally, the questions in ScienceQA can be categorized into three domains: natural science (NAT), social science (SOC), and language science (LAN).
Implementation. We conduct zero-shot experiments on L.MSs including ChatGPT [37] and GPT- 3 [3]. Since LLMs are unable to accept images as visual input, convert images into captions
7

---

Model
Report
Sizel, GT-R
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Avg
Heuristic baselines
Random [31]
NeurlPS22
40.28
46.13
29.25
47.45
40.08
33.66
39.35
40.67
39.83
Human [31]
NeurlPS22
90.23
84.97 87.48
89.60
87.50
88.10
91.59
82:42
88.40
Few-shot models
GPT-3(CoT) [31]
NeurlPS22
175B
<
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
75.17
Cha) [33]
arXivas
175B
78.82
70.98
83.18
77.37
67.92
86.15
80.72
74.03
78.31
ChamcleonChaalGP) [33]
arXivas
175B
81.62
70.64
84.00
79.77
70.80
86.62
81.86
76.53
79.93
Zero-shot models
Ours(GPT-3)
175B
78.60
73.90 80.45
77.27
69.96
82.93
80.65
73.50
78.09
Ours(ChaIGPT)
175B
80.15 76.72 82.82
78.89
72.53
85.02
82.86
75.21
80.15
Fine-tuning models
LLaMA based Fine-tuning models
LMAdapter [69]
arXiv23
7B
84.37 88.30 84.36
83.72
80.32
86.90
85.83
84.05
85.19
TS based Fine-taming models
UnifiedQA [31]
22
223M
71.00 76.04 78.91
66.42
66.53
81.81
77.06
68.82
74.11
MMCoT [71]
arXiv
223M
87.52
77.17 85.82
87.88
82.90
86.83
84.65
85.37
84.91
UnifiedQA [31]
NeurlPS22
223M
68.16 69.18 74.91
63.78
61.38
77.84
72.98
65.00
70.12
MMCoT! [71]
arXiv
223M
82.51 77.73 82.82
81.09
75.11
85.30
81.64
80.95
81.40
Ours
223M
88.72 86.84 84,91
87.59
83.34
88.08
888.58
85.10
87.34
+6.21 +9.11 +2.09
+6.50
+8.23
+2.78
+6.94
+4.15
+5.94
Table 1: Main results (%). Size backbone model size. GT-R means models are trained with ground truth rationales. Question classes: NAT natural science, SOC = social science, LAN language science, TXT = text context, IMG image context, NO no context, G1-6 grades 1-6, G7-12 grades 7-12. f denotes implementation by removing ground truth rationales when fine-tuning.
via BLIP-2 [26] as extra input to LLMs. Furthermore, following previous works [71, 31J, we adopt UnifiedQA [20] our base model for fine-tuning. We CLIP ViT-L/14 [41] as the visual encoder to extract the global visual input V, and the local visual input Vi. The hyperparamters Np. N, and C. are 3, 16 and 4, respectively. We train our model for 30 epochs with learning rate of 1e-4 and batch size of 16. All experiments are implemented by PyTorrch [39] and HuggingFace [61] and conducted on NVIDIA Tesla A40 GPUs. We employ accuracy our evaluation metric and furnish comprehensive results across each domain.
4.1 Comparison with State-of-the-Arts in Zero-shot Prompting and Fine-tuning Learning
Table I shows the comparison of our DDCoT with the kate-of-the-an models [31, 33, 71] zero-shot and fine-tuning benchmarks. Our approach consistently achieves superior performance compared to previous methods. Note that we further report the results of the recent works [33, 71, 69] which have not been published but have been preprinted in the research community.
Zero-shot Prompting. Regarding zero-shot prompting, our zero-shot approach outperforms the pub- lished state-of-the-eee tft few-shot method [31] by 2.92% on GPT-3 and 1. 1.84% on ChatGPT, respectively, Compared with the concurrent work Chameleon [33], which incorporates a vast array of external tools, our method maintains comparable performance, even attaining 1. .73% enhancement on the IMG split. These suggest that our DDCoTT, integrated with negative- space prompting, effectively curtails factual errors, guarantecing the accuracy of multimodal reasoning in LLMs. Furthermore, we observe that the performance of GPT-3 (67. (77% and ChatGPT (67. 66799%% in few-shot manner is similar, and the amplification of our technique in the IMG branch escalates as the performance of the language models strengthens (2.53% for GPT-3 and 4.61% for ChatGPT), The possible reason is language models rarely acquire an innate understanding of dense image information, while explicit rationales guidance triggers the inherent reasoning capability of such models.
Fine-tuning Learning. As shown in Table 1, our DDCoT has achieved exceptional accuracy on fine-tuning benchmarks. DDCoT significantly surpasses the base model UnifiedQA [20] by 17.22% and 21. 21.96% on avg split and IMG split, respectively. These demonstrate that our rationales not only enhance the model's multimodal reasoning ability in zero-shot setting but also help the fine-tuning of
8

---

LMs to achieve multimodal alignment and joint inference. In comparison to the Co based approach MM-CoT [71] which uses annotated rationales for training, our multimodal rationales, generated through zero-shot CoT prompting, still enhance performance by an average of 2.43%.
(b) Analysis
IMG
TXT
Avg
(a) Generalization
NAT
SOC
LAN
busehie(BB
72.93
85.84
79.7
MMCoT
50.98
77.73
80.00
B+caption
75.26
83.64
79.6
Ours
66.43
83.69
83.09
B+origin img
62.82
75.94
69.70
MMCoT
78.86
62.20
76.82
B+DLP&RCVE
75,16
85,25
80,45
Ours
86.5
71.77
85.09
R
75.16
85.25
80.45
MMCoT
79.53
76.38
55.82
w/ naive w/ our R
75.06
89.61
82.96
Ours
86.59
86.73
68.00
83.34
91.23
87.34
Table 2: The results of (a) Generalization and (b) Modality and Rationale Analysis.
Generalization. As shown in Table 2(a), conduct supplementary experiment to evaluate the model's generalization capabilities after fine-tuning. By designating two domains for visibility during training, we report the accuracy of the questions in the unseen remaining domain within the test set. Across all three divisions, our approach consistently surpasses MM-CoT by 15.5%, 9.6%, and 12.2%, respectively. These results demonstrate the excellent scalability of our rationales when handling out-of-distributin data.
4.2 Analysis Effects of Visual Modality, Rationale Generation, and Fine-tuning Components
The effects of different modalities of visual information. Table 2 (b) presents the effects of different forms of image information the fine-tuning models: (1) We start by directly adding captions or image inputs to the language only T5 based baseline. The former results in an improvement of 2.33% on IMG split. However, the absence of additional components and the direct usage of images as input leads to poor performance. The result suggests that models with a confined number of parameters encounter difficulties aligning images with text on small datasets. (2) Nevertheless, with the aid of our well-designed components, including leep-layer prompting (DLP) and rational-compresse visual embedding (RCVE), the language model effectively harnesses the information gleaned from images, thereby achieving the comparable performance of 75.16% on IMG split with the model using captions as inputs. (3) Despite both methods enhancing performance on IMG split, the improvement remains unsatisfactory due to the language model's 's difficulty in comprehending image information without guidance, which is also observed the zero-shot scenario delineated in Section 3.1. Simultaneously, observe slight degradation in the performance of the text split, which is potentially attributed to the disparity in inputs between IMG and TXT splits in training data.
The effects of rationale generation compo- nents. As shown in Table 2(b) and Table 3, we further demonstrate the effectiveness of the components of our DDCoT prompting for ra- tionale generation. (1) With naive rationales obtained directly from prompting GPT-3 with "let's think step by step", we noted that the IMG branch garners no benefits, even evidencing a
IMG
Avg
naive R
75.06
82.96
decomposed question R duty distinct w/o uncertainty R
75.61
83.09
78.19
85.15
duty distinct w/ uncertuinty R
83.34
87.34
w/o integrate to joint reasoning
77.49
83.75
Table 3: Ablation study on components of DDCoT. slight degradation in performance. It indicates that introducing rationales can facilitate the language model in comprehending and reasoning within the context, while image-agnostic rationales do not contribute to the model's multimodal understanding. (2) Compared to the naive rationale [25], deconstruction [72, 22] and joint reasoning without specialized designs result in a tiny improvement (row 2 in Table 3). The limited improvement be attributed to the fact that LLMs still perform combined reasoning and recognition task, leading to a suboptimal execution of the recognition task. (3) Hence, our duty distinct design, implemented through negative-space prompting, achieves 2.58% and 2.06% performance gain on IMG and average splits (row 3 in Table 3). It alleviates the reliance on the reasoning capability of the off-the-shelf visual model and generate more reliable multimodal rationales. (4) Moreover, by explicitly emphasizing uncertainty (row 4 in Table 3), we observe a notable improvement of 5.15% on IMG split, while also contributing to an average enhancement of 2. 1.9. It achieves remarkable enhancement of correctness, which also significantly contributes to fine-tuning learning.

---

Quantitative Analysis of Hallucinations. Additionally, we conduct human evaluation of the authenticity to assess the phenomenon of hallucinations [24] in the rationale gener- ation process. As shown in the Table 4, in- troducing interleaved visual information exac- erbates the hallucinations and diminishes the authenticity of generated rationales by 28.1% 1% compared with naive prompting devoid of any visual information input. Our duty- dett-dditinnt design significantly mitigates the impact of hallucinations. More- over, the further suppression of hallucinations is observed with the explicit emphasis on uncertainty. These feedbacks align with the findings in Section 3.1.3 and the results presented in Table 3.
Method
Context
Authenticity
naive R
w/o visual
0.883
naive R
interleaved
0.602
duty distinct w/o uncertainty R interleaved
0.783
duty distinct uncertainty R.
interleaved
0.855
Table 4: Human evaluation on hallucinations.
The roles of our fine-tuning components. We also vali- date the impact of our proposed deep-layer prompting and comppessed visual embedding. We replace the rational-compressed embedding with origin visual features, resulting in reductions of 1.28% and 3.02% in average per- formance and IMG split, respectively. Similarly, removing deep-layer prompting which assists alignment leads to a 0.49% and 0.99% reduction. Additionally, we further conduct ablation studies the hyperparameters of No and N,. Detailed results and analysis will be presented in the supplementary material.
IMG
Avg
Ours w/ RCVE w/o DLP&RCVE
87.34 83,34
80.32
86.06
79.33
85.57
Table 5: Ablation study on fine-tuning components.
Comparation with GT-R
Human Evaluation
Method
w/o GT-R
B-1
B-4 R-L Sim.
Relevant
Correct
Complete
Coherent
Explainable
MMCOT
0.970 0.930 0.970 0.990
70.83%
67.99%
64.81% %%
57.94%
58.73%
GPT-3
0.075 0.018 0.249 0.548
81.01%
75.99%
65.54%
61.64%
60.32%
Ours
0.147 0.041 0.287 0.601
92.00%
86.38%
85.71%
84.33%
83.26%
Table 6: Automatic metrics and human evaluation of explainability.
Explainability. We further exhibit the explainability evaluation including the automatic metrics and human evaluations in Table 6. In automatic metrics, we adopt the BLEU- 1/4 [38]. ROUGE- L [38], and Sentence Similarity [42] metrics. Note that the automatic evaluation solely reflects the similarity between the generated rationales and the annotated ground truth. In the course of human evaluation, annotators are required grade each rationale on the criteria of Relevance, Correctness, Completeness, Coherence, and Explainability. It is worth noting that although previous methods [31, 71] achieve decent results in terms of Relevant and Correct, they perform particularly poorly in other areas, especially Explainability. In contrast, despite modest automatic metric results, our rationales generated by DDCoT significantly surpass other methods across all aspects of human evaluation, which is more valuable for interpretable studies.
5 Conclusion
This paper proposes a novel DDCoT prompting for multimodal reasoning tasks with language models. Not only achieving state-of-the-art performance on zero-shot learning, our model, with the novel techniques of deep-layer prompting and rational-compressed visual embedding. also demonstrates significant reasoning ability on the ScienceQA benchmark. The experimental results demonstrate the superiority and generalization ability of our proposed DDCoT for both zero-shot learning and fine-tuning. Ethical Statement. Our model is based on currently available pre-trained language models. In our experiments, we have not utilized any information concerning personal privacy, data security, or ethical hazards. However, it is crucial to acknowledge that large language models may introduce certain biases, which can originate from the training datasets. We must recognize that entirely climinating biases from the model is a challenging task, and current technology may not achieve it completely. Therefore, we encourage users to be mindful of these potential biases when utilizing our model and to comprehend that the model's outputs are not infallible but require interpretation and understanding in conjunction with real-world circumstances.
Acknowledgment: This work was supported by the National Natural Science Foundation of China (222611744 Shanghai Pujiang Program (No.21 PJ141000)), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), MoE Key Laboratory of Intelligent Perception and Human-Machinn Collaboration (ShanghaiTed) University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
10

---

References
[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9948-8957, 2019.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[4] Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870-7881, 2020.
[5] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Discntangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.
2022.
[6] Jaemin Cho, Lei. Hao Tan, and Mohit Bansal. Unifying vision-and- languan tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.023111 2022.
[8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instraction-fimetune language models, arXiv preprint arXiv:22 10. 1416, 2022.
[9] MJ Collins. tutorial on dual decomposition and lagrangian relaxation for inference in natural language processing. Journal of Artificial Intelligence Research, 33:305-362, 2012.
[10] Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- bassd prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720. 2022.
[11] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211. 10435, 2022.
[12] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijic Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama- Llama-adapter v2: Parameter-effieient visual instruction model. arXiv preprint arXiv:2304. 15010. 2023.
[13] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. arXiv preprint 22X 2022.
[14] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 328-339 2018
[15] Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu. Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. arXiv preprint 2309 14494, 2023.
[16] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need; Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
[17] YEJIN BANG ISHII, ANDREA MADOTTO, and PASCALE FUNG. Survey of hallucination in natural language generation. ACM Comput. Surv, 1(1), 2022.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[19] Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, and Qi Wu. Overcoming language priors in vqa via decomposed linguistic representations. In Proceedings of the AAAI conference on artificial intelligence. volume 34, pages 11188, 2020.
11

---

[20] Daniel Khashabi, Sewon Min. Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguisticx; EMNLP 2020, pages 1896-1907, 2020.
[21] Muhammad Uzait Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[22] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. arXiv preprint axXiv22100024006, 2022.
[23] Ryan Kiros, Yukun Zhu, Russ R Salakhuidinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Advances in neural information processing systems, 28, 2015.
[24] Philipp Koehn and Rebecca Knowles, Six challenges for neural machine translation. arXiv preprint arXiv:1766.00 0100000372.222 2017.
[25] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2222 2022.
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language- kinuaae-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[27] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Man-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02338 2022.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint Si4iv:2304004483 2023.
[29] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10.0066,, 2021.
[30] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-uning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 61-68, 2022.
[31] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances Neural Information Processing Systems, 35:2507-2521 2022.
[32] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Mang-Chun& Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022.
[33] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXivx2304..0000 098-2, 2023.
[34] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206- 5215, 2022.
[35] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410. 2022.
[36] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings the 58th Anmual Meeting of the Association for Compitational Linguistics, pages 1906-1919, 2020.
[37] Long Ouyang, Jeffrey Wa, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 06:27730-27744, 2022.
[38] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
12

---

[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chaman, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[40] Fabio Petroni, Tim Rocktäscell, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lunguage Processing (EMNLP-IJCNLP),, pages 2663,243,, 2019.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[42] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:/908, 10084, 2019.
[43] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426 0-265, 2020.
[44] Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.0066 2021.
[45] Vasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metries. In Proceedings of the Seventh Workshop on Building Educa- rional Applications Using NLP, pages 157-162, Montréal, Canada, 2012. Association for Computational Linguisties.
[46] Shikhar Sharma, Layla El Asti, Hannes Schulz, and Jeremie Zumer. Relevance unsupervised metries ceie-orieted dialogue for evaluating natural language generation. arXiv preprint arXiv: 09799, 2017.
[47] Yongliang Shen, Kaitao Song, Xu Tan. Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580, 2023.
[48] Cheng Shi and Sibei Yang. Spatial and visual perspective-taking via view rotation and relation reasoning for emhodied reference understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 201-218. Springer, 2022.
[49] Cheng Shi and Sibci Yang. Edadet: Open-vocabulary object detection using early dense alignment. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2023.
[50] Cheng Shi and Sibei Yang. Logoppompt:synheetic text images can be good visual prompts for vision- language models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2023.
[51] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-lahel recognition with limited annotations. Advances in Neural Information Processing Systems, 35:30569-30582, 2022.
[52] Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Contrastive grouping with transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 23570-23580, 2023.
[53] Jiajin Tang, Ge Zheng, and Sibei Yang. Temporal collection and distribution for referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15466- 15476, 2023.
[54] Jiajin Tang, Ge Zheng, Jingyi Yu, and Sibei Yang. Cotdet: Affordance knowledge prompting for task driven object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3068-3078, 2023.
[55] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2302 13971, 2023.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
13

---

[57] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[58] Xingyao Wang, Sha Li, and Heng Ji. Code4struct: Code generation for few-shot structured prediction from natural language. arXiv preprint arXiv:2210.12810 2022.
[59] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. /1171,, 2022.
[60] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le. Denny Zhou, al. Chain-of- thought prompting elicits reasoning in large language models, Advances Neural Information Processing Systems, 35:24824-2483 2022.
[61] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the the- natural language processing, In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[62] Chenfei Wu, Shengming Yin, Weizhen Qi. Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint 27X1423303,0661, 2023.
[63] Yu Wu, Lu Jiang, and Yi Yang. Switchable novel object captioner. IEEE Transactions on Pattern Analysis 102-111121111711 2022.
and Machine Intelligence,
[64] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1653, 2017.
[65] Jiacheng Yang. Mingxuan Wang, Hao Zhou, Chengqi Zhao, Weinan Zhang. Yong Yu, and Lei Li. Towards making the most of bert in neural machine translation. In Proceedings of AAAI conference on artificial intelligence, volume 34, pages 9378 9385, 2020.
[66] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:22/0.03629, 2022.
[67] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[68] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476 15488, 2022.
[69] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient finc-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[70] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.0343 2022.
[71] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 2023.
[72] Denny Zhou. Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans. Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
[73] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[74] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vasion-language models. International Journal of Computer Vision, 130(9):2377-235 2022.
[75] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint Xr0vv:2304. 10592, 2023.
14

---

Appendix
In this section, present additional implementation details, experiment results and discussions. The content structure is outlined as follows:
Section Additional Results
Section A. Insights for Different GPT Models
Section A.2 Quantitative ablations
DDCoT and visual components
Section A.3 Hyperparameters for Fine-tuning Components
Section A.4 Additional experiments the effectiveness of our DDCoT with existing
pre-trained VLMs and multimodal reasoning models
Section A.5 - Additional experiments on Captioning and Video Question Answering
tasks
Section Human Evaluation
Section C- Detailed Prompts
Section D Case Studies
Section Limitations
A
Additional Results
Question
Sparse Caption
Two sets of magnets are of equal size. The magnets in pair feet apart, magnets pair feet apart.
Think about the magnetic force between the magnets in each pair. Which of the following statements true?
GPT-3:
ChatGPT: A
Context:
Dense Caption
The images below show two pairs of magnets. The magnets in different pairs affect each other. All magnets shown are made of the same meterial.
the figure, there are two of magnets: pairl the left and pair2 on the right. In pairl, the left magnet and
pole the right, while the right magnet has the pole on the left and pole the right. Similarly, in pair2,
the
Options:
left magnet has the S pole on the left and the N pole on the right, and the right magnet has the pole on and the pole on the right.
The magnetic force stronger Pair 2.
B. The magnetic force is stronger in Pair 1.
The strength the magnetic force same in both pairs.
Pairl consists of straight of the left magnet to the left end of the right magnet. distance labeled
connecting
right
Pair
Pair
in. Similarly, pair2, the straight line between the end the left magnet and the left end the right magnet is labeled as 2 in.
GPT-3:
ChatGPT:
Question
Sparse Caption
text shows The best in class for over years!!
rhetorical appeal primarily
GPT-3:
ChatGPT:
Context: N/A
Dense Caption
rider depicted riding into the distance on dirt road, with green fields on both sides, In the lower right comee, white text, proudly declares "The best its class for yearst. upper right corner, white text reads "STIMCOM", accompanied by black text stating "Hip Replacements".
Options: A, ethos (character)
BE ITS CLASS FOR OVER YEARS
B. logos (reason)
C. pathos (emotion)
GPT-3:
ChatGPT:
Figure Cases that GPT-3 and ChatGPT face the difficulty in understanding dense image information.
A.1 Insights for Different GPT Models
In the submitted paper we have presented several findings concerning the L.MMs. this section, we aim to provide additional illustrative instances for GPT-3 [3] and the recent and potent ChatGPT [37].
15

---

Difficulty in understanding dense image information. Figure 6 presents additional instances of LLMs failing to comprehend dense image information. For sparse captions like the example in Figure 2 of the submitted paper, we observe that both GPT-3 and ChatGPT may struggle to comprehend image information in captions. Additionally, when prompted with more detailed captions, their difficulties in understanding become more pronounced. This highlights the challenges that even the most versatile and powerful language models currently available face in comprehending dense image information.
Question
w/o rationale
following fossils older? more likely answer.
N/A
Context:
GPT-3:
ChatGPT: C
This diagram shows fossils in an undisturbed sedimentary rock sequence.
Caption:
correct rationale
figure displays three sedimentary rock layers. uppermost layer contains palm leaf fossils, the second layer contains root fossils, and the bottom layer contains wood fossils.
The shallower, the older
GPT-3:
ChatGPT:
Options: root
wrong rationale
palm leaf
The deeper, the older
C. wood
GPT-3: B
ChatGPT: B
Question
w/o rationale
Complete the text describe diagram. Solute particles moved both directions across the permeable membrane. But more solute particles moved across the membrane
N/A
GPT-3: A
ChatGPT:
correct rationale
Context:
As can be seen from the image, the initial state particles the right brane and three particles membrane. As time passes, the left side beco- mes four particles, i.e., one particle gainedi the right side also becomes four particles, one particle is losti Therefore, the direction of motion of the particles is to left.
When there was an equal concentration on both sides, the particles reached equilibiim. The diagram below solute. Each solute particle is represented by green ball. The solution fills a closed container that divided in half by membrane. The membrane, represented by dotted line, is permeable to the solute particles. The diagram shows how the solution can change over time during the process of diffusion.
the mem- the left side of the
solution with
Caption:
GPT-3: A
ChatGPT: A
The figure illustrates the state transition of box. the initial state, there are three green particles to the left of the box's center line and five green particles to the right. the subsequent state, the box shows four green particles on each side of the center line
wrong rationale
can be seen from image, has five particles the right side of the mem- brane and three particles on the left side of the membrane, time passes, four particles, i.e., particle is losti the right side also becomes four particles, lee, one particle is gained, Therefore, the direction of motion of the particles right.
state
beco-
Options:
to than right
right than the left
GPT-3:
ChatGPT:
Figure 7: Cases of rationale-sensitive reasoning with GPT-3 and ChatGPT.
Rationale-sensitive reasoning. Figure 7 shows more examples where the reasoning of GPT-3 and ChatGPT sensitive to the input rationales. The correct answers obtained without using rationales indicate that the LLMSs possess commonsense knowledge to respond to the question. However, incorrect inputs can lead to misleading outcomes for both GPT-3 [3] and ChatGPT [37].
Note that the challenges involved in understanding dense image information, coupled with the difficulty of obtaining high-quality captions in practical scenarios, limit the universal applicability of reasoning without using rationales, although it may be feasible in the presented examples.
Hallucinations during generation. Figure 8 illustrates several instances where both GPT-3 and ChatGPT fall into hallucinations during rationale generation.
Our findings indicate that in cases where the provided caption information is insufficient, such as in the first example, both GPT-3 [3] and ChatGPT [37] tend to imagine image details to respond to the question. Even when provide ample information, demonstrated in the second and third
16

---

Question
Compare the average kinetic energies of the particles in each sample. Which sample has higher temperature?
Rationale: The particles in sample B have more kinetic energy average than particles in sample This means that sample has a higher temperature.
Context:
The diagrams below show two pure samples of ges Identical closed. rigid containers. Each colored ball represents one gas particie. Both samples have the same number of particles.
ChatGPT
Rationale:
The average kinetic energy of gas particles is directly proportional to temperature
Caption:
gas. Sample average kinetic energy because its particles are moving faster and range velocities compared Sample Therefore, Sample B has higher temperature than Sample A.
higher
The diagrams helow show
two pure samples and
Options:
A. sample
sample
Ours DDCoT w/o Caption
neither, the samples have same
The average kinetic energy of gas particles is directly proportional temperature, particles sample al much faster speed than the particles in sample B, Both samples
Rationale:
moving
temperature
Mass aatis/44
particles. Therefore, sample has a higher temperature than sample B.
700ell
sevt 780m)
Question
GPT-3
Rationale: continent highlighted Europe.
Which continent highlighted?
because Europe the only continent that is shaped as an Inverted sharp corner right a monned shape on the upper left)
triangle
Context: N/A
Caption:
ChatGPT
The picture shows a map of the world. In it, the area right in the middle highlighted. It roughly shaped inverted triangle. It has a sharp corner on the right and rounded shape on the upper left. is connected to peninsula the top right and detached
Rationale: Based on the description of the highlighted area, we the Arabian Peninsula. The peninsula is connected to detached Island at the bottom right, which is the island Socotra. The peninsula also connected to larger landmass at the top right, which is the continent of Asia. Therefore, the correct answer Asia.
bottom right.
Options:
Europe
B. Africa
Ours DDCoT w/o Caption
C. Asia
The highlighted desert as its geographical feature. The neighboring countries are not relevant to the question. By looking the image and considering the supolementary information, we can determine that the highlighted area Africa.
Rationale:
cirde and has Sahara
South
Question
GPT-3
The Punnett square only shows the possible genotypes offspring, the phenotype. determined by the body size allele, which controlled by the B The B allele dominant, the phenotype the offspring determined by the B allele. The allele is present in out of the possible genotypes, the probability that the offspring will have normal-sized body 3/4.
Rationale:
What the probability that a rat produced by this cross will body?
case, phenotype
Context:
In group of rats, some individuals have a normal-sized body others dwarf body. In this group, the gene the body size trait has two alleles. The allele for dwarf bady (b) recessive to the allele for normal-sized sized body (B). This Punnett cross
ChatGPT
square
rats.
Rationale: The Punnett square shows the possible genotypes of the offspring. determine the probabiiity of rat having dwarf body, need look at the genatype of the rat. least
Caption:
The image shows table of genes. The horizontal axis Bb, and the vertical axis BB
has at that
of the recessive allele (bb), It will have a dwarf Punnett square, we
body. Looking four possible genotypes of the offspring: 88, Bb, Bb, and bb. Only last genatype (bb) one way to get bi genatype, which by crossing two Bb ratsi Therefore, the probability dwarf body is 1/4 or 25%..
B
b
Options: 3/4
dwarf
B. 0/4
having
BB
Bb
C. 2/4
D. 1/4
E. 4/4
Ours DDCoT w/o Caption
The Punnett square that genotypes parent rats are Bb and Bb. The possible genatypes of offspring bb.
BB
Bb
body (b) recessive the allele for normal-size sized body (B). Therefore, only the bb ge- notype will result dwarf body. The probabilty
offspr-ing inheriting the recessive allele for dwarf body (bb) 1/4 or 25%.
Figure 8: Cases of hallucinations when generating rationales with GPT-3 and ChatGPT.
17

---

examples, we observe that both language models still may fall into ballucinations that do not align with the provided information.
In contrast to generating rationales solely based on the caption and question information, our approach alleviate the hallucinations some extent by decomposing the questions into simple recognition tasks and emphasizing the uncertainty of image-related aspects. It is worth noting that while the former direct methods rely on manually crafted high-quality rationales, our approach utilizes BLIP- 2 [26] as visual question answering (VQA) model, serving as a visual component. However, our approach outperforms the former methods in terms of performance and interpretability.
Unfortunately, is very challenging to entirely solve the issues of hallucinations. Although we alleviate hallucinations, we still encountered difficulties in certain cases, such as the third example, and further details regarding this limitation will be explored in section E.
IMG
TXT
Avg
baseline(B)
72.93
85.84
79.70
B + our R B gt R our model our model our
75.81
82.40
82.83
81.07
84.07
85.97
75.16
85.25
80.45
83.34
91.20
87.34
our model gt R
84.43
92.09
88.00
Table 7: Quantitative ablations
DDCoT and visual components.
A.2 Quantitative ablations on our DDCoT and visual components
We conduct additional ablation study on the extent of impact exerted by our DDCoT prompting and visual components, as shown in Table 7. DDCoT prompting and visual components cooperatively facilitate inducing visual information to language models for multimodal reasoning. We can observe that rationales generated by our DDCoT and visual components individually exhibit certain gains in terms of IMG improvement. However, when combined, they yield substantial gains.
Besides, please note that the annotated ground truth rationales within the ScienceQA dataset inherently encompass the final prediction, i.e., correct answers. To ensure fair comparison, manually exclude the answers from these annotations, using the remaining text as input rationales for fine-tuning. Under this configuration, our proposed rationale achieves fine-tuning performance comparable to the annotated rationales.
N
5
N,
8
16
32
Avg
86.72
87.34
86.02
Avg
86.63
87.34
86.30
Table 8: Ablation of N, and N,.
A.3 Hyperparameters for Fine-tuning Components
Table presents the results of our ablation studies on Np, which denotes the number of learnable prompt tokens, and N,, which represents the number of low-rank vectors. Regarding Np, we conducted experiments using values of 1, 3, and 5. Increasing the number of prompts introduces more learnable parameters and provide stronger guidance align vision and language, while too many prompts may disrupt the model's comprehension performance. Considering N,, we investigated the influence of different filtering intensity in the Rational-Compressse Visual Embedding process by experimenting with values of 8, 16, and 32. The experimental results indicate that selecting 3 for N₂ and 16 for N,, yields the best performance.
visual features and result in a decline in
A.4 Additional experiments on the effectiveness of our DDCoT with existing pre-trained VLMS and multimodal reasoning models
We also validate the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models, as shown in Table We observe that rationales generated by our proposed DDCoT is compatible with such pre-trained VLMs. Without correct rationales, existing pretrained
18

---

Model
NAT
SOC
LAN
TXT
IMG
NO
Avg
Flamingo [2]
21.89
52.41
20.27
23.50
39.11
19.02
27.87
Flamingo with our R
39.20
48.93
30.90
39.68
45.81
32,40
39.01
MiniGPT-4 [75]
43.83
48.59
43.36
55.01
42.84
41.67
44.71
MiniGPT-4 with our R
57.37
62.32
46.82
65.91
56.72
48.57
55.67
Table 9: The effectiveness of our DDCoT with Flamingo [2] and MiniGPT-4 [75]
VLMs and multimodal reasoning models have difficulty in complex reasoning tasks. Fortunately, the generalizable rationales generated by our DDCoT prompting can help existing VLMs to comprehend visual information and reason with rich knowledge, achieving significant improvement of 1.44%
and 10.96% based on Flamingo [2] and Mini GPT-4 [75] as Table shows.
NoCaps
MSVD-QA
CIDEr SkipThoughtCS EmbeddinnsAverageCSS GreedyyMatchnggScore
Acc
BLIP-2
76.15
49.84
89.20
77.94
34.4
Ours
46.26
84.78
92.35
79.12
39.3
Table 10: Addition experiments on NoCaps and MSVD-QA.
A.5
Additional experiments on Captioning and Video Question Answering tasks
We extend our approach to more appropriate datasets. While other existing datasets may not fully exploit the benefits of our approach, we venture into exploring the captioning task on NoCaps [1] and the video question-anss answering task on MSVD-QA [64], as shown in Table 10.
For captioning, we prompt the LLM [37] to solve sub-problems derived from a simple caption, aiming optimize and enrich it using the corresponding sub-answers. The substantial knowledge within LLM enables the generation of semantically enriched captions, leading to improvements in metrics evaluating sentence semantics, i.e. 34.94%, 3. 3.15%, and 1.18% in terms of SKipThoughtCS [23], Embedding AverageCS [46], and Greedy MatchingScore [45]. Note that the CIDEr [57] metric to evaluate ours is limiting. It is designed to measure the similarity between the tested caption and reference captions without considering the diversity and high-level semantics.
For video question answering. LLM deconstructs problems like the decomposition step ScienceQA. We sample video frames for VQA recognition and integrate frame information for multimodal rationale and answers. Leveraging the sequence understanding in LLM and visual information returned by the VQA model, we achieve a 4.9% improvement over BLIP-2 [26].
Note that we randomly evaluated only 1000 images from NoCaps and 1000 videos from the MSVD test dataset zero-shot setting.
B Human Evaluation
This section introduce the details of our human evaluation. Figure 9bb) shows example of our question. Each sample comprises the question, context, options, and image. Evaluators are asked to rate the rationales generated by GPT [3], MM-COT [71], and our method in five aspects: relevance (related to the question), correctness (accuracy reasoning and answer), completeness (logical reasoning's 's comprehensiveness) coherence (consistency of reason). and explainability (interpretability of reasoning and answer). The rating scale ranges from 0 to Additionally, the questions and rationales organized into 12 groups, with each group assigned to three evaluators. Finally, we average the scores for each aspect of each rationale, resulting in overall scores and ratios relative to the maximum score.
C Detailed Prompts
This section introduce more details about our zero-shot DDCoT prompting. We employ ChatGPT[37] the most important component of our rationale generator. Specifically, is utilized in breaking
19

---

User study on Rationale Generation
Does Asimine vrooaa howe sell: that howe rucleve pou
8.60
your proprix Lenttrneee
Owerveww
giver gives
Rattorate:: Auamina mambe the ddagion Part calchheea
aadeus Thwrstor, Address tniiss bax oel hawe rucler.
bye machin loaning algotthes tountuu
deams wh madiin luaning vewwart, complitt, sulccen liity
propooul wdemant take
whiked the
O
O
poppoaal connet iil picos the surent caswer and cornect ooereeecoaaaaaat oxplanation
proponal comp the explonations, it explane the anoover
propocal when liies lopically connected fow ainochly
0
aloviing plerr undertiar cible prevenations
Comples
0
0
0
0
0
Explinatiity
0
0
(b)
Figure 9: Interface of human evaluation. Figure(a) shows the instructions, figure(b) shows one example our question.
duties of reasoning and recognition step and joint reasoning step. Figure 10 shows complete prompts for zero-shot DDCoT prompting.
D Case Studies
To better understand the effectiveness of our proposed method in generating rationales, randomly selected several cases from the test set along with the process of rationale generation. Figure showcases several map-related questions, demonstrating how our method integrates simple visual features (such as the shape of highlighted areas) with common knowledge to obtain correct reasoning and answers. the examples presented in Figure 12, our method successfully identifies within the images, acquiring relevant knowledge. Figure 13 illustrates four more complex questions, where our method leverages information obtained from the images to perform intricate reasoning. However, when it comes to the complex interaction between images and textual context, our method still fall into hallucinations, leading erroneous reasoning and answers.
E
Limitations
While have succeeded in mitigating portion of the hallucination problem arising from multimodal inputs, it is important to acknowledge that this problem is not entirely resolved. As illustrated in Figure 13, model is also susceptible to the risk of hallucinations. Investigating methods to suppress hallucinations a potential topic for further research and exploration. In addition, we did not extra image-text pairs pre-train the alignment between vision and language modalities. Such pre-trianing is expected to further improve the alignment for joint reasoning.
As approach involves zero-shot prompting of the LLM to generate rationales, there exists potential risk inheriting social biases from the LLM. These biases, which encompass cultural, ethical, and various other dimensions, might be reflected in the generated rationales, potentially leading to adverse effects on users. To mitigate this issue in the future, potential solutions could involve designing constraints at each prompting stage utilizing more advanced LLMs trained on unbiased resources.
20

---

DDCcT
Breaking Reasoning Down to Recognition Steps with Negative-Space Prompting
do your guide choosing the right answer the question, Note that insufficient information to answer questions common, because do not have Inform- ation about the picture. The final answer should be one of the options.
System: helpful, highly intelligent guided
User: Given the context, questions and options, please think step-hy-steg about the preliminary knowledge answer the question, deconstruct the question as completely as possible down to necessary sub-suestions based on context, questions options. Then with the aim helping humans answering form is as follows:
original question, try to sub-questions, The expected
Sub-questions:
<sub-question
2. <sub-question 2>
Sub-answers:
'Uncertain'
2. <sub-answer 2> or 'Uncertain'
Answer: <One the options> or 'Uncertain'
For question, assume that you do not have any information about the picture, but try to answer the sub-questions and prioritize whether your general knowledge can answer it, and then consider whether the context can help. sub-questions can answered, then answer
short sentence possible, If susssssioonos cannot determined without information images, please formulate corresponding sub-answer into "Uncertain",
Only use "Uncertaln" as an answer appears in the sub-answers. All answers are expected as concise as passible. Here is attempt:
Context: (context)
Has An Image: (has Image)
Question: (question)
Options: (option)"
Visual Recognition to Obtain Visual Complements
Aribitrary Question Answering (VQA)
Integrate to Joint Reasoning
System: You helpful, highly intelligent teacher. You will not only your to guide humans the correct answer, will also give the rationales reference.
User: Given the context, questions, options, supplementary information, think step by siep and answer the questions. Please
note that we need not only the answer, but more importantly the rationales of getting the answer. The expected answering form
follows:
Rationale: <rationale>
Answers: <one of the options>>
Please note the supplementary information given may not always valid. Please Information the
rationale and choose the relatively correct option your answer.
Here is an attempt:
Context: (context)
Has Image: (has_ image)
Question: (question)
Options: (option) Supplementary information: (Suuplleeentar__infordaiii
The complete prompts of our zero-shot DDCoT prompting. The underlined parts are introduced in Section 3.2 of the submitted paper. The yellow background represents designs for critical thinking, and the blue background represents designs for differentiate duties of reasoning and recognition. "()" denotes the corresponding inputs, and ctcpllyeentary_iffrrmatte represents the sub-questions and sub-answers obtained before.
Figure 10:
21

---

Queson
Ours Outputs
captal Carolinr?
Sub-question:
(1) What is location of North Caroline?
Contect: Select
(2) What the capital of North Carolin??
Sub-answers (1) North Carolina state located the southcastern region the United
Options:
States. (2) The cap tal of North Caroline Raleigh.
A. Saint Paul Fargo Raleigh D. Charlotte
VQA: Understandin (ind questions, Image)
located the southeleste region the Unitd States. (2) The capital tal North Caraina Raliighi
Sub arbbarrs (1)
Rationali:: question asking capital Carolina. supplementtry information
confirms that North Canoline is state located in the southeestem region the United States and capttal Carolina falligh.
GPT3(COT)
BECAUSE: Rileigh ceptall North Carolini
Question
Outpucs
Sab -ue--uuestoon
Which property maidi this osjeet?
What the shape the highlihted area?
geogroprical
Contest:
highlightted
Options:
Sub answers: (1) Uncertain (2) Uncortain (3) Uncertain tain
VQA: weaeeddigggggggssioooon quu--qussoonss image]
Antarctica
Australia
Sub ansswrrs' didd (2)a mountain attanticc ocean, pacific occan, and artiic occan
Rationate: highiighted
surronnded Attantti
Australla
highighhed
the Rodky Hountains, which North
GPT3(COT)
BECAUSE:
fitm largest continent. also the coldest, windlest, driest
Question
Ours Outputs
Sub Sub-questoon
Which country is highlighted?
(1) What the image showing? (2) the distingusshing festures highlighted country?
Context: Select the beteer answer.
the geographica locations countries the options?
Options:
Sub answers: (1) Uncertain (2) Uncertain (3) Uncertain
VQA: Understandin iortoonddgggseeee (uloo--u one, image]
answers' the
can bbean southeastem panihauls
(3) the
western hamaphere
Rationale: showing
information
highlignted country southeaster coest
Uasse only the penireuls, Howeree,
perirsule
C(CT) Outputs
BECAUSE: Nighighteed Sant Vincent Grenadines
Whish Question
Outputs
qud-qquetion::
prima
(3)
meriden??
(2) Which continents does the prime meridian pass through?
Contest:: Swerd the better
(3) the prime mendan shown the image?
Options: South America Asia C. Antarctica
longitude designated as longitude. the Iine that divicies the Earth into the Easter and Western ftemtspeere, (2) prime mendlen passes throus thruugh Europe, Africa, and Antarttica, (3) Unce
Sub-answers: (2) prime
VQA: unskestani to (uub-quaaioon, imsge)
Sub-answerrs's
The piime meridion
designated
It
divids
Hemispheress (20The prime
meridian passes through Europe, Arkka, and Antarcica, (3)
desigrated de longitude. the Ins that divides the Earth into the Castem and Westem Hemisphenes: The supolementary
Rationale:
The prime morid dian is the liee of long
informaion states that the prime mendlen passes through Europe, Airiss and Antardica The image does not show the prime moridann
GPTD(COT) Outputs
BECAUSE: The prime merid meridaan Intersedss with Aeal
Figure 11: Cases that need to understand the map.
22

---

Question
Ours Outputs
sub-question:
Select
(1) Does the image show mamnill
Context:
Mammals have Sub-answers: (1) (2)
animal the
dart rabelt?
young çorfila mammall.
Options: A, poison dart frog
VQA: anserstaddijjaee (oub-i qusttons, Image]
B. rebbitt
Sub-answers'
The given corfeett states that mammat i have hair or for and feed their yourg milk, The supplementary information contimm that the animal the mage mammal. Therefore, we need select the option that mpposents a mammall. The option (soison daif firdt suno mammall as not hawe har fur and does not fead young mil. On the other hand, the option jabaer is mammal as it for and feeds young milk
Rationale:
(PTT)) Outpurs
BECAUSE & poison dart mommall
Dutputs
Select the fash belw
(1)
Context:
Fish live undenwater. They have fins,
(2) woodlecker fish?
not Imbs. ager shark an exampte of fish.
Sub-enswers: (3) Yes Uncertain
Options:
VQA: Undennanndee boguuadddor, imsape]
salmon
waadpecier
answerss Yes (2)88 alrd
Rationale: undorwater
limbs, given image supplementary
option
bird.
GPT (COT)) Outputs
BECAUSE Silmon are Woodpeckers are not fasl
Question
Ours Outputs
Which material this teapet made of?
(1) the teapot
(2) What are the characte kotics of porcalsin and rock?
better answer.
answere: (1) Unoertain (2) Porcelain type coramic material
translaentt
It made by teating monorias ganerally including clay in the form of laolin, in kiin to tomparatures
Options: A. parcelein
between 1,200 and 1,400 "c. Rock naturally occumting soldd substance composed minerals.
B. rock
Sub -answers't (1) pink and green teapot with pink roses (2) Parosiain a type of coramic material that is often white and translucent. It mode by heating materials, generally including clay in the form of "C. Rock neturally uccurting
kaolin, in temperstures between 1,200
substance
Rationate:
teapot, and the options porczlain transtucent, rock naturalty accumiin composed minerals, suppementtry information describes porcolain biinng heating bak, generally day
type of ceramic material
green kaolin, roses, temperaturs appeorance porociain trapets. Therefore, the mate tespot most likaly por celai...
1,200
1,400 image
GPT S(COT) Outputs
BECAUSE:
made from mettriels, incloding metel gless, day
Question
Outputs
property matches this
question: (1) What color of the object
Context: Select the better answer.
(2) What the shape of the abject?
answers: Uncertain Uncertan
Options: transparent.
boundy
VQA: Underdarddngg woa cub-cuestion, image]
Sub answers': (1) white (2) glass bottle
Rationale: The object glass
information, proporty
matches object T "bansparenn" as glass is bransare and allowss light poss through
GPT3(COT)
BECAUSE: The definiion of transperent "able be seen through with clarity The definition of bouncy Table to rebound spring back into shape after baing comoressed.
glass
Figure 12: Cases that need identify objects and acquire relevant knowledge.
23

---

Quczioon
Ours Outputs
Comporr the awerage kinsli enargies the partdes rpin.
Sub sib-question:
(3) a the relatenaip betwoan the avarage kanet enargy gau partiles and tamperatarr any
tempenttur?
firrrnnee speed
Contort The diogra no below show heo purd sang samples gas stott col close,, right conzalt crt.
sample?
Sub-anowers: The temperature (2) lt unceriai senccher there & d Birence spood mass the pariides in each sample. (1) toth samples heve the some number perticies
hetii
particiss dieetty proportonsl
gas
particles.
Oytione:
VQL www.ggeeeoooooo insoe)
the same temparature
nathher: sampffes have
Sub aroowrr'' (1) ave age ieneti energy of gas partdiss directly proportional temperttioe (2)
speed
tame
particas.
Rationale:: The average sinetic energy of partides directly propo Tooal to temperoture The partcis sample A are movin at much faster spead than the perticio sample S. soth samples have the same sumber of particls, Thardfore, sample has hightr terppssturd than sample B.
CPT3/COTT Outouts
BECAUSE The tworage neti energy of the sortices in sample B Highor than avolage
Mintic ene Ty the pa ea somple
This moans Sample
terpprrtturi
Question
Outpuis
Witch solutio has hightr concert traton of blue parides?
Sub- fubbqquattio::
(1) what do the buc balls represont the digian?
blue ball represerta one puricle of soluk.
Contret: The dingren in marel soll fore, Fach
(2) there diferrnce the number of blue balls Solution A and Solution B7
Sub-answers: The Mue bells represent particles soline. Uncertain
A. somple B. sample netther;
viy Undentandi (ai-qqud twe, Image]
Sub- aebbannwwentt (1) The Mlue balls reprement partidess solute. (2)
represent
there Bee the numbter hlue ls both solutions, concentmation..
Blln parsities both setltions ware
3 Dutputs
BECAUSE
solutz
solvent, concentration of
call be usood in szecil wass, One way to dep tio it the number of moles of solute per sckiioon Another way in vaprevs number of grems wrilee
molin "Tameiow, Soltttoo
aonommniion blue pariicess.
Which lowest average
inoipittion London?
preciptation month? (3) which month has the lowest avardge procihation?
what gloph
graph quedtion beloww.
enswene: (1) The grach shoos the predpiitton lewns lordon for ruch mooth the yean We
mine the axcage pres plation for each month by calculating prode that month divding years
by
everage preciptation
redabouaation, image)
A. July B. November C. February
London photion that
yeast3 divitingg
detwrmine
predpitation each morth by rlving
by the number years in the satasct. (3) The monim
lowist avarage prode lation London Fibnaard
evinge
Rationate: We need
deterntine month
lowert everage predpitation london.
From the grapn, can see that the bar for July &s the shat indicating the lowst proditiaiin lovdl To confirm this, we coliiiatee muerage precpiationn with morth by du dlng the sotal preciptation for eath morth Deing so,
years
ieenaae
CPPT FICD) Ducputs
BECAUSE graph
January December The months
1sted acress the bottom m the groph. This precpitation meacured mi indeo's and la shown by the bluz bans with telled Tike four shanei that the mozeh with the "igged euriage pine prodioo ansmeee
average precipilatton rileuarg
Question
Ours Outputs
Sub-question:
What the probubiltty that produced by this croes have body?
the genalypes paevet ruta?
possible genotypes ottiring?
probabil
abile for dwarf
Exntwet in group of nets, some and vokore hoow in normal- s2ed bock and others have dvarf boos. In this group. the pone for body
Sub- answers: ot The Punnctt square shows that the penotypes of the parcnt ars are db and Bb. (2) The poss
penotypes of offtoring are 66, Bb, and bb. (3) The probaiiltty fyy of an ofteping imnerting the recessive
alliee fora
damer Iealy (1o) 1/4
loddy
VQA: inverrteein doaeesatioonn
genckypes
The
The
oftiping immettng vecessive slidle
Biser baady 0580 (356)
b
Options:
The Punnett sq pare shows that the genstypes of the parent réss are Bb and to. The possibl genobypes offzzing ane sb, and dllee a dwart body recessive panshppa Bir body
Rationale:
maa uoed bock (6).
B
BB
Bb
A:3/4
RL1/4
GPT3(COT) Outputs
result cross in thg Purret Square, Bie3 for the body saaa bra are Shown the top and the left sida ane squaiss
BECAUSE:
BB
Bb
E.M
The alicks for the body sae bol 211 8 and la Wihhe alice normal sood body (8) is doninart to the sliele for
dvert
nooooo---o body (8) rearesorted a capital yithi
dvarl body (e)
esemed 159 lithee wllet for normue-eee -eseedeeeeeedeeeee body
noroeesstse body
allele
Figure 13: Complex and Challenging cases.
24


Reviews:
[{"review_id": "Ko4CmGrVWP", "reviewer": "Reviewer_ka68", "summary": "1. The paper studied the challenges and limitations in rationale generation for multimodal problems. Then the authors propose a Duty-Distinct Chain-of-Thought Prompting (DDCoT) to collect language-related or visual-related information and select valid information to generate the rationale.\n2. The rationale can be used in zero-shot and fine-tuning settings for question answering. The authors designed a fine-tuning framework with deep-layer prompting and rationale-compressed visual embedding. \n3. The experiment results demonstrate the effectiveness of the rationale generated by the proposed method with SOTA results on the ScienceQA benchmark. ", "questions": "1. The baseline(B) model in Table 2 is not explained. \n2. Does 'B+our img' in Table 2 use rational-compressed visual embedding or not? What is the difference between 'B+our img' and 'w/ our R'? In line 308, it seems that the 'B+our img' includes rational-compressed visual embedding. \n3. Does 'w/o DLP' in Table 3 include RCVE? ", "limitations": "The limitation is not discussed in the paper. Societal impact is discussed. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed prompting design to separate the text reasoning and visual information for multi-modal QA problems is novel. \n2. The result on the ScienceQA dataset is significant in zero-shot and fine-tuning settings, the ablation shows the effectiveness of different parts of the model. ", "weaknesses": "1. Some experiment settings are not clearly explained or confused. See questions. \n2. It's better to validate the effectiveness of 'Integrate to Joint Reasoning' part of prompting in the ablation if it's an important part of the method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review. We carefully address your questions and comments below.\n\n> **Some experiment settings** are not clearly explained or confused. See questions.\n\nThank you for the valuable feedback, and we will clearly explain experimental settings in the revised paper and address the confusion. Below, we explain the questions one by one.\n\n> **The baseline(B) model in Table 2** is not explained.\n\nFollowing [1], we adopt the T5-Base with only text input (i.e. questions, contexts, and options) to predict answers as our baseline (B). All image-related information is omitted, given that T5 operates solely as a language model. Thanks for pointing out this, and we will include the explanation in the revision.\n\n[1] “Multimodal Chain-of-Thought Reasoning in Language Models” Zhang, Zhuosheng, et al. *arXiv preprint arXiv:2302.00923* (2023).\n\n> Does **'B+our img' in Table 2** use rational-compressed visual embedding or not? What is the difference between 'B+our img' and 'w/ our R'?\n\n- 'B+our img' in Table 2(b) uses both rational-compressed visual embedding and deep-layer prompting. The first four experiments in Table 2(b) focus on how to utilize visual information in different modalities. In comparison to captions, image modality presents a more challenging scenario for the model to comprehend visual information. Our proposed deep-layer prompting and rational-compressed visual embedding facilitate the model's understanding and learning of the alignment between different modalities, ultimately enabling the extraction and utilization of image features.\n- 'B+our img' in the first four lines is identical to 'no R' in the below three lines. The below three lines intend to elucidate the effects of rationales generated from different methods. Utilizing 'B+our img' as a foundation, the rationale produced by our DDCoT aids the model in better comprehending and reasoning for the multimodal context, resulting in notable performance improvement in the IMG split ('w/ our R').\n\n> Does **'w/o DLP' in Table 3** include RCVE?\n\nThe 'w/o DLP' in Table 3 does not include RCVE. The notation 'w/o DLP' refers to the condition 'w/o DLP & w/o RCVE'. We will replace the notation 'w/o DLP' with 'w/o DLP & w/o RCVE' in our revised paper.\n\n> **Ablations on 'Integrate to Joint Reasoning':** It's better to validate the effectiveness of 'Integrate to Joint Reasoning' part of prompting in the ablation if it's an important part of the method.\n\nThanks for the valuable feedback. We agree that validating the effectiveness of 'Integrate to Joint Reasoning' is essential, given its integral role within our approach.  We present the ablation studies for this component in the following table:\n\n|                  | IMG   | TXT   | Avg   |\n| ---------------- | ----- | ----- | ----- |\n| No R             | 75.16 | 85.25 | 80.45 |\n| naive R | 75.06 | 89.61 | 82.96 |\n| sub-qa as R      | 77.49 | 82.36 | 83.75 |\n| our R            | 83.34 | 91.2  | 87.34 |\n\nAs shown in the table, the absence of the 'Integrate to Joint Reasoning' part also leads to a decline in performance, specifically -5.85% for the IMG split and -3.59% for overall performance. The decline is attributed to the model’s struggle to comprehend scattered facts instead of coherent reasoning chains, coupled with its limited reasoning capabilities.\n\nThanks for the suggestion, and we will include the ablation and analysis of 'Integrate to Joint Reasoning' part of prompting in the revised paper.\n\n> **The limitation** is not discussed in the paper. Societal impact is discussed.\n\nWe discuss the limitations across the following aspects (see Appendix. E for details): (1) The challenge of hallucinations in multimodal reasoning remains partially unresolved. (2) Cross-modality pretraining is anticipated to enhance the efficacy of our methods further. (3) The social biases introduced by the LLMs."}, {"review_id": "3gstrercMw", "reviewer": "Reviewer_oVaJ", "summary": "The paper proposes Duty-Distinct Chain-of-Thought (DDCoT) Prompting for multimodal reasoning problems (e.g. VQA). Despite CoT's success in language-only reasoning problems, authors argue that multimodal reasoning challenges CoT as the rationale part is sensitive to the input information. Since image caption is the only source of image information for LLM, once the caption is generated poorly, rationale will intensity hallucinations. \n\nTo this end, DDCoT proposes two insights: \"keep critical thinking\" and \"disentangle reasoning and recognition\". Specifically, the pipeline is like:\n1. Given input question, zero-shot prompt LLM: “please think step-by-step and deconstruct the question down to necessary sub-questions\"\n2. Determine if each subquestion can be answered without visual information with zero-shot prompt: “Assume that you do not have any information about the picture, try to answer the sub-question and formulate the corresponding sub-answer as ‘Uncertain’ if the sub-question cannot be determined”. This helps deal with hallucination (i.e. LLM makes up facts about images).\n3. Use an off-the-shelf VQA model to answer each subquestion \"with negative space\" (my guess is, each subquestion with 'Uncertain' as answer?)\n4. Given subquestions and answers, LLM is prompted to perform CoT reasoning, generating a rationale and a final answer.\n\nThe paper also considers a finetuning setup with some architecture inventions.\n\nResults on ScienceQA show\n- zero-shot prompting: 1-3% improvement from previous few-shot CoT baselines\n- finetuning: more impressive improvements and generalization from baselines.\n\nAnalysis on visual information source, rationale generation process, fine-tuning components, and explainability are also conducted to further justify DDCoT.\n", "questions": "I think the paper can benefit from better writing, and experiments from different datasets.", "limitations": "Appendix E talks about hallucination and social bias. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Multimodal reasoning (e.g. VQA) is an important research direction, and LLM prompting is an emerging and promising approach to it. The paper highlights limitations of existing CoT-based methods for VQA (sensitive to caption; rationale does not improve performance), and propose reasonable solutions to these problems.\n\n- DDCoT proposes an interesting heuristic way to combine LLM and off-the-shelf VQA models as a tool: LLM decomposes questions, decide which parts to call VQA model, and aggregates information back for answer. The approach could be potentially valuable for other tasks (involving video, code, webpages...)\n\n- The consideration of both prompting and finetuning setups add to technical depth. \n\n- Experiments show improvements, and their design looks solid. I appreciate comprehensive and diverse ablations and analysis.", "weaknesses": "- The presentation is a bit poor. The concrete problem is just \"how to best put visual information in text, so that rationale can help\", and the main insight is just \"use VQA models across subquestions to best induce visual information, instead of off-the-shelf captioning\". But it takes me some time to get it. I don't think \"Flexibility/generalizability/explainability\" makes too much sense to me --- I don't think few-shot CoT is really labor-intensive (just 3 examples??), inflexible, or hard to explain. The main motivation should be performance and generalization. 3.1.1 seems commonly known and a bit redundant. So intro, 3.1, 3.1.1, 3.1.2 are like four different stories of motivation, and a bit confusing to me. Why not sticking to one single story, instead of writing a story in 4 different ways/views/places?\n\n- Continuing on the presentation issue, I don't get Figure 1,2,3,4,5 --- all are examples, and in fact, Figure 5 should better be the teaser given it actually tells a bit about how the method works. Fig 1's example doesn't really tell much, as MM-CoT and UnifiedQA's rationales are omitted. etc.\n\n- Continuing on the presentation issue, some terms are confusing (or at least lack explaining), e.g. \"negative space prompting\".\n\n- While finetuning performances seem stronger, prompting performance is only slightly better than few-shot CoT baselines, and there are not a lot of prompting baselines to begin with. In the long run, seems prompting performance will be more important than finetuning performance (one can also imagine multi-modal GPT-4 just solves ScienceQA and beats every model, making it less worthwhile to study...)\n\n- The results are only on ScienceQA, not only how general the findings are, given original CoT is evaluated on various problems, and multimodal CoT should be able to solve a lot of different tasks as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and the concerns about the presentation. We carefully address your questions and comments below and will improve the writing accordingly.\n\n> **The presentation is a bit poor (part 1)**: The concrete problem and main insight. The motivation and organization structure of Sec 3.\n\nThanks for your detailed review. As the reviewer points out, our core problem is **how to best put visual information in the text to generate helpful rationales.** In order to address this problem, the below two explorations on helpful rationales and visual information are needed, which we present in Sec 3.1:\n\n- What kind of **rationales is helpful** and generalizable? (see Sec 3.1.1 for details)\n- What challenges and issues will there be in **putting visual information** for rationale generation? (see Sec 3.1.2 for details)\n\nBased on the explorations, we summarize key principles to generate helpful rationales and propose the concrete DDCoT prompting method in Sec 3.2:\n\n- As the reviewer points out, using VQA models across subquestions to best induce visual information is one of the main insights in DDCoT prompting.\n- In addition, the “sub-questions with negative-space prompting” (Line 191) and “integrate to joint reasoning” (Line 214) are also the main and crucial parts. The former is used to mitigate hallucination, and the latter help to generate coherent and logical rationale rather than scattered factual sub-questions and answers.\n\nThe writing structure in “Section 3 Method” is presented from concept exploration (Sec 3.1) to concrete method design (Sec 3.2). We appreciate your valuable feedback, and we will provide an overview at the beginning of Section 3 to introduce our core problem, organizational structure, and main content of each subsection. Also, we will carefully revise the writing in each subsection accordingly.\n\n> **The presentation is a bit poor (part 2)**: I don't think few-shot CoT is really labor-intensive, inflexible, or hard to explain.\n\nIt is labor-intensive for fully-supervised methods that require ground-truth rationale annotations. For few-shot CoT methods, we agree that providing a few examples of rationales for one case is not expensive. However, automatically searching for appropriate examples for a large set of diverse questions takes work (another promising direction, but not the scope of this paper), as CoT methods are greatly sensitive to the reasoning complexity and task-specific demonstration. That is also why we believe generalizability is crucial and currently focus on the zero-shot setting, which can generalize to different questions.\n\n> **Continuing on the presentation issue:** The information in Figures 1,2,3,4,5; Fig 5 should better be the teaser. Fig 1's example doesn't really tell much.\n\nAs the reviewer raises in the previous weakness, “the main motivation should be performance and generalization,” we also agree that generalization and performance are very crucial aspects, which thus be presented in Fig 1 to be emphasized. Fig 1 (a) shows a simple out-of-distribution example to illustrate the poor generalization ability of previous methods compared to ours. And Fig 1 (b) shows the performance comparison in different settings. Figures 2, 3, and 4 are examples to help better demonstrate and understand our insights in Sec 3.1, 3.1.1, and 3.1.2, respectively.\n\nThanks for the feedback and suggestion. We will combine our most crucial insights in Fig 4 and the method illustration in Fig 5 to redesign and update a more appropriate teaser.\n\n> **Continuing on the presentation issue:** Some terms are confusing (or at least lack explaining), e.g. \"negative space prompting\".\n\nThe \"negative space prompting\" refers to our prompting method, including decomposition and uncertainty. We decompose multi-modal CoT into multiple sub-questions with \"spaces\", where the \"space\" are \"positive\" if LLMs can answer the sub-question, and otherwise, the \"space\" is \"negative\" to be filled, i.e. the uncertainty. We intend to include both decomposition and uncertainty in the name. If the explanation is still confusing, we will replace \"negative space prompting\" with \"uncertainty prompting\". Thank you for your feedback. We will carefully polish the paper and modify the confusing terms to facilitate understanding.\n\n> **Finetuning performances seem stronger.** Prompting performance will be more important.\n\nThe performance improvement on finetuning is more significant than on zero-shot setting. This is understandable as the complete information of an image cannot be entirely translated into the text input for LLMs. Note that our prompting method with ChatGPT surpasses the few-shot CoT baseline by 4.61% in the IMG split (Line 285), which is significant and improves more than in the Avg split. Besides, we observe that our technique can benefit more when the used LLMs strengthen (2.53% for GPT-3 and 4.61% for ChatGPT) (Line 284).\n\n> **Multimodal CoT should be able to solve a lot of different tasks as well.**\n\nWe appreciate the reviewer’s recognition of the potentially valuable of our approach, and we agree that multimodal CoT could be applied to various tasks as well. Per your advice, we further conduct experiments on captioning and video question answering tasks. The below table presents the general effectiveness of our approach across various tasks.\n\n|||| NoCaps|| MSVD-QA |\n| - | - | - | - | - | - |\n|| CIDEr | SkipThoughtCS | EmbeddingAverageCS | GreedyMatchingScore | Acc|\n| BLIP-2 | 76.15 | 49.84| 89.20| 77.94| 34.4|\n| Ours   | 46.26 | 84.78| 92.35| 79.12| 39.3|\n\nPlease refer to our third response to the Reviewer kQGJ for the analysis. Thanks for the valuable suggestion, and we will explore various tasks in zero-shot and finetuning settings in the revised paper."}, {"review_id": "1x7os60Ct3", "reviewer": "Reviewer_kQGJ", "summary": "The work focuses on solving multimodal reasoning task. In the zero-shot setting, the work prompts LLM to conduct step-by-step reasoning. To avoid hallucination due to the lack of image features, LLM is asked to leave the answer to sub-questions as “uncertain” if they involve images. Then the corresponding sub-questions are answered by the visual components. In the fine-tuning setting, the generated rationales are used to train an LM augmented with a visual encoder. Experiments show that the elicited multimodal rationales can improve performance in both settings. ", "questions": "1.\tHow do you sample the rationales for fine-tuning? Do you only sample those leading to the correct predictions?", "limitations": "The authors have clearly stated the risks/biases brought by LLMs which are the common problems for LLMs. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tOriginality: The work adopts careful prompt engineering that encourages LLM to (1) offload the step of visual recognition to the visual component and (2) discard errors in the intermediate results. Both are shown to be effective in mitigating hallucination and robust respectively.\n-\tQuality: The work investigates both zero-shot evaluation and fine-tuning to validate the effectiveness of the generated multimodal rationales. Comprehensive experiments and ablation studies are conducted to justify the design choices.\n-\tClarity: The paper is well-written with illustrative running examples and figures. The intuitions behind each design choice are clearly stated, which helps readers to understand both the technical problems and motivations.\n-\tSignificance: The proposed method brings significant improvement to a particular dataset, and also shows better generalization.\n", "weaknesses": "1.\tThe key ideas the paper tries to convey are somewhat scattered. The prompt engineering part and the visual component can be individually stand-alone as independent work. Right now, it is also hard to tell which part matters more. Perhaps the authors can add experiments where (1) a baseline model is trained on the rationales generated by the proposed method and (2) the proposed augmented model is trained on human-annotated rationales.\n2.\tOnly one dataset is chosen. And many questions in ScienceQA do not require images to be answered indeed. I would encourage the authors to explore more suitable datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and helpful comments. Our detailed responses are available below for your consideration.\n\n> **Somewhat scattered key ideas (part 1).** The prompt engineering part and the visual component can be individually stand-alone as independent work.\n\nWe appreciate the review’s acknowledgment of our DDCoT prompting and visual components. Both DDCoT prompting and visual components facilitate inducing visual information to language models for multimodal reasoning. The former focuses on generating high-quality multimodal rationales. The latter aims at optimizing the utilization of those generated rationales and visual features, as visual information in images cannot be fully converted to the text inputs of LLMs. Thanks for your valuable feedback, and we will include a comprehensive perspective on the two technologies at the outset of Section 3.2.\n\n> **Somewhat scattered key ideas (part 2).** Which part of this paper matters more? Perhaps add experiments where (1) a baseline model is trained on the rationales generated by the proposed method and (2) the proposed augmented model is trained on human-annotated rationales.\n\nThanks for the valuable suggestion. We also agree with the necessity to demonstrate the extent of impact exerted by the two components, and we present the results in the following table:\n\n|                   | IMG   | TXT   | Avg   |\n| ----------------- | ----- | ----- | ----- |\n| baseline          | 72.93 | 85.84 | 79.7  |\n| baseline + our R  | 75.81 | 82.40 | 82.83 |\n| baseline + gt R   | 81.07 | 84.07 | 85.97 |\n| our model         | 75.16 | 85.25 | 80.45 |\n| our model + our R | 83.34 | 91.2  | 87.34 |\n| our model + gt R  | 84.43 | 92.09 | 88.00 |\n\nAs shown in the table, we observe that our R and visual components individually exhibit certain gains in terms of IMG improvement. However, **when combined, they yield substantial gains.** This observation aligns with our response above.\n\nBesides, please note that the annotated ground truth rationales within the ScienceQA dataset inherently encompass the final prediction, i.e., correct answers. To ensure a fair comparison, we manually exclude the answers from these annotations, using the remaining text as input rationales for fine-tuning. Under this configuration, our proposed rationale achieves a fine-tuning performance comparable to the annotated rationales.\n\nThanks for pointing out this, and we will add the experiments and analysis in the revised paper.\n\n> **Only one dataset is chosen.** And many questions in ScienceQA do not require images to be answered indeed. I would encourage the authors to explore more suitable datasets.\n\nThanks for your valuable feedback. The ScienceQA indeed includes some questions that do not require images to answer, but we achieve significant improvement in its IMG split (requiring images to answer). Currently, ScienceQA is a suitable dataset as it incorporates multimodality and needs for complex reasoning.\n\nWe agree that extending our approach to more appropriate datasets would be beneficial. While other existing datasets may not fully exploit the benefits of our approach, we venture into exploring the captioning task on NoCaps and the video question-answering task on MSVD-QA:\n\n|      |  |  | NoCaps |                                                       | MSVD-QA |\n| ------ | ------ | ------------- | ------------------ | ------------------- | ------- |\n|        | CIDEr  | SkipThoughtCS | EmbeddingAverageCS | GreedyMatchingScore | Acc     |\n| BLIP-2 | 76.15  | 49.84         | 89.20              | 77.94               | 34.4    |\n| Ours   | 46.26  | 84.78         | 92.35              | 79.12               | 39.3    |\n\n**For captioning,** we prompt the LLM to solve sub-problems derived from a simple caption, aiming to optimize and enrich it using the corresponding sub-answers. The substantial knowledge within LLM enables the generation of semantically enriched captions, leading to improvements in metrics evaluating sentence semantics, i.e. 34.94%, 3.15%, and 1.18% in terms of SkipThoughtCS, EmbeddingAverageCS, and GreedyMatchingScore. Note that the CIDEr metric to evaluate ours is limiting. It is designed to measure the similarity between the tested caption and reference captions without considering the diversity and high-level semantics.\n\n**For video question answering,** LLM deconstructs problems like the decomposition step on ScienceQA. We sample video frames for VQA recognition and integrate frame information for multimodal rationale and answers. Leveraging the sequence understanding in LLM and visual information returned by the VQA model, we achieve a 4.9% improvement over BLIP-2.\n\nDue to time limitations, we randomly evaluated only 1000 images from NoCaps and 1000 videos from the MSVD test dataset in a zero-shot setting. In the revised paper, we intend to extend our experiments to encompass additional tasks in both zero-shot and fine-tuning settings.\n\n> **How do you sample the rationales for fine-tuning?** Do you only sample those leading to the correct predictions?\n\nFor fine-tuning, we employ all the available rationales without resorting to sampling."}, {"review_id": "xlWTYptxgR", "reviewer": "Reviewer_aDZq", "summary": "This paper proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors deeply analyze the challenges and insights in multimodal CoT for the rationale generation.\n- The authors propose a novel DDCoT prompting to maintain a critical attitude and identify reasoning and recognition responsibilities  through the combined effect of negative-space design and deconstruction.\n- The experiments show the superiority of the proposed method.", "weaknesses": "- Are the analysis results of the single case  in section 3.1.1 and 3.1.2  applicable to most samples in the data set? In addition to chatgpt, whether gpt-3 can reach similar conclusions?\n- On what basis are the three principles in 3.2 considered？\n- Why the “uncertainty”  are called as a negative space?\n- The statement ‘’learnable prompts to facilitate the alignment  of visual and linguistic semantics at a shallow level but also utilizes explicit rationale to jointly encode multimodality by learning different prompts for each encoder layer.’’ should cite related papers.\n- There are some minor errors in this paper. For example, “exploring to introduce programming approach” should be “exploring to introduce a programming approach”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Please find our responses below.\n\n> **General insights:** Are the analysis results of the single case in section 3.1.1 and 3.1.2 applicable to most samples in the data set?\n> \n\nOur findings in 3.1.1 and 3.1.2 are applicable to most samples. The applicability of “the different roles of rationales in zero-shot and fine-tuning learning” (3.1.1) possesses an intuitive nature and extends to generic scenarios. Similarly, the discovery “interleaved information without careful design exacerbates hallucinations” in section 3.1.2 remains pertinent for a substantial portion of the dataset, making it a noteworthy phenomenon. We present the quantitative analysis of hallucinations (see Appendix A.2 for details) and a new experiment for the discovery in 3.1.2 as depicted below:\n\n| Method | Authenticity  |\n| --- | --- |\n| w/o visual information | 0.883 |\n| w/ visual inforamtion | 0.602 |\n| w/ visual information & duty distinct w/o uncertainty  | 0.783 |\n| w/ visual information & duty distinct w uncertainty R | 0.855 |\n\nAs shown in the table, introducing visual information exacerbates the hallucinations and diminishes the authenticity of generated rationales by 28.1% compared with naive prompting devoid of any visual information input. Our \"duty distinct\" and \"uncertainty\" designs effectively alleviate the hallucinations, elevating authenticity to a level comparable to scenarios without visual information input. \n\nThanks for the valuable comments, and we will add the user study and analysis in the revised paper.\n\n> **Can gpt-3 reach similar conclusions as chatgpt?**\n> \n\nGPT-3 can reach similar conclusions as ChatGPT, in terms of ”difficulty in understanding dense image information” (3.1), ”rationale-sensitive reasoning” (3.1.1) and ”intensified hallucinations with visual information” (3.1.2). We provide additional illustrative instances in Appendix A.3, and the similar performance of our method when using GPT-3 and ChatGPT can also demonstrate that. \n\n> **On what basis are the three principles in 3.2 considered？**\n> \n\nWe consider the three principles from the following insights:\n\n- For the principle “utilize LLMs’ intrinsic knowledge to generate multimodal rationales”: Rationales are necessary for multimodal comprehension with LLM because LLM has difficulty jointly reasoning multimodal information (3.1). Besides, flexible rationales are required to be knowledge-enriched (3.1.1), so LLMs become a natural choice for rationale generation.\n- For principle \"explicitly cue the LLMs to differentiate the responsibilities of reasoning and recognition step by step\": The challenge of hallucinations is exacerbated when introducing interleaved multimodal information as LLMs would fabricate visual information (3.1.2). Explicit duty distinct prompting is an optional solution for fabricated visual information, compelling the LLMs to identify the recognition responsibility of off-the-shelf visual models and visual models to obtain visual facts. Similarly, the responsibility for reasoning needs to be assigned to LLMs with reasoning capabilities.\n- For principle “explicitly mark the negative space for uncertain parts, emphasizing critical thinking in rationale generation”: Considering the rationale-sensitive reasoning for zero-shot prompting (3.1.1), the authenticity of rationales becomes significant. Uncertainty facilitates the LLMs to maintain a critical attitude towards questions and supplementary visual information, thereby enhancing the authenticity of the generated rationales.\n\nThanks for the valuable feedback, and we will revise the overview of 3.2 to add the above clarification. \n\n> **Why the “uncertainty” are called as a negative space?**\n> \n\nThe \"negative space prompting\" refers to our prompting method, including decomposition and uncertainty. The multi-modal CoT in our approach is decomposed into multiple sub-questions with \"spaces\", where the \"space\" that can be answered by the LLM is \"positive\", and otherwise the \"space\" is \"negative\" to be filled, i.e. the uncertainty. We intend to incorporate both decomposition and uncertainty into the name. If the clarification remains unclear, we will replace \"negative space prompting\" with \"uncertainty prompting\". Thank you for your feedback. We will carefully polish the paper and modify the confusing terms to facilitate understanding. \n\n> **Citation:** The statement ‘’learnable prompts to facilitate the alignment of visual and linguistic semantics at a shallow level but also utilizes explicit rationale to jointly encode multimodality by learning different prompts for each encoder layer.’’ should cite related papers.\n> \n\nThanks for the valuable feedback, and we will cite and discuss the shallow prompting [1,2,3,4] and deep prompting [5,6,7,8] methods. \n\n> **Minor errors:** There are some minor errors in this paper. For example, “exploring to introduce programming approach” should be “exploring to introduce a programming approach”.\n> \n\nThanks for pointing out this, and we will polish the paper in the revision.\n\n[1] Learning to prompt for vision-language models. Zhou, Kaiyang, et al. IJCV2022.\n\n[2] Conditional prompt learning for vision-language models. Zhou, Kaiyang, et al. CVPR2022.\n\n[3] Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Sun, Ximeng, Ping Hu, and Kate Saenko. NeurIPS2022\n\n[4] Prompt distribution learning. Lu, Yuning, et al. CVPR2022.\n\n[5] Visual prompt tuning. Jia, Menglin, et al. ECCV2022.\n\n[6] Maple: Multi-modal prompt learning. Khattak, Muhammad Uzair, et al. CVPR2023.\n\n[7] P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. Liu, Xiao, et al. ACL2022.\n\n[8] P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. Liu X, Ji K, Fu Y, et al. arXiv:2110.07602, 2021."}, {"review_id": "WdYV4KOUOO", "reviewer": "Reviewer_ndag", "summary": "This paper explore utilizing the rationales to achieve the multimodal reasoning based on LMs.\nThe author analyzes the challenges on using the LM to perform the multimodal reasoning such as Hallucination problem. \nBased on the preliminary observations, the authors propose jointly exploit the reasoning ability in LLMs and the image understanding capability of visual question-answering models for general multimodal rationale generation. \nExperimental results demonstrate that the proposed method named DDCoT is effective from the experimental results.", "questions": "See the above strength and weakness parts", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper clearly expresses their motivation and analyzes the common problems when exploiting language models for multimodal reasoning. \n1. The motivation is clear and the analysis of hallucination problem is comprehensive.\n2. The performance of the proposed DDCoT is demonstrated to be superior to some common baselines.", "weaknesses": "The method proposed in this paper seems to be a simple performance combination of large language model and existing visual question answering model, and mainly utilizes the ability of large language model to decompose questions. Such a kind of method has already been explored in the research direction of using large language models to call external tools, such as Visual Chatgpt, Augmented Language Models. \n1. Although the logic of the proposed method is clear, the technical novelties seem to be insufficient. The proposed method is similar to the Self-asking COT, or ``Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP’’. For the multimodal information interaction, this paper does not present the detailed description. \n2. The claims in lines 75-76 show that this paper is the first to explore the zero-shot multimodal reasoning. However, they do not give the enough analysis of this point and do not consider the performance of pretrained VLMs such as OFA or Flamingo. In addition, similar methods such as ModCR[1], Mini GPT-4, BLIP-2, and others should be compared. \n3. The proposed module Rational-Compressed Visual Embedding (RCVE) aims to compress visual input embeddings according to the multimodal rationales by filtering visual features. I am confused by this calculation. For example, Formula 1, it adopts global visual features as query vectors and rational information as Key and Value. This process is the integration of different information, which could be not in line with its purpose.  \n4. Deep-Layer Prompting (DLP) is similar to the Prompt tuning v2 method [2]. The whole approach should be improved. Missing the corresponding citation.\n[1] Liu X, Ji K, Fu Y, et al. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks[J]. arXiv preprint arXiv:2110.07602, 2021.\n[2] Guo J, Li J, Li D, et al. From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models[J]. arXiv preprint arXiv:2212.10846, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. Please find our responses below.\n\n> **Response to the overall weaknesses:**\n> \n\nWe thank the reviewer for the rigorous consideration. But we have to claim that our contributions and technical novelties are far beyond the “simple combination of LLMs and VQA models” (simple combination) and the “utilization of the ability of LLMs to decompose questions” (simple decomposition):\n\n1. **We are the first to (1)** study and achieve zero-shot multimodal CoT rationale generation, **(2)** discover LLMs’ hallucinations are intensified with interleaved multimodal inputs and thus deeply analyze how to put visual information in the text to generate helpful rationales, and **(3)** propose a novel prompting method (DDCoT) to generate rationales works for both zero-shot and finetuning learning. Previous works and the mentioned Visual ChatGPT and Augmented Language Models do not explore these aspects. \n2. **The above simple combination and decomposition cannot achieve multimodal rationale generation**. \n    1. Such a simple combination predisposes LLMs towards hallucination, generating fabricated rationales (see Lines 172-183). As a result, rationales simply generated from LLMs with captions offer significantly constrained assistance for multimodal reasoning (73.33 on IMG and 82.15 on average) in comparison to our approach (83.34 on IMG and 87.34 on average). \n    2. Also, the simple decomposition without considering “negative-space prompting” (Lines 194-205) or “integrating to joint reasoning” (Lines 214-223) cannot work well, i.e., -7.73% (Lines 15-19 in the Appendix) for the former, and -5.85% (refer to the respond table to the Reviewer ka68) for the latter on the IMG split, in comparison to our DDCoT.\n3. Our DDCoT not only **significantly outperforms** previous methods but also exhibits impressive generalization ability and explainability. \n\n> **Technical novelties** seem to be insufficient. The proposed method is similar to the Self-asking COT, or Demonstrate-Search-Predict.\n> \n\nRegarding the concern about our novelties, please refer to our “response to the overall weaknesses”. \n\nThe significant differences compared to Self-asking COT and Demonstrate-Search-Predict are:\n\n1. **Focuses and motivations:** The Self-asking COT and Demonstrate-Search-Predict work solely on the language modality, and they have different focuses compared to our multimodal CoT rationale generation. In contrast, the new challenges and corresponding solutions brought by multimodality are what we focus on.\n2. **Technicals**: Self-asking COT and Demonstrate-Search-Predict have not explored at least two of our core ideas: (1) distinct duties by considering the uncertainty, (2) integrating sub-question and answers to generate one coherent rationale instead of final prediction.\n\n> **Multimodal reasoning:** The claims in lines 75-76 show that this paper is the first to explore the zero-shot multimodal reasoning. However, they do not give the enough analysis of this point and do not consider the performance of pretrained VLMs such as OFA or Flamingo. In addition, similar methods such as ModCR[1], Mini GPT-4, BLIP-2, and others should be compared.\n> \n1. This work is the first to study the “zero-shot **multimodal rationale generation”** (Lines 75-76) rather than “multimodal reasoning”. \n2. In fact, our proposed rational generation is compatible with such pretrained VLMs and methods of multimodal reasoning. Without correct rationales, existing pretrained VLMs and multimodal reasoning models have difficulty in complex reasoning tasks. Fortunately, **the generalizable rationales generated by our DDCoT prompting can help existing VLMs** to comprehend visual information and reason with rich knowledge, achieving significant improvement of **11.14**% and **10.96**% based on Flamingo and Mini GPT-4 as the below table shows. Thanks for your feedback, and we will include the analysis and experiments. \n    \n    \n    |  | NAT | SOC | LAN | TXT | IMG | NO | Avg |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | OFA | 5.91 | 0.11 | 13.18 | 13.05 | 0.30 | 11.85 | 6.58 |\n    | Flamingo | 21.89 | 52.41 | 20.27 | 23.50 | 39.11 | 19.02 | 27.87 |\n    | Flamingo with our R | 39.20 | 48.93 | 30.90 | 39.68 | 45.81 | 32.40 | 39.01 |\n    | Mini GPT-4 | 43.83 | 48.59 | 43.36 | 55.01 | 42.84 | 41.67 | 44.71 |\n    | Mini GPT-4 with our R | 57.37 | 62.32 | 46.82 | 65.91 | 56.72 | 48.57 | 55.67 |\n    | BLIP-2 | 67.40 | 56.36 | 52.45 | 67.01 | 62.77 | 53.58 | 61.21 |\n    | Ours | 88.72 | 86.84 | 84.91 | 87.59 | 83.34 | 88.08 | 87.34 |\n\n> **Calculation in RCVE:** The proposed module Rational-Compressed Visual Embedding (RCVE) aims to compress visual input embeddings according to multimodal rationales by filtering visual features. I am confused by this calculation.\n> \n\nFormula 1 and Formula 2 work together to form the RCVE module. Global visual features are updated based on the similarity to the input rationales in Formula 1. In formula 2, the updated visual features first reshape to $N_r$  low-rank intermediate vectors and then capture local visual information through the attention mechanism. These processes culminate in the compression of local visual information into $N_r$ visual embeddings.\n\n> **Citation:** Deep-Layer Prompting (DLP) is similar to the Prompt tuning v2 method [2]. The whole approach should be improved. Missing the corresponding citation [1][2].\n> \n\nThanks for your valuable feedback. We will cite and discuss the shallow prompting [1,2,3,4] and deep prompting [5,6,7,8] methods, which are common in prompting learning. Our DLP is used to facilitate the alignment of visual and linguistic semantics at a shallow level and to combine with RCVE to utilize rationale to encode multimodality jointly at each layer. \n\nSee our response to Reviewer aDZq for the citations [1-8]."}]

Author rebuttal:
We really appreciate all reviewers for their valuable feedback. Our code will be made public upon acceptance.

We are encouraged by the reviewers’ recognition of our novel/interesting contribution (ka68, oVaJ, aDZq), solid and robust technical design (oVaJ, kQGJ, aDZq), compelling performance improvement (ka68, oVaJ, kQGJ, aDZq, ndag) and diverse ablation studies (ka68, oVaJ, kQGJ).

------

In our individual replies, we attempted to address specific questions and comments as clearly and detailed as possible. Here, we briefly summarize these additional experiments and evaluations:

- Ablations on the 'Integrate to Joint Reasoning' part.
- Additional experiments on Captioning and Video Question Answering tasks.
- Quantitative ablations on our DDCoT and visual components.
- Additional user study for validating the generalization of the discovery in 3.1.2.
- Quantitative comparison with existing pre-trained VLMs and multimodal reasoning models: Our proposed rationale generation is compatible with such pre-trained VLMs.

------

We hope that these additional results further strengthen DDCoT’s position as the state-of-the-art multimodal Chain-of-Thought (CoT) approach:

- We are the **FIRST** to study and achieve **zero-shot multimodal rationale generation**, considering flexibility, generalizability, and explainability.
- We deeply analyze the challenges of **putting visual information in the text to generate rationales** (Sec 3.1) and **induct the critical principles** (Lines 187-190) in generating flexible and generalizable multimodal rationales using LLMs: (1) Drawing on the LLMs' intrinsic knowledge, (2) differentiating the responsibilities of reasoning and recognition, and (3) emphasizing critical thinking in the face of uncertainty.
- We propose zero-shot DDCoT prompting to generate multimodal rationales that significantly improve the multimodal reasoning abilities of LMs in both zero-shot prompting and fine-tuning learning while exhibiting impressive generalization ability.

Meta-review: The paper proposes a novel method to generate multimodal rationales for question answering using language models and visual models. The method improves the performance and robustness of the system and reduces the hallucination problem. The paper also conducts extensive experiments and analysis to validate the effectiveness of the method. Some questions were raised by reviewers and the authors have already answered them in the rebuttal phase, such as the definitions of some experimental settings, the differences between this work and previous work, the claim of the key contribution (i.e., the 1st to study and achieve zero-shot multimodal rationale generation), the results of the proposed method on more datasets, etc. Based on the consensus got from all reviewers, I recommend an ACCEPT to this paper.
Decision: Accept (poster)