SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ge Zheng $^{1*}$  Bin Yang $^{1*}$  Jiajin Tang $^{1*}$  Hong-Yu Zhou $^{2}$  Sibei Yang $^{1\dagger}$
$^{1}$ ShanghaiTech University  $^{2}$ The University of Hong Kong
Project Page: https://toneyaya.github.io/ddcot/
A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.
One of the fundamental aspirations of AI systems is to address complex tasks with reliability and efficiency that are aligned with human capabilities [31, 48, 58, 11, 5, 52, 53, 49, 50]. In tackling such complex tasks, humans rely on multi-step reasoning that integrates information from various modalities. Recently, language models (LMs) [3, 37, 55, 8, 7] have shown remarkable progress in a range of multi-step reasoning tasks by prompting [60, 25, 70, 15, 54] or fine-tuning [35, 13, 59, 27] with the chain of thought (CoT) that mimics the human reasoning process.
However, most studies on CoT reasoning have focused solely on the language modality [44, 10, 32, 72, 22, 5, 68, 66], with minimal attention to multimodal contexts. UnifiedQA [31] and MM-CoT [71] are the pioneering works of eliciting CoT reasoning in vision-and-language multimodality. They employ multimodal inputs, as illustrated in Figure 1(a), necessitating training for the generation of the intermediate reasoning steps (i.e., rationales) either as an explanation accompanied by an answer or as reasoning prior to inferring the answer. Despite the benefits of these multimodal CoT reasoning methods in enhancing the ability of LMs to answer multimodal science questions, several significant challenges have impeded their deployment: (1) Labor-intensive annotation: manual annotation for rationales is time-consuming, costly, and challenging to ensure consistency and completeness in
†Corresponding author. yangsb@shanghaiitech.edu.cn
Figure 1: Comparison of existing multimodal CoT methods and our DDCoT on (a), (c), (d) and (e) an out-of-distribution example to illustrate the generalizability and (b) performance of zero-shot and fine-tuning learning, demonstrating that we are the first to generate general multimodal rationales.
multi-step CoT reasoning, thereby restricting the utilization of these approaches. (2) Flexibility: the effectiveness of multimodal rationales generated by existing methods [31, 71] is limited to either zero-shot prompting or fine-tuning learning with LMs, as demonstrated in Figure 1(b). Our observations indicate that the rationales generated by MM-CoT fail to provide any benefits for zero-shot prompting, while the rationales generated solely by GPT-3 do not enhance the reasoning ability of the fine-tuning model. (3) Generalizability: the current multimodal CoT reasoning approaches [31, 71] exhibit limited generalizability to problems that require novel and unseen reasoning paths. As shown in Figure 1(d) and (e), basic science questions that are out-of-distribution pose a challenge for these models. Moreover, models trained on a specific subset, which includes arbitrary two subjects of questions in natural science, social science, and language science, perform worse when applied to another subject of questions (see Section 4.1). (4) Explainability: the objective of multimodal CoT extends beyond inferring answers to include providing explanations, yet the interpretability of current generated rationales still requires further improvement.
This study aims to explore overcoming the aforementioned challenges and develop a zero-shot, generalizable and explainable approach for generating rationales to enhance the reasoning abilities of LMs in both zero-shot and fine-tuning learning. To achieve this, we first probe the incorporation of rationales into multimodal reasoning and determine a two-step rationale generation and utilization process. Then we explore (1) the role of rationales in the utilization phase, where they serve as authentic knowledge providers, and (2) the challenges of intensified hallucinations in rationale generation phase. Based on these factors, we formulate the following approach.
To generate general multimodal rationales, we propose a novel multimodal CoT prompting approach called Duty-Distinct Chain-of-Thought Prompting (DDCoT), which generate multimodal rationales using language-only LLMs [3, 37], considering the explored factors above. For the first factor, our key insight is "critical thinking - keeping a healthy dose of skepticism": explicitly indicating the uncertainty in the rationale generation process is critical since it helps improve the correctness of the generated rationales, which serve as key inputs guiding LMs in the thinking process of inferring answers, particularly in the zero-shot setting. For the second factor, our key insight is to "let everyone do their jobs - divide labor and work together": it is necessary to prompt LLMs to explicitly identify the reasoning and recognition responsibility of LLMs and off-the-shelf visual models, in order to overcome the language hallucination issue that arises when directly generating rationales from interleaved multimodal inputs. Based on this, we propose a sequential process of negative-space prompting, visual recognition, and joint reasoning to perform interleaved reasoning and recognition. More detailed analysis of the observed factors and motivation behind the two insights are elaborated in Section 3.1 and Section 3.2.
To utilize the generated rationales to facilitate multimodal question answering of LMs, we leverage them as inputs to explicitly guide the LMs' chain of thought and the attention to multimodal inputs in both zero-shot prompting and fine-tuning learning. We combine generated rationales with problem
statements as inputs to the LLMs for zero-shot learning. For fine-tuning learning, we propose deep-layer prompting (DLP) and rational-compressed visual embedding (RCVE) to utilize the rationales better to filter, encode, and jointly infer over interleaved multimodal inputs.
To sum up, our contributions are three-fold: (1) This work is the first to study the zero-shot multimodal rationale generation. We deeply analyze the challenges and insights in multimodal CoT for the rationale generation: rationale sensitive in zero-shot prompting, the knowledge-required in fine-tuning due to catastrophic forgetting, and hallucinations intensified due to interleaved multimodal inputs. We hope these insights could benefit future research work. (2) We propose a novel DDCoT prompting to maintain a critical attitude and identify reasoning and recognition responsibilities through the combined effect of negative-space design and deconstruction. The resulting rationales can directly be part of multimodal inputs to improve the reasoning abilities of LMs in both zero-shot prompting and fine-tuning learning. (3) With the rationales, our methods consistently outperform the state-of-the-art LMs, improving both GPT-3 and UnifiedQA by  $+2.53\%$  and  $+8.23\%$  respectively on questions that have image context, while exhibiting impressive generalizability and explainability.
CoT Reasoning of LLMs. LLMs have been demonstrated successful in natural language processing. Recently, zero-shot [25] and few-shot [60, 44] multi-step thinking prompts have been found to significantly improve LLMs' reasoning ability, thus such chain-of-thought (CoT) methods have attracted growing interest. Some works are interested in example selection in terms of similarity [44, 32], diversity [70], and complexity [10]. Other methods optimize the reasoning pipeline, exploring to introduce programming approach [5], explicit decomposition of problems [72, 22] or calibration coordination across multiple rationales [59, 27, 10]. Inspired by these works, we focus on extending CoT reasoning to multimodality, whilst tackling inherent complications that emerge from therefrom.
Transferring Specialized Reasoning Skills to Small Models. In addition to research on CoT reasoning in LLMs, some studies conduct CoT on models with a smaller number of parameters. The works [35, 13] distill the CoT capabilities of LLMs into smaller models, facilitating the performance of smaller models in specific tasks. Unlike them to generate rationales in inference, our work utilizes the generated rationales via CoT reasoning from LLMs to explicitly guide the understanding of the image for multimodal reasoning.
Cross-modal CoT Reasoning. The pioneering work [31] for multimodal CoT proposes ScienceQA, a dataset consisting of scientific questions involving multimodality with annotated rationales. They perform zero-shot prompting on GPT-3 and fine-tuning learning on UnifiedQA [20] to generate rationales and answers simultaneously.
The following work MM-CoT [71] devises a two-stage framework in which the model initially learns to generate rationales based on the ground-truth annotations, and then utilizes all available information to generate the final answer. However, their generated rationales can only exclusively benefit either zero-shot or fine-tuning learning.
Integrate Visual Modality to Language Models. With the demonstrated capability of LLMs to incorporate open-world commonsense knowledge [40, 43], an increasing number of studies are focused on making visual modality available to well-established LLMs for solving complex visual and multimodal problems. Some approaches [16, 26, 67, 28, 69, 6, 12, 63] incorporate additional training data to align image features with linguistic space. Others [33, 62, 47] take advantage of the scheduling ability of LLMs to dynamically integrate off-the-shelf vision models, extracting image information into text form explicitly. Unlike them directly integrating multimodal inputs once, we explicitly identify the uncertainty in rationale generation and complement visual information step by step.
Our work focuses on how to best put visual information in the text to generate generalizable and explainable rationales. In this section, we first introduce the concepts and motivation in exploration (Sec 3.1), and then present our concrete method designs for rationale generation (Sec 3.2) and utilization (Sec 3.3).
Figure 2: An example shows the importance of input rationales in multimodal reasoning as well as the disparate roles of rationales in zero-shot and fine-tuning scenarios.
3.1 Motivation: Leverage Rationales for Multimodal Reasoning
The success of rationales in unimodal reasoning [25, 60, 44] motivates us to leverage rationales to enhance both reasoning capabilities and interpretability in multimodal reasoning. In this section, we first investigate the incorporation of rationales into multimodal reasoning in Section 3.1.1, and then examine the roles of rationales in enhancing multimodal reasoning in Sections 3.1.2, while also illuminating the challenges to generate general rationales with LLMs in Section 3.1.3.
3.1.1 Two-step Reasoning Process: Multimodal Rationales Generation and Utilization
Following [31, 71, 69], we first prompt LLMs to generate answers accompanied by rationales and observe that it cannot improve multimodal reasoning capabilities, i.e.,  $74.04\%$  accuracy of generating only answers and  $75.17\%$  accuracy of generating both rationales and answers [31].
Moreover, as Figure 2(a) illustrates, despite sufficient image information provided in the form of image caption, the LLM, GPT-3 [3] in this case, fails to jointly reason the question and the caption to infer answer. The possible reason is that LLMs have difficulty understanding dense image information.
Motivated by that people usually extract key information from images in the context of questions, rather than indiscriminately focusing on all dense information, we explore to provide the rationale as a structured logical chain to explicitly guide the understanding of the image. As shown in Figure 2(a) and (d), incorporating the rationale as input facilitates the model's multimodal reasoning capabilities. Therefore, we employ a two-step reasoning process: multimodal rationale generation, and their subsequent utilization.
3.1.2 Roles of Rationales Differ in Zero-shot and Fine-tuning Learning
In further exploration, we find that the effect of rationale varies in zero-shot and fine-tuning models.
Rationale-sensitive reasoning for zero-shot prompting. Upon conducting zero-shot prompting on LLMs, such as ChatGPT [37], we find that the model tends to reason in line with the input rationales. For example, without referring to additional rationales, the common sense knowledge encoded in ChatGPT allows it to answer the basic biological question "which nutrients are mainly provided by the foods in the image", as shown in Figure 3(c). However, the assertion "oranges are known to be a good source of fats" in misleading rationale results in a failure of ChatGPT to answer the same question (see Figure 3(a)). Given language models' tendency to reason primarily based on input rationales, the accuracy of rationales becomes crucial for zero-shot prompting on LLMs.
Figure 3: An example for disparate roles of rationales in zero-shot and fine-tuning scenarios, with question depicted in Figure 1.
Figure 4: An example of the hallucination challenge which is exacerbated by providing interleaved multimodal information at once in generating rationales with LLMs.
Knowledge-required reasoning for fine-tuning. In contrast to zero-shot prompting learning, the fine-tuning models not only require the correctness of rationales but also exhibit a strong reliance on comprehensive prior knowledge embedded in rationales. As shown in Figure 3(d) and 2(b), lacking common sense knowledge, such as the nutrient mainly provided by oranges and factors affecting magnetic force, the fine-tuning model fails to answer these two questions. Compared to zero-shot prompting, the fine-tuning model showcases enhanced error tolerance (as shown in Figure 2(c)), while concurrently showing increased susceptibility to knowledge deficiencies. This stems from the catastrophic forgetting during fine-tuning in LMs [14, 4, 65].
3.1.3 Challenge Lies in LLMs with Multimodality: Hallucinations Intensified
To generate general rationales simultaneously fulfill the above two roles for assisting language models in understanding multimodal information, we first analyze the state-of-the-art method [71], which is trained to generate rationales relying on manual annotations. However, it generates rationales without considering their correctness, thereby leading to the risk of misleading language models. In addition, based on training with manually annotated rationales, it lacks the ability to generate rationales that encompass sufficient relevant knowledge for out-of-distribution questions, leading to a significant performance drop (see Figure 3(b) and the left table in Table 2). Unlike MM-CoT [71], we resort to LLMs to generate rationales with zero-shot prompting, leveraging their intrinsic capacity for generalization. We further explore the strategies to generate rationales.
Uni-modal rationales have limited effect. We start by directly prompting "let's think step by step" without any image information, resulting in visually irrelevant rationale (see Figure 4(a)) and poor performance (see row "w/ naive R" in the right table in Table 2).
Interleaved information exacerbates hallucinations. To generate multimodal rationales involving image information, the challenge lies in mitigating the language hallucinations [24, 36, 17, 2], which is exacerbated by providing interleaved multimodal information at once. A naive attempt is to extract the image caption and combine it with the question as a joint prompt to generate rationales. Despite the integration of image information, the generated rationales remain suboptimal for image-related questions, which is not aligned with our initial expectations. Upon analyzing various cases, we observe that interleaved information predisposes LLMs towards hallucination, generating fabricated visual information. As illustrated in Figure 4, we extracted the caption "a food web" from the image, which lacks the necessary information. Consequently, the language model resorts to imagining image-related information such as "the primary consumers are the zooplankton and the kelp" (see Figure 4(b)), leading to difficulty in distinguishing reliable knowledge from the language hallucinations.
3.2 Zero-shot DDCoT Prompting for Multimodal Rationale Generation
We generate general multimodal rationales based on the following key insights, which are derived from Section 3.1:
(1) Utilize two-step reasoning process, considering that LLM has difficulty jointly reasoning multimodal information directly (Section 3.1.1).
Figure 5: An overview of our DDCoT and its utilization to improve the multimodal reasoning of LMs. Note that although errors encounter in the second sub-problem during visual recognition, the language model rectifies this error in the joint reasoning step with critical thought.
(2) Generate rationales that meet the requirements of both zero-shot and fine-tuning learning, filled with knowledge and requiring fidelity (Section 3.1.2).
(3) Alleviate the intensified hallucinations from interleaved information (Section 3.1.3).
Accordingly, we propose Duty-Distinct Chain-of-Thought Prompting (DDCoT), which encompasses three steps: (1) We utilize LLMs' intrinsic knowledge to generate multimodal rationales. (2) We explicitly cue the LLMs to differentiate the responsibilities of reasoning and recognition step by step. (3) We explicitly mark the negative space for uncertain parts, emphasizing critical thinking in rationale generation. Our DDCoT jointly exploits the reasoning ability in LLMs and the image understanding capability of visual question-answering models for general multimodal rationale generation.
Breaking Reasoning Down to Recognition Steps with Negative-Space Prompting.
First, with the given question, context, and options, we prompt the LLM [37] to deconstruct the input question into a sequence of basic sub-questions, breaking the complex reasoning chain into simple steps, as shown in Figure 5. Diverging prior works in NLP community [9, 19], we introduce the following prompting designs: (1) We utilize a single-stage deconstruction to simplify the problem-solving process. Specifically, we employ the instruction "please think step-by-step and deconstruct the question down to necessary sub-questions" to obtain the sub-question sequence at once. (2) Then we explicitly prompt the LLMs to determine whether each sub-question can be answered without visual information. In cases where sub-questions involving visual recognition are unanswerable, the LLMs are instructed to answer "uncertainty" as a negative space. We provide the model with the following prompt: "Assume that you do not have any information about the picture, try to answer the sub-question and formulate the corresponding sub-answer as 'Uncertain' if the sub-question cannot be determined". By adopting a critical stance towards sub-questions and introducing the explicit assumption of invisibility, we successfully mitigate hallucination in LLMs when dealing with sub-questions involving images, reducing factual errors.
Visual Recognition to Obtain Visual Complements. However, the obstruction caused by negative space prevents LLMs from directly activating the chain-of-thought reasoning. In order to fill the negative space, we leverage the image understanding capability of the off-the-shelf models to acquire visual information as visual complements. Specifically, we employ the visual question answering (VQA) model [26] to individually answer the sub-questions with negative space, which correspond to simple visual recognition problems. Note that our approach can solely leverage the basic visual recognition capability of the VQA model and robust to its possible inference errors thanks to the following joint reasoning integration.
Integrate to Joint Reasoning. With a sequence of complete sub-answers including visual recognition results, we resort to LLMs again to integrate the information and engage in reasoning processes
to derive the rationales. Incorporating our obtained sub-questions and corresponding sub-answers as supplementary information, we prompt LLMs "think step by step" to perform joint reasoning with linguistic and visual information and generate multimodal rationales. Furthermore, we prompt explicitly with "note that the supplementary information given may not always be valid" and "select valid information to form the rationale" to encourage a critical attitude towards the supplementary information and thoughtful analysis. By explicitly highlighting uncertainty, LLMs are able to pay attention to the origin question information and intrinsic knowledge, effectively filtering and even rectifying supplementary knowledge to derive more reasonable answers, as shown in Figure 5.
3.3 Utilization of Multimodal Rationales in General Scenarios
In this section, we introduce the utilization of our rationales to achieve multimodal reasoning, both for zero-shot and fine-tuning learning. For zero-shot learning, we consider prompting the model with rationales, including established knowledge and a reasoning chain of thought. Additionally, for fine-tuning, our proposed rationales, in combination with our proposed deep-layer prompting and rationale-compressed visual embedding, improve deep multimodal understanding and reasoning.
Utilization for Zero-shot Prompting. As shown in Figure 5, for LLMs like ChatGPT, we utilize zero-shot prompts that combine rationales generated by Section 3.2 with our problem statements as inputs to the model. Our rationales generated by explicitly questioning the uncertainty with improving reliability can facilitate LLMs to make more precise assessments and exhibit fewer hallucinations.
Utilization for Fine-tuning Learning. The framework of the fine-tuning process is shown in Figure 5. During fine-tuning, we introduce learnable prompts in both shallow and deep layers to align cross-modal information and facilitate multimodal reasoning. In addition, instead of directly inputting the entire image into the LM, we use the multimodal rationales to guide the filtering of key image features as visual input embeddings for the LM.