SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Deep-Layer Prompting (DLP) is designed to assist in the alignment and joint reasoning of multimodal inputs at multiple levels. It employs not only learnable prompts to facilitate the alignment of visual and linguistic semantics at a shallow level [74, 73, 51, 34] but also utilizes explicit rationales to jointly encode multimodality by learning different prompts for each encoder layer [18, 21, 30, 29]. Specifically, we randomly initialize learnable prompts  $P \in \mathbb{R}^{L \times N_p \times C}$  for  $L$  encoder layers. For the  $l$ -th layer, we load the prompt  $P_l \in \mathbb{R}^{N_p \times C}$  at the begging and end of the visual input embeddings.
- Rational-Compressed Visual Embedding (RCVE). Instead of inputting visual features into the LM directly, we compress visual input embeddings according to our multimodal rationales. Specifically, we leverage the rationales to jointly comprehend the textual and visual contexts, serving as the prior knowledge to filtering visual features. Given the text embeddings  $T \in \mathbb{R}^{N_t \times C}$  that contains context and rationale, the global visual input  $V_g \in \mathbb{R}^C$ , and the local visual inputs  $V_g \in \mathbb{R}^{N_v \times C}$ , we first update the global visual feature based its similarity to text embeddings as follows,
$$
V _ {t} = \operatorname {A t t e n t i o n} \left(V _ {g}, T\right), \tag {1}
$$
where  $V_{t}\in \mathbb{R}^{C}$  is the updated visual feature and Attention  $(\cdot ,\cdot)$  is the standard multi-head crossattention module [56]. Next, instead of directly using the updated visual feature  $V_{t}$  to capture relevant local visual inputs  $V_{I}$ , we introduce low-rank intermediate vectors as crucial mediators for filtering local inputs as follows:
$$
V _ {r} = \operatorname {r e s h a p e} \left(\operatorname {M L P} \left(V _ {t}\right)\right), V = \operatorname {A t t e n t i o n} \left(V _ {r}, V _ {l}\right), \tag {2}
$$
where  $\mathrm{MLP}(\cdot)$  denotes a three-layer linear layer with activation function,  $V_{r}\in \mathbb{R}^{N_{r}\times C_{r}}$  represents  $N_{r}$  low-rank vectors, and  $V$  is the final visual embeddings to input to the LM's encoder.
ScienceQA benchmark [31] is the first multimodal science question-answer dataset comprising 21,000 questions with multiple choices and images. Following previous works [71, 31], we divide ScienceQA into training, validation, and test sets, which contain 12,726, 4,241, and 4,241 examples, respectively. Additionally, the questions in ScienceQA can be categorized into three domains: natural science (NAT), social science (SOC), and language science (LAN).
Implementation. We conduct zero-shot experiments on LLMs including ChatGPT [37] and GPT-3 [3]. Since LLMs are unable to accept images as visual input, we convert images into captions
Model	Report	Size↓	GT-R	NAT	SOC	LAN	TXT	IMG	NO	G1-6	G7-12	Avg
Heuristic baselines
Random [31]	NeurIPS22	-	-	40.28	46.13	29.25	47.45	40.08	33.66	39.35	40.67	39.83
Human [31]	NeurIPS22	-	-	90.23	84.97	87.48	89.60	87.50	88.10	91.59	82.42	88.40
Few-shot models
GPT-3(CoT) [31]	NeurIPS22	175B	✓	75.44	70.87	78.09	74.68	67.43	79.93	78.23	69.68	75.17
ChatGPT(CoT) [33]	arXiv23	175B	✓	78.82	70.98	83.18	77.37	67.92	86.13	80.72	74.03	78.31
Chameleon(ChatGPT) [33]	arXiv23	175B	✓	81.62	70.64	84.00	79.77	70.80	86.62	81.86	76.53	79.93
Zero-shot models
Ours(GPT-3)	175B	78.60	73.90	80.45	77.27	69.96	82.93	80.65	73.50	78.09
Ours(ChatGPT)	175B	80.15	76.72	82.82	78.89	72.53	85.02	82.86	75.21	80.15
Fine-tuning models  $\nabla$
LLaMA based Fine-tuning models
LMAAdapter [69]	arXiv23	7B	84.37	88.30	84.36	83.72	80.32	86.90	85.83	84.05
T5 based Fine-tuning models
UnifiedQA [31]	NeurIPS22	223M	✓	71.00	76.04	78.91	66.42	66.53	81.81	77.06
MMCoT [71]	arXiv23	223M	✓	87.52	77.17	85.82	87.88	82.90	86.83	84.65
UnifiedQA [31]	NeurIPS22	223M	68.16	69.18	74.91	63.78	61.38	77.84	72.98	65.00
MMCoT† [71]	arXiv23	223M	82.51	77.73	82.82	81.09	75.11	85.30	81.64	80.95
Ours	223M	88.72	86.84	84.91	87.59	83.34	88.08	88.58	85.10
+6.21	+9.11	+2.09	+6.50	+8.23	+2.78	+6.94	+4.15
Table 1: Main results (\%). Size = backbone model size. GT-R means models are trained with ground truth rationales. Question classes: NAT = natural science, SOC = social science, LAN = language science, TXT = text context, IMG = image context, NO = no context, G1-6 = grades 1-6, G7-12 = grades 7-12. † denotes implementation by removing ground truth rationales when fine-tuning.
via BLIP-2 [26] as extra input to LLMs. Furthermore, following previous works [71, 31], we adopt UnifiedQA [20] as our base model for fine-tuning. We use CLIP ViT-L/14 [41] as the visual encoder to extract the global visual input  $V_{g}$  and the local visual input  $V_{l}$ . The hyperparameters  $N_{p}, N_{r}$  and  $C_r$  are 3, 16 and 4, respectively. We train our model for 30 epochs with a learning rate of 1e-4 and batch size of 16. All experiments are implemented by PyTorch [39] and HuggingFace [61] and conducted on NVIDIA Tesla A40 GPUs. We employ accuracy as our evaluation metric and furnish comprehensive results across each domain.
4.1 Comparison with State-of-the-Arts in Zero-shot Prompting and Fine-tuning Learning
Table 1 shows the comparison of our DDCoT with the state-of-the-art models [31, 33, 71] on zero-shot and fine-tuning benchmarks. Our approach consistently achieves superior performance compared to previous methods. Note that we further report the results of the recent works [33, 71, 69] which have not been published but have been preprinted in the research community.
Zero-shot Prompting. Regarding zero-shot prompting, our zero-shot approach outperforms the published state-of-the-art few-shot method [31] by  $2.92\%$  on GPT-3 and  $1.84\%$  on ChatGPT, respectively. Compared with the concurrent work Chameleon [33], which incorporates a vast array of external tools, our method maintains comparable performance, even attaining a  $1.73\%$  enhancement on the IMG split. These suggest that our DDCoT, integrated with negative-space prompting, effectively curtails factual errors, guaranteeing the accuracy of multimodal reasoning in LLMs. Furthermore, we observe that the performance of GPT-3  $(67.43\%)$  and ChatGPT  $(67.92\%)$  in a few-shot manner is similar, and the amplification of our technique in the IMG branch escalates as the performance of the language models strengthens  $(2.53\%$  for GPT-3 and  $4.61\%$  for ChatGPT). The possible reason is language models rarely acquire an innate understanding of dense image information, while explicit rationales guidance triggers the inherent reasoning capability of such models.
Fine-tuning Learning. As shown in Table 1, our DDCoT has achieved exceptional accuracy on fine-tuning benchmarks. DDCoT significantly surpasses the base model UnifiedQA [20] by  $17.22\%$  and  $21.96\%$  on avg split and IMG split, respectively. These demonstrate that our rationales not only enhance the model's multimodal reasoning ability in a zero-shot setting but also help the fine-tuning of
LMs to achieve multimodal alignment and joint inference. In comparison to the CoT-based approach MM-CoT [71] which uses annotated rationales for training, our multimodal rationales, generated through zero-shot CoT prompting, still enhance performance by an average of  $2.43\%$ .
(a) Generalization	NAT	SOC	LAN
MMCoT	50.98	77.73	80.00
Ours	66.43	83.69	83.09
MMCoT	78.86	62.20	76.82
Ours	86.5	71.77	85.09
MMCoT	79.53	76.38	55.82
Ours	86.59	86.73	68.00
(b) Analysis	IMG	TXT	Avg
baseline(B)	72.93	85.84	79.7
B+caption	75.26	83.64	79.6
B+origin img	62.82	75.94	69.70
B+DLP&RCVE	75.16	85.25	80.45
no R	75.16	85.25	80.45
w/ naive R	75.06	89.61	82.96
w/ our R	83.34	91.23	87.34
Generalization. As shown in Table 2(a), we conduct a supplementary experiment to evaluate the model's generalization capabilities after fine-tuning. By designating two domains for visibility during training, we report the accuracy of the questions in the unseen remaining domain within the test set. Across all three divisions, our approach consistently surpasses MM-CoT by  $15.5\%$ ,  $9.6\%$ , and  $12.2\%$ , respectively. These results demonstrate the excellent scalability of our rationales when handling out-of-distribution data.
4.2 Analysis Effects of Visual Modality, Rationale Generation, and Fine-tuning Components
The effects of different modalities of visual information. Table 2 (b) presents the effects of different forms of image information on the fine-tuning models: (1) We start by directly adding captions or image inputs to the language only T5 based baseline. The former results in an improvement of  $2.33\%$  on IMG split. However, the absence of additional components and the direct usage of images as input leads to poor performance. The result suggests that models with a confined number of parameters encounter difficulties aligning images with text on small datasets. (2) Nevertheless, with the aid of our well-designed components, including deep-layer prompting (DLP) and rational-compressed visual embedding (RCVE), the language model effectively harnesses the information gleaned from images, thereby achieving the comparable performance of  $75.16\%$  on IMG split with the model using captions as inputs. (3) Despite both methods enhancing performance on IMG split, the improvement remains unsatisfactory due to the language model's difficulty in comprehending image information without guidance, which is also observed in the zero-shot scenario delineated in Section 3.1. Simultaneously, we observe a slight degradation in the performance of the text split, which is potentially attributed to the disparity in inputs between IMG and TXT splits in training data.
The effects of rationale generation components. As shown in Table 2(b) and Table 3, we further demonstrate the effectiveness of the components of our DDCoT prompting for rationale generation. (1) With naive rationales obtained directly from prompting GPT-3 with "let's think step by step", we noted that the IMG branch garners no benefits, even evidencing a
Table 2: The results of (a) Generalization and (b) Modality and Rationale Analysis.
IMG	Avg
naive R	75.06	82.96
decomposed question R	75.61	83.09
duty distinct w/o uncertainty R	78.19	85.15
duty distinct w/ uncertainty R	83.34	87.34
w/o integrate to joint reasoning	77.49	83.75
Table 3: Ablation study on components of DDCoT.
slight degradation in performance. It indicates that introducing rationales can facilitate the language model in comprehending and reasoning within the context, while image-agnostic rationales do not contribute to the model's multimodal understanding. (2) Compared to the naive rationale [25], deconstruction [72, 22] and joint reasoning without specialized designs result in a tiny improvement (row 2 in Table 3). The limited improvement can be attributed to the fact that LLMs still perform a combined reasoning and recognition task, leading to a suboptimal execution of the recognition task. (3) Hence, our duty distinct design, implemented through negative-space prompting, achieves  $2.58\%$  and  $2.06\%$  performance gain on IMG and average splits (row 3 in Table 3). It alleviates the reliance on the reasoning capability of the off-the-shelf visual model and generate more reliable multimodal rationales. (4) Moreover, by explicitly emphasizing uncertainty (row 4 in Table 3), we observe a notable improvement of  $5.15\%$  on IMG split, while also contributing to an average enhancement of  $2.19\%$ . It achieves remarkable enhancement of correctness, which also significantly contributes to fine-tuning learning.
Quantitative Analysis of Hallucinations. Additionally, we conduct a human evaluation of the authenticity to assess the phenomenon of hallucinations [24] in the rationale generation process. As shown in the Table 4, introducing interleaved visual information exacerbates the hallucinations and diminishes the
Method	Context	Authenticity
naive R	w/o visual	0.883
naive R	interleaved	0.602
duty distinct w/o uncertainty R	interleaved	0.783
duty distinct w uncertainty R	interleaved	0.855
authenticity of generated rationales by  $28.1\%$  compared with naive prompting devoid of any visual information input. Our duty-distinct design significantly mitigates the impact of hallucinations. Moreover, the further suppression of hallucinations is observed with the explicit emphasis on uncertainty. These feedbacks align with the findings in Section 3.1.3 and the results presented in Table 3.
The roles of our fine-tuning components. We also validate the impact of our proposed deep-layer prompting and rational-compressed visual embedding. We replace the rational-compressed embedding with origin visual features, resulting in reductions of  $1.28\%$  and  $3.02\%$  in average performance and IMG split, respectively. Similarly, removing deep-layer prompting which assists alignment leads to a
Table 4: Human evaluation on hallucinations.
IMG	Avg
Ours	83.34	87.34
w/o RCVE	80.32	86.06
w/o DLP&RCVE	79.33	85.57
$0.49\%$  and  $0.99\%$  reduction. Additionally, we further conduct ablation studies on the hyperparameters of  $N_{p}$  and  $N_{r}$ . Detailed results and analysis will be presented in the supplementary material.
Table 5: Ablation study on fine-tuning components.
Method	w/o GT-R	Comparison with GT-R	Human Evaluation
B-1	B-4	R-L	Sim.	Relevant	Correct	Complete	Coherent	Explainable
MMCOT	0.970	0.930	0.970	0.990	70.83%	67.99%	64.81%	57.94%	58.73%
GPT-3	✓	0.075	0.018	0.249	0.548	81.01%	75.99%	65.54%	61.64%	60.32%
Ours	✓	0.147	0.041	0.287	0.601	92.00%	86.38%	85.71%	84.33%	83.26%
Table 6: Automatic metrics and human evaluation of explainability.
Explainability. We further exhibit the explainability evaluation including the automatic metrics and human evaluations in Table 6. In automatic metrics, we adopt the BLEU-1/4 [38], ROUGE-L [38], and Sentence Similarity [42] metrics. Note that the automatic evaluation solely reflects the similarity between the generated rationales and the annotated ground truth. In the course of human evaluation, annotators are required to grade each rationale on the criteria of Relevance, Correctness, Completeness, Coherence, and Explainability. It is worth noting that although previous methods [31, 71] achieve decent results in terms of Relevant and Correct, they perform particularly poorly in other areas, especially Explainability. In contrast, despite modest automatic metric results, our rationales generated by DDCoT significantly surpass other methods across all aspects of human evaluation, which is more valuable for interpretable studies.
This paper proposes a novel DDCoT prompting for multimodal reasoning tasks with language models. Not only achieving state-of-the-art performance on zero-shot learning, our model, with the novel techniques of deep-layer prompting and rational-compressed visual embedding, also demonstrates significant reasoning ability on the ScienceQA benchmark. The experimental results demonstrate the superiority and generalization ability of our proposed DDCoT for both zero-shot learning and fine-tuning. Ethical Statement. Our model is based on currently available pre-trained language models. In our experiments, we have not utilized any information concerning personal privacy, data security, or ethical hazards. However, it is crucial to acknowledge that large language models may introduce certain biases, which can originate from the training datasets. We must recognize that entirely eliminating biases from the model is a challenging task, and current technology may not achieve it completely. Therefore, we encourage users to be mindful of these potential biases when utilizing our model and to comprehend that the model's outputs are not infallible but require interpretation and understanding in conjunction with real-world circumstances.
Acknowledgment: This work was supported by the National Natural Science Foundation of China (No.62206174), Shanghai Pujiang Program (No.21PJ1410900), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
In this section, we present additional implementation details, experiment results and discussions. The content structure is outlined as follows:
Section A - Additional Results
- Section A.1 - Insights for Different GPT Models
- Section A.2 - Quantitative ablations on our DDCoT and visual components
- Section A.3 - Hyperparameters for Fine-tuning Components
- Section A.4 - Additional experiments on the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models
- Section A.5 - Additional experiments on Captioning and Video Question Answering tasks
Section B - Human Evaluation
Section C-Detailed Prompts
Section D - Case Studies
Section E - Limitations
Figure 6: Cases that GPT-3 and ChatGPT face the difficulty in understanding dense image information.
A.1 Insights for Different GPT Models
In the submitted paper we have presented several findings concerning the LLMs. In this section, we aim to provide additional illustrative instances for GPT-3 [3] and the recent and potent ChatGPT [37].
Difficulty in understanding dense image information. Figure 6 presents additional instances of LLMs failing to comprehend dense image information. For sparse captions like the example in Figure 2 of the submitted paper, we observe that both GPT-3 and ChatGPT may struggle to comprehend image information in captions. Additionally, when prompted with more detailed captions, their difficulties in understanding become more pronounced. This highlights the challenges that even the most versatile and powerful language models currently available face in comprehending dense image information.
Which of the following fossils is older? Select the more likely answer.
Context:
This diagram shows fossils in an undisturbed sedimentary rock sequence.
Caption:
The figure displays three sedimentary rock layers. The uppermost layer contains palm leaf fossils, the second layer contains root fossils, and the bottom layer contains wood fossils.
A. root
B. palm leaf
C. wood
w/o rationale
GPT-3: C
ChatG
correct rationale
The shallower, the older.
ChatGPT: C
wrong rationale
The deeper, the older.
GPT-3: B
ChatGPT: B
Complete the text to describe the diagram. Solute particles moved in both directions across the permeable membrane. But more solute particles moved across the membrane
When there was an equal concentration on both sides, the particles reached equilibrium. The diagram below shows a solution with one solute. Each solute particle is represented by a green ball. The solution fills a closed container that is divided in half by a membrane. The membrane, represented by a dotted line, is permeable to the solute particles. The diagram shows how the solution can change over time during the process of diffusion.
The figure illustrates the state transition of a box. In the initial state, there are three green particles to the left of the box's center line and five green particles to the right. In the subsequent state, the box shows four green particles on each side of the center line
A. to the left than to the right
B. to the right than to the left
Figure 7: Cases of rationale-sensitive reasoning with GPT-3 and ChatGPT.
GPT-3: A
ChatGPT: A
As can be seen from the image, the initial state has five particles on the right side of the membrane and three particles on the left side of the membrane. As time passes, the left side becomes four particles, i.e., one particle is gained; the right side also becomes four particles, i.e., one particle is lost. Therefore, the direction of motion of the particles is to left.
As can be seen from the image, the initial state has five particles on the right side of the membrane and three particles on the left side of the membrane. As time passes, the left side becomes four particles, i.e., one particle is lost; the right side also becomes four particles, i.e., one particle is gained. Therefore, the direction of motion of the particles is to right.
Rationale-sensitive reasoning. Figure 7 shows more examples where the reasoning of GPT-3 and ChatGPT is sensitive to the input rationales. The correct answers obtained without using rationales indicate that the LLMs possess commonsense knowledge to respond to the question. However, incorrect inputs can lead to misleading outcomes for both GPT-3 [3] and ChatGPT [37].
Note that the challenges involved in understanding dense image information, coupled with the difficulty of obtaining high-quality captions in practical scenarios, limit the universal applicability of reasoning without using rationales, although it may be feasible in the presented examples.
Hallucinations during generation. Figure 8 illustrates several instances where both GPT-3 and ChatGPT fall into hallucinations during rationale generation.
Our findings indicate that in cases where the provided caption information is insufficient, such as in the first example, both GPT-3 [3] and ChatGPT [37] tend to imagine image details to respond to the question. Even when we provide ample information, as demonstrated in the second and third
Compare the average kinetic energies of the particles in each sample. Which sample has the higher temperature?
The diagrams below show two pure samples of gas in identical closed, rigid containers. Each colored ball represents one gas particle. Both samples have the same number of particles.
The diagrams below show two pure samples A and B.
Sample A
Mass of each particle: 44 u
Sample B
Mass of each particle: 44 u
Average particle speed:  $760~\mathrm{m / s}$
Rationale: The particles in sample B have more kinetic energy on average than the particles in sample A. This means that sample B has a higher temperature.
Rationale: The average kinetic energy of gas particles is directly proportional to the temperature of the gas. Sample B has a higher average kinetic energy because its particles are moving faster and have a wider range of velocities compared to Sample A.
Therefore, Sample B has a higher temperature than Sample A.
Ours DDCoT w/o Caption
Rationale: The average kinetic energy of gas particles is directly proportional to temperature. The particles in sample A are moving at a much faster speed than the particles in sample B. Both samples have the same number of particles. Therefore, sample A has a higher temperature than sample B.
Which continent is highlighted?
Context: N/A
The picture shows a map of the world. In it, the area right in the middle is highlighted. It is roughly shaped as an inverted triangle. It has a sharp corner on the right and a rounded shape on the upper left. It is connected to a peninsula at the top right and a detached island at the bottom right.
A. Europe
B. Africa
C. Asia
D. South America
Figure 8: Cases of hallucinations when generating rationales with GPT-3 and ChatGPT.
What is the probability that a rat produced by this cross will have a dwarf body?
In a group of rats, some individuals have a normal-sized body and others have a dwarf body. In this group, the gene for the body size trait has two alleles. The allele for a dwarf body (b) is recessive to the allele for a normal-sized body (B). This Punnett square shows a cross between two rats.
The image shows a table of genes. The horizontal axis is Bb, and the vertical axis is BB
A.  $3/4$
B.  $0/4$
C.  $2/4$
D.  $1 / 4$
E.  $4/4$
Rationale: The continent that is highlighted is Europe. This is because Europe is the only continent that is shaped as an inverted triangle and has a sharp corner on the right and a rounded shape on the upper left.
Rationale: Based on the description of the highlighted area, we can identify that it is the Arabian Peninsula. The peninsula is connected to a detached island at the bottom right, which is the island of Socotra. The peninsula is also connected to a larger landmass at the top right, which is the continent of Asia. Therefore, the correct answer is Asia.
Rationale: The highlighted area is a circle and has the Sahara desert as its geographical feature. The neighboring countries are not relevant to the question. By looking at the image and considering the supplementary information, we can determine that the highlighted area is in Africa.