SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rationale: The Punnett square only shows the possible genotypes of the offspring, not the phenotype. In this case, the phenotype is determined by the body size allele, which is controlled by the B gene. The B allele is dominant, so the phenotype of the offspring will be determined by the B allele. The B allele is present in 3 out of the 4 possible genotypes, so the probability that the offspring will have a normal-sized body is  $3/4$ .
Rationale: The Punnett square shows the possible genotypes of the offspring. To determine the probability of a rat having a dwarf body, we need to look at the genotype of the rat. If a rat has at least one copy of the recessive allele (bb), it will have a dwarf body. Looking at the Punnett square, we can see that there are four possible genotypes of the offspring: BB, Bb, Bb, and bb. Only the last genotype (bb) will result in a dwarf body. There is only one way to get a bb genotype, which is by crossing two Bb rats. Therefore, the probability of a rat produced by this cross having a dwarf body is 1/4 or  $25\%$ .
Rationale: The Punnett square shows that the genotypes of the parent rats are Bb and Bb. The possible genotypes of offspring are BB, Bb, and bb. The allele for a dwarf body (b) is recessive to the allele for a normal-sized body (B). Therefore, only the bb genotype will result in a dwarf body. The probability of an offspr- ing inhering the recessive allele for dwarf body (bb) is  $1 / 4$  or  $25\%$
examples, we observe that both language models still may fall into hallucinations that do not align with the provided information.
In contrast to generating rationales solely based on the caption and question information, our approach can alleviate the hallucinations to some extent by decomposing the questions into simple recognition tasks and emphasizing the uncertainty of image-related aspects. It is worth noting that while the former direct methods rely on manually crafted high-quality rationales, our approach utilizes BLIP-2 [26] as a visual question answering (VQA) model, serving as a visual component. However, our approach outperforms the former methods in terms of performance and interpretability.
Unfortunately, it is very challenging to entirely solve the issues of hallucinations. Although we alleviate hallucinations, we still encountered difficulties in certain cases, such as the third example, and further details regarding this limitation will be explored in section E.
IMG	TXT	Avg
baseline(B)	72.93	85.84	79.70
B + our R	75.81	82.40	82.83
B + gt R	81.07	84.07	85.97
our model	75.16	85.25	80.45
our model + our R	83.34	91.20	87.34
our model + gt R	84.43	92.09	88.00
A.2 Quantitative ablations on our DDCoT and visual components
We conduct additional ablation study on the extent of impact exerted by our DDCoT prompting and visual components, as shown in Table 7. DDCoT prompting and visual components cooperatively facilitate inducing visual information to language models for multimodal reasoning. We can observe that rationales generated by our DDCoT and visual components individually exhibit certain gains in terms of IMG improvement. However, when combined, they yield substantial gains.
Besides, please note that the annotated ground truth rationales within the ScienceQA dataset inherently encompass the final prediction, i.e., correct answers. To ensure a fair comparison, we manually exclude the answers from these annotations, using the remaining text as input rationales for fine-tuning. Under this configuration, our proposed rationale achieves a fine-tuning performance comparable to the annotated rationales.
Table 7: Quantitative ablations on our DDCoT and visual components.
Np	1	3	5	Nr	8	16	32
Avg	86.72	87.34	86.02	Avg	86.63	87.34	86.30
Table 8: Ablation of  ${N}_{p}$  and  ${N}_{r}$  .
A.3 Hyperparameters for Fine-tuning Components
Table 8 presents the results of our ablation studies on  $N_{p}$ , which denotes the number of learnable prompt tokens, and  $N_{r}$ , which represents the number of low-rank vectors. Regarding  $N_{p}$ , we conducted experiments using values of 1, 3, and 5. Increasing the number of prompts introduces more learnable parameters and provide stronger guidance to align vision and language, while too many prompts may disrupt the model's comprehension of visual features and result in a decline in performance. Considering  $N_{r}$ , we investigated the influence of different filtering intensity in the Rational-Compressed Visual Embedding process by experimenting with values of 8, 16, and 32. The experimental results indicate that selecting 3 for  $N_{p}$  and 16 for  $N_{r}$  yields the best performance.
A.4 Additional experiments on the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models
We also validate the effectiveness of our DDCoT with existing pre-trained VLMs and multimodal reasoning models, as shown in Table 9. We observe that rationales generated by our proposed DDCoT is compatible with such pre-trained VLMs. Without correct rationales, existing pretrained
Model	NAT	SOC	LAN	TXT	IMG	NO	Avg
Flamingo [2]	21.89	52.41	20.27	23.50	39.11	19.02	27.87
Flamingo with our R	39.20	48.93	30.90	39.68	45.81	32.40	39.01
MiniGPT-4 [75]	43.83	48.59	43.36	55.01	42.84	41.67	44.71
MiniGPT-4 with our R	57.37	62.32	46.82	65.91	56.72	48.57	55.67
VLMs and multimodal reasoning models have difficulty in complex reasoning tasks. Fortunately, the generalizable rationales generated by our DDCoT prompting can help existing VLMs to comprehend visual information and reason with rich knowledge, achieving significant improvement of  $11.14\%$  and  $10.96\%$  based on Flamingo [2] and Mini GPT-4 [75] as Table 9 shows.
Table 9: The effectiveness of our DDCoT with Flamingo [2] and MiniGPT-4 [75]
CIDEr	SkipThoughtCS	NoCaps EmbeddingAverageCS	GreedyMatchingScore	MSVD-QA Acc
BLIP-2	76.15	49.84	89.20	77.94	34.4
Ours	46.26	84.78	92.35	79.12	39.3
Table 10: Addition experiments on NoCaps and MSVD-QA.
A.5 Additional experiments on Captioning and Video Question Answering tasks
We extend our approach to more appropriate datasets. While other existing datasets may not fully exploit the benefits of our approach, we venture into exploring the captioning task on NoCaps [1] and the video question-answering task on MSVD-QA [64], as shown in Table 10.
For captioning, we prompt the LLM [37] to solve sub-problems derived from a simple caption, aiming to optimize and enrich it using the corresponding sub-answers. The substantial knowledge within LLM enables the generation of semantically enriched captions, leading to improvements in metrics evaluating sentence semantics, i.e.  $34.94\%$ ,  $3.15\%$ , and  $1.18\%$  in terms of SkipThoughtCS [23], EmbeddingAverageCS [46], and GreedyMatchingScore [45]. Note that the CIDEr [57] metric to evaluate ours is limiting. It is designed to measure the similarity between the tested caption and reference captions without considering the diversity and high-level semantics.
For video question answering, LLM deconstructs problems like the decomposition step on ScienceQA. We sample video frames for VQA recognition and integrate frame information for multimodal rationale and answers. Leveraging the sequence understanding in LLM and visual information returned by the VQA model, we achieve a  $4.9\%$  improvement over BLIP-2 [26].
Note that we randomly evaluated only 1000 images from NoCaps and 1000 videos from the MSVD test dataset in a zero-shot setting.
B Human Evaluation
This section introduces the details of our human evaluation. Figure 9(b) shows an example of our question. Each sample comprises the question, context, options, and image. Evaluators are asked to rate the rationales generated by GPT [3], MM-COT [71], and our method in five aspects: relevance (related to the question), correctness (accuracy of reasoning and answer), completeness (logical reasoning's comprehensiveness), coherence (consistency of reason), and explainability (interpretability of reasoning and answer). The rating scale ranges from 0 to 5. Additionally, the questions and rationales are organized into 12 groups, with each group assigned to three evaluators. Finally, we average the scores for each aspect of each rationale, resulting in overall scores and ratios relative to the maximum score.
C Detailed Prompts
This section introduces more details about our zero-shot DDCoT prompting. We employ ChatGPT[37] as the most important component of our rationale generator. Specifically, it is utilized in breaking
Figure 9: Interface of human evaluation. Figure(a) shows the instructions, figure(b) shows one example of our question.
duties of reasoning and recognition step and joint reasoning step. Figure 10 shows complete prompts for zero-shot DDCoT prompting.
D Case Studies
To better understand the effectiveness of our proposed method in generating rationales, we randomly selected several cases from the test set along with the process of rationale generation. Figure 11 showcases several map-related questions, demonstrating how our method integrates simple visual features (such as the shape of highlighted areas) with common knowledge to obtain correct reasoning and answers. In the examples presented in Figure 12, our method successfully identifies within the images, acquiring relevant knowledge. Figure 13 illustrates four more complex questions, where our method leverages information obtained from the images to perform intricate reasoning. However, when it comes to the complex interaction between images and textual context, our method still falls into hallucinations, leading to erroneous reasoning and answers.
While we have succeeded in mitigating a portion of the hallucination problem arising from multimodal inputs, it is important to acknowledge that this problem is not entirely resolved. As illustrated in Figure 13, our model is also susceptible to the risk of hallucinations. Investigating methods to suppress hallucinations is a potential topic for further research and exploration. In addition, we did not use extra image-text pairs to pre-train the alignment between vision and language modalities. Such pre-trianing is expected to further improve the alignment for joint reasoning.
As our approach involves zero-shot prompting of the LLM to generate rationales, there exists a potential risk of inheriting social biases from the LLM. These biases, which encompass cultural, ethical, and various other dimensions, might be reflected in the generated rationales, potentially leading to adverse effects on users. To mitigate this issue in the future, potential solutions could involve designing constraints at each prompting stage or utilizing more advanced LLMs trained on unbiased resources.
DDCoT
Breaking Reasoning Down to Recognition Steps with Negative-Space Prompting
System: You are a helpful, highly intelligent guided assistant. You will do your best to guide humans in choosing the right answer to the question. Note that insufficient information to answer questions is common, because you do not have any information about the picture. The final answer should be one of the options.
User: Given the context, questions and options, please think step-by-step about the preliminary knowledge to answer the question, deconstruct the question as completely as possible down to necessary sub-questions based on context, questions and options. Then with the aim of helping humans answer the original question, try to answer the sub-questions. The expected answering form is as follows:
Sub-questions:
1.<sub>sub-question 1</sub>
2.<sub>sub-question 2</sub>
Sub-answers:
1.<sub>sub-answer 1</sub> or 'Uncertain'
2. <sub>sub-answer 2> or 'Uncertain'
... Answer: <One of the options> or 'Uncertain'
For a question, assume that you do not have any information about the picture, but try to answer the sub-questions and prioritize whether your general knowledge can answer it, and then consider whether the context can help. If sub-questions can be answered, then answer in as short a sentence as possible. If sub-questions cannot be determined without information in images, please formulate corresponding sub-answer into "Uncertain".
Only use "Uncertain" as an answer if it appears in the sub-answers. All answers are expected as concise as possible.
Here is an attempt:
Context: {context}
Has An Image: {has_image}
Question: {question}
Options: {option}"
Visual Recognition to Obtain Visual Complements
$\sim$  Arbitrary Visual Question Answering (VQA) Model  $\sim$
Integrate to Joint Reasoning
System: You are a helpful, highly intelligent teacher. You will not only do your best to guide humans to the correct answer, but you will also give the rationales as a reference.
User: Given the context, questions, options, supplementary information, think step by step and answer the questions. Please note that we need not only the answer, but more importantly the rationales of getting the answer. The expected answering form is as follows:
Rationale: <rationale>
Answers: <one of the options>
Please note that the supplementary information given may not always be valid. Please select valid information to form the rationale and choose the relatively correct option as your answer.
Options: {option}
Supplementary information: {supplementary_information}
Figure 10: The complete prompts of our zero-shot DDCoT prompting. The underlined parts are introduced in Section 3.2 of the submitted paper. The yellow background represents designs for critical thinking, and the blue background represents designs for differentiate duties of reasoning and recognition.  $\{\}$  denotes the corresponding inputs, and  $\{$  supplementary_information\} represents the sub-questions and sub-answers obtained before.
What is the capital of North Carolina?
A. Saint Paul B. Fargo C. Raleigh D. Charlotte
Ours Outputs
Sub-question:
(1) What is the location of North Carolina?
(2) What is the capital of North Carolina?
Sub-answers: (1) North Carolina is a state located in the southeastern region of the United States. (2) The capital of North Carolina is Raleigh.
VQA: Understanding [sub-questions, image]
Sub-answers': (1) North Carolina is a state located in the southeastern region of the United States. (2) The capital of North Carolina is Raleigh.
Rationale: The question is asking for the capital of North Carolina. The supplementary information confirms that North Carolina is a state located in the southeastern region of the United States and the capital of North Carolina is Raleigh.
GPT3(COT) Outputs
BECAUSE: Raleigh is the capital of North Carolina.
Which property matches this object?
A. North America
C. Antarctica
B. South America
D. Australia
(1) What is the shape of the highlighted area?
(2) What are the geographical features of the highlighted area?
(3) What are the surrounding bodies of water of the highlighted area?
Sub-answers: (1) Uncertain (2) Uncertain (3) Uncertain
VQA: Understanding[sub-questions, image]
Sub-answers': (1) a circle (2)a mountain range (3) atlantic ocean, pacific ocean, and arctic ocean
Rationale: The highlighted area is a circle with a mountain range, and it is surrounded by the Atlantic Ocean, Pacific Ocean, and Arctic Ocean. Based on this information, we can eliminate Antarctica and Australia as they are not surrounded by these three oceans. The mountain range in the highlighted area is likely the Rocky Mountains, which are located in North America.
BECAUSE: Antarctica is the world's fifth largest continent. It is also the coldest, windiest, and driest continent.
Which country is highlighted?
A. Haiti B. Saint Lucia
C. Saint Vincent and the Grenadines
(1) What is the image showing?
(2) What are the distinguishing features of the highlighted country?
(3) What are the geographical locations of the countries in the options?
Sub-answers': (1) the location of the caribbean (2) the southeastern coast is a peninsula
(3) the caribbean islands are located in the western hemisphere
Rationale: The image is showing the location of the Caribbean. The supplementary information tells us that the highlighted country has a southeastern coast that is a peninsula. Based on the options, only Haiti and Saint Lucia have a southeastern coast that is a peninsula. However, Haiti does not have a peninsula that is as pronounced as the one in the image.
BECAUSE: The country highlighted is Saint Vincent and the Grenadines.
Which of these continents does the prime meridian intersect?
A. South America B. Asia C. Antarctica
Figure 11: Cases that need to understand the map.
(1) What is the prime meridian?
(2) Which continents does the prime meridian pass through?
(3) Is the prime meridian shown in the image?
Sub-answers: (1) The prime meridian is the line of longitude that is designated as  $0^{\circ}$  longitude. It is the line that divides the Earth into the Eastern and Western Hemispheres. (2)The prime meridian passes through Europe, Africa, and Antarctica. (3) Uncertain
Sub-answers': (1) The prime meridian is the line of longitude that is designated as  $0^{\circ}$  longitude. It is the line that divides the Earth into the Eastern and Western Hemispheres. (2)The prime meridian passes through Europe, Africa, and Antarctica. (3) no
Rationale: The prime meridian is the line of longitude that is designated as  $0^{\circ}$  longitude. It is the line that divides the Earth into the Eastern and Western Hemispheres. The supplementary information states that the prime meridian passes through Europe, Africa, and Antarctica. The image does not show the prime meridian.
BECAUSE: The prime meridian intersects with Asia.
Select the mammal below.
Context: Mammals have hair or fur and feed their young milk. A gorilla is an example of a mammal.
A. poison dart frog B. rabbit
(1) Does the image show a mammal?
(2) Is the animal in the image a poison dart frog or a rabbit?
Sub-answers: (1) Uncertain (2) Uncertain
Sub-answers': (1) Yes (2)poison dart frog
Rationale: The given context states that mammals have hair or fur and feed their young milk. The supplementary information confirms that the animal in the image is a mammal. Therefore, we need to select the option that represents a mammal. The option 'poison dart frog' is not a mammal as it does not have hair or fur and does not feed its young milk. On the other hand, the option 'rabbit' is a mammal as it has fur and feeds its young milk.
BECAUSE:A poison dart frog is not a mammal.
Select the fish below.
Context: Fish live underwater. They have fins, not limbs. A tiger shark is an example of a fish.
A. salmon
B. woodpecker
(1) Is salmon a fish?
(2) Is woodpecker a fish?
Sub-answers: (1) Yes (2) Uncertain
Sub-answers': (1) Yes (2)no, it is a bird.
Rationale: Fish live underwater and have fins, not limbs. The given image shows a fish, which means we need to select an option that is also a fish. The supplementary information confirms that salmon is a fish, while woodpecker is a bird.
BECAUSE:Salmon are fish. Woodpeckers are not fish.
Which material is this teapot made of?
A. porcelain
B. rock
(1) What does the teapot look like?
(2) What are the characteristics of porcelain and rock?
Sub-answers: (1) Uncertain (2) Porcelain is a type of ceramic material that is often white and translucent. It is made by heating materials, generally including clay in the form of kaolin, in a kiln to temperatures between 1,200 and  $1,400^{\circ}\mathrm{C}$ . Rock is a naturally occurring solid substance composed of minerals.
Sub-answers': (1) a pink and green teapot with pink roses (2) Porcelain is a type of ceramic material that is often white and translucent. It is made by heating materials, generally including clay in the form of kaolin, in a kiln to temperatures between 1,200 and  $1,400^{\circ}\mathrm{C}$ . Rock is a naturally occurring solid substance composed of minerals.
Rationale: The question asks about the material of the teapot, and the options are porcelain and rock. Porcelain is a type of ceramic material that is often white and translucent, while rock is a naturally occurring solid substance composed of minerals. The supplementary information describes porcelain as being made by heating materials, generally including clay in the form of kaolin, in a kiln to temperatures between 1,200 and  $1,400^{\circ}\mathrm{C}$ . The image shows a pink and green teapot with pink roses, which is consistent with the appearance of porcelain teapots. Therefore, the material of this teapot is most likely porcelain.
BECAUSE:Teapots can be made from a variety of materials, including porcelain, metal, glass, and clay.
A. transparent.
B. bouncy
glass bottle
Figure 12: Cases that need to identify objects and acquire relevant knowledge.
(1) What is the color of the object?
(2) What is the shape of the object?
Sub-answers': (1) white (2) a glass bottle
Rationale: The object is a glass bottle and the color is white. Based on this information, the property that matches the object is "transparent" as glass is a transparent material and allows light to pass through it.
BECAUSE: The definition of transparent is "able to be seen through with clarity." The definition of bouncy is "able to rebound or spring back into shape after being compressed."
A.sample B.B.sample A C. neither; the samples have the same temperature
Sample A  
Mass of each particle: 44 u  
Average particle speed: 770 m/s
Sample B:  
Melt of each particle:  $44\mathrm{~m}$   
Average particle velocity:  $760\mathrm{~m / s}$
(1) What is the relationship between the average kinetic energy of gas particles and temperature?  
(2) Is there any difference in the speed or mass of the particles in each sample?
(3) Is there any difference in the number of particles in each sample?  
Sub-answers: (1) The average kinetic energy of gas particles is directly proportional to temperature. (2) It is uncertain whether there is any difference in the speed or mass of the particles in each sample. (3) Both samples have the same number of particles.
Sub-answers': (1) The average kinetic energy of gas particles is directly proportional to temperature. (2) the particles in sample a are moving at a much faster speed than the particles in sample b. (3) Both samples have the same number of particles.
BECAUSE:The average kinetic energy of the particles in sample B is higher than the average kinetic energy of the particles in sample A. This means that sample B has a higher temperature.
Which solution has a higher concentration of blue particles?
Context: The diagram below is a model of two solutions. Each blue ball represents one particle of solute.
A.sample A B.sample B C. neither; their concentrations are the same
Solvent volume:  $20~\mathrm{mL}$  Solution A
Solvent volume:  $20~\mathrm{mL}$  Solution B
(1) What do the blue balls represent in the diagram?
(2) Is there a difference in the number of blue balls in Solution A and Solution B?
Sub-answers: (1) The blue balls represent particles of solute. (2) Uncertain
Sub-answers': (1) The blue balls represent particles of solute. (2) no
Rationale: The blue balls represent particles of solute in both Solution A and Solution B. As there is no difference in the number of blue balls in both solutions, the concentration of blue particles in both solutions is the same.
BECAUSE: The concentration of a solution is the amount of solute in a given amount of solvent. In other words, it is the amount of solute per unit volume of solvent. The concentration of a solution can be expressed in several ways. One way is to express it as the number of moles of solute per liter of solution. Another way is to express it as the number of grams of solute per liter of solution. The concentration of Solution A is 0.5 moles per liter. The concentration of Solution B is 1.0 moles per liter. Therefore, Solution B has a higher concentration of blue particles.
Which month has the lowest average precipitation in London?
Context: Use the graph to answer the question below.
A. July B. November C. February
(1) What does the graph show?
(2) How can we determine the average precipitation for each month?
(3) Which month has the lowest average precipitation?
Sub-answers: (1) The graph shows the precipitation levels in London for each month of the year.(2) We can determine the average precipitation for each month by calculating the total precipitation for that month and dividing it by the number of years in the dataset. (3) The month with the lowest average precipitation in London is February.