IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
Authors: Haoxuan You
Rui Sun
Zhecan Wang
Long Chen
Gengyu Wang
Hammad Ayyubi
Kai-Wei Chang
Shih-Fu Chang
Venue: EMNLP 2023 Findings (2023)
Keywords: Vision-Language Learning
Vision-Language Model
Large Language Model
Zero-Shot Evaluation
TL;DR: We introduced IdealGPT, an innovative vision-language reasoning framework that decomposes questions into sub-questions, uses VLMs for answering, and employs an LLM to determine whether to finalize the answer or re-initiate the iteration.

Abstract: The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT.

IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
Haoxuan You'; Rui Sun'; Zhecan Wang'; Long Chen', Gengyu Wang' Hammad A. Ayyubi', Kai-Wei Chang', Shih-Fu Chang!
Columbia University HKUST University of Califormia, Los Angeles
(hy2612, rs4110, zw2627, gengyu wang, ha2578, sc250)acolunbia.
longchen@ust. hk, kwchangêcs. ucla.edu
Abstract
End-to-End Methods:
Answer They are walking along streets
The field iision-nnd-languaae (VL) under- standing has made unprecedented progress with end-to-end large pre-trane mre-trained VL models (VLMs). However, they still fall short zemo shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely domain-specific sub-question decom- posing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, framework that iteratively decomposes VL rea- soning using large language models (LLMs). Specifically, IdealGPT utilizes an L.I. to gen- erate sub-questions, VLM to provide corre- sponding sub-answers. and another LLM to reason to achieve the final answer. These three modules perform the divide-and-congue- pro- cedure iteratively until the model is confident about the final answer to the main question. evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by absolute 10% on VCR and 15% on SNLI-VE. Code is avail- able at https//gittabbco///yoouelaaaaa
End-to-End VLMS
Main Question: What are the man and woman doing here?
Ours:
Answerer (VLMs)
Questioner
Question
Sub- Answer 1:
The wearing long white dress, man is formal artire
weaaing?
-Sub-Suuueee Questieetoo 2:
Where are the people located?
Answer
At wedding
Sub- Que--uuetionn 3: How are the mao and woman interacting with each other?
Ano-er They are holding handss.
Unconfident
New Iteration
We
If Confident
Reasoner
Sp
Finish
They are going to get
mariied because...
Figure I: Comparisons between the pipelines of preva- lent end-to-end VLMs (Upper) and our proposed Ideal- GPT (Below) for VL reasoning tasks.
VL tasks. However, they still struggle to address zero-shot VL reasoning tasks that require intricate or multi-step inferencing such as visual common- sense reasoning (VCR) (Zellers et al., 2019), exemplified in Figure 1. Despite the overall suc- cess of these models, the difficulties with zero-shot reasoning settings represent a serious challenge in the current state of VL research.
1 Introduction
The field of -ssion-and-languaaa (VL) understand- ing has witnessed a prolifcration of pre-trained VL models (VLMs) (Yu et al., 2022; You et al., 2023; Alayrac et al., 2022; Zhu al., 2023b; Liu al., 2023). They are usually pre-trained and fine- tuned in end-to-end fashion, i.e., these models always directly make final predictions in single step. With abundant pre-trained knowledge, they already achieve impressive results in comparison to human performance across many downstream
In comparison, humans are able to excel in these tasks effortlessly. As in Figure 1, to answer the question, "What are the man and woman doing here?", ; we humans would intuitively approach it in a seride-and-conquee fashion by starting with simple questions like "What are they dressed as?" to identify people in the image first. Then we may advance further to wonder about their interactions, "How do they interact?" and the location, "Where
Equal Contribution

---

are they at?". Even though it may seem challenging answer the original question directly, it is much easier to answer the three sub-questions. After an- swering the three simpler sub questions, we can understand the image comprehensively, and utilize commonsense knowledge to obtain conclusion: "they are going to gel married". This step-by-step procedure can be explicitly formulated as de- compositional reasoning process of three key steps: (I) dividing the main question into multiple sub- questions focusing on visual details, (2) answering casier sub-questions, and (3) reasoning upon the sub-anse answers to address the main question.
quently, the Reasoner analyzes cumulative infor- mation extracted from sub-answers to infer the pos- sible answer to the main question. If the Reasoner ascertains that the evidence gathered so far is in- sufficient to confidently answer the main question (either due to uninformative sub-questions or noisy sub-answers), it loops back to the Questioner with its analysis of the gathered evidence. Hence, Quest questioner would purposely try generate more targeted sub-questions obtain more informative evidence. These iterations of the QA loop would continue to be initiated until Reasoner is confident of resolving the main question or the number of iterations reaches a predefined threshold.
Inspired by this divide-and-conquer manner, sev- eral works solve VI. reasoning with this composi- tional pipeline: collecting the sub-questions dataset (Selvaraju et al., 2020), generating sub- sub-questooss for the main question (Uchara et al., 2022; Wang et al., 2022b,d), and utilizing sub-questions to help answer the main question (Selvaraju al., 2020; Uehara et al., 2022; Wang et al., 2022b). Nonethe- less, several drawbacks still exist and hinder their practice: 1) Existing methods rely on lask-specific trained models to generate sub-questions, which are not generalizable. 2) Existing methods im- practically assume that the sub-questions and sub- answers generated in one round can guarantee suffi- cient evidence to address the main question, which is profoundly incorrect in practice. For instance, the generated sub-questions might be not infor- mative enough or deviate from the main question. These predicted sub-answers may be noisy and raise conflicts due to possible misprediction. There- fore, existing methods may cause irrational final predictions be forced to learn spurious bias to guess the final answer.
Compared with previous compositional and end- to-end methods, the proposed IdcalGPT brings sev- eral significant benefits: 1) Transparency and In- terpretability. It is straightforward to pinpoint which sub-answer or reasoning step results in the undesired final answer. Meanwhile, multi-round interactions allow models to showcase their un- derstanding and reasoning process step by step which leads to the final answer. 2) Modularity. With the rapid development of LLMs and VLMs, Quest t /Ress/eese and Answerer can easily be updated to the more powerful LLM and VLM to improve performance. 3) Robustness. Exist- ing models still heavily suffer from problems like superficial biases, inconsistent predictions, or hal- lucination. All of these can lead to conflicted and noisy evidence during reasoning steps. Our multi- rounds can robustly consider models' both noisy and accurate predictions to converge to the most confident answer. 4) Generalizability. IdealGPT can be seamlessly applied to multiple tasks. This is because of the fact that various VL tasks require reasoning skills and can be inherently formatted question-ansen answer tasks. Moreover, IdealGPT illus- trates strong zero-shot ability with no training or finetuning on specific tasks.
To address these above-mentioned issues, we proposed a new framework IdealGPT, which Iteratively decomposes vision and language reason- ing with large language models (LLMs). Specif- ically, IdeaIGPT employs two LLMs (e.g., GPT (Ouyang et al., 2022; OpenAI, 2023) in our experi- ments) as the Questioner and the Reasoner, and a pretrained VLM as the Answerer. In this frame- work, these three agents interact with each other to perform the dividc-and-conquee procedure itera- tively until finding confident answer to the main question. As shown in Figure 1, the Questioner first raises sub-questions decomposed from the main question, and the Answerer subsequently replies with the corresponding sub-an answers. Subse-
We quantitatively evaluate IdealGPT on several challenging VI. reasoning tasks in a zero-shot set- ting, including VCR and Visual Entailment (SNLI- VE) (Xie et al., 2019). Since zero-shot VCR and SN are too challenging for most of the previous VLMs (Li et al., 2023; Yu et al., 2022; Alayrac al., 2022), we found only GPT-4-like models based on instruction-tume LLMs, such MiniGPT4 (Zhu et al., 2023b) and LLAVA (Liu et al., 2023), are capable of tackling the tasks. Com- pared with the above-mentions mentioned GPT-4-like mod-

---

els, IdealGPT outperforms the best by an absolute 10% in VCR and 15% in SNLI-VE.
modality gap. Inspired by InstructGPT (Ouyang et al., 2022), improve the generalization perfor- mance of LLMs to unseen tasks and align users' in- tentions, some VL models finetune the pre- ire-trained models using extra instruction-tuning data. V. (Liu et al., 2023) leverages the trainable projection layer to project the output from the visual encoder to the LLM and utilizes VL conversational data generated by GPT-4 (OpenAI, 2023) finetune the LLM. MiniGPT4 (Zhu et al., 2023b) employs the pre-trained visual encoder and Q-Former from BLIP-2 and uses image captions generated by Chat- GPT to perform training on the LLM and the single linear projection layer.
2 Related Works
2.1
Compositional QA in Vision/Language
Answering multi-hop reasoning questions directly be challenging in NLP. Press et al. (2022) in- vestigate the ability of language models to perform compositional reasoning tasks where the final solu- tion depends on correctly composing the answers. Wang et al. (2022c) exploit a self-consistency method to sample diverse set of reasoning paths and then filter and aggregate by choosing the most consistent answer. Yoran et al. (2023) sample multi- ple reasoning chains and mix information between them to select the most relevant facts in generating an explanation and predicting the answer. In VQA, in order to investigate the reasoning process and promote the reliability of models, SQuINT (Sel- varaju et al., 2020) collect VQA-introspect dataset providing low-level perception sub-questions to answer the complex reasoning questions. Some methods (Uehara et al., 2022; Wang et al., 2022b) decompose the original complicated questions into several informative sub-questions. By answering these questions, they can help to answer the original questions. These existing methods rely on task-specific trained models and generate sub- questions and sub-answers in one round, which prevents their generalizability and reasoning abil- ity in practical problems. However, IdealGPT can be seamlessly utilized in different tasks by slightly adjusting the prompt. Moreover, our iterative ap- proach can efficiently solve challenging tasks such as VCR and SNLI-VE without further training.
2.3 GPT-Aided Visual Reasoning
LLMs are pre-trained colossal corpus they can attain rich prior knowledge and strong reason- ing ability. Recently, a trend has emerged that lever- ages LLMs in combination with a range of vision or multimodal models. These approaches create a system capable of addressing various multimodal tasks without the need for additional training. MM- React (Yang et al., 2023), HuggingGPT (Shen et al.. 2023), Chameleon (Lu et al., 2023), Visual Chat- GPT (Wu al., 2023) regard GPT (i.e., ChatGPT, GPT-4) as a controller to coordinate and collaborate with other models (e.g., visual foundation models) to tackle complicated multimodal tasks. VisProg (Gupta and Kembhavi, 2022) and ViperGPT (Surís et al., 2023) exploit GPT-3 (Brown et al., 2020) and CodeX (Chen et al., 2021) to generate a program to solve VL tasks in one-round query answering. ChatCaptioner (Zhu et al., 2023a) lets ChatGPT and BLIP-2 interact to accomplish image caption- ing in dialogue approach. All of them borrow the strong reasoning ability from LLMs and boost performance in a wide range of VL tasks. Different from ViperGPT, VisProg, and ChatCaptioner, Ide- alGPT solves VL tasks iteratively in multi-round manner and our proposed method can be conve- niently deployed in diverse set of VL tasks such as VCR and SNLI-VE. More details are in Sec. 3.4.
2.2 End-to-End Vision-Laanguage Models
VL pre-training models (Li al., 2019; Chen et al., 2020; Li et al., 2022: You et al., 2022: Yu al., 2022) are pre-trained on large-scale image-text pairs, which enable these models' joint understand- ing between different modalities. Recently, there is a trend to utilize the knowledge from LLMs and align visual features to the text space. Flamingo (Alayrac et al., 2022) inserts cross-attention layers into LLMs to import visual features and employs billions of image-text pairs to pre-train the new layers. BLIP-2 (Li al., 2023) is powered by the pre-trained visual encoder and LLMss. It uses a lightweight Querying Transformer (Q- (Q-Former) following a two-stage pre-training to bridge the
3 Method
In this section, we introduce the proposed Ideal- GPT. Our focus is on the tasks of open-domain VQA, where a question q asked about an im- age T. There are three components IdealGPT framework: a Questioner, an Answerer, and a Reasoner. each iteration, based on q and

---

Answer of 2nd Iteration
Main Question:
Analysis: Because they are sed,
Is the person on left and person on right dating?
Done
Answer. No, they are not dating
Answer of 1st Iteration
Analysie: Watching vague action, so We need more information about Answer: not
determine is romantic nat.
More
Ittrationn
Caption
Genersted Analysis
Questioner
Reasoner
All Existing
Sub-Que Sub-Questions
Sub- Sub-Answers
What is the body language the man?
Sub-Question 1:
2nd Ieeraioo
Who is the person on left?
Sub-Que Sabb-uuetion
1st Iteration
Sub Questions
Generated
Sub- A00-Answrrs
Exieting
Sub-Answer 1:
1st Iteration
Nun
He looking
Sub-Answer
2nd fieration
the nun
Sub- Question 2: person right?
Sub-Question 2: What the facial expression of the man?
Sub-Answer 2 Man
Sub-Answer Sed
Sub- Question
Sub- Sub-Question 3:
What is the physical contact batween person on left and person right
Sub Bub-Answer& 3: Watching
Sub-Answer 3: Sad
What is the facial
Answerer (VLMs)
expresalon of the nun?
Path of 1st Iteration
Path of 2nd Iteration
The pipeline of proposed IdeaIGPT. We use example in VCR validation set for illustration, which is finished in 2 iterations. In the initial iteration, the main question is decomposed into multiple sub-questions. which are answered by Answerer, and then the Reasoner summarizes generated sub-answers to decide whether a confident answer can be deduced. Since Reasoner is not sure about the answer in the 1st iteration, it generates the analysis, and all existing information is input into Questioner again for amother new iteration. The iterative process ends when Reasoner confident about answer or the maximum number of iterations is reached.
Figure 2:
I, Questioner first raises a set of sub-questions SubQ = { .s., s (Sec. 3.1), then Answerer generates the corresponding sub-answers SubA {sa1,..., } (Sec. 3.2), and Reasoner analyzes both SubA and SubQ to decide if confident an- swer to the main question q can be derived (Sec. 3.3). If a confident answer cannot be inferred in the current iteration, Questioner is prompted to ask additional supplementary sub-questions, and another ation is triggered. This loop keeps iterating until the Reasoner finds a confident final answer or the number of iterations reaches predefined maxi- mum. The overall pipeline is shown in Figure 2.
2023) have demonstrated a strong ability to fol- low instructions and reason with human knowl- edge. Additionally, some works (Surís et al., 2023; Wu et al., 2023; Zhu et al., 2023a) have utilized LLMs to aid visual tasks, which demonstrates that LLMs have acquired diverse visual knowledge certain degree as well. Inspired by these find- ings, we prompt GPT as a Questioner to raise sub-questions. By default, ChatGPT (Ouyang et al.. 2022) is used. While GPT-4 (OpenAI, 2023) is a stronger alternative, it is costlier.
iter-
The imput in VQA tasks usually includes a main question and an image I, and sometimes answer candidates A {ai, ..... a, } if the task is format- ted multiple-choice QA problem. The target of the Questioner is first to recognize the evidence needed to address q and then decompose into sub-evidences. To acquire those sub-evidences, Ques i oner would then generate a set of sub- questions SubQ = {sq1,..., sq:}. For achieving quality results, we also design prompt Py as an instruction for GPT understand the objective and the desired output. For each task, the prompt is slightly different to accommodate the task de- scriptions With solely the main question q and prompt Pg input into the Questiones we empiri-
3.1 Questioner
In previous works (Uehara et al., 2022; Wang et al., 2022b), sub-questions are generated by models trained on specific sub-questions data (Selvaraju et al., 2020). However, since the training data are specifically annotated for the samples in Antol et al. (2015), these sub-question generators cannot scale and generalize to different domains and complex reasoning tasks. Annotating sub-question dataset covering all types of visual reasoning tasks and image domains is also infeasible. Recently, LLMs (Ouyang et al., 2022; OpenAI, 2023; Anil et al.,
'Sec the detailed prompts in Appendix D

---

cally found that the generated sub-questions from initial itcrations tend be too generic and uninfor- mative. This could be because LLMs don't It see the images and as such are devoid of the visual input. To facilitate the Questiones to understand the im- age and generate more informative questions, we provide short caption C₁ generated by a VLM as an additional input the Questioner. Therefore, in the first iteration, the sub-question generation can be formulated as follows:
end-to-end VLMs but also VL systems such Surís et al. (2023); Gupta and Kembhavi (2022); Shen et al. (2023).
3.3 Reasoner
GPT-like LLMs (Ouyang et al., 2022; OpenAI, 2023; Anil et al., 2023) have shown impressive summarization and reasoning ability with common- sense knowledge. Therefore, like Questioner, we also choose ChatGPT as the Reasoner but prompt differently. Specifically, the input to Reasoner contains main question g, caption CN. all exist- ing sub-questions SubQ1: and corresponding sub- answers SubA1.t. And the Reasoner is prompted to generate both the analysis and the final answer with its prompt PA
SubQ1 = (2,
As mentioned before, there may not be sufficient evidence for the Reasoner to address the main question after only one iteration. This can be due to common issues like the sub-questions not in- formative enough conffict/noise existing among sub-ansoe answers. In this case, IdealGPT would prompt the Reasoner to generate an explanation/anaalsii regarding why it may not have sufficient evidence to address the main question. Subsequently, we would loop back to the Questioner to generate additional supplementary sub-questions. In the t- th iteration (t > 1), Questioner accepts all pre- vious sub-questions SubQ1:-1 and sub-answers SubAra 1, and the previous analysis E, 1 from Reasoner (c.f.,, Sec. 3.3) as additional input:
Es, a = ChatGPT(SubQ SubA I. CI, PR).
If the Reasoner is not confident about the final answer, it is instructed to faithfully indicate that by generating a specific response such as "We are not sure". If this particular response is detected, we start another iteration by asking the Questioner to add supplementary sub-questions. The above procedure forms a loop among the three agents, which will stop if the Reasoner can find confident answer or the number of iterations reaches a pre- defined bound (a hyperparameter).
SubQu =ChaiGPT(g, Pq.
SubQu:t 1, SubAA 1, Ee. 1), where SubQ1:t11 = {SubQ1 U... SubQu-1} and SubA == (SubA, U...U SubA, 1). Pre- vious sub-questions and sub-answers can inform Questioner what has been asked and solved, and the analysis can guide Questioner to generate specific sub-questions, such sub-questions collect more informative evidence about spe- cific visual object and so on.
3,4 Comparison with Other Methods
E.s. ViperGPT/VisProg. VisProg (Gupta and Kem- bhavi, 2022) and ViperGPT (Surís al., 2023) uti- lize LLMs to decompose VL tasks into steps of ex- plicit conditional logic in coding based on low-level visual-spatial detection. IdealGPT shares a simi- lar idea of decomposition or divide-and-conquer However, IdealGPT can iteratively go through the divide diiide-anddcoonuerr process until collecting suffi- cient evidence to generate a confident answer. This multi-pass process involves the continuous refine- ment of the set of sub-questions and even re- correctifying mechanism to the final answer pre- diction. Conversely, ViperGPT/VisProg performs programs in one pass regardless of whether the predicted answer is confident or not. This differ- ence also applies Yang et al. (2023); Shen et al. (2023); Lu al. (2023). Moreover, ViperGPT is limited by the set of available APls, VisProg is lim- ited by the set of available commands, and Neuro- symbolic VQA limited by the set of primitive operations the Domain Specific Language.
3.2 Answerer
Given the generated sub- questiusssnnns SubQ, the goal of Answerer to answer them correspondingly to provide evidence for answering the main ques- tion. In IdealGPT, the Answerer is a pre-trained VLM without finctuning on any dataset to keep intact its generalization ability. Each sub-question is answered separately:
sa, VLM(sqq, I),
where sqi E SubQ and sa, E SubAA is noted that theoretically, the Answerer can not only be

---

4.1 Experimental Setup
You sesidiant who has rich vieui commonsense knowolddee and strong manoning abiinte...
Visual Entailment (VE)
Datasets. In this paper, employ two VL datasets, VCR (Zellers et al., 2019) and SNLI-VE (Xie et al., 2019) as they represent the two typi- cal VL reasoning tasks, visual question answering and visual entailment. Different from the tradi- tional VQA (Antol et al., 2015) task, VCR needs commonsense knowledge to understand the visual scenes and requires multiple-steps reasoning to an- swer the question. Also, in terms of the task for- mat, it is multiple-choice QA problem. SNLI-VE originates from Stanford Natural Language Infer- ence (SNLI) (Bowman et al., 2015), a text entail- ment (TE) task based on Flicker30k (Young et al., 2014). It extends TE into the visual domain and asks the model whether the image is semantically entaileddneutral/conttaiice to the text hypothesis, thus it can be treated three-category classifi- cation task. In VCR and SNLI-VE, we randomly select 5000 samples from the val/dev split of the dataset and evaluate our method in the zero-shot scenario with the accuracy for evaluation.
goel to effectively predict whether the somantically entails the hypothesis....
Inperfact Caption A crane on the side of buildng.
Image Captioning
Hypothesss: construction working
Answerer (VLMs)
Questioner
Aab-Annwwrrrr 1:
Sub- Question T:
ladder.
What is the crans doing?
Sub- On a laddlrr
2
What the postion the construction workers?
Sub-Answer 3: It morning.
Sub- Question 3: weather and day image?
Reasoner
Analysis:
Prodicted Answer:
lifting ladder
Entaliment
Addiltionall, the time of day is morning and the weather is sunny, which are both typical conattions for construction work So, the workers are (ibely working on the bualding
Figure 3: The illustration of how proposed IdealGPT works in SNLI-VE. IdealGPT exhibits its perceptional ability in sub-question 1 and 2. Plus, it shows strong commonsense reasoning ability in sub-question 3.
Models. We choose ChatGPT to act as the Reasoner and Questioner and access it via "gpt- 3.5-turbo' API. It should be noted that set the temperature as 0 reduce the randomness so that we can have a stable result to see how our pro- posed method performs in different tasks. As for the Answerer, three pre-trained VLMs (BLIP-2, MiniGPT4, and VA) are selected to serve for comprehensive comparison. It's noted that all VLMs we use are pre-trained-only without being finetuned on any dataset guarantee zero-shot generalizability. We design three simple prompts for these models respectively (more details can be found in the appendix D) to help them better answer sub-questions. Further, in practice, these VLMs are also used to produce image captions so that the ChatGPT can reserve a general understand- ing of the unseen image initially.
F.s. ChatCaptioner ChatCaptioner (Zhu et al., 2023a) lets ChatGPT and BLIP-2 interact with each other to caption images dialogue system. Ide- alGPT shares the idea of utilizing ChatGPT to gen- erate questions and VLMs to answer them. But IdealGPT focuses on the VL reasoning tasks and incorporates the iterative design for generalized zero-shot vision language reasoning.
4 Experiments
In this section, we evaluate our method quantita- tively two main tasks Visual Commonsense Reasoning (VCR) and Visual Entailment (SNLI- VE). We introduce datasets and models first. Then, we show superior zero-shot performance on VCR and SNLI-VE compared to other existing mod- els. Afterward, ablation studies are conducted to confirm the effectiveness of our proposed method, with error analysis for deep dive into our method. In addition to VCR and SNLI-VE, case the experimental results of Augmented Out- side Knowledge Visual Question Answering (A- OKVQA) (Schwenk et al., 2022). Since it is more a knowledge-basee task requiring less complex reasoning ability, we put it in Appendix C.
4.2 Visual Commonsense Reasoning
VCR covers various commonsense in diverse vi- sual scenes, including temporal, spatial, causal, and mental commonsense, etc. It's formatted as multiple-choice answering problem, and to find the correct answer, the model is expected reason among four answer candidates to find the correct answer. In VCR, often happens that there are multiple persons in one image, and if the question mentions one of them, the bounding box will be
also show-

---

Acc.(%)
Acc.(%)
Random Guess
25
Random Guess
33.3
R2C (Zellers al., 2019)
63.8
EVE-Image (Xie al., 2019)
71.6
Sup.
Sup.
VisualBERT (Li et al., 2019)
70.8
UNITER (Chen et al., 2020)
79.4
MerlotReserve (Zellers et al., 2022)
84.0
OFA (Wang et al., 2022a)
91.0
BLIP-2 (Li et al., 2023)
MiniGPT44 (Zhu et al., 2023b)
35.1
MiniGPT4 (Zhu et al., 2023b)
40.6
LLaVA (Liu et al., 2023)
40.3
LLaVA (Liu al., 2023)
28.3
IdealGPT(ours)
55.3
IdealGPT(ours)
50.7
Table Accuracy of SNLI-VE (ZS: Zero-Shot, Sup: Supervised)
Table I: Accuracy of VCR Q->A task (7S: Zero-Shot, Sup: Supervised)
answers of sad expressions and distant body lan- guage allow the Reasoner ensure confident final answer.
used to distinguish it from others. However, most existing VLMs cannot understand bounding boxes in the text input, making them hard to perform VCR directly. To alleviate that, we conduct a pre- processing describe the mentioned person's 's spa- tial coordinate in words easy-to- to-ders-nddd Please see details in Appendix E.
4.3 Visual Entailment
Visual Entailment requires the model to predict whether the image semantically entails the text. each sample, there is a pair of an image and text hypothesis along with three answer candidates (i.e., entailment, neutral, and contradiction). The model needs select one of them represent the rela- tionship between the image and the text hypothesis. is challenging to solve the three-category clas- sification task in a zero-shot manner. in Fig.3, to begin with, we make some rules and introduce the goal of this task to ChatGPT' to en- sure it can understand our instructions and respond reasonably. We utilize VLMSs generate the image caption and inject it into the prompt. Thereafter, Quest Questioner decomposes the original hypothesis into several sub-questions. After answering all sub- questions by VLMs, Reasoner will summarize the image caption, hypothesis, all sub-questions, and corresponding sub-answers together to provide a comprehensive analysis. In Fig.3, we should notice that not only does IdealGPT exhibit the percep- tional ability in sub-question 1 and 2, but also it shows the strong commonsense reasoning ability in sub-question 3.
Although a lot of models can be finetuned on this task (Zellers et al., 2019; Li et al., 2019), there is hardly any model that tackles it in the zero-shot fashion. We empirically tried BLIP-2, MiniGPT4, and LLaVA and found that only the GPT4-like models can functionally perform zero-shot VCR, while other models such BLIP-2 fail to under- stand the long context and the instruction of finding the correct answer. We present the experimental result in Tab. 1, where zero-shot results are obtained from the randomly sampled 5000 data in the validation set. As we can see, IdealGPT can outperform the best GiT4-like model, MiniGPT4 by over 10%. It's noted that in IdealGPT reported here, BLIP2-FlanTS, 55 5-XL is used the Answerer. Please refer to Sec. 4.4 for ablations the choice of Answerer.
shown
In F2g.2, showcase an example of how Ideal- GPT solves an example in VCR successfully. As we can see, in the first pass, the generated sub- questions and predicted sub-answers are not in- formative enough to support a solid conclusion be- cause their identities and interaction of "wwtcciing"" cannot quite indicate whether they are dating or not. Further, in the second pass, after inputting the anal- ysis from the Reasoner and existing sub-questions and sub-answers, the Questioner is prompted to ask additional supplementary sub-questions to col- leet more evidence about their expressions and body language. As a result, the updated sub-
Supervised methods for SNLI-VE have been well-studied (Xie et al., 2019; Chen et al., 2020; Wang et al., 2022a), while zero-shot approaches are less explored. We tried LLaVA and MiniGPT4 to do zero-shot SNLI-VE. In Tab. 2, can observe that compared to MiniGPT4 and LLa VA, IdealGPT consistently surpasses them by large margin, 20%, and 15% respectively. This result shows that not only can our method understand long instructions

---

Model
Max. #lterations
Acc.(%)
Model
Answerer
Acc.(%)
1
49.2
BLIP-2
55.8
2
53.2
IdealGPT MiniGPT4
52.8
IdealGPT
4
55.8
LLaVA
53.2
6
57.3
7
57.4
Table 4: Ablation of choice of Answerer.
Table 3: Ablation of iterative decomposing. Max. #lter- ations=1 means deterministic answering in one round without iterative decomposing.
Model
Caption from
Acc.(%
BLIP-2
55.8
IdcalGPT
MiniGPT4
48.4
L.LVA
51.2
but also it is able to handle different task formats (SNLI-VE and VCR have distinctively different task formats. The former is image-hypothen but the latter is the guestion-ansseriin format). From Tab. 2, we can also notice that the perfor- mance of MiniGPT4 is the near random- random-guessing level. More details and discussion about SNLI-VE can be found in Appendix B.
Table 5: Ablation of generated captions.
pair
visual questions. We mainly ablate three VLMs: BLIP-2, MiniGPT4, and LLaVA. The result is shown in Tab. 4. Although MiniGPT4 and LLaVA can follow instructions and understand the longer contexts, they are worse than BLIP-2 when an- swering questions related to detailed visual facts, This also echoes the limitation mentioned their papers about hallucinations and the lack of spa- tial/geometric understanding. Note that the BLIP- 2 use is BLIP2-FIanT5-XL which in our ex- periments, gives similar performance as BLIP2- FlanT5-XXL.
4.4 Ablation Studies
In this section, we ablate both the design choice and component choice. We first demonstrate the necessity of iterative design in IdcaIGPT. Then ablate different VLMs for generating captions and performing as Answerer. In all ablations, we use randomly sampled 500 data set from VCR, which in our findings is cnough to distinguish different model choices.
Image Captions. To efficiently search for the best caption in our method, we fix the VLM as BLIP2- F5anT5-XL and go through pre-trained VLMs with- out fine-tuning on any caption dataset. From the ex- perimental results shown in Tab. 5, exhibits the best performance. is interesting ob- serve that both MiniGPT4 and LLaVA have shown impressive captioning ability in their papers/demos, but they fail in our framework when compared with BLIP-2. We further go through many cap- tions generated from MiniGPT4/LLaVA and BLIP- 2 and find that MiniGPT4/LLaVA tends to generate longer, more informative but less accurate (or more hallucination) captions. In contrast, BLIP-2 tends to generate short, less informative, but also less mistaken captions. So the hallucinations/mistaess in TimiGPT4/LLaAAggnneeee mislead the Questioner and Reasoner to wrong answers.
Iterative Decomposing, A key design in Ideal- GPT is that if the model (Reasoner) is not about the final answer, will trigger new pass of QA to ask additional aub-questions and therefore provide more visual facts to the Reasoner. The iter- ative decomposing will continue until the Reasoner is confident to make decision or the number of passes reaches a pre-defined bound, which is a hy- perparameter. We evaluate IdealGPT with iterative decomposing and without iterative decomposing (i.e., Max. #Iterations=)) in Tab. 3. We see that iterative decomposing design can boost the ac- curacy by as high as around 8%. It's noted that more passes also mean more inference time, and we find setting the maximum number of iterations to 4 achieves good trade-off between efficiency and effectiveness, which is used as default in all other experiments. Under the above setting, the average number of passes across sampled data is 1.8.
BLIP-2
captions tend
4.5 Error Analysis
Since our method consists of multiple components, would interesting to see where the error comes from. We went through 50 failure samples of Ideal-
Answerer Choice. As we mentioned before, Answerer can be any VLM capable of answering

---

Source
Type
Ratio.(%
framework ensures better interpretability of VLMs' reasoning and robustness re-correctifying pre- dictions such as hallucination. Additionally, the generalizability of our framework can be illustrated by the modularity and our superior zero-shot per- formance. Our extensive experiments prove the effectiveness of each component in IdealGPT and verify that it can outperform the best existing GPT- 4-like models by absolute 10% in VCR and 15% in SNLI-VE.
Questioner
Not Relevant Sub-questions
16
Answerer
Wrong Sub-answers
52
Hallucination
24
Reasoner
Misunderstandingg
8
Table Error Analysis of different components
GPT on VCR validation set and conducted an error analysis about different types of eπrоrs from differ- ent components, thanks to the good transparency and interpretability of IdealGPT. The quantitative result is shown in Tab. 6. Below more detailed analyses.
Acknowledgements
This work is supported by DARPA MCS program under Cooperative Agreement
Limitations
Questioner: We found that the Questioner sometimes generates sub-questions that are not quite relevant to the four answer choices, thus not helpful in distinguishing the correct answer. Be- sides the capability of LLM itself, we found it's also heavily influenced by the caption we provided. If the caption is more precise with less hallucina- tion, then the Questioner would be more precise too.
Although we have witnessed the strong perfor- mance of the proposed method, we still have some limitations. Firstly, answering key sub-questions correctly plays significant role in the success of our system. Therefore, our final results are largely bottlenecked by the performance of pre-trained VLMs. Additionally, we just employ the general image caption to give ChatGPT an idea of what the image looks like. However, dense captioning might be more informative and better help ChatGPT deal with the unseen image. Besides, the prompts are manually designed by us and it is difficult to find the optimal prompt in a specific situation. It will be better if the prompt can be generated automatically. Last but not least, compared with end-to-end solu- tions, ours naturally takes more time and has higher latency because of multiple feedforward passes of Ques tioner//nswerer//eessoo
Answerer: Half of the errors come from the Answerer providing wrong aub-answers. As illus- trated in Tab. 4, we can find that the choice of Answerer is quite important. A good Answerer should understand various types of questions and generate answers with less hallucination.
Reasoner: There are types of errors from Reasoner: a). Hallucination: Sometimes Reasoner predicts the answer only with unclear visual clues, thus leading to wrong prediction. In those cases, one more round of the decomposition loop might help to find more clear visual evidence. This also accords with our observation that more decomposi- tion iterations help. b). Misunderstanding: Some- times, the LLM misunderstands the answer candi- dates and gives wrong predictions, which mainly due to the LLM's capacity.
Ethics Statement
ChatGPT is pre-trained the colossal corpus which likely contain potential racial and gen- der bias. Therefore, if someone finds our work interesting and would like to use it in a specific en- vironment, we strongly suggest the user check the potential bias before usage. In addition, is hard to control the generation of LLMs like ChatGPT. We should be aware of the potential problems caused by hallucinations.
5 Conclusion
In this work, we identify critical problems with ex- isting VQA methods, especially the lack of address- ing zero-shot reasoning tasks and the false assump- tion potentially forcing models to answer questions without sufficient information. To address these issues, we propose IdealGPT utilize LLMSs to construct multiple-passes framework among a Questioner, a Answerer, and a Reasoner. This

---

References
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language- -imagg pre- training with frozen image encoders and large lan- guage models, arXiv preprint xrXiv:2301.22500
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven
Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Ma- chine Learning, pages 22888-12900 PMLR.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305. 10400
I Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning, arXiv preprint arXiv:2304.08488
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023, Chameleon: Plug-and-plaa compo- sitional reasoning with large language models. arXiv preprint arXiv:2304.09842
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1508.05326
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
OpenAL. 2023. Gpt-4 technical report. arXiv.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint N.Xiv:2007.0337..
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.033:
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCY 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162. Springer.
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 28, 2020, Proceedings, Part XXX, pages 104- 120. Springer.
Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh. Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Kamar. 2020. Squinting at vqa models: In- trospecting vqa models with sub-questions. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10003-10011 I.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends huggingface. arXiv preprint arXiv:2303.1 1758.
Tanmay Gupta and Aniruddha Kembhavi. 2022. Vi- sual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211. 115599
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. 2022. Webly supervised concept expansion for gen- eral purpose vision models. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Is- rael, October 23-27, 2022, Proceedings, Part XXXVI, pages 662-681. Springer.
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.088 001288.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint arXiv:

---

Kohei Uchara, Nan Duan, and Tatsuya Harada. 2022. Learning to ask informative sub-quee questions for visual question answering. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recog- nition, pages 4681 1681-4690.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. Ofa: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sqquene learning framework. In Inter- national Conference Machine Learning, pages 23318 213440. PMLR.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung. Mojiaha Seyedhosseini, and Yonghui Wu, 2022. Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:2205.019)
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Vi- sual commonsense reasoning, In Proceedings of the IEEF/CVE conference on computer vision and pat- tern recognition, pages 6720-6731.
Ruonan Wang, Yuxi Qian, Fangxiang Feng, Xiaojie Wang, and Huixing Jiang. 2022b. Co-vqa: Answer- ing by interactive sub question sequence. In Find- ings of the Association for Computational Linguisties: ACL 2022, pages 2396 2408.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quac Lc, Ed Chi, and Denny Zhou. 2022c. Self-consistency improves chain of thought reasoning language models. arXiv preprint arXiv:2203.11177
Zhecan Wang, Haoxuan You, Yicheng He, Wenhao Li, Kai-Wei Chang. and Shih-Fu Chang. 2022d. Understanding me? multimodal evaluation for fine-grained visual commonsense. arXiv preprint arXiv:2211.05895
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023a. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.0653
Chenfei Wu, Shengming Yin, Weizhen Qi, Xi- aodong Wang. Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and edit- ing with visual foundation models. arXiv preprint arXiv:2303.04671.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023b. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint ar34.:2304. 10592.
Ning Xie. Farley Lai, Derek Doran, and Asim Ka- dav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706
Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. Mm- react: Prompting chatgpt for multimodal reasoning and action. Xiviv preprint Xrv:::2303. 2203.133388 1138I.
Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023. Answering multiple chains thought. arXiv preprint arXiv:2304. 13007.
questions by meta-reasoning
Haoxuan You, Mandy Guo, Zhecan Wang. Kai-Wei Chang, Jason Baldridge, and Jiahui Yu. 2023. Cobit: A contrastive bi-directional image-text generation model. arXiv preprint ar Xiv::2300.. 13455.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. 2022. Learning visual representation from modality- sharred contrastive language-ing image pre-training. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviy, Israel, October 23- 27, 2022, Proceedings, Part XXVII. pages 87. Springer.

---

A
Data Statistics
Acc.
Random Guess
25.0
VL Tasks
Number of Samples
LXMERT (Tan and Bansal, 2019)
51.4
Sup.
GPV-2 (Kamath et al., 2022)
60.3
C
1664
InstructBLIP (Dai et al., 2023)
81.0
N
1672
SNLI-VE
E 1664
MiniGPT4 (Zhu et al., 2023b)
49.4
N
LLaVA (Liu et al., 2023)
30.0
All
5000
IdealGPT(ours)
62.6
VCR
5000
Table Accuracy of A-OKVQA in Multiple-Choiee setting(ZS: Zero-Shot, Sup: Supervised)
Table 7: Statistics of SNLI-VE and VCR (C: Contradic- tion, N: Neural, E: Entailment, VL: Vision-Language)
We randomly select 5000 VCR and SNLI-VE samples from respective val and dev split. Zero- shot learning is conducted on these samples.
Since it difficult to evaluate the generated an- swer in the open-vocabulary setting, we choose the MC evaluation setting. We design the prompt for A-OKVQA² and directly input the question, four answer choices, and the general image cap- tion into the Questioner to put forward several sub-questions. Afterward, the VLMs can reply the Questioner and Reasoner is able to collect and analyze the sub-questions and sub-answers output prediction when sure about its answer. In Tab. 9, we investigate zero-shot A-OK VQA by using MiniGPT4 and LLaVA. IdealGPT can sur- pass MiniGPT4 and LLaVA by 13.2 % and 32.6 % respectively, which confirms the stronger reasoning ability of our proposed method. Moreover, we out- perform some early proposed supervised methods like LXMMETT (Tan and Bansal, 2019) and GPV-2 (Kamath al., 2022). Since we implement our method without any further training, we still can't 't exceed InstructBLIP (Dai et al., 2023), which is one the current best-performing supervised mod- els on A-OKVQA.
B Details of Visual Entailment Results
C
E
All
Random Guess
33.3
EVE-Image (Xie al., 2019)
71.0
70.6
73.1
71.6
Sup.
UNITER (Chen et al., 2020)
79.4
(OFA (Wang al., 2022a)
91.0
MiniGPT4 (Zhu al., 2023b)
3.2
3.2
99.0
35.1
ILLVVA (Liu et al., 2023)
12.0
31.3
77.6
40.3
(doulG)T(ours)
83.4
25.9
56.7
55.3
Table 8: Accuracy of SNLI-VE (ZS: Zero-Shir Shot, Sup: Supervised, C: Contradiction, Neutral, E: Entail- ment)
From the results shown in Tab.8, can observe that our proposed method outperforms MiniGPT4 and LLaVA by large margin. More- over, MiniGPT4 exhibits near-chance level perfor- mance. When is faced with different samples, always replies entailment, which indicates that it doesn't 't obtain a strong reasoning ability to process and understand the visual entailment task. How- ever, IdealGPT can achieve strong zero-shot results and even surpass the supervised EVE--mmaee model in contradiction category.
D Details of Prompts Used
The prompts of IdealGPT used in the VCR task are shown in Fig. As for SNLI-VE, the prompts are shown Fig. It's noted that the [placeholder] means we will replace with the corresponding text of the instance, such as main question, cap- tion, and four choices.
C Augmented Outside Knowledge Visual Question Answering
E VCR Pre-Processing
We exploit two different approaches to pre-process VCR and select the better one for our experiments in the main text. The first pre-processing is to di- vide the image region from left to right into three
A-OKVQA is challenging benchmark for knowledge-required visual question answering, which demands world knowledge that goes beyond the image. It provides Multiple-Choice Choice (MC) as well as Direct Answer (DA) evaluation settings.
Prompts for A-OKVQA are very similar to VCR in Ap- pendix D

---

Acc.(%)
MiniGPT4 (Zhu al., 2023b)
42,4
LLaVA (Liu et al., 2023)
31.0
DB. LLaVA (Liu et al., 2023)
MiniGPT4 (Zhu et al., 2023b)
30.8
28.6
Table 10: Comparison of two different pre pre-processing
ways for VCR (TB: Three Bins, DB: Drawn-on Boxes)
bins and check if the mentioned object's center point belongs which bin. If it's in the most left bin, it's renamed as "person on the left". Similarly, "person in the middle" is in the middle bin, and "person on the right" is in the right bin. Since most QAs in VCR mentioned less than three persons, it can cover most cases. The second approach is to follow past work (Zellers et al., 2022) in 'draw- ing on' the annotated detection tags to the image. We select seven different colors (i.e., red, green, blue, orange, purple, eyan, and yellow) represent the different persons mentioned the question on answer. Assume there are two people one sam- ple, when doing the inference, 'personl' will be replaced by person in the red bounding box', and "person2' will be replaced by 'person in the green bounding box'. Moreover, the model can see the red and green bounding boxes drawn in the im- age. As mentioned above, most cases in VCR mentioned less than three people. Therefore, our implementation can cover the majority of the cases. The comparison of these two methods can found in Tab. 10. We randomly select 500 samples from VCR val split to conduct zero-shot learning by us- ing LLaVA and MiniGPT4 to see which method is better. We can observe that the former approach is better, utilize this setting in the main text.

---

Questioner Prompt:
System Prompt
You axsiders who has rish visual commorsenie knowledge neasonin widl
You provied with
Aimougg you won't be able direcity Niw the image. will rood peneral copton that might not ent preciee out provide overa desorition.
grat
To offectivily analace fou iragee and select the correct arcowst the quastden, you anourd break cown Shu quesidon no several aub-caesiiion that aderuss Inu key
image
Here ore the nites you chould helow when lisinn the sub quesiions,
shouldm ddn't mention
lee subwn wettone in the inlowwnng tormee "Subwyjunet restination to ?t Subbqqeetton 2:
with The subiqussiion are recossary vary to dasanguich cornec, arnowar.
quertion olddd
understand
Exampie
cuention What hepparing in imsgee?
image?
Srab-tue ape-suettionn idhal actioos eoente in the perton deang?
umottorre de ungrasssara woman??
Bub--swation d. Whha brannd car?
System Prompt of Following Iterations:
You are aspiatont who has rich visual comoossene knowledge and strong reasoning abl
unetion sbouu image
arswen cand camee
Allooug you wonth bo ablo to dinecdy viswe Bhe trace you will rood vo per capton thdt might roc endivlly prouse but will provide on overa doaac pldan, by
comesponcing
andiaily preciss
aradya whether vive aub-queedaas sud-ernowess help sctve the main quasttor.
The nument aubb-questons and sub-enswerr ame not surt caent selue main queeteon your gost
Bases cutsiigg ingiggg quosiion and analysia you should
questorn, gather information and necessary solve qusstion.
Here are the roloe you afould low wi listng uddiliona fonal sub sub-queddors.
List the sub buesstione the followinn formet "Aastioonal Sub eustinnn P. Additooaal Sub b.bbqquede question 2:..."
with The sichq metion recensarry in dedingstien cormact anowwer
understand.
Format Enample::
Adddltion Sub suossion ooox
Sup-cteettion on ueus
Additionaa
Input Prompt 1st Iteration:
Main Quieeion: [Sheshoide)
Chokoss: ploolddaa
Pleate the
requirement mentoned
Input Prompt of Following Iterations:
Main Queetoon: [Hbrcholde]]
glowndderr
questons
(taseholded
Aracyade jpacet slow]
VCR Answerer Prompt:
Queedtor: [[tccshcket]] Arewne
VCR Reasoner Prompt:
System Prompt of
Last Ittration:
are
who rich commonsenna knoolledee altong ressoring abdl provided with
main queetion about an traqee and four arawer card carees
Aithogh you decompoaed from
direcly image. you peneral coption
preciee
desoriptee
comespancing
provided voust mobeet thai the not
aritrely precien
your gosl
sub-cuasdiote commsponding answerr, you ahould
liwer aroweer form answer cardkdatass
Here ane the riles should hellow your ressorse:
paragraph If you have found more anower, conciudo correct answer in fomnat of "oore Likely Answeon 12/091 Othoreise, corelude widl "Moru Lively Anosin We not mure option commot
Response Format:
Analydic
Arooser: 10U1N
System Prompt of Last Iteration:
You are assistann who has rish visul commonsenie knowledge and strong reasoning ubil
questoon about mmae asuranewer
Allhoag you worth be able to disecly vieid the inso, you will noce ve penenal captian lisi might rot entiwily preciss but will provide an overa cleac polon. questoon comespencing
cuestions decomposed
provided
Rases on rub ree-quetions cnmpepeeding anowem you must find the more Bonyy answer from the anownn canditeees
follow
All lini. cemorallate your meeecring and inteerro oe procese vitton one penegraph Steet with the formet "Anaipala."" glow gueeengg
Resporse Format:
Mone kely Anower 103M
Input Prompt:
Mann Quustioo: lolochholdel
Cholpe:: praoxmaltees
[pacceholder]
Pleese folloe the above-tertinn Lande inxtuction be Anahysa and More L-bdly Arcever.
Figure 4: The prompts of IdealGPT in VCR task.

---

SNLI-VE Questioner Prompt:
System Prompí of 1st lteration:
You will be prov rid with Mtoough won': to ablo dirodty vew the rood gonera caption that right not entretyy predise but provide on overa cesorittor.
Your good
etbethely prodict whether the imag semonticaly entaie trstut hypotheds and serect the roswer from enta lnnen neuito, controotiooonoooono controticion you should
you
suboquattonn.
Eraure ood solion infepurstent, mere dottosskrre anoukle maniion udd-uussiors.
scb-qaanionn fore beeing sub--qusdion fon Subegassion
Each sdocal Whet"
Each oue quesion should short casy to undorsland
Hypothosac
along Sub What otjocts sutueda prosectia in iesarreee ai wate antione
mooks.
people maing?
Sub
vibldnn?
Sub quesi What's the gender of group poople?
Prompt Following Iterations:
andtrontt whoo haee rish voout commonseraa troweedig and strong meseoring bng Briw.
A sotual freechheae aloos, imagee and these uroner cun cadakea
3 Bome sub questions procoded prodictng whather the imago somantially entals tostal potthess. the conosponding riginggg anowers are provited by An ansissia whallier enswers oroicy procss.
quastom
prodict whathee semaniko untall the komaal hyyotfros
todu hypothos, Based on cxising sup suestonn and andlylo, you should poso ooddorna mosstors, that can gather more informaton mutson and are recessory prockct whother
The qurren questions
procks
somanticall
typerheis
you
quertions
Brsure sedi sub-qu indeparstert. mesri aubequestora ahoulde mrsenioon uddqueeliiors
the subequeetinn
rowingg Vuddiionne tutt-question Additiona Suboquaction
Bach should 'hher"
Each wub cucalaan stnclld be alant saay to undersland.
Format Example:
un Aukiiiordd Subiquaed
Addilonaal Sub question 3 box
Input Prompt of 1st Iteration:
Insorfect Oupioo (pkasnoklad)
Trree Chrises: t accho der
Please qustooro 'olowinn requirement
Input Prompt of Following Iterations:
[plesenakted]
Hyprhacce: [plosttldor
Trem Clazess: bracahslon
sesooee: [plesentter)
Pleste rub-quettone
requbwment meniioned hehwn
SNLI-VE Answerer Prompt:
Quest on placeholder Answer:
Prompt Prompt:
Iteration:
You will be prow aod with
hopethess Atbough wont: able ciradty niw the
answer will general oastion that right bu ontredy roly presive but will provide do an overa cesoritior. propossd and predicing ka whalhe uae nemiicallyy untata tunkau typoti sone comeacoriing arswers are provided noted
Rlvodd on aub-quettions and comwapoeding enswers you ahould rivd the more lhvy aessue from the thee aonoerr candidetes.
Hero are the náoos you chou to low your nournesponne:
demmmendee your reseoeingg irfenence procees trin cow panagroth the "Amabyde""
2. If hase
ansowes
corned in formal which opeion commot..
Anewer erbellmeeton wwwpre/wwwetettteddeeeee. Otteriie annclude
with Lony Ansoest
Reeacrom Formet
More Livel Answer ontzaling www.hhhuoooooooiiieee.....
System Prompt
You assistant
krowindig string reasoning
with
frppcthees atout respe and three semove can cideeee
Althw
cention that might entirel prrctio provide coouttation.
3. Bome sad- queuttrr proposed for predecing whecher the image semunbicllly by sntaii the kunkual hopot sota. and the ooneaporeing unoveers are provided by
Busodl on cucstiona and comnaponding answers, must andd more fieeyy answer from three answer cundicutes.
should yournesponee:
At demonszaate your roosonning nn and inference procoss with thin paragroph Stai with the formtt Woodysis"
Woroe
Sven
wdu matest gleaking
Ressonse Format
Aradivata pia aessnc
Anawwer andedimeet totdiio
Input Prompt:
Insortent Caproon: [plssemakded]
(
Tiee Choilos: 3i oocho der
Pleaee belno www.mmenntrred Inmmrtton Annlysto Morn Boty Answee
Figure 5: The prompts of IdealGPT ANLI-VE task.


Reviews:
[{"review_id": "PJivttcsaG", "reviewer": "Reviewer_MWQz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method surpasses other methods on VCR and SNLI-VE tasks.\n2. This paper is well written and easy to follow.", "reasons_to_reject": "1. Comparative methods are limited. The authors should include other baselines (such as VQG [Uehara et al., 2022], Co-VQA [Wang et al., 2022] that use sub-questions to predict final answers to evaluate the performance of the proposed method even though they are not zero-shot methods. \n2. More evaluation datasets are required. The proposed method should be also tested on other tasks such as visual question answering or OK-VQA to show generalizability of the method. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposed IdealGPT to utilize LLMs to construct a multiple-passes framework for zero-shot VQA tasks.\nThe proposed method outperforms other zero-shot methods.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oZQNCm4PqN", "reviewer": "Reviewer_NAya", "summary": "", "questions": "Question A: How does the performance of IdealGPT change when using different Answerers, especially when using VLMs with different levels of performance or specific strengths and weaknesses?\n\nQuestion B: Can you provide more insights into the trade-offs between the number of iterations and the accuracy of IdealGPT? For example, how does the performance change with more than 4 iterations, and is there a point of diminishing returns?\n\nQuestion C: Have you considered exploring automatic prompt generation or optimization techniques to improve the performance of IdealGPT without relying on manually designed prompts?\n\nQuestion D: Are there any strategies or methods to mitigate the potential biases and hallucination issues introduced by ChatGPT when used as the Questioner and Reasoner in IdealGPT?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. IdealGPT enables a clear understanding of the reasoning process by decomposing the main question into sub-questions and sub-answers, making it easier to pinpoint any issues in the model's predictions.\n2. The framework can be easily updated with more powerful LLMs and VLMs to improve its performance, as it separates the components of Questioner, Answerer, and Reasoner.\n3. Generalizability: IdealGPT can be seamlessly applied to various tasks by slightly adjusting the prompt and demonstrates strong zero-shot ability without training or fine-tuning on specific tasks.\n4. Empirical results showing that IdealGPT outperforms state-of-the-art GPT-4-like models in zero-shot settings on challenging VL reasoning tasks, providing a new benchmark for future research.", "reasons_to_reject": "1. The success of IdealGPT relies heavily on the Answerer's ability to correctly answer key sub-questions. The final results are limited by the performance of the pre-trained VLMs used as Answerers.\n2. IdealGPT requires multiple feedforward passes for the Questioner, Answerer, and Reasoner, which may increase latency and computational costs compared to end-to-end solutions.\n3. The use of ChatGPT as the Questioner and Reasoner might introduce biases or hallucinations present in the pre-trained LLM, affecting the overall reasoning process.\n4. The prompts for the LLMs are manually designed, making it challenging to find the optimal prompt for a specific situation.\n5. The dependency on the performance of the Answerer (VLM) may limit the generalizability of the framework when dealing with different tasks or situations where the pre-trained VLMs perform poorly.", "questions_to_authors": "Question A: How does the performance of IdealGPT change when using different Answerers, especially when using VLMs with different levels of performance or specific strengths and weaknesses?\n\nQuestion B: Can you provide more insights into the trade-offs between the number of iterations and the accuracy of IdealGPT? For example, how does the performance change with more than 4 iterations, and is there a point of diminishing returns?\n\nQuestion C: Have you considered exploring automatic prompt generation or optimization techniques to improve the performance of IdealGPT without relying on manually designed prompts?\n\nQuestion D: Are there any strategies or methods to mitigate the potential biases and hallucination issues introduced by ChatGPT when used as the Questioner and Reasoner in IdealGPT?", "paper_topic_and_main_contributions": "This paper addresses the problem of zero-shot vision-and-language (VL) reasoning tasks that require multi-step inferencing. The main contribution is the proposed IdealGPT framework, which iteratively decomposes VL reasoning using large language models (LLMs). IdealGPT consists of three modules: a Questioner (LLM) to generate sub-questions, a pre-trained VL model (VLM) as the Answerer to provide corresponding sub-answers, and another LLM as the Reasoner to infer the final answer. These modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. IdealGPT is evaluated on multiple challenging VL reasoning tasks under a zero-shot setting, outperforming the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iB6whMSDZm", "reviewer": "Reviewer_duK1", "summary": "", "questions": "1. What types of the errors did the model make? Are the errors from the reasoner, questioner or answerer?\n2. If using different VLMs, what would be the effects? Does this method heavily depend on the capabilities of VLMs?\n3. For the ZS. performance, the baseline models, such as MiniGPT4 or LLaVA have very low performance. Why do these baseline models have bad performance on these tasks? Did you carefully work on the prompt engineering for LLaVA or MiniGPT4?\n4. In VCR dataset, some questions will use [obj1] or [obj2] to indicate the objects in the image. How did you process this in the IdealGPT framework? \n5. Does the LLM ask wrong or irrelevant questions?\n6. I realize that some baselines such as LLaVA have very low performance. One of the reason might be that under zero-shot setting, LLaVA does not have strong context learning ability. What would be the performance if the chatGPT directly answer the questions?\n7. Have you compared your results with ChatCaptioner as you mentioned in the related work? Also, some of the works use the similar tricks and we might want to compare with them, such as GPT + CoT.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The framework is novel and easy to reproduce. It could be useful for various vision-language tasks.", "reasons_to_reject": "The framework is a zero-shot setting and the performance is influenced significantly by the capabilities of the VLMs and the LLMs. The paper did not explore too much on how different VLM could influence the performance. Also, the paper lacks some essential baselines such that the performance improvements are not convinced. ", "questions_to_authors": "1. What types of the errors did the model make? Are the errors from the reasoner, questioner or answerer?\n2. If using different VLMs, what would be the effects? Does this method heavily depend on the capabilities of VLMs?\n3. For the ZS. performance, the baseline models, such as MiniGPT4 or LLaVA have very low performance. Why do these baseline models have bad performance on these tasks? Did you carefully work on the prompt engineering for LLaVA or MiniGPT4?\n4. In VCR dataset, some questions will use [obj1] or [obj2] to indicate the objects in the image. How did you process this in the IdealGPT framework? \n5. Does the LLM ask wrong or irrelevant questions?\n6. I realize that some baselines such as LLaVA have very low performance. One of the reason might be that under zero-shot setting, LLaVA does not have strong context learning ability. What would be the performance if the chatGPT directly answer the questions?\n7. Have you compared your results with ChatCaptioner as you mentioned in the related work? Also, some of the works use the similar tricks and we might want to compare with them, such as GPT + CoT.", "paper_topic_and_main_contributions": "This paper introduces a zero-shot framework, IdealGPT, on open-domain visual question-answering task. Specifically, the framework IdealGPT iteratively decomposes VL reasoning using large language models. The framework has three components, a questioner, an answer and a reasoner. Given an image and a question, the reasoner checks whether it is confident to answer the question. If not, the questioner asks more questions and the answer answers them. The reasoner will use new information to answer the original question again. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new model called IdealGPT to utilize LLMs to construct a multiple inference steps for zero-shot VQA tasks. The proposed method outperforms other zero-shot methods.

The reviewers were all in agreement that the method is sound with good experimental studies, strong results, and interesting ideas. Most of the concerns regarding soundness were minor, regarding more discussion of complexity, ablation studies, experimental details, and also asking for more comparisons and datasets regarding different LLM reasoning abilities, risk of hallucination, and choice of prompts. The reviewers adequately addressed these concerns during the discussion period. Overall, the reviewers were mixed in their excitement regarding the paper, with 2 ambivalent and 1 finding it strong. The reviewers who were ambivalent largely found that the proposed method mostly inherits known limitations of language models, which might limit its use due to existing issues wrt reasoning abilities, risk of hallucination, and sensitivity to prompts.
Decision: Accept-Findings