Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
Authors: Ziyue Wang
Chi Chen
Peng Li
Yang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: visual question answering
knowledge reasoning
in-context learning
TL;DR: Propose a framework where LLMs proactively ask questions for more image information for knowledge-based VQA, together with filtering models to refine the information.

Abstract: Large Language Models (LLMs) demonstrate impressive reasoning ability and the maintenance of world knowledge not only in natural language tasks, but also in some vision-language tasks such as open-domain knowledge-based visual question answering (OK-VQA). As images are invisible to LLMs, researchers convert images to text to engage LLMs into the visual question reasoning procedure. This leads to discrepancies between images and their textual representations presented to LLMs, which consequently impedes final reasoning performance. 
To fill the information gap and better leverage the reasoning capability, we design a framework that enables LLMs to proactively ask relevant questions to unveil more details in the image, along with filters for refining the generated information. We validate our idea on OK-VQA and A-OKVQA. Our method continuously boosts the performance of baselines methods by an average gain of 2.15\% on OK-VQA, and achieves consistent improvements across different LLMs.

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions
Ziyue Wang', Chi Chen', Peng Li Yang Liu
DDppartment of Computer Science and Technology, Tsinghua University, Beijing, China Insstitute for Al Industry Research (AIR), Tsinghua University, Beijing, China SShanghai Artificial Intelligence Laboratory, Shanghai, China (wangziyue22, chenchi (atnnnchh11)mmaaam 19)@mails..mmmmm fealls.ssinng tsinghua. .du. cn
l iipengeair.tsinghua.. edu. cn liuyang201 1 edu. cn
Abstract
Large Language Models (LLMs) demonstrate impressive reasoning ability and the main- tenance of world knowledge not only in natural language tasks, but also in some vision-language tasks such as open-domain knowledge-based visual question answering (OK-VQA) As images are invisible to LLMs. researchers convert images to text to engage LLMs into the visual question reasoning proce- dure. This leads to discrepancies between im- ages and their textual representations presented LLMs,, which consequently impedes final reasoning performance. To fill the information gap and better leverage the reasoning capahil- ity, design framework that enables LLMs to proactively ask relevant questions to unveil more details in image, along with filters for refining the generated information. We validate our idea on UK-VQA and A-OKVQA. Our method continuously boosts the performance of baselines methods by average gain of 2.15% on OK-VQA- and achieves consistent improvements across different LLMs!.
Question: If gets cold enough what will happen to the area being stepped over?
VinVL Caption
PromptCap Caption
A woman on skis in the snow near a tree.
woman skis the snow
snow X
snow will cover X
(A)
SF
What does the woman step over?
What else needed to onswer visual question?
LLM
Water
VLM
What happens water
when gets cold?
LLM
It freezes
Image Information & Knowledge
Coption:
woman on skis in the snow near a tree.
Complementory information:
The women steps
Water freezes when gets cold.
freeze
(B)
Figure An example of framework (B) compared to baselines (A). In the two methods in (A), the caption models do not provide precise information of "what is being stepped over", resulting in hallucinated answers. Our method (B) empowers the LLM to actively seek and acquire missing information by querying the VLM.
1 Introduction
Large-scale language models (LLMs) have exhib- ited impressive capabilities in terms of their world knowledge and reasoning abilities, leading to re- markable achievements in various Natural Lan- guage Processing (NLP) tasks such as common- sense reasoning (Tamborrino et al., 2020; Wei et al., 2022) and open-domain question answering (Li et al., 2022; Kamalloo et al., 2023). Building upon the success in the realm of text, recent re- search has explored the utilization of pre-trained LLMs in Vision- Vision-Language (VL) tasks. These stud- ies have shown promising performance, especially for knowledge-intensing tasks such as knowledge- based Visual Question Answering (VQA) (Marino
et al., 2019; Schwenk al., 2022), where both image understanding and external knowledge are imperative for answering open-ended questions. The key challenge of leveraging LLMss in VL tasks is to bridge the gap between images and text, i.e., enabling LLMs to understand images. To ad- dress this challenge, two approaches have been investigated. The first approach extends the LLM with visual perception modules form Vision- Language Pre-training (VLP) model (Alayrac al., 2022; Chen al., 2022; Li al., 2023). Despite the high performance, this approach requires training on large-scale image-text pairs, which can be com-
E "Corresponding authors: Peng Li and Yang Liu. 'Code will he released at https://git thwbb.ggtttt...t/// h THUNLP-MT/FIIG.

---

putationally expensive, particularly for LLMs with massive parameters such as GPT-3 (Brown al.,
the converted textual information.
We achieve an average gain of 2.15% on OK- VQA over baselines, and attain consistent im- provements across different LLMs.
2020) and PaLM (Chowdhery et al., 2022). The second approach transforms images into textual representations, which are then used as prompts for the LLMs (Yang et al., 2022; Hu et al., 2023a; Chen et al., 2023; Guo et al., 2023). This training-free approach is more cost-effective and enables LLMs to be utilized in more flexible paradigm, allowing for easy adaptation different tasks.
Related Work
2.1 In-Context Learning for OK-VQA
In-context learning refers to the ability of LLMs to adapt to new tasks given only few examples without parameter tuning (Brown et al., 2020). To exploit the knowledge possessed by LLMS and ex- tend this ability to OK-VQA, PICa (Yang et al., 2022) converts images into captions and tags, and applies in-context learning with GPT-3. A follow- ing work, PromptCap (Hu et al., 2023a), fine-tunes question-guide captioning model, urging the tar- get messages appear in the captions. Instead of describing an image by single caption, Chen al. (2023) represent an image as the combination of its regions in a Chain-of-Thought (CoT) style (Wei et al., 2022). Prophet (Shao et al., 2023) argues that indistinct captions lead to aimless predicting, and proposes to provide answer candidates with cor- responding confident scores as references. These methods select in in-context examples according the similarities between training and test instances. However, there are unexpected information loss during the conversion from images to text. These methods conduct a compressive one-time conver- sion to turn images into text, while we prompt the LLM to iteratively ask for detailed information. Our method is orthogonal to these approaches and can continually improve their performances.
However, as discussed in the works of Yang et al. (2022) and Hu et al. (2023a), general image cap- tions may lack the subtle information required to answer visual questions. To resolve this, Yang et al. (2022) compromise captions with image tags, and Hu et al. (2023a) propose incorporating questions into the caption generation process. Despite their successes, it remains impractical to reveal every subtle detail necessary to answer visual questions in a single concise description. As illustrated in Figure/ 1, the captions fail to spot the "area being stepped over" as the "water", resulting in halluci- nated answers. Two primary concerns exist regard- ing existing image-to-text conversion approaches for VQA: (1) The converted textual descriptions might be insufficient to solve the visual questions, or could contain misleading content; (2) Existing methods convert images to text as preprocessing step of the input. This one-off conversion is a lossy compression of the conveyed information, and does fully provoke the reasoning ability of LLMs.
In this paper, we present a framework where LLMs proactively interact with Vision-Language Models (VLMs) to gather specific information of interest, as depicted in Figure 1. This interaction is aimed at automatically seeking and regaining details that may be omitted during the image-to- text conversion. To enhance the informativeness of the generated questions and the correctness of their corresponding answers, we design a refinement module summarize only the useful information for generating the final answers. We validate our approach on OK-VQA and A-OKVQA datasets and conduct experiments across different L.MMs. Our contributions are as follows:
2.2 New Question Generation for VQA
In VQA tasks, some questions are ambiguous and might have different answers (Bhattacharya et al., 2019). Uehara et al. (2022) propose to gener- ate new questions to assist the reasoning of the original questions. They train a visual question generator with supervision from human annotated dataset (Selvaraju et al., 2020). It is evaluated on VQA dataset (Antol et al., 2015), and not ex- tensible to open- npen-domain knowledge-baid based VQA. Img2prompt (Guo et al., 2023) describes zero- shot approach OK-VQA by generating ex- new question for each instances imitate the few-shot setting without actually using multiple in-context examples. Its question generation proce- dure is irrelevant to the images. Instead of depend- ing on human annotations, given the question and
We design model agnostic framework that al- lows LLMs to proactively interact with VLMs to unveil missing information.
Our method can rectify inaccurate information generated during the image-to-text transforma- tion process and minimize the ambiguity of

---

malio sapture
magg encoder
skinng the
weating skis Usiik foxt;
probur 2 proby
croe country doing
popule actirity whi slting
What down the step ever?
Question encoder
Filter:
Caption:
Question: Wh person doing?
Answer: croee country untry do
dding ennweyyood roed
S
M captioning
What happors to the Water it gets
It
soneiing
one esuapmmnnt options for
Ma
encoder
artivtyy show conittion
required
daggreen
Captton: person sollin down srowey hill On what parson transding on? Antwer
S
Tronen
information: The over the water Water trocze when gats
gets enough what will happen to the area being stepped
The women steps
area will change wtenn..
g'x
waner.
Mater Freenes
5.
Question: cold enough
cold
happan lines being whappggeepeeieeeeeo oxer?
q
(A) Inquiry Module
(B) Refinement Module
(C) Answering Module
Figure 2: Our proposed framework consists of three modules. First, in the inquiry module ($3.1), we prompt the LLM generate new questions for the missing image information required answer the original question, and obtain answers from a VLM. Then, refinement module ($3.2) is adopted to summarize the questions and answers, filtering and extracting useful information from them. Finally, in the answering module ($3.3), the LLM is prompted predict the final answer with the augmented image information.
image information, we directly prompt LLMs to generate new questions targeting the missing infor- mation. This is not constrained on pre-annotated datasets and allows to uncover more potential information and knowledge.
question, the inquiry module prompts an LLM to generate new questions that seek for missing image information required to answer the original ques- tion, and uses a VLM to answer them based the image ($3.1). Then, we adopt refinement mod- ule to summarize information from the generated questions and answers, filtering and extracting the relevant and useful information from them ($3.2). Finally, the answering module prompts the LLM to predict the final answer to the original question with the augmented image information ($3.3).
2.3 Incorporating LLMs and VLMs for Vision-Language Tasks
Some concurrent works have also incorporated LLMs and VLMs for VL tasks. ChatCap- tioner (Zhu et al., 2023), for instance, aims to cap- ture richer details from images by progressively generating questions with LLM and then answer- ing these questions using VLM. The resulting an- swers are summarized form image descriptions. However, this approach places a stronger emphasis on the quantity of detected image details rather than their correctness. In some VL tasks, such as VQA, this inevitably introduces noise, leading to inaccu- rate image descriptions that may result in incorrect model predictions. AVIS (Hu et al., 2023b) also explores the decomposition of visual questions, but its primary focus lies in the planning procedures for tool-use and their corresponding executions.
Before delving into the details of our method, we shall declare some important notations. We use I, c, q to denote the image, its caption, and the original question, respectively. The caption c is generated by the image captioning model Ma:
c=M((!,
which is used as the preliminary image informa- tion presented LLMs for question generation. The generated new questions are denoted as q' ig': d2i..... 4gg, with the corresponding answers as a' = [a4; az; ; ; where K represents the total number of questions.
3 Method
3.1 Prompting LLM to Proactively Ask for Information
Our overall framework, as illustrated in Figure comprises three key modules: inquiry, refinement and answering. Given the image caption and the
We leverage the reasoning capabilities of the LLM to identify the essential image information that may

---

be lost during the image-to-text conversion pro- cess, Specifically, given the image caption e and the original question q, we first prompt the LLM to generate K new questions q' that inquire about the additional image information necessary to an- swer q. Suppose that the k-th question of q' has L tokens, denoted as 4/ = (ut+ ........ vE),, the decoding process can be formulated as:
Selection
Original
All
Random
Best
Accuracy
45.17
45.25
41.31
63.52
Table I: Preliminary experiments on the effects of the generated QA pairs. We report the average accuracy scores of prompting the LLM to answer each OK-VQA question with: (1) "Original": the original question y only; (2) "ALL": all the QA pairs; (3) "Random": one randomly selected QA pair. "Best" refers to the best- performing QA pair (i.e., the upper bound).
5/e arg max PILLM of
(th Vh Pv. 2:
(2)
3.2 Refining the Generated Information
where Pq is the instruction prompt. The outline of
the prompt Pq for LLM is follows:
Inspired by the preliminary experiments in $3.1, we design refinement module that can extract useful information for the LLM to answer the original question from the noisy generated QA pairs. Our refinement module includes a summarization stage and a filtering stage.
Instruction for the decomposition task w/ Please decompose the TARGET-QUESTION into K sub questions:
n in-context examples +/ TARGET-QUESTION: q² \n Catpion: c'
Sub questions: 1. 41 2. 4/2,
Firstly, we summarize 9 and the corresponding answer d'. into a narrative description s/.- Denoting s as L-token target sequence, we have:
TARGET-QUESTION: 9 An Caption:
Sub questions:
Then, employ visual question answering model M, answer these new questions based on the original image as:
12l arg max PLLM (3/ Ps: Ph. at). (4)
=a(kk(
(3)
where / is the I-th token of the summary sy and ps the instruction prompt. The complete templates used in this section are listed in Appendix F.
where de. refers to the answer to the k--th question.
To better understand the role of the generated questions and answers, we conducted a preliminary experimental analysis. Specifically, concatenate each question o% to its answer a'. to form QA pair 0/ lok; af.. and prompt the LLM answer OK-VQA questions given this representation. We investigate the results of prompting L.L with dif- ferent contexts: 1) the original question only; 2) all the QA pairs; 3) one randomly selected QA pair and 4) the best-performing QA pair. For the last case, for each question we calculate the accuracy scores when prompting with each QA pair, and then select the maximum score among them, which rep- resents an upper bound on the performance. These accuracy scores are calculated by the soft scores following Goyal et al. (2017).
Then, we apply a filter to assess the helpfulness of summary 5/ to the final prediction. The output is a contribution score of s/ given the image I, the question g, and different combinations of text inputs including question g/-- answer d'z. summary s and question-answer pair [4/: af))-
Specifically, our refinement module consists of two types of encoders, text encoder Euc, and im- age encoder Enc,, and a 3-layer multilayer per- ceptron (MLP). We use the pre-trained weights of CLIP (Radford et al., 2021) to initialize our text and image encoders. Given I, q and s'g. we first generate the visual features hvisual and the textual features h as:
From the results presented in Table 1, we can draw two key conclusions. First, the generated questions and answers indeed contain information that helps to answer the original question, compar- ing the results of Best and Original. Second, the generated QA pairs are noisy, as neither using all QA pairs nor randomly selecting one improves the performance. This highlights the necessity an in- formation refinement process to filter out irrelevant or misleading information in the generated pairs.
hysenal Fanco (T);
(5)
h = Enc;(t), t
{g. ghe des daake sk)
(6)
h = Avg Avg(h=l44.440004
hm=g)s
(7)
where h the average of the features of each kind of textual inputs. Then we calculate the fused features of the image and the summary 8 as:
MLPP( Rnisuat).
(8)

---

Method
LLM
Image Information
OK-VQA
A-OKVQA
Results with accessible LLMs
PICa
davinci
Captions (G) + Tags
48.0*
PICal
Captions (G) + Tags
49.67
49.18
hoxt-davinci-09
+ Ours
Captions (G) + Tags + refined details
53.76 +4.09
51.13 +1.95
PromptCapt
Captions (Q)
53.50
52.99
text-davinci-002
Ours
Captions (Q) + refined details
54.42 +0.92
53.21 +0.22
Prophet
Caption (G) + Candidates
57.91
58.2*
text-davinci-000
Ours
Caption (G) + Candidates + refined details
59.34
+1.43
59.79* +1,59
Resudts with unavailable L.Mss
PromptCap
code-davinci-002
Captions (Q)
60.4
56.3
Table 2: Direct answer accuracy on OK-VQA (test) and A-OKVQA (val). We use f to denote our reproduced versions, because PrompiCap and PICa use different GPT-3 engines in their paper. PICa uses davinci; PrompiCap uses -ode-davinci-022 which is deprecated. refers to multi-query ensemble because the single query results of Prophet is not reported by Shao et al. (2023). The Captions (G) refers to the caption generated by purposs VLP, such as VinVL. The Captions (Q) refers to the PromptCap caption. Candidates refers to answers candidates predicted by VQA model. Refined details refer to our regained information.
To optimize the filter, we directly use the ground- truth VQA accuracy Mgk of each training instance intermediate
where a" is the ground-truth answer of question g" and n refers to the number of shots used for in-context learning. We denote the template for the test data Template Quteee, which is follows:
(I, q, s'k, Vz*) in OK-VQA pervision signal. The final loss is formulated as:
C --y log(p=e) ( +1 (1-1111-100)) y2*) log(1 -p++]], (9)
Template-Query
Image information:
Caption: C'An Question: q'nn Answer:
where P_= = =22²) is the contribution score of & for answering given J. Please refer to Ap- pendix A for detailed data construction process, During inference, exploit the refinement mod- ule generate the refined information S. We first calculate the contribution score the original ques- tion g as P29 with the trained filter. Then we select all the summarics sy that have larger contribution scores Pa* than pæo to form the refined information set S as:
Please refer to Appendix F for the complete prompt templates with instructions.
4 Experiments
We validate our method on open-domain knowledge-base VQA, and conduct experiments on OK-VQA and A-OKVQA. Implementation details are described in the following sections.
4.1 Dataset
S= (pealpp 2 pro p=o.R=12a.....
(10)
OK-VQA (Marino al., 2019) is open-domain knowledge-based VQA dataset with about 9k train- ing and 5k testing questions. The images come from MSCOCO (Lin et al., 2014). The annotated questions are all open-ended, and each of them is associated with 10 groundtruth answers. Due to the deprecation of dataset v1.0, we use v1.1 in this paper. A-OKVQA (Schwenk et al., 2022) augments OK-VQA by the scale, tasks and extra rationales. This dataset has three splits, training (17k), validation (1.4K) and test (6.7k). A-OKVQA includes two tasks, direct answer (DA) and mul- tiple choices (MC). The DA tasks is the same as OK-VQA VQA, which requires to answer open-ended
3.3 Answer Reasoning
The answering module prompts a frozen LLM predict the final answer with both the con- verted image information such as the caption c and our regained S. For the prompt template, we mostly follow PICa (Yang et al., 2022) as the other methods do, but add a new section for our refined image information as follows (denoted as Fempsate-Fee-shho).
/+ Template- aettttt----ssot shot */ Image information: S
Caption: C" Question: Vn Answer: a"

---

questions with external knowledge. While the MC task asks choose an answer from close set with choices. focus on the open-domain setting and evaluate our method OK-VQA and the DA task A-OKVQA. Both datasets employ the soft accu- racy (Antol et al., 2015) as the evaluation metric.
Method
OK-VQA
A-OKVQA
Multimodal tuning
LXMERT
30.7
BLIP-2 (5lan))-XLL)
40.7
VLC-BERT GPV-2
43.1
45.0
48.6
Flamingo (80B)
57.8
InstructBLIP (Vicuna-7B)
62.1
64.0
M-L (562B)
66.1
4.2 Experimental Settings
Methods querying external KBs (w/o LLMs)
KRISP
38.9
33.7
We generate 3 new questions for each and apply the ensemble filters to refine the generated informa- tion. The ensemble filters contains all types of inputs, 4 deg 5/2 and g// d/].
RA-VQA
54.5
REVEAL
59.1
52.2
Methods with LLMs
Ing2Prompt175B
45.6
42.9
PICa-Full
48.0
TRiG
50.5
Baseline methods. We apply our method upon existing approaches of the same in-context learn- ing paradigm, including PICa (Yang et al., 2022), PromptCap (Hu et al., 2023a) and Prophet (Shao et al., 2023). The image information of PICa comes from captions and tags (Microsoft Azure tagging API-). PromptCap follows similar set- tings, but replaces the captioning model by fine- tuned question-aware caption model. Apart from the captions, Prophet supplies LLM with extra answer candidates obtained from their fine-tuned VQA models.
KAT (ensemble)
54.4
REVIVE
58.0
assistGPT
44.3
PrompiCap
60.4
59.6
Prophet
57.9
Ours
59.3 +1.4
57.9
Prophet (ensemble)
61.1
58.2
Ours (ensemble)
61.3 +0.2
59.8 +1.59
Table Comparisons the previous methods of on OK-VQA (test) and A-OKVQA (val). The highest ac- curacy of methods using LLMs are bolded. The sota of the other two scopes are underlined. Please refer Appandix C for the full results.
The best results of PICa and Prophet reported on the 16-shot and 20-shot settings, respectively. They employ multi-query ensemble further en- hance the performance, where they prompt the LLM for 5 times with different in-context examples. We implement our method upon these baselines fol- lowing the same settings, where the number of shot is 16 for PlCa+Ours and PromptCap+Ours, and 20 for Prophet+Ours.
VLMs. We use BLIP-2 (Li et al., 2023) to predict answer af for d.. The choice of VLMs for obtain- ing caption C' varies alone baselines. Specifically, PICa and Prophet uses VinVL (Zhang al., 2021) to convert images into captions, and PrompiCap fine-tunes question-aware caption model based on OFA (Wang et al., 2022).
4.3 Results
LLMs. For answer reasoning, the three base- lines employ differrntt LLMs. Prophet employs the LLM engine text-davinci -002, which is an InstructGPT model (Ouyang et al., 2022). PICa uses davinci, a GPT-3 model (Brown et al., 2020). PromptCap uses code-davinci-002 but is now deprecated and the authors suggest to replace it by -ext-davinci-0021 in the published code and model Considering the accessibility of LLMs and for fair comparisons, we employ the LLM engine used in Prophet (text-davinci-002) and reproduce the other two using the same engine. The gpt 5-tuurbo-00001 engine is employed for question generation and summarization.
Table shows the results on OK-VQA and A- OKVQA. With the accessible LLMs, we achieve the highest accuracy of 59.34% under single query setting, and enhance the accuracy of baselines by 2.15% on average. PromptCap OK-VQA
achieves accuracy of 60.45% with code-davinci -002 engine, which is cur- rently inaccessible. While for the reproduction with text-davinci-002. we improve the perfor- mance of PromptCap by 0.9%. For A-OKVQA. we consistently observe improvements, and the av- erage increment is 1..5%.
Table 3 lists the results of previous methods on OK-V OK-VQA and A-OKVQA. where we report our en- semble results in accordance Prophet. Within the realm of methods using LLMs, we further improve the best result by 0.2% for both dataset. Specifi-
Azure Computer Vision API v3.2 for tagging:
tttps:/wwstus www.:tuusdde...oii.... wtw..coonnneeeeeioooooorooo wic...ioot.o..omm....... servicces/comppueer-viiioo---- "https://giithun com/Yushi Hu/PromptCap

---

Methods
7-shot
Caption
Image
Tag
Refine-
Rehne-F
ACC
PromptCap
Refined information
54.42 (a)
PrompiCap
Refined information
52.25 (b)
Ours
4
PromptCap
Refined information
50.19 (c)
4
OFA- large
Refined information
48.43 (d)
4
OFA large
Refined information
47.63 (e)
4
OrA-large
All the questions + BLIP-2
43.25 (1)
Ours w/o refinement
4
OFA-large
Original questions + BLIP-2
45.17 (g)
Ours w/o BLIP-2
4
OFA-large
Original questions
44,86 (h)
BLIP-2
0
40.70 (i)
Table 4: The ablation study of how different information affect the reasoning performance on OK- OK-VQA. Ours: our proposed pipeline. Caption: the models used to generate image captions. Image Info.: the resources of our image information, the "BLIP-2" in this column refers to the BLIP-2 answers, "Refined information" applying refining on "All the questions + BLIP-2 answers". Tag: whether to use image tags. Refine-Q:: applying refining to the "Image information 5 in Template-Quuss Quiery-Quuruuu in $3.3. Refine-/t: applying refining the "Image information S in Femplate-Few-shot Feo-shot (n in-context examples) in $3.3. Line () refers to directly using BLIP-2 for OK- VKKV and the result taken from et al. (2023). Please refer to $4.4 for detailed explanation and comparison.
Scheme Accuracy
Accuracy
Method
LLM
Accuracy
T
Ensemble
59.34
Ours
Prophet
Propher
Single-a
58.53 -0.81
T=1
59.3
57.9
Prophet
57.52
text text-davinci davinci-002
Single-s
58.48 -0.86
T-2
60.5
+ Ours
58.54 +1.02
Single-qa
58.50
T=3
61.1
Prophet
50.76
ILAMA-13B
Single-q
58.10-1.2
T=4
61.2
+ Ours
53.08 +2.32
All
58.03 -1331
T=5
61.3
61.1
Prophet
44.28
LLaMA-7B
(a)
(b)
+ Ours
49.47 +5.19
PrompiCap
Table 5: Ablation studies on OK-VQA Results of method in the two table refer to Prophet+Ours with 20-shot setting. (a): Comparison of refining schemes. Ensemble: 4-model ensemble. Single: -a/-s/-qa/-q re- fer to filter model trained using a', s', q'o a' and q' respectively. All: using all information refining. (b): Results of varying the number (T) of en- semble queries. T=1: no ensemble.
PrompiCap
53.50
text-davinci-002
+ Ours
54.42 +0.92
PromptCap
47.45
LLaMA-13B
Ours
48.72 +1.27
PrompiCap
44.37
LLaMA-7B
+ Ours
44.59 +0.22
s') without
PICa
PICa
49.67
text-davinci-002
+ Ours
53.76 +4.09
PICa
42.96
LLaMA-13B
+ Ours
46.25 +3.32
PICa
39.20
cally, we achieve +1 +1.59% on A A-OKVQA compared to Prophet. For A-OKVQA, PrompiCap achieves the previous highest accuracy, 59.6%. But can- directly apply our method upon it because its employed LLM engine is inaccessible now. For multimodal tuning setting, PaLM-E (Driess al., 2023) and InstructBLIP (Dai et al., 2023) present the state of the art on OK-VQA and A-OKVQA, re- spectively. While PaLM-E is trained upon 540B language model, PaLM (Chowdhery et al., 2022), it is over threefold of the largest LLM we use (175B).
+ Ours
LLaMA-7B
42.68 +3.48
Table 6: Results our method with different L.MMs on OK-VQA. We use 16-shot for methods in this table input of 16-shot is hitting the maximum input length of LLaMA.
questions; 4) to analyse the impact of multi-query ensemble; and 5) to examine the consistency of our method across different LLMs.
Appropriately integrated information helps to boost the performance. We summarise the paradigm to solve VQA task into two categories. A VLM paradigm that directly uses VLM (e.g., BLIP-2 in Line (i) in Table 4), and an LLM-based paradigm where LLMs collaborate with VLMs answer visual questions. Following the LLM-based paradigm, we progressively integrate new informa- tion and apply refining methods from Line (h) to (a) in Table 4, and observe continual enhancements.
And InstructBLIIP employs instruction tuning.
4.4 Ablation Study
We conduct the following ablation studies: 1) to verify that the gain of our method comes from properly integrating more image information; 2) to compare the effectiveness of different selection schemes in refinement module; 3) to investigate the impact of scaling up the number of generated

---

PromptCap
Caption: chicken sandwich
plato
What is in the sandwich?
Predict: chicken
X
LLM
ham
(1)
Prophet
Captien: a white plate topped with french fries and sandwich.
S
the sandweihh or grilled?
VLM
Candidates: cheese/o. 66), teastedij od), 03), prllledoo.o cakeeda 01), fry(0.03), ham)).0t. sandwichhis 01), cheeselo 00), ham cheese|0.00 80)
grilled
VUM
ham
(+ PrompiCap) Predict: ham O
What kind sandwich this? Groundtruth: ham, reuben, grilled cheese
Question:
(+ Prophet) Predikt: grilled cheese
PromptCap
Caption: a bathraom with shower enclosure made of glass
What are features glass
Predict: frasted
X
(2)
Prophet
transparent
Caption: bathroom with a sink, tollet and shower stall
the clear or frosted?
KLM
Candidates: frostedip 581, cleario 12), fancy88.31, largel0.02, thickjo.co american 01), 20throom(00001), doont 01), winelo 01), whiteto.1))
LLM
Question: What kind of plass is used to make shower enclasure?
= Predict: frosted X
|+ PromptCap) Predict: ckea O
(+ Prophet) Predict: clear O
Groundtruth: tempered, clcar, pane
Figure 3: Cases compared Prophet and PromptCap without applying method. The frames titled by "Prompt- Cap"/"Propher" depict the results given by these two baselines in our reproduced version. The information leading to incorrect answers are marked red.
Line (h) represents results for 4-shot setting fol- lowing the PICa template. While Line (g) imple- ments our template, Template-Few-shor with n 4. The BLIP-2 answers are included in "Image infor- mation" Template-Few-ee Few-shot and Template- Humpla The participation of BLIP-2 raises the accuracy by 0.31%. When we feed all generated information (Line (f), the accuracy only marginally improves by 0.07%. Notably, applying the refinement mod- ule to the query instance (Line (c)) results in sig- nificant 2.38% accuracy boost. Line (d) introduces image tags and contributes an additional 0.8% im- provement compared Line (e). Extending to 16-shot setting further boosts the result by 2.06%. While the previous lines only apply information refining to the query instance, we go further in Line (a) by refining the in-context examples. This contributes to increment of 2.17% compared to Line (b), and is more influential than adding tags, increasing the number of shots and replacing the caption type.
As shown in Table 5 (a), filters help improve the performance against directly selecting all the questions without filtering. However, single filters make only minor contributions to the final accuracy because each of them addresses only a single aspect of the criteria.
Scaling up the number of generated questions. Intuitively, scaling the amount of generated ques- tions will contribute extra gain. To verify this, we doubled the number of generated questions to 6. By applying the same selection strategy under 20-shot single query setting, we obtain a final ac- curacy of 59.68% on OK-VQA, which is slightly higher (+0.34%) than generating only 3 questions.
Impact of multi-query ensemble. PICa and Prophet exhibit improvements of multi-query en- semble by prompting LLM for 5 times with differ- ent in-context examples. We investigate the influ- ence of multi-query ensemble on our method. As shown in Table 5 (b), although the accuracy of our method increases along with the number of ensem- ble queries, the gap between ours and Prophet's are narrowed. As the in-context examples are ranged according to the relevance to test instance, the more examples we use, the less coherent to the test instance they will be. Thereby, noises could be introduced with the progressive increase of en- semble queries. Similarly, Chen et al. (2023) also observe a decline in performance when continu- ously increase the number of ensemble queries.
In conclusive, the performance benefits from adding more image information. Our method to seek and refine image information makes a substan- tial contribution in this regard.
Effectiveness of filtering schemes. We evaluate the performance of different selection schemes in the refinement module, including (1) Ensemble: se- lecting according ensemble filters as proposed in $3.2, (2) Single: selecting according single filter, including filters trained using answers, question, summaries, and question-answer pairs (qa), and (3) All: selecting all generated questions o' and the original question g without filtering.
Consistency of our method across LLMss Dif- ferent LLMs largely affect the results (Shao et al., 2023; Hu et al., 2023a). We investi-

---

gate if our method is generalizable across LLMs trained in different ways and with different scales, including LLaMA-7B, LLaMA-13B, and text-davinci-002 (1 (175B). Table 6 proves the ef- fectiveness of our method on both InstructGPT deext-davinci-09 engine) and LLaMA. Results also demonstrate the robustness of our method on the scales of LLM, ranging from 7B to 175B. We notice that our method introduce less improvement on PrompiCap compared to Prophet and PICa. We suppose that the captions of PrompiCap are de- rived from the question, and they seem to be more determinate as shown in Figure 3, which easily dominates the reasoning and impair the attendance of other information.
fit the image. However, the LLM does not follow the given confidence and selects the least confident one. contrast, Figure 3 (2) demonstrates situa- tion where the most confident candidate is not the correct answer. this two scenarios, our method supports the correct answer with more detailed in- formation.
Conclusion
In this paper, we focus on open-domain knowledge- based VQA and propose model agnostic frame- work that successfully unveils missing detail during the image-to-text transformation. Our method ac- quires the ability to rectify inaccurate information generated by captioning models, and the ability to minimize the ambiguity of the converted tex- tual information for further improvements. Our method can be applied upon existing baselines, and achieves average gains of 2. 2.15% on OK-VQA and 1.25% on A-OKVQA over the baselines. Abla- tion studies show that our method attain consistent improvements across different LLMs.
5
Case Study
We conduct comprehensive analysis of the cases, and observe that method exhibits the following capabilities: 1) unveil better level of detail as stated in the $3.1; 2) to rectify inaccurate caption information; and 3) to minimize the ambiguity of provided information, e.g., captions and candidates. Cases are demonstrated in Figure 3. We compared the results of baselines, PtomptCap and Prophet, with and without applying our method.
Limitations
Unveiling Missing Details. Our generated ques- tions bridge the information gap between the ques- tion and the image caption by revealing missing details necessary to answer the question. An shown in (2) of Figure 3, the question asks about the "kind of glass", but relevant features are not included in the PromptCap caption. The absence of detail leads to improper prediction, "frosted". How- ever, the two questions in our method pinpoint the detailed features, "transparent" and "clear", and contribute to a correct prediction. These imply the effectiveness of our generated questions.
In this work, we demonstrate the effectiveness of our framework on OK-VQA and A-OKVQA, and show consistent improvement across different LLMs. However, we do not verify the feasibility of our idea other vision-language tasks that also quire knowledge, such as visual commonsense rea- soning. Intuitively, the paradigm to prompt LLMs to uncover missing image details can applied to a wild range of VL tasks. While the questions our framework are generated independently, further challenges include to progressively ask informative question upon the previous questions and acquire the accurate answers. We hope that our work will encourage further investigations that explore the capabilities of LLMs the VL realm.
Rectifying Inaccurate Information. Our method rectify the misleading information provided in the captions. In the first case shown in Figure 3, we correct the wrong message given by PromptCap that it not a "chicken" sandwich by the question "what in the sandwich" with answer "ham".
Acknowledgement
Minimizing the Ambiguity. By engaging more image information, method provides evidence support the correct candidates in Prophet and the proper captions of PromptCap, thereby enhanc- ing the confidence and the reliability of accurately provided information in these baselines. In Fig- ure 3, the candidates of Prophet in (1) properly
This work is supported by the National Key R&D Program of China (2022ZD0160502) and the Na- tional Natural Science Foundation of China (No. 61925601, 62276152). We thank Siyu Wang for her participation in this work, and appreciate all the reviewers for their insightful suggestions.

---

References
Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez. Brica Mor- eira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. ArXiv preprint, Abs/2204.02311.
Jean- Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for fhw-shot learning. Advances in Neural Information Processing Systems, 35:23716-255
Douglas Eck, Jeff Dean,
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. 2015. VQA: visual question an- swering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. pages 2425-2433. IEEE Com- puter Society.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung. and Steven Hoi. 2023. InstructBLIP. Towards general-purpose vision- language models with instruction tuning. ArXiv preprint, abs/2305.06500.
Nilavra Bhattacharya, Qing Li, and Danna Gurari. 2019, Why does a visual question have different answers? In 2019 IEEE/CVF International Conference on Com- puter Vision, ICCV 2019, Seoul, Korea (South), Octo- ber 27- November 2, 2019, pages 4270-4279. IEEE
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Tchter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Pete Florence. 2023. Pal An embodied multimodal language model. ArXiv preprint, abs/2303.03378.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yifan Du, Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Zero-shot visual question answering with language model feedback. ArXiv preprint, abs/2305. 17006.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. 2023. AssistGPT: general multi-modal assistant that can plan, execute, inspect, and learn. ArXiv preprint. abs/2306.08640.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. 2022. PaLI: jointly-scaled mul- tilingual language-image model. ArXiv preprint, fbs/2209.06794,
Feng Gao, Qing Ping, Govind Thattai, Aishwarya N. Reganti, Ying Nian Wu, and Prem Natarajan. 2022. Natural language-
Transform -eee--eeeteeeeeeeeeee centric mutside-knowlede visual question answering. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5057-5067 IEEE
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong. Hao Zhang. and Chuang Gan. 2023. See, Think, Confirm: Interactive prompting between vi- sion and language models for knowledge-based vi- sual reasoning, ArXiv preprint, abs/2301 1022222...522..... ,05226.
François Gardères, Maryam Ziaeefard, Baptiste Abe- loos, and Freddy Lecue. 2020. ConceptBert: Concept-aware representation for visual question swering. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020, pages 489 498, Online. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim,
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 69133
Liangke Gui, Borui Wang, Qiuyuan Huang, Alexan- der Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer

---

for vision-an -oooo-and-iae and--aanddddaaaa -anguggg In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 956-968, Seattle, United States. Association for Computational Lin- guistics.
Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral. 2021. Weakly-supervisee vittae-re reader for knowledge-based question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6417-6431.
Jiaxian Guo, Jumnan Li, Dongxu Li. Anthony Meng Huat Tiong. Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedingg of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10867-10877.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. KRISP: inte- grating implicit and symbolic knowledge for open- domain knowledge-baat based VQA. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 1021, virtual, June 19-25, 2021. pages 14111-14121. Computer Vision Foundation / IEEE.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2023a. PromptCap: Prompt-guided task-aware image captioning. ArXiv preprint, abs/221 1. .09699.
Kenneth Marino. Mohammad Rastegari, Ali Farhadi. and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring externa knowledge. In JEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195-3204. Com- puter Vision Foundation IEEE.
Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. 2023b. AVIS: Autonomous visual in- formation secking with large language models.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Newral Information Processing Systems, 35:27730-2777 277444.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2022. REVEAL: Retrieval- augmented visual-language pre-training with multi- source multimodal knowledge memory. ArXiv preprint, abs/2212.05221.
Vivek S. Pai, Peter Druschel, and Willy Zwaenepoel. 2000. IO-Lite: A unified I/O buffering and caching system. Acm Transactions Computer Systems, 18(1):37-66.
Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiii. 2023. Evaluating open-domain ques- tion answering in the era of large language models. ArXiv preprint, abs/2305.06984.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishinn, Jack Clark, Gretchen Krueger, and Nlya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. 2022. self-prompting large language models for open- domain QA. ArXiv preprinf, abs/2212.08633
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. ArXiv preprint, abs/2301.12521
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014, Microsoft COCO: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer.
Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. 2023. VLC-BERT: Visual question answering with contextualized common- sense knowledge. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vi- sion ( (WACV), pages 1155 1165.
Weizhe Lin and Bill Byrne. 2022. Retrieval augmented visual question answering with outside knowledge. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11238- 11254, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark. Kenneth Marino, and Roozbeh Mottaghi. 2022. A-OKVQA: benchmark for visual question answer- ing using world knowledge. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162. Springer.
Yuanze Lin, Yujia Xie. Dongdong Chen, Yichong Xu. Chenguang Zhu, and Lu Yuan. 2022. REVIVE: Re- gional visual representation matters in knowledge- based visual question answering. In Advances in Neural Information Processing Systems.
Ramprasaath R. Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Túlio Ribeiro, Be- smira Nushi, and Ece Kamar. 2020. SQuINTing at VQA models: Introspecting VQA models with

---

sub-questions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seartle WA, USA, June 13-19, 2020, pages 10000-10008, IEEE.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of GPT-3 for few-shot knowledge- based VQA. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourih Con- ference Innovative Applications of Artificial In- telligence, TAAI 2022, The Twelveth Symposium Educational Advances in Artificial Intelligence, EAAI 2022 Viriual Event, February 22 March 1, 2022. pages 3081-3089, AAAI Press.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023, Prompting large language models with answer heuris- tics for snowledge-hasel visual question answering. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Puttern Recognition, pages 14974- 14983.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. VinVI.: Revisiting visual representa- tions in vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579-5588. Computer Vision Foundation IEEE.
Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pan- nier, Pascal Voitot, and Louise Naudin. 2020. Pre- training is (almost) all you need: An application commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878 3887, Online. Association for Computational Linguistics.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023. ChatGPT asks, BLIP-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXi::2333.06594
Kohei Uchara, Nan Duan, and Tatsuya Harada. 2022. Learning to ask informative sub-questions for visual question answering. In IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19-20, 2022, pages 4680.4689.. IEEE
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: unifying ar- chitectures, tasks, and modalities through simple sequence-to-sequene learning framework. In Inter- national Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 13318-23340 PMLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe Le, Denny Zhou, et al. 2022. soning in large language models. In Advances in Neural Information Processing Systems.
Thhouhhh prompting elicits rea-
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022a. GRiT: A generative agiio-tootex: trans- former for object understanding. arXiv preprint 2.Xiv:2212000280
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022b. Multi-modal answer validation for knowledge-based VQA. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022. Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial In- telligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 271 2712-2721. AAAI Press.
Jialin Wu and Raymond Mooney, 2022. Entity-focused dense passage retrieval for outside- nutside-knowledee knowledge visual question answering. Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 8061-8072, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.

---

A Details for Gathering Training Data for Refinement Module
C Full List of Results
We provide results of previous methods on OK- VQA and A-OKVQA in Table 7 and Table 8. The current state-of-the-ar on OK-VQA is 66.1 and is contributed by 562B pre-trained multimodal model, including a 540B language model and 22B vision encoder. We achieve the highest score in both methods querying external KBs and meth- ods with LLMs. InstructBLIP (Dai al., 2023) achieves the SOTA A-OKVQA with multimodal instruction tuning.
The process for constructing the training data for refinement module described in $3.2. We denote set I, con-
generated image information taining the four types of generated image informa- tion, q's a', q'a' and s'. is the of all the origi- nal visual questions, and refers the corresponding images. ys is resulting training labels used in $3.2, where Uz refers to the label for single generated information. The soft accuracy score fc is computed following Goyal et al. (2017).
Method
Accuracy
Muliimodal pre-train
Algorithm Pipeline for Supervision Gathering Imput: Image, Question and Generated Image In- formation (I,Q.I)
LAMOCHI (Du et al., 2023)
40.3
BLIP-2 (5-1nT5-XI.) (l.i ct al., 2023)
40.7
BLIP-2 (FlanTS-XXLL) (Li et al., 2023)
45.9
OFA-Iarge (Wang ct al., 2022)
49.4
Output: Image Information (I), Original Ques- tions (Q) and Images (I) with labels (ys)
Unified-1O (2.8B) (Pai et al., 2000)
54.0
Flamingo (80B) (Alayrac et al., 2022)
57.8
Require: Pi,g is the prompt for answer reasoning regarding question q and image
InstructBLIP (Dai et al.. 2023)
62.1
PaLM-E (562B) (Driess al., 2023)
66.1
Methods querying external KBs
1: procedure GET LABELS((:))
ConceptBERT (Garderes al., 2020)
33.7*
KRISP (Marino al., 2021)
38.9
2: for Q Q and i and I C I do
Vis- Vis-DPR (Luo et al., 2021)
39.2
3: a LLM (Pr,(
MAVEx (Wu al., 20226)
40.3
4:
acCq - Ac
VLC-Bert (Ravi al., 2023)
43.1
TRiG (Gao al., 2022)
50.5
5: if accy then
RA-VQA (Lin and Byrne, 2022)
54.5
6:
y/ <-
REVEAL (Hu al., 2022)
Methods with Img2Prompt175B (Guo et al., 2023)
7: else
8:
V/2 < 0
KAT (ensemble) (Gui et al., 2022)
54.4
KAT-full EnFoRe (ensemhle)
55.2
(Wu and Mooney, 2022)
PICa-Full (Yang et al., 2022)
48.0
B The Declaration of Trainable Modules in Our Pipeline
REVIVE (Lin al., 2022)
58.0
PrompiCap (Hl et al., 2023a)
60.4
Prophet (Shao al., 2023)
57.9
These models are frozen in our entire pipeline: the captioning model Me. the VQA model Ma and the LLMs (for question generation, summarization and reasoning).
Prophet+ours
59.3 (+1.4)
Prophe (ensemble) (Shao al., 2023)
61.1
Prophet+ours (ensemble)
61.3 (+0.2)
Table 7: Comparisons to other methods on OK- UKAA
The refinement module in Figure 2 requires train- ing, which includes parts:
indicates results reported on OK-VQA v1.0. + refers method with currently deprecated LLMs. Instruct- BLIP refers to CaaecetBII(Viiuna--/i accuracy within methods using LLMs bolded and the highest accuracy within the other two scopes are underlines.
The highest
An image encoder (for image): to encode the original VQA image;
A question encoder (for text): encodes the original VQA question;
An information encoder (for text): to encode our generated image information (denoted as "Info encoder" in Figure 1);
filter: an ML.P-based network that evaluates the helpfulness of the obtained image informa- tion towards answering the visual questions.

---

Method
Accuracy
ble setting. For Prophet, the gain of the ensemble setting over the single setting is 3.19% (Line 5), which more than doubled compared to results on Line to Line 4, and is nearly doubled compared to Line 6. This shows an inconsistency regarding the gain of ensemble setting over single setting, and supports our hypothesis that Prophet benefits more from the ensemble setting compared to other methods.
Pythia (Schwenk et al., 2022)
25.2
ViLBERT (Schwenk et al., 2022)
30.6
LXMERT (Schwenk et al., 2022)
30.7
ClipCap (Schwenk et al., 2022)
30.9
Fine-tune
KRISP (Marino et al., 2021)
33.7
LAMOC (Du ct al., 20233
37.9
VLC-BERT (Ravi et al., 2023)
45.0
GPV-2 (Schwenk ct al., 20222
48.6
Unified-10 (Pai et al., 2000)
REVEAL (Hu ct al., 2022)
52.2
InstructBLIP (Dai al., 2023)
64.0
Img2Prompt1 758 (Guo et al., 2023)
42.9
In-context
assistGPT (Gao et al., 2023)
44.3
Methods
Single
Ensemblc
Gain
PromptCap o (Hu et al., 2023a)
PICa
49.67
51.36
1.69
Prophet (Shao al., 2023)
+ Ours
53.76 (+4.09)
54.91 (+3.55)
1.15
Prophettours
PrompiCap
53.50
54.16
0.66
Ours
54.42 (+0.92)
55.01 (+0.85)
0.59
Table 8: Comparisons other methods A-OKVQA
Prophet
57.91
61.10
3.19
DA task. & refers to method with currently deprecated LLMs. InstructBLIP refers to The highest accuracy is bolded and the second best is underlines.
Ours 59.34 (+1.43) 61.30 (+0.20)
1.96
Table 10: Single and Ensemble performances of our
methods and baselines. Gain refers to the increment of ensemble result over single result.
D Ensemble with different numbers of shots
E Experiments with Dense Captions
The results of ensemble queries are listed in Table 9. We notice that our method improved the result of Prophet by 1.43% in 20-shot single query setting, But the increment decreases to 0.2% when applying ensemble. Also, similar pattern can be found in 16- shot setting. The increments from T=1 to T=3 relatively significant, which are 1.8% (20-shot) and 2.1 11% (16-shot). While continuously increasing the number ensemble queries from T=3 to T=5, the increments decrease to 0.2% and 0.1%. Because the in-context examples are arranged according to the relevance to the test instances, these phe- nomenon could result from engaging irrelevance examples when enlarging the number of queries.
In our framework, the LLM perceives the image from two parts of the prompt: "Caption" and "Im- age Information" (as the templates described in $3.3). We incorporate the dense captions (denoted as GRiT (Wu et al., 2022a)) and conduct types of experiments with the PICa pipeline:
Dense captions directly "Caption":
Dense captions as a type of "Image Informa- tion".
As shown Table 11, the first two lines show the influence of dense captions and general cap- tions. Simply replacing the OSCAR captions by GRiT captions reduces the accuracy from 49.67% to 46.16% 16%. Adding GRiT to PlCa as the "Image Information" (line 4) also impairs the original PICa performance (line 1) by 0.24%.
20-shot
16-shot
T
Ours
Prophet
Ours
Prophet
T=1
59.3
57.9
58.5
57.5
1
660.5
60.1
T=3
61.1
60.6
T=4
61.2
60.8
Lines 3, 4 and 5 present the performances of different types of "Image Information". Notably, replacing our information by GRiT'S dense cap- tions decreases the performance by 4.33% (com- paring line 4 to line 3). While combining Ours and GRiT captions as the "Image Information" (line 5) reduces the accuracy by 1.21 % compared to using only Ours "Image Information" (line 3).
T=5
61.3
61.1
60.9
60.8
Table 9: 20-shot 16-shot results of the number ensemble queries (T) OK-VQA. T=1 means semble.
We conduct further examination the ensem- ble behavior on OK-VQA dataset and find that Prophet indeed benefits more from ensemble com- pared to the other baselines. Table shows the results of the single query setting and the ensem-
conclude, dense captions introduce noise and degrade the accuracy, regardless of the role as ei- ther "Caption" or "Image Information". In contrast,

---

utilizing the image information obtained and re- fined by our method consistently yiclds the best results.
An Example for Question Generation
Please decompose the TARGET-QUESTION into 3 questions that can be answered via commonsense knowledge. The sub-questions should not mention another sub-questions. You information from the CAPTION. An
Methods
Caption
Image Info.
ACC
PICa
OSCAR
49.67
PICa
GRiT
46.16 (-3.15)
PARGET-QUESTIN what is the hairstyle of the blond called?\n
+Ours
OSCAR
Ours
53.76
+D.C.
OSCAR
GRIT
49.43 (-4.33)
Caption: Two
tennis players
+D.C.+Ous
OSCAR
GurT+Ours
52.55 (-1,21)
tennis court.An
It this hairstyle long short? 2. What the notable features of the hairstyle? 3. what hairstyle common for women player when they are playing tennis\n
Sub questions:
Table 11: Experimental results with dense captions (GRiT). "Caption" refers the type of caption used in Template-Few-tfee Feo-shot and Template- Query in $3.3. "Im- age Info." refers to type image information used in Template-Fer shw-shot and Template-Quer in $3.3.
TARGET-QUESTION: How old you have to be in canada
this?An
Caption: couple of people are holding up drinks. \n
Sub questions: 1. Why are people holding up drinks? 2. what is the restriction age to drink in Canada? 3. what are people drinking?\n
F Prompt Templates
TARGET-QUESTION: When was this piece of sporting equipment invented?\n Caption: man in a wetsuit carrying surfboard to the water. \n Sub questions: 1. what is the man carrying with him? 2. What is the purpose of the sporting equipment? 3. What is the history of the invention of the sporting equi pment?\n
In this section, we provide detailed prompt tem- plate for question generation, summarization, and in-context learning for visual question reasoning. Since the input length of LLMs limited, to present LLMs with more potential image infor- mation and relevant knowledge, we remove the separators ("===") used in PICa and its followers.
TARGET-QUESTION: what hair style does the child have?\n
Caption: little girl with short hair talking cell phone. An
Sub questions:\n
F.1 Prompt Templates for Question Generation
F.2 Prompt Templates for Summarization
Here is the prompt template and an example for question generation described in $3.1. The number of questions to generate is 3, and the test instance is marked in blue. The template and example are follows:
In the refinement module, we summarize the gener- ated questions with corresponding answer into nar- rative expressions further process. Here is the prompt template and an example for information summarization described in $3.2, the test instance is marked in blue:
Prompt Template for Question Generation
Please decompose the TARGET-QUESTION into questions that can be answered via commonsense knowledge. The sub-questions should not mention another sub-questions. You information from the CAPTION. TARGET-QUESTION: qn
Prompt Template for Summarization
Please summarise the following question and corresponding answer into description sentence. An
Caption: C"
Sub questions:
di
q" An A: anAn Summary: TT. 2. 4
TARGET-QUESTION: q\n Sub questions:
g\n anAn Summary:

---

An Example for Summarization
An Example for Reasoning with PICa
Please summarise the following question and corresponding answer into description sentence.
Answer the questions using the provided image information, captions and extra commonsense knowledge. Answers should be longer than words:\n
Q: What is the specific type of drink be? A: mar tini. An Summary: People drinking martinis. An
Image information: the person is skiing; the person is wearing skis their feet; country skiing is popular activity while skiing. An
Q: What is the legal age consume alcohol in Canada?\n
country skiing through forrest winter. winter, tree, sky, outdoor recreation, piste, blizzard, ski resort, outdoor, snow, skiing\n Question: What this person doing?\n Answer: cross country ski
Caption:
is
A: 18.\n
Summary: People should least 18 consume alcohol in Canada.
Q: What type of drinks are on table?\n
A: soda.An
person wearing skis; cross country skis are one the equipment options for this activity; Snow condi tindi impact travel safety during this activity.
Image information:
Summary: There a soda on the table. An
Q: How is this beverage made?\n
A: it coffee drinkAn
Summary:
Caption: A man on skis riding through the snow. country skier, footwear, mountain, mountain guide, snowshoe, winter, glacial landform, standing, ski equipment, ice cap'nn
Question: What is this person doing?\n Answer: ski
F.3 Prompt Templates for Reasoning
The women steps over the water; Water freezes when it gets cold;
Image information:
The area will change when the temperature reaches 0 degrees. \n
We employ few-shot in-context learning for answer reasoning. Here the prompt template described $3.3, the test instance is marked in blue:
Caption: woman skis in the snow near a tree. cross-country skier, footwear, outdoor recreation, blizzard, freezing, snowshoe, winter sport, winter, snow, trekking poleln
Question: If it gets cold enough what will happen to the area being stepped over?\n Answer:
Prompt Template for Reasoning
Answer the questions using the provided image information, captions and extra commonsense knowledge. Answers should be no longer than 3 Wn
Image information: S"
Caption: C"An Question: vn\nn Answer: d" Image information: S\n
Caption: C'An Question: q'n Answer:
We implement our method with different base- lines according to the their default settings for im- age representation. PICa employs captions with tags image representation; PromptCap uses thire question-aware captions; and Prophet provides ex- tra answer candidates. There are the examples for PICa+ours, PrompiCap+ours and Prophet+ours, our refined information is bolded in the template, and the test instance is marked in blue:

---

An Example Reasoning with CrompiCap
An Example for Reasoning with Prophet
Answer the questions using the provided image information, captions and extra commonsense knowledge. Answers should be no longer than 3 words:\n
Answer the questions using the provided image information, captions, candidate answers and extra commonsense knowledge. Each candidate answer is asseciated with confidence score within bracket. The true answer may not be included in the candidate answers. Answers should be longer than words:An
the person is skiing; the is wearing skis their feet; country skiing popular activity while skiing. An
Image information:
Caption: person skiing snowy road. Question: What this person doing?\n
the person is skiing; the person is wearing skis their feet; crass country skiing is popular activity while skiing.
Image information:
Answer: cross country
Image information: the person is wearing skis; cross country are one the equipment options for this activity; Snow conditions impact travel safety during this activity.
Caption:
is cross country skiing through a forrest winter.
Question: What is
person doing?\n
Candidatew: ski (0.98), cross country ski (0.63), skiis (B.13), hike (0.11),
Caption: person skiing down a snowy hill.An An
snow (0.09), cross country (0.02), skiing (8. (8.81), snowboard (0.00), camp (0.00),
Question: What this person doing?\n Answer: ski
cold weather (0. (n) Answer: cross country ski
Image information: the person is wearing skis; cross country skis are one of the equipment options for this activity; Snow conditions impact travel safety during this activity. An
Image information:The women steps over the water; Water freezes when it gets cold; The area will change when the temperature reaches degrees. \n
Caption: man on skis riding through the snow. \n
Caption: a woman on skis in the snow)n Question: If it gets cold enough what will happen to the area being stepped over?\n Answer:
Question: What is this person doing?\n
Candidatew: ski (0.99), snow (0.66) sky (0. (0.15). water (0.03), skiis (0.02), ski pole (0.01), downhill (0.01), snowboard (0. 80), hill (0.00), commuter (0. in
Answer: ski
Image information: The women steps over the water; Water freezes when it gets cold; The area will change when the temperature reaches 0 degrees. \n
Caption: woman skis in the
tree.
Question: If it gets cold enough what will happen the area being stepped over?\n (0.02),
Candidatew: fall (0.04), break (0.01), avalanche (8.01), death (0.01), cold (0.00), freeze (0.00), autumn
(0. (0.00), oxygen (0.00), drown (8.00) 00)
Answer:


Reviews:
[{"review_id": "yB2wLxzrDs", "reviewer": "Reviewer_QtV8", "summary": "", "questions": "Question A: Why not try dense captioning to obtain more image information? (this question has been included in the Reasons To Reject section)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: Differences between our method and ChatCaptioner (\"ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions\")**\n\n**Response 1:**\n\n- We focus on different aspects of image information. ChatCaptioner aims to capture richer details from images as image captions. However, this approach inevitably introduces noise, i.e., inaccurate descriptions of the images. For many vision-and-language (VL) tasks, such as VQA, this noise could lead the model to make incorrect predictions. Therefore, our focus is not only on capturing more image information but also on effectively extracting the useful parts from this information for specific VL tasks.\n- We design a refinement module to successfully select proper information that will have positive effect on answering the questions. As reported in Table 2 and Table 5(a) in our paper, using all the information **without filtering** (\"All\" in Table 5(a)) results in an accuracy of only 58.03%, while it is 57.91% without our image inforamtion (\"Prophet\" in Table 2). By introducing our designed refinement module, the accuracy increases to 59.34% (\"Prophet+Ours\" in Table 2). A similar phenomenon can be observed from the dense caption experiments below (Response 4). \n- We appreciate your suggestions and will include ChatCaptioner in our revised version. \n\n**Question 2: Efficacy compared with Prophet**\n\n**Response 2:**\n\nWe would like to further emphasize the improvements of our method compared to Prophet in Table 3 in our paper as follows:\n\n- For A-OKVQA, our single query result (57.9%, \"Prophet+Ours\" in Table 3) is already comparable to the ensemble result of Prophet (58.2%, \"Prophet (ensemble)\") with a significantly lower inference cost. Furthermore, our ensemble performance reaches 59.8% (\"Prophet+Ours (ensemble)\" in Table 3), marking a 1.6% improvement.\n\n- For OK-VQA, we obtain an increment of 1.4% over Prophet in the single query setting (\"Prophet+Ours\"). In the ensemble setting, we can observe a performance improvement of 0.2% (\"Prophet+Ours (ensemble)\"). \n    - One factor that might contribute to this reduced improvement in the ensemble setting is the inherent characteristics of the Prophet method. Prophet relies on a fine-tuned VQA model to generate answer candidates from a closed-set of predefined answers (all the answers present in the training dataset). \n    - This constraint is particularly noteworthy for OK-VQA, given its lack of answer diversity compared to A-OKVQA. The OK-VQA dataset has 11,508 distinct answer categories, with each question having a maximum of 5 different answers. On the other hand, the A-OKVQA dataset contains 29,631 unique answer categories, with each question having up to 10 distinct answers.\n    - By integrating more few-shot examples, the answer candidates will present a dataset bias. For OK-VQA dataset, we hypothesize that Prophet can benefit from the biased candidates in the ensemble setting. \n\n- We conduct further examination on the ensemble behavior on OK-VQA dataset and find out that Prophet indeed benefits more from ensemble compared to the other baselines. The table below shows the results of the single query setting and the ensemble setting. For Prophet, the gain of the ensemble setting over the single setting is 3.19% (Line 5), which is more than doubled compared to results on Line 1 to Line 4, and is nearly doubled compared to Line 6. This shows an inconsistency regarding the gain of ensemble setting over single setting, and supports our hypothesis that Prophet benefits more from the ensemble setting compared to other methods. \n    \n    |No.|    Method     | Single        | Ensemble      | Gain of Ensemble over Single|\n    |:--|:--------------|:--------------|:--------------|:-----------------:|\n    | 1 | PICa          | 49.67         | 51.36         | 1.69             |\n    | 2 | PICa+ours     | 53.76 (+4.09) | 54.91(+3.55)  | 1.15             |\n    | 3 | PromptCap     | 53.50         | 54.16         | 0.66             |\n    | 4 | PromptCap+ours| 54.42 (+0.92) | 55.01 (+0.85) | 0.59             |\n    | 5 | Prophet       | 57.91         | 61.10         | **3.19**         |\n    | 6 | Prophet+ours  | 59.34 (+1.43) | 61.30 (+0.20) | 1.96             |\n\n- Therefore, we consider the reduced improvement in the OK-VQA + ensemble setting to be a special case, whereas in most other cases our approach leads to significant improvements compared to Prophet.\n\n**Question 3: Experiments on new tasks that are more different from OK-VQA and A-OKVQA**\n\n**Response 3:**\n\n- Exploring other VL tasks, such as VCR, is a highly insightful suggestion. We are now working on it. However, due to the extensive transfer work required, we regret that we were unable to provide the results within the given time constraints. We will strive to include the relevant findings in subsequent versions.  \n \n**Question 4: Dense caption experiments**\n\n**Response 4:**\n\n- In our framework, the LLM perceives the image from two parts of the prompt: \"Caption\" and \"Image Information\" (as in the template shown between Line 288-289 in our paper). We incorporate the dense captions (denoted as GRiT) and conduct the following two types of experiments with the PICa pipeline:\n    - dense captions directly as \"Caption\" (Line 2);\n    - dense captions as a type of \"Image Information\" (Lines 4 and Line 5).\n\n    |No.|                   Method   | Caption | Image Information | ACC   |\n    |:-|:----------------------------|:--------|:------------------|:------|\n    |1 | PICa                        | OSCAR   | -                 | 49.67 |\n    |2 | PICa                        | **GRiT**| -                 | 46.16 (-3.51) |\n    ||\n    |3 | PICa + Ours                 | OSCAR   | Ours              | 53.76 |\n    |4 | PICa + Dense Caption        | OSCAR   | **GRiT**          | 49.43 (-4.33) |\n    |5 | PICa + Ours + Dense Caption | OSCAR   | Ours + **GRiT**   | 52.55 (-1.21) |\n\n- In the table above, Line 1 and Line 2 show the influence of dense captions and general captions. Simply replacing the OSCAR captions by GRiT captions reduces the accuracy from 49.67% to 46.16% (-3.51%). Adding GRiT to PICa as the \"Image Information\" (Line 4) also impairs the original PICa performance by 0.24% (comparing Line 4 to line 1).\n- Lines 3, Line 4 and Line 5 compare the performances of different types of \"Image Information\". Comparing Line 3 and Line 4, replacing our information by GRiT's dense captions decreases the performance by 4.33%. While combining Ours and GRiT captions as the \"Image Information\" reduces the accuracy by 1.21% compared to using only Ours \"Image Information\" (comparing Line 5 to Line 3).\n- To conclude, dense captions introduce noise and degrade accuracy, regardless of their role as either \"Caption\" or \"Image Information\". In contrast, utilizing the image information obtained and refined by our method consistently yields the best results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of this paper is relatively interesting, proposing a method that leverages the interaction between LLM (Language Large Models) and VLM (Vision-Language Models) to obtain finer-grained image information, which in turn aids visual reasoning tasks. It offers the research community a new perspective on interactive use with large models. Simultaneously, it addresses, to some extent, the current limitation of LLMs being unable to access visual information. \n\n2. The experiments in the article are comprehensive, with rich visualization results. These contents validate the points proposed in the paper. The performance comparisons and ablation studies demonstrate the efficacy of the proposed method. Furthermore, tasks like OKVQA and AOKVQA are inherently challenging. The method presented in this paper achieves competitive results on these tasks, further underscoring its effectiveness.\n\n3. The method put forward in the paper is not only effective but also intuitive, easy to replicate, and clear-cut.", "reasons_to_reject": "1. Generally speaking, I believe the novelty is somewhat lacking. For instance, earlier this year (published on arxiv on March 12th, more than three months before the EMNLP deadline), there was a paper titled \"ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions\" that employed an approach similar to what you've proposed, aiming to capture more visual information to produce image captions with richer details. Admittedly, in terms of the overall workflow, the structure of your paper seems more comprehensive, encompassing broader considerations. However, on the whole, I don't feel surprised when I read your paper, as the core content bears resemblances to previous work.\n\n2. I believe you should explore one or two additional vision-language tasks. As you've acknowledged in the limitation section, your experiments solely focus on OKVQA and AOKVQA tasks. Essentially, these two tasks are quite similar, with AOKVQA being a more advanced version of OKVQA. Based on the results in Table 3, the performance of OKVQA Prophet (ensemble) only improves by a mere 0.2% with the addition of your proposed method, which hardly convinces me of its efficacy in this context. A 0.2% increment could even be attributed to minor experimental errors, rendering the results somewhat dubious. Hence, rather than tackling both AOKVQA and OKVQA, it would be more sensible to solely concentrate on AOKVQA. On the one hand, the outcomes from Table 3 for AOKVQA appear reasonable and credible; on the other hand, it avoids the redundancy of undertaking two similar tasks. If a wider array of tasks were addressed, incorporating both wouldn't be an issue, but with just these two, their similarity is more glaring. Therefore, if you decide to pursue two tasks, I'd recommend focusing on Visual Commonsense Reasoning (VCR https://visualcommonsense.com/) and AOKVQA. I noticed that you mentioned VCR in the limitation section as well. Personally, I'm more intrigued by the combination of VCR and AOKVQA. I believe readers could glean more insights from these two tasks, given the significant differences in task format (direct generation vs. multiple-choice) and textual content between VCR and AOKVQA. Such diverse content seems far more engaging to me.\n\n3. Given the need to extract more detailed image information for enhanced visual reasoning, why haven't you considered dense captioning? Would using dense captioning yield favorable results? (You can refer to the approach on https://github.com/JialianW/GRiT) Naturally, integrating dense captioning might introduce the problem of hallucination, making the outcomes potentially noisier. Despite the possibility of such issues arising, I'd like to see some experiments incorporating dense captioning, whether in the main content or the appendix.\n\n4. This is a frequently discussed question. Your work stems from the perspective that LLM cannot visualize images, which is indeed correct at present. However, with the evolution of MLLM and the anticipated unveiling of GPT-4's multimodal capabilities, the value of your work is bound to face significant challenges. From my viewpoint, I'd expect authors working on LLM Aided visual reasoning to proactively address the challenges presented by GPT-4 and its implications on their research. However, I haven't observed this aspect in your work.", "questions_to_authors": "Question A: Why not try dense captioning to obtain more image information? (this question has been included in the Reasons To Reject section)", "paper_topic_and_main_contributions": "Paper Topic:\nThis paper proposes a method to utilize LLM to ask more questions about the image so that LLM can obtain more details about the image to fill the image information gap. After obtaining more visual information, LLM can better handle visual reasoning tasks like OK-VQA and A-OKVQA. In order to generate useful image information, the authors introduce a framework consisting of three modules, i.e., inquiry, refinement, and answering.\n\nMain Contributions:\n1. The authors design a model-agnostic framework that can be applied in different LLMs to help them proactively interact with VLMs to unveil missing image information.\n2. The proposed method can correct wrong information and minimize the ambiguity of converted textual information.\n3. The proposed method can improve baseline models and attain improvements across different LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny. ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions. I think this paper should be included in the reference list (this paper was released in ArXiv on 03/12/2023, which is more than three months before the EMNLP deadline). This work also introduces an approach to make LLM and VLM interact to generate more questions so that we can obtain more visual information in the end. After summarization, we can have a caption containing more details about the image.\n\n2. Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A. Ayyubi, Kai-Wei Chang, Shih-Fu Chang. IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models. This paper is similar to your work to some extent. I hope it can be included in the 'New Question Generation for VQA' section in related work. Since it is contemporaneous work, I just suggest you do so."}}, {"review_id": "FQmhsFMBJx", "reviewer": "Reviewer_Ka9D", "summary": "", "questions": "- It is unclear what parts are trained and what aren’t. What parts of what models are trained when training the refinement using the VQA annotations? Is the entire pipeline trained end-to-end? \n- For the baselines, are the VLM models also trained on the VQA datasets used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1: What parts are trained when training the renement module?**\n\n**Response 1:**\n\n- These models are frozen in our entire pipeline: the captioning model ($M_c$), the VQA model ($M_a$) and the LLMs.\n\n- The refinement module in Figure 2 requires training, which includes 4 parts:\n    - An image encoder(for image): to encode the original VQA image;\n    - A question encoder(for text): to encodes the original VQA question;\n    - An information encoder(for text): to encode our generated image information (denoted as \"Info encoder\" in Figure 2 in our paper);\n    - A filter: an MLP-based network that evaluates the helpfulness of the obtained image information towards answering the visual questions. \n\n**Question 2: Is the entire pipeline trained end-to-end?**\n\n**Response 2:**   \n  \nNo. The LLMs and VLMs in our entire pipelin are frozen. Only the refinement module requires training (as stated above). For the training of the refinement module, we first construct target labels according to Appendix A in our paper. And then we train the encoders and the filter with BCE loss (please refer to Equation 9 on Line 269 in our paper). \n\n**Question 3: For the baselines, are the VLM models also trained on the VQA datasets used?**\n\n**Response 3:**  \n\nYes, Prophet and PromptCap also require training on the VQA datasets, but they train VLMs for different purposes. Prophet trains different VQA models on OK-VQA and A-OKVQA datasets. PromptCap does not involve the training of VQA models, but its image captioning model are trained on the VQA datasets. \n\n**Question 4: Clarification of the results**\n\n**Response 4:**\n\nSorry for causing the confusion. We carefully revise Table 4 in our paper. Here is the reformed version:\n\n| Method | Number of Shots| Caption | Image Information | Tags | Refinement on Query Instance | Refinement on Few-shot Examples | ACC |\n|---------------------|---|-----------|---------------------------|-----|-----|-----|----------|\n| Ours                | 16| PromptCap | Refined information    |&#10004;|&#10004;|&#10004;|  54.42(a)|\n| Ours                | 16| PromptCap | Refined information       |&#10004;|&#10004;| -   |  52.25(b)|\n| Ours                | 4 | PromptCap | Refined information       |&#10004;|&#10004;| -   |  50.19(c)|\n| Ours                | 4 | OFA-Large | Refined information       |&#10004;|&#10004;| -   |  48.43(d)|\n| Ours                | 4 | OFA-Large | Refined information       | -   |&#10004;| -   |  47.63(e)|\n| Ours w/o refinement | 4 | OFA-Large | All the questions + BLIP-2 answers| -   | -   | -   |  45.25(f)|\n| Ours w/o refinement | 4 | OFA-Large | Original question + BLIP-2 answers| -   | -   | -   |  45.17(g)|\n| Ours w/o BLIP-2     | 4 | OFA-Large | Original question         | -   | -   | -   |  44.86(h)|\n| BLIP-2              | 0 | -         | -                         | -   | -   | -   |  40.70(i)|\n\nRecalling the prompt template for answer reasoning in Section 3.3 in our paper, the template consists of two parts: \n\n- Few-shot in-context examples (Line 287 to Line 288 in our paper), denoting as ***Template-Few-shot***:\n    \n        Image information: S^n \\n Caption: C^n \\n Question: q^n \\n Answer: a^n\n\n- A query instance (Line 290 to Line 291 in our paper), denoting as ***Template-Query***:\n\n        Image information: S \\n Caption: C \\n Question: q \\n Answer: \n    \nWith the in-context examples, the LLM is required to predict the answer for the question q in the query instance. \n\nHere are some explanations of the table above:\n\n- \"Caption\": the models used to generate the image captions; \n- \"Image Information\": the resources of our image information, \"Refined information\" = applying refining on \"All the questions + BLIP-2 answers\"; \n- \"Tags\": whether to use image tags. According to [PICa](https://arxiv.org/pdf/2109.05014.pdf), image tags are place right after the captions in the template above;\n- \"Refinement on Query Instance\": applying refining to the \"Image information S\" in ***Template-Query*** mentioned above;\n- \"Refinement on Few-shot Examples\": applying refining to the \"Image information S^n\" in ***Template-Few-shot*** mentioned above;\n    \nFor the question of **“Line (f) converts Line (h) to few-shot setting”. What does it exactly mean? Fewer context? Fewer VQA training data points? How are they sampled?** \n\n- It should be Line (g) (previous Line (f)) vs Line (i) (previous Line (h)) in this new table. Line (g) and Line (i) belong to different paradigms. Line (i) follows the VLM paradigm that directly applies the VLM (BLIP-2) for OK-VQA task. Line (g) belongs to the LLM-based paradigm where LLMs cooperate with VLMs for OK-VQA task. \"Line (g) converts Line (i) to few-shot setting\" means that the LLM is making use of BLIP-2 answers for the final reasoning in a few-shot learning way.\n- Line (i) reports the zero-shot BLIP-2 result taken from [the BLIP-2 paper](https://arxiv.org/pdf/2301.12597.pdf). While Line (g) is a 4-shot setting where each query instance is provided with 4 examples (following ***Template-Few-shot***, n=4). These examples are selected according to the relevance of images and questions to the query instance. The BLIP-2 answers are included in the \"Image information S^n\" in ***Template-Few-shot*** and \"Image information S\" in ***Template-Query***.\n\nHere are the main conclusions of this table in brief: \n\n- Line (e) vs Line (f): refinement module significantly improves the accuracy (+2.38%);\n- Line (g) vs Line (h): BLIP-2 answers help (a bit) the LLM to reason for answers (+0.31%);\n- Line (d) vs Line (e): tags are helpful (+0.80%);\n- Line (b) vs Line (c): increasing the number of shots significantly improves the accuracy (+2.06%);\n- Line (a) vs Line (b): applying refinement on the few-shot examples markedly boost the accuracy (+2.17%).\n    \nThese are the main results of the ablation study that claim the importance and necessity of every details in our framework. \n\n**Question 5: How much the BLIP and filtered information is helping the LLM？**\n\n**Response 5:**\n\n- How much the BLIP is helping the LLM?\n      \n    We compare the results of with and without BLIP-2 under 4-shot setting. Line 1 in the table below is the baseline reuslt of using just the captions (without \"Image Information\"). Introducing BLIP-2 improves the accuracy by 1.96%.\n         \n    |No.|BLIP-2|Caption|ACC|\n    |:-|:-----|:------|:--|\n    |(1)| No  | OFA-Large |43.21|\n    |(2)| Yes | OFA-Large |45.17 (+1.96)|            \n\n- How much the filtered information is helping the LLM?\n      \n    Including all the information increases the accuracy by only 0.08% (Line 1 vs Line 2), while using refinement module to produce the filtered information increases the accuracy by 2.46% (Line 1 vs Line 3). The refinement module alone contributes +2.38% to the final performance (Line 2 vs Line 3).\n         \n    |No.|Refinement|Image Information|ACC|\n    |:-|:-----|:--------|:--|\n    |(1)| No  |Original question + BLIP-2| 45.17 |\n    |(2)| No  |All the information|45.25|\n    |(3)| Yes |Filtered information| 47.63 |\n\n**Question 6: Missing references**\n\n**Response 6:**\n    \nWe appreciate for your suggestions and will include them in the Relate Work section. In the first paper, \"Language Models are Visual Reasoning Coordicators\", the LLM simultanuously reads the output of two VLMs (BLIP and OFA) to predict the answer. In our pipeline, different VLMs are progressively made use of by the LLM. It could be our futher work to build a framework where LLMs automatically decides the paradigm of how to utilize different VLMs for reasoning. The second paper \"Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation\" focuses on generating logically consistent question-answer pairs. In our paper, we encourage the LLM to explore different knowledge fields that are somewhat helpful to answering the questions. But the logical consistency is a great insight for us to further improve the generation of new questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting design choice requiring very minimal training of interfacing LLM and VLMs, instead of training adapters for image input to llm like llava etc.\n\n- The accuracies on the VQA datasets look promising. \n\n- They show their approach holds with various LLMs. ", "reasons_to_reject": "- The results section could be organized better. Some lines like “Line (f) converts line (h) to few-shot setting” - are lacking details. What does it exactly mean? Fewer context? Fewer VQA training data points? How are they sampled? \n\n- Another baseline to include is the accuracy of the LLM to answer the question without any image information or just the caption. Table 6 compares the accuracy to using BLIP-2 in answering the question. Since their final method has the LLM answering the question, it would be nice to see how much the BLIP and filtered information is helping the LLM. \n", "questions_to_authors": "- It is unclear what parts are trained and what aren’t. What parts of what models are trained when training the refinement using the VQA annotations? Is the entire pipeline trained end-to-end? \n- For the baselines, are the VLM models also trained on the VQA datasets used?", "paper_topic_and_main_contributions": "Authors design a pipeline that prompts LLMs to ask clarification questions to a vision-language model for the LLM to answer a question accurately about an image. They show that this outperforms simply using the vision-language models for answering and some other similar LLM-based methods on the OK-VQA and AOK-VQA datasets. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See the weaknesses section."}, "extra_text": {"Missing_References": "Some related work that might be important to look at:\n- https://openreview.net/pdf?id=kdHpWogtX6Y  - LLM coordinates information from VLMs. \n- https://arxiv.org/abs/1909.04696 - they generate entailed questions to the question asked to improve VQA consistency. "}}, {"review_id": "iL0PiBaWya", "reviewer": "Reviewer_cxzi", "summary": "", "questions": "How do you judge the improvement compared with complexity of the method?\n\nPlease answer my concers in `Reasons To Reject` section, if most of question are resovled, the score could be increased.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions 1: Complexity vs performance**\n\n**Response 1:**\n\n- The complexity: \n    - Prophet also needs a frozen captioning model and requires to train different VQA models for different benchmark. \n    - In our framework, we train the refinement module in Figure 2, but freeze the VQA and captioning models in the inquery module in Figure 2. We conduct ablation study to show the effectiveness and necessity of our refinement module. As reported in Table 2 and Table 5(a) in our paper, using all the information **without refining** (\"All\" in Table 5(a)) results in an accuracy of 58.03% (only +0.12% compared to Prophet). By introducing our refinement module, the accuracy increases to 59.34% (\"Prophet+Ours\" in Table 2, +1.43% compared to Prophet). \n    \n- The inference speed:\n\n    The inference speeds of Prophet and our method are nearly identical. When requesting a OpenAI API (for GPT-3.5), both Prophet and our method cost 1.1-1.2 second/instance in the 20-shot, single query setting. In the ensemble setting, both speeds increase to 4.5-5.2 second/instance for ensemble setting. \n    \n    Excluding the influence of API, both the VQA model in Prophet and the refinement module in our method require approximately 0.02 second/instance.\n\n- The improvement of our method: \n\n    We would like to further emphasize the improvements of our method compared to Prophet in Table 3 in our paper as follows:\n\n    - For A-OKVQA, our single query result (57.9%, \"Prophet+Ours\" in Table 3) is already comparable to the ensemble result of Prophet (58.2%, \"Prophet (ensemble)\") with a significantly lower inference cost. Furthermore, our ensemble performance reaches 59.8% (\"Prophet+Ours (ensemble)\"), marking a 1.6% improvement.\n\n    - For OK-VQA, we obtain an increment of 1.4% over Prophet in the single query setting (\"Prophet+Ours\"). In the ensemble setting, we can observe a performance improvement of 0.2% (\"Prophet+Ours (ensemble)\"). \n        - One factor that might contribute to this reduced improvement in the ensemble setting is the inherent characteristics of the Prophet method. Prophet relies on a fine-tuned VQA model to generate answer candidates from a closed-set of predefined answers (all the answers present in the training dataset). \n        - This constraint is particularly noteworthy for OK-VQA, given its lack of answer diversity compared to A-OKVQA. The OK-VQA dataset has 11,508 distinct answer categories, with each question having a maximum of 5 different answers. On the other hand, the A-OKVQA dataset contains 29,631 unique answer categories, with each question having up to 10 distinct answers.\n        - By integrating more few-shot examples, the answer candidates will present a dataset bias. For OK-VQA dataset, we hypothesize that Prophet can benefit from the biased candidates in the ensemble setting. \n\n    - We conduct further examination on the ensemble behavior on OK-VQA dataset and find out that Prophet indeed benefits more from ensemble compared to the other baselines. The table below shows the results of the single query setting and the ensemble setting. For Prophet, the gain of the ensemble setting over the single setting is 3.19% (Line 5), which is more than doubled compared to results on Line 1 to Line 4, and is nearly doubled compared to Line 6. This shows an inconsistency regarding the gain of ensemble setting over single setting, and supports our hypothesis that Prophet benefits more from the ensemble setting compared to other methods. \n    \n        |No.|    Method     | Single        | Ensemble      | Gain of Ensemble over Single|\n        |:--|:--------------|:--------------|:--------------|:-----------------:|\n        | 1 | PICa          | 49.67         | 51.36         | 1.69             |\n        | 2 | PICa+ours     | 53.76 (+4.09) | 54.91(+3.55)  | 1.15             |\n        | 3 | PromptCap     | 53.50         | 54.16         | 0.66             |\n        | 4 | PromptCap+ours| 54.42 (+0.92) | 55.01 (+0.85) | 0.59             |\n        | 5 | Prophet       | 57.91         | 61.10         | **3.19**         |\n        | 6 | Prophet+ours  | 59.34 (+1.43) | 61.30 (+0.20) | 1.96             |\n\n    - Therefore, we consider the reduced improvement in the OK-VQA + ensemble setting to be a special case, whereas in most other cases our approach leads to significant improvements compared to Prophet.\n\n**Question 2: Questions about the method and experiments**\n\n**Response 2:**\n\n- **(1) Line 214, the method for calculating scores in Table 1 in our paper.**\n\n    The reported scores are the soft accuracy of VQA according to the paper [VQA: Visual Question Answering](https://arxiv.org/pdf/1505.00468.pdf), and are implemented according to [github link](https://github.com/GT-Vision-Lab/VQA). The scores in Table 1 in our paper are calculated under the 4-shot setting. Each column represents:\n\n    - \"Original\": reasoning with the original question only; \n    - \"All\": using all the information for reasoning, including the summarization of the original question and its BLIP-2 answer and the summarizations of all the generated question-answer pairs for an instance; \n    - \"Random\": randomly select one summarization from the \"All\" setting above; \n    - \"Best\": the upper bound score among information in the \"All\" setting. We use the highest accuracy of each instanse as the \"Best\" score. \n\n   This table demonstrates the effectiveness of the generated questions together with BLIP-2 answers, and the necessity of a refinement module. \n\n- **(2) Lines 264-265 and details in Appendix A in our paper.**\n\n    Please refer to Appendix D.3 for examples of the prompt $P_{i,q}$. The $ACC_{soft}$ for each intance is calculated as:\n\n     $ Acc_{soft}(ans) = min(\\{ \\frac{number \\quad of \\quad matching \\quad answers}{3}, 1\\}) $\n         \n    We will move the relevant content back to Section 3 in our paper for better understanding if permitted extra space. \n\n- **(3) Lines 275-276, how is $z_q$ calculated?**\n        \n    $z_q$ is the contrubition score of the original question *q* with BLIP-2 answers. It is not calculated with *q* only. We use the summarization of the original question *q* together with its BLIP-2 answer as the textual input for the filter. \n\n- **(4) The example prompt is not consistent with Figure 2, where the figure does not display caption.**\n     \n    Sorry for causing the confusion. We omit the caption in Figure 2 because it is a common field among our method and the baselines. We will put the caption back in the answering module in Figure 2 in our revised version. Thanks for pointing that out.\n\n- **(5) The inconsistent results in Tables 2 and 3 in our paper.**\n    \n    Most of the results in Table 2 in our paper are reported with the single query setting, except for those marked by '\\*' because we cannot find any single query setting results publicly. Since most of the methods report their single query results, we also report the single query results in Table 2 in our paper for fair comparison. We stated in the caption of Table 2 in our paper that '\\*' refers to multi-query ensemble results. In Table 3 in our paper, the single query results of Prophet (57.9%) and Prophet+ours (59.3%) are the same as those in Table 2.  \n\n- **(6) For Table 4, the application of answers from 'Blip2' remains unclear.**  \n        \n    Sorry for the unclear design of the table. We reform Table 4 as below:\n        \n    | Method | Number of Shots| Caption | Image Information | Tags | Refinement on Query Instance | Refinement on Few-shot Examples | ACC |\n    |---------------------|---|-----------|---------------------------|-----|-----|-----|----------|\n    | Ours                | 16| PromptCap | Refined information    |&#10004;|&#10004;|&#10004;|  54.42(a)|\n    | Ours                | 16| PromptCap | Refined information       |&#10004;|&#10004;| -   |  52.25(b)|\n    | Ours                | 4 | PromptCap | Refined information       |&#10004;|&#10004;| -   |  50.19(c)|\n    | Ours                | 4 | OFA-Large | Refined information       |&#10004;|&#10004;| -   |  48.43(d)|\n    | Ours                | 4 | OFA-Large | Refined information       | -   |&#10004;| -   |  47.63(e)|\n    | Ours w/o refinement | 4 | OFA-Large | All the questions + BLIP-2 answers| -   | -   | -   |  45.25(f)|\n    | Ours w/o refinement | 4 | OFA-Large | Original question + BLIP-2 answers| -   | -   | -   |  45.17(g)|\n    | Ours w/o BLIP-2     | 4 | OFA-Large | Original question         | -   | -   | -   |  44.86(h)|\n    | BLIP-2              | 0 | -         | -                         | -   | -   | -   |  40.70(i)|\n\n    Recalling the prompt template for answer reasoning in Section 3.3 in our paper, the template consists of two parts: \n\n    - Few-shot in-context examples (Line 287 to Line 288 in our paper), denoting as ***Template-Few-shot***:\n    \n            Image information: S^n \\n Caption: C^n \\n Question: q^n \\n Answer: a^n\n\n    - A query instance (Line 290 to Line 291 in our paper), denoting as ***Template-Query***:\n\n            Image information: S \\n Caption: C \\n Question: q \\n Answer: \n    \n    With the in-context examples, the LLM is required to predict the answer for the question q in the query instance. \n\n    Here are some explanations of the table above:\n\n    - \"Caption\": the models used to generate the image captions; \n    - \"Image Information\": the resources of our image information, \"Refined information\" = applying refining on \"All the questions + BLIP-2 answers\"; \n    - \"Tags\": whether to use image tags. According to [PICa](https://arxiv.org/pdf/2109.05014.pdf), image tags are place right after the captions in the template above;\n    - \"Refinement on Query Instance\": applying refining to the \"Image information S\" in ***Template-Query*** mentioned above;\n    - \"Refinement on Few-shot Examples\": applying refining to the \"Image information S^n\" in ***Template-Few-shot*** mentioned above;\n    - Line (i): BLIP-2 refers to directly using BLIP-2 for OK-VQA.\n    - **Response to the questions of the application of BLIP-2 answers: Are they used for few-shot examples? If so, how does 'ours-w/o' function?** \n        - The BLIP-2 answers are not used for few-shot examples. They are used for generating the image information (\"S^n\" and \"S\" in the template above) used in the \"Image information\" field. \n        - The \"ours-w/o\" in the original Table 4 in our paper is moved to Line (h) as \"Ours w/o BLIP-2\" in the nre table above. The image information (\"S^n\" and \"S\" in the template above) refer to the summarization from quesitons and the corresponding BLIP-2 answers. While in Line (h), we directly place the original questions without BLIP-2 answers in the \"Image information\" field in both ***Template-Query*** and ***Template-Few-shot***. \n\n    Here are the brief conclusions of this table:\n\n    - Line (e) vs Line (f): refinement module significantly improves the accuracy (+2.38%);\n    - Line (g) vs Line (h): BLIP-2 answers help (a bit) the LLM to reason for answers (+0.31%);\n    - Line (d) vs Line (e): tags are helpful (+0.80%);\n    - Line (b) vs Line (c): increasing the number of shots significantly improves the accuracy (+2.06%);\n    - Line (a) vs Line (b): applying refinement on the few-shot examples markedly boost the accuracy (+2.17%).\n    \n    This is the main results of the ablation study that claims the importance and necessity of every details in our framework. \n\n- **(7) Lines 414-415, an improvement from 40.7 to 47.63 seems unreasonable.**\n\n    Sorry for causing the confusion. As we stated in our paper in Line 414-416, \"there is a continuous gain from 40.70% to 47.63% along with the participating of more detailed information\". It is not a straightforward comparison bewteen the two different paradigms. \n\n    Reffering to the new table above, Line (i) (\"BLIP-2\", 40.70%) refers basis of the VLM paradigm that directly uses the BLIP-2 for OK-VQA task; Line (h) (\"Ours w/o BLIP-2\", 44.86%) refers to the basis of the LLM-based paradigm that cooperates VLMs together with LLMs for OK-VQA task. As we integrate the BLIP-2 answers into our \"Image Information\", we provide the vanilla BLIP-2 accuracy in this table as the starting point. Seen from the new table above, we progressively make used of accessible resources and increase to 47.63% from the basis of the VLM paradigm (Line (i), 40.70%) and the basis of the VLM+LLM paradigm (Line (h), 44.86%) as follows:\n\n    - Line (i) vs Line (h): the comparison of baseline scores for the two paradigms. The VLM paradigm directly uses BLIP-2 for the OK-VQA task (Line (i), 40.70%), and the LLM-based paradigm cooperates VLMs together with LLMs for the OK-VQA task (Line (h), 44.86%);\n    - Line (h) to Line (g): including BLIP-2 answers in the \"Image Information\" to the vanilla VLM+LLM paradigm in Line (h) (from 44.86% to 45.17%);\n    - Line (g) to Line (f): further adding image information obtained from our generated questions with BLIP-2 answers (from 45.17% to 45.25%);\n    - Line (f) to Line (e): applying refining on the \"Image information\" (from 45.25% to 47.63%).\n\n- **(8) Explanation of 'n' and '$ \\mathcal{E} $*'.**\n    - 'n' refers to the number of shots. It is renamed as \"Number of Shots\" in the new table above.\n    - '$ \\mathcal{E} $*' refers to the format engineering on the few-shot examples, it means to apply filters to refine the image information in the few-shot examples from the training set. We rename it as \"Refinement on Few-shot Examples\" in the new table above.\n\n**Question 3: The case study**\n\n**Response 3:**\n\n- The reasoning procedure of Prophet involves two stages: \n    - Stage 1: the VQA model produces 10 candidates with confidences; \n    - Stage 2: the LLM predicts the final answer considering the provided candidates. \n- In Figure 3 (1) in our paper, the actual prediction of Prophet is \"ham and cheese\". \n\n    The \"grilled cheese (0.66)\" is just one of the intermediate results from \"Stage 1\" above, while the \"ham and cheese\" is the final prediction of Prophet in this case. This case implies that even if the VQA model produces correct candidates (\"grilled cheese\") with reasonable confidence (\"0.66\"), the LLM may not follow the suggestions when performing reasoning. \n\n    But in our method, we can rectify this wrong LLM prediction and generate \"grilled cheese\" (prediction of Prophet+ours) instead of \"ham and cheese\" (prediction of Prophet).\n\n**Question 4: Missing reference**\n\n**Response 4:**\n\n- \"Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering\" is a contemporaneous work to ours, but we'll include it in the revised version.\n- \"From Images to Textual Prompts\" is already cited and is mentioned as \"Img2Prompt\" in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper convincingly illustrates the necessity of providing nuanced, query-specific information to complement the coarse, generalized image captions diretly output from caption model without any query information. It recognizes a shortcoming in existing approaches that convert image to text for application in large language models to solve visual question-answering tasks. The one-off conversion process often loses important details, underscoring the importance of query-related information to infuse sufficient visual details into the Language Learning Model .\n\nThe proposed framework is thoughtfully and intuitively designed, reflecting an in-depth understanding of the problem at hand.\n\nThe research is supported by thorough and comprehensive experiments. It includes a comparative analysis with most related works, and an exhaustive ablation study, all of which affirm the effectiveness of the proposed method.", "reasons_to_reject": "The study presents a commendable motivation and idea, yet the implementation of the method seems disproportionately complex given the relatively marginal improvement. The method encompasses several modules, including a caption model, VQA model, filter, and large language models. However, the advancement over 'Prophet' does not appear to warrant the complexity and increased inference time of the method.\n\nThe articulation of the method and experiments is not clear. For instance, (1) at line 214, the method for calculating scores in Table 1 is unclear; (2) lines 264-265 lack sufficient detail, relegating crucial information to the appendix and leaving the motivation and methods for obtaining ground-truth VQA accuracy unclear; in Appendix A, P_{i,q} and ACC_{soft} aren't sufficiently explained; (3) at lines 275-276, it is unclear how z_q is calculated. While Equation 8 appears to use all information, it remains unexplained if only 'q' is used to get z_q; (4) in Section 3.3, the example prompt is not consistent with Figure 2, where the figure does not display `caption`; (5) the 'Prophet' results in Tables 2 and 3 are inconsistent; (6) for Table 4, the application of answers from 'Blip2' remains unclear. Are they used for few-shot examples? If so, how does 'ourts-w/o' function? (7) Table 4 should provide an explanation for the meaning of 'n' and 'E' in its caption; (8) the statement at lines 414-415 claiming an improvement from 40.7 to 47.63 seems unreasonable, as 40.7 directly uses 'Blip2', a VLM, while 47.63 uses a different method involving LLM. The improvement could be attributed not just to more detailed information but also enhanced reasoning ability.\n\nIn the case study, Figure 3 (1) suggests that 'Prophet's' output of 'grilled cheese' with a probability of 0.66 should indeed be the correct answer", "questions_to_authors": "How do you judge the improvement compared with complexity of the method?\n\nPlease answer my concers in `Reasons To Reject` section, if most of question are resovled, the score could be increased.", "paper_topic_and_main_contributions": "This paper concentrates on knowledge-based Visual Question Answering tasks. It enhances existing methods by introducing a novel framework that extracts intricate, question-related caption from images. The core strategy involves deconstructing questions into subquestions based on the given captions and query, extracting answers for each subquestion, and thereafter generating a comprehensive caption that incorporates all this information. This enhanced information, coupled with the original question and caption, is fed into the Language Learning Model (LLM). The infusion of this supplemental image information amplifies the LLM's capacity for reasoning, enabling it to generate more precise and informed answers.\n\nThe primary contributions are a unique, comprehensive framework and a experimental results that provide extensive validation of the methodology proposed.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Appendix A, line 851, should be y_{z^k} instead of y{_z}^k"}, "extra_text": {"Missing_References": "There are certain works that directly use caption as additional information and use the same VLM (instead of LLM) to answer the vqa tasks should be metioned somehow.\n\n@misc{awal2023investigating,\n      title={Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering}, \n      author={Rabiul Awal and Le Zhang and Aishwarya Agrawal},\n      year={2023},\n      eprint={2306.09996},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n\n@misc{guo2023images,\n      title={From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models}, \n      author={Jiaxian Guo and Junnan Li and Dongxu Li and Anthony Meng Huat Tiong and Boyang Li and Dacheng Tao and Steven C. H. Hoi},\n      year={2023},\n      eprint={2212.10846},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}"}}]

Author rebuttal:

Meta-review: This paper introduces a framework that enables LLMs to proactively ask relevant questions to unveil more details in the image and evaluate the method on two visual QA datasets, OK-VQA and A-OKVQA. I think the idea explored in this paper is interesting, the performance gain given the complexity of additional modules seems somewhat limited, as pointed out by the reviewers. I am also curious whether this method can give further improvements on top of existing models on other tasks, as suggested by the reviewer QtV8
Decision: Accept-Findings