TOA: Task-oriented Active VQA
Authors: Xiaoying Xing
Mingfu Liang
Ying Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge-based visual question answering
task-oriented
active image understanding
large language model
visual reasoning
multi-round dialogue
TL;DR: We propose a method that actively collects visual information in a task-oriented manner.

Abstract: Knowledge-based visual question answering (VQA) requires external knowledge to answer the question about an image. Early methods explicitly retrieve knowledge from external knowledge bases, which often introduce noisy information. Recently large language models like GPT-3 have shown encouraging performance as implicit knowledge source and revealed planning abilities. However, current large language models can not effectively understand image inputs, thus it remains an open problem to extract the image information and input to large language models. Prior works have used image captioning and object descriptions to represent the image. However, they may either drop the essential visual information to answer the question correctly or involve irrelevant objects to the task-of-interest. To address this problem, we propose to let large language models make an initial hypothesis according to their knowledge, then actively collect the visual evidence required to verify the hypothesis. In this way, the model can attend to the essential visual information in a task-oriented manner. We leverage several vision modules from the perspectives of spatial attention (i.e., Where to look) and attribute attention (i.e., What to look), which is similar to human cognition. The experiments show that our proposed method outperforms the baselines on open-ended knowledge-based VQA datasets and presents clear reasoning procedure with better interpretability.

TOA: Task-oriented Active VQA
Xiaoying Xing Mingfu Liang Ying Wu
Northwestern University
Department of Electrical and Computer Engineering
(xiaoyingxing2026, mingfuliang202000n nowthwestern.co edu
yingsu@horthwwstorm.. edu
Abstract
Knowledge-bard baaeed visual question answering (VQA) requires external knowledge to answer the question about an image. Early methods explicitly retrieve knowledge from external knowledge bases, which often introduce noisy information. Recently large language models like GPT-3 have shown encouraging performance as implicit knowledge source and revealed planning abilities. However, current large language models can not effectively understand image imputs, thus it remains an open problem extract the image information and input large language models. Prior works have used image captioning and object descriptions to represent the image. However, they may either drop the essential visual information to answer the question correctly or involve irrelevant objects to the task-of-interess To address this problem, we propose to let large language models make initial hypothesis according to their knowledge, then actively collect the visual evidence required verify the hypothesis. In this way, the model can attend to the essential visual information a task-oriented manner. We leverage several vision modules from the perspectives of spatial attention (i.e., Where to look) and attribute attention (i.e., What to look), which is similar to human cognition. The experiments show that our proposed method outperforms the baselines on open-ended knowledge-basse VQA datasets and presents clear reasoning procedure with better interpretability.
Introduction
The problem of knowledge-based visual question answer (VQA) 21 requires open-world knowledge to answer the question about image, which is more challenging than traditional VQA tasks 3 since the model needs to extract relevant external knowledge and then perform joint reasoning on question, image, and the knowledge. To this end, most existing solutions explicitly retrieve external knowledge from various sources like Wikipedia [4] [5] 6. 7 and ConceptNet [8]
4 However, mainly limited by the explicit knowledge-retrieval stage [10]. First, the
their performances required knowledge may not be successfully retrieved, leading to insufficient information to answer the question. Second, the retrieved knowledge may be noisy and introduce irrelevant information,
hence degrading the performance of the joint reasoning stage.
Recent works [ 10 12
explore to use the large language models (LLMs) like GPT-3 [14]]]
as implicit knowledge base, since they have shown powerful abilities in knowledge retrieval [[] and reasoning [16]. However, it remains challenge to enable a pretrained LLM to further exploit the visual information, as this requires large computation resources to fine-tune the LLM on vision- language tasks [8]. To sidestep this challenge, captioning- based methods translate the image into a caption such that the LLM can understand image information like a textual prompt. Despite the significant improvement compared to the explicit retrieval-based methods, their performances still limited because the caption may usually drop essential visual information required to answer the question. On the other hand, the detection-based based method leverages object detectors to extract all
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the candidate visual concepts in the image and provide the LLM with textual object tags [ 6 or brief description [2]. In this way, the LLM is provided with more exhaustive visual information. Nevertheless, many of the detected objects are irrelevant to the question and introduce noisy visual information to the language model. Another category of works utilizes the in-context learning ability of LLLM to decompose the input questions into modular programs and execute on set of pre-defined vision functions 19 20]. However, they directly generate the answers through the execution of the program with simple symbolic logic, which limits their application to knowledge-basee based VQA.
To help the LLM better understand the image, we propose a task-oriented active (TOA) VQA method to actively and progressively acquire the visual information according to the task, as depicted in Figure [] In general, our TOA method imitates the human cognitive process of reasoning, hypothesis, and verification. It consists of LLM as the scheduler to make task planning and decision, and visual models to execute the order from the scheduler. Based on the understanding of the input question and the open- gpen-world knowledge obtained during pretraining, the LLM makes hypotheses on the answer and actively acquires relevant visual evidence from the visual model to verify the hypothesis. As a consequence, the LLM in our TOA method can integrate open- world knowledge and the collected visual evidence to answer the question, in contrast to previous works that directly obtain the answer only by generated program 20 19]. On the other hand, our method extracts the visual information in a task-oriented manner instead of looking at the whole image aimlessly. hence attending to the essential visual information more accurately and reducing the irrelevant information. Moreover, differing from the existing methods that generate a complete program at once and solely based on the question [9], the LLM in our TOA interacts with the vision modality progressively through multi-round dialogue and decides the next instruction based on the previous interaction experience. When the visual evidence conflicts with the hypothesis, the LLM can make new hypothesis or ask for other visual evidence to make the final decision. Therefore, even when the vision model makes a mistake, our method can still verify the hypothesis from a different angle.
To verify the effectiveness of the proposed method, we evaluate our methods and other competitive methods OK-VQA 2 and A-OKVQA (2 datasets, where both datasets ask questions that require open-world knowledge beyond the image to obtain the correct answer. The empirical results show that our method outperforms the comparison methods open-ended question answering, indicating our capability to better leverage implicit knowledge and capture essential visual information.
Our contributions are summarized as follows:
We propose a new method to actively collect visual information in a task-oriented manner, which can more accurately attend to the essential information in the images and reduce the introduction of irrelevant information.
We design a human-like cognitive process, i.e., reasoning, hypothesis, and verification, to better activate the reasoning abilities of the LLM through clear problem-solving procedures and high-quality visual information.
We develop multi-round dialogue approach to solve the problem progressively and decide the next step dynamically, which has clear answering process and better tolerance of mistakes in previous steps.
2
Related Work
Knowledge-based Visual Question Answering. Early works 21
22 2 24 retrieve supporting
knowledge from fixed knowledge hases annotated with the required facts to answer the questions.
Subsequent open-knowledge VQA tasks [2] [ need to acquire open-world knowledge instead of
fixed knowledge bases. One category of solutions to open-knowledge VQA [4] 7 g 9 explicitly
retrieves external knowledge from various sources, such as Wikipedia (25]. ConceptNet (26) and
Google Images 4. The performances of the explicit retrieval- based methods are mainly limited by the knowledge retrieval stage, since it may either fail retrieve the required knowledge or
introduce noisy information. Another line of works [O, 12] 13 leverages LLM as an implicit knowledge source. They often translate the visual information into captions
[ or object-level significantly outperforms
descriptions [1]. Although the pioneering caption- explicit retrieval-based methods, it is still limited since the generic caption may drop some essential
method PlCa [I]
visual information required to answer the question. To remedy this issue, PromptCAP [13 generates

---

Task Scheduler
Question:
What property that allows images seen on an object's surface like the ornament'ss
Vision Executor
The caption does not provide useful Information to answer the question. We need to collect more visual information.
Task analysis
Caption: person holding an orange front of Christmas tree
Reasoning: Since the question asks about ornament, first locate image.
locate(ornament)
Hypothesis: None
Verification: locate(ornament, ment)
Visual verification
Located ornament.
Reasoning: The property that allows images to be seen on an objects surface is called reflection. We can verify if the ornament has a reflective surface.
filter (ornamett,,,, reflective)
Hypothesis: [reflection]
Verification: filterforammnt, reflective)
Yes, the ornament reflective.
Reasoning: Since the omamenn has reflective surface, it verifies the hypothesis.
Answer decision
Answer: reflection
The overview of our task-oriented active (TOA) VQA method. TOA consists of task scheduler and vision executor which communicates through multi-round dialogue. The task scheduler actively analyzes the existing information. Then it either requires specific visual information to help make further decision, or makes hypotheses on the answer and asks for visual evidence to verify its hypothesis. On the other hand, the vision executor follows the instruction from the scheduler to perform different kinds of visual verification and then returns the execution results to help the scheduler come up with the next decision. The gray arrows indicate the interaction process.
Figure 1:
more relevant image captions conditioned on the input question. However, how to efficiently extract image information and translate into textual prompts for LLMs remains an open problem. Different from the previous works that prompt LLMs with passively extracted visual information, we propose to let the LLM actively acquire the essential visual information task-oriented manner.
Promping Large Language Model for Reasoning. Recent works have found that LLMs emerge reasoning abilities (27 [6 28 29). When prompted with in-context examples of chain-of-though rationales, LLMs have shown impressive performance on tasks that require reasoning abilities [16] 29 30). Subsequent work (28) further explores the zero-shot reasoning ability of language models by simply prompting with 'Let's think step by step'. Another family of works excavate the task- planning ability of large language models 19 20
They prompt the LLM to generate programming over sub-tasks which are executed by pre-defined sub- sub-modules. Among them Visual Programming [19] achieves impressive performance with good interpretability on GQA dataset [4), which is designed to examine the compositional reasoning ability. However, since the answer is generated through the program execution, it has limited application on open-knowledge VQA.
32 3
Vision-language Model. Vision-language pre-training models have made significant progress in 39 40)
They first pretrain the model on large-scale image-text data to learn good image and language representation, then fine-tune on various down-streaming tasks. They exhibit strong transfer ability and achieve cate-of-the-art on various vision-language tasks. Another line of works train the model by contrastive learning, aiming to align visual concepts and language representations
recent years 35 36 37
38
[ 42. Besides, BLIP-2 43] proposes a pre-training strategy first learn vision- lasiun-language representation learning from a frozen image encoder, then learn vision-to-languaae generative learning from a frozen language model. However, these vasion-language pre-training models require large vision-language datasets and computation resource to fine-tune. Our method uses some pre-trained vision- language models to attend to the relevant region on the image and further collect the required visual information.
3

---

3 Method
In this section, we introduce our task-oriented active (TOA) VQA which simulates the cognitive process of humans involving three essential parts: reasoning, hypothesis, and verification. To this end, we design two agents, the task scheduler and the visual executor. These two agents engage in multi-round dialogues to communicate with each other. The scheduler parses the question, analyzes the required visual information, makes proper hypotheses to the visual executor for verification, and decides on the final answer with comprehensive consideration. The visual executor consists of set of vision functions and collects visual information according to the requests from the scheduler. The illustration of the proposed method is shown in Figure T
3.1 Task Scheduler
The task scheduler leverages the natural language understanding and chain-of-thought [T6] reasoning abilities of LLM, which plays the role of task analysis, planning, and decision-making Given the imput question and brief description of the image generated by a captioning model [13. the LLM first parses the question and the information provided by the caption. Then it either makes an initial hypothesis and collects visual evidence to verify the hypothesis, or acquires more visual information when there is insufficient information to make any hypothesis. In the subsequent steps, the LLM makes further analysis considering the collected visual information. It can adjust the current hypothesis according to the previous conversation and ask for further verification. When the hypothesis is confirmed by the visual evidence, it decides the final answer and ends the dialogue.
Since the language model generates natural language based on a probabilistic model, they tend to generate diverse contents. To facilitate the interaction between the language modality and the vision models, we restrict the output format to combination of free-form natural language and symbolic representation. Specifically, the output format consists of four parts: REASONING, HYPOTHESIS, VERIFICATION, ANSWER the following template:
REASONING: r' HYPOTHESIS: [h,, h2, ...] VERIFICATION: function(args) ANSWER:
At each step, the instructions generated by the scheduler contain all four parts above to guarantee format consistency. Each of them can be None when inapplicable. REASONING is the analysis basis of the subsequent parts. It contains a chain-of-the -ohthh reasoning process that illustrates the hypothesis and requires visual verification. HYPOTHESIS is list of possible answers the question. There can be multiple hypotheses at each time, while when the current information is insufficient to make any hypothesis, then HYPOTHESIS would be marked as None. VERIFICATION specifies the instruction to call the vision functions. The instruction is constructed in Python similar to [9], which consists of the name the function and the specified args. ANSWER is the prediction result to the question. is only specified when the scheduler has decided on the final answer, otherwise marked as None.
3.2 Visual Executor
Despite the strong reasoning and language understanding abilities of LLMs, they have limited capability for image understanding as fine-tuning on vision-language data requires large computation resource. To alleviate this issue, we leverage several vision models to execute the requests from the scheduler and collect the required visual evidence to verify the hypothesis. By doing so, the LLM can actively acquiring specific visual information that it believes to be crucial for enriching its understanding of the image to make an appropriate decision. Moreover, our proposed method extracts the visual information in a task-oriented manner, i.e., tailored to the question, which can more efficiently attend to the useful information and reduce the noisy information than existing methods.
Motivated by human cognition, we correspondingly design our visual executor to have the task- oriented visual understanding module from two aspects, spatial attention (i.e., Where to look) and attribute attention (i.e., What to look). Instead of looking at the whole image aimlessly, our task- oriented image understanding module first locates the region of interest that is most relevant to the question. Then it extracts different image textures according to the query, including attribute-level features and semantics-level features. The functions of the vision executor are shown in Figure We leverage several pre-trained vasion-language models to implement the visual executor module, which will be elaborated in Section 4.1

---

Spatial
locate(object)
Returns an image patch containing the object
relatc(object, relation)
Locates the region-of-intesn -intte-of-interesttt according to the relationship with the specified object
crop(direction)
Crops image the specified direction
Attribute
attribute@object, attribute)
Returns the queried attribute of the object
filter(objcct, property)
Clussifies the object possesses the property
locate(clephant)
relate(elephant. left)
exist(object)
Detects the target object exists
query(property)
Answers simple queries about the image region
count(abject)
Returns the number of the target object
crop(above)
Figure 2: Description of the spatial and attribute vision modules (left) and execution examples of spatial function (right).
3.3 Multi-rounds Interaction
In our proposed method, the task scheduler and the visual executor can interact with each other through multi-round dialogue. In this way, the scheduler in method acquire the essential information progressively, and can also dynamically decide the next step based on the previous conversation. When there multiple hypotheses that can be the possible answers to the question, the scheduler will verify each of them sequentially. of the hypotheses, then this hypothesis is excluded from all the hypotheses. However, sometimes all the hypotheses may be excluded by the visual verification. This may be caused by mislcading wrong. In
the visual verification conflicts with one
results of the vision model or because all the hypotheses made by the scheduler such cases, the scheduler needs to jointly consider the common sense knowledge and the collected visual information. Thus we let the scheduler consider all the information hand to either ask for more visual information to make a new hypothesis, or maintain the initial hypothesis and give the answer with the highest probability. Compared the methods that directly generate the answer the program execution 19 20 and fully rely the results of the vision models, our proposed method encourages the L.M to actively seek essential visual evidence oriented to the question a sequential ceantheis--veificattin procedure. This may reduce the impact of the inferior results of vision models especially when asked to perform challenging vision tasks, since the LLM can make an alternative request which is more feasible to vision models.
When the scheduler confirms the answer through visual verification, stops the dialogue and outputs the answer. Since the scheduler decides the cessation of the dialogue accordingly, in contrast to the previous works that end the dialogue after fixed rounds [44,, our hypothesis-veriie verification process through the dialogue can be more efficient and reduces redundant interactions.
3.4 Prompting Strategy
Now we introduce our prompting strategy to activate the reasoning ability of LLM. It consists of brief description of the task requirements, instructions leverage the available vision models, and a few in-context examples to enforce the LLM to generate the output in consistent format. and prompts the task
Specifically, the task description emphasizes the idea of arpothesis-verifccatinn scheduler to leverage the vision functions to gather information. It is designed as follows:
have question about an image. Each time you need to make a hypothesis and verify by collecting visual information. You can use the following visual functions to gather information. You can only call one vision function at a time. The answer should be very concise.
The instruction for leveraging the available vision models Python-like pseudo code that describes the vision function, inspired by the API design in ViperGPT (20). In this way, the scheduler can
5

---

better understand the function of the vision models, and it facilitates the scheduler to call the required vision functions. An example is shown follows:
def filter Lobject:strr,
presupposes the existence of -bbjcc.
return True if object possesses the .propertt else False.
We also provide the large language model with a few in-context examples of the multi-round dialogue to guarantee the format consistency of the textual output. Previous works [45 [T] have indicated that in-context example selection is essential to the result. To find the most representative examples, we extract the image and question embedding using CLIP model 41. Then we cluster the image-question pairs with the feature embedding and select the examples that are closest to the cluster centers as the representative examples. In the inference stage, we follow the in-context example selection approach in [[]] For each input data, we compute its cosine similarities with the available examples in the embedding space and select the top ls examples with the highest similarities. Specifically, for the training data X = {wu, 2i we denote the fused image and question embeddings
as = (eimab Lemb), and the clustering centers as C = {Ci, C2, ... ,m The representative examples of each cluster oj are selected as follows:
arg max 24
|15||2||2111
Then given a testing input image-question pair (v, ②) and 2 (Vemb. 9emb), the top in-context examples with maximum similarities are:
ZTr
I == argTopK ie (1. | |rsl12
(2)
4 Experiment
4.1 Experiment Settings
Datasets. We mainly evaluate our proposed method on knowledge-based VQA dataset VQA MI and conduct several experiments on A-OKVQA [2] supplementary. OK-VQA is commonly used knowledge-hased VQA dataset that contains 14,055 image-question pairs associated with 14,031 images from MSCOCO dataset [46). The questions manually filtered to ensure all questions require outside knowledge to answer. Each question is annotated with ten open-ended answers. We evaluate our method on the test set. A-OKVQA is an augmented knowledge-related dataset containing 25K image-question pairs. It also provides supporting facts and rationales to obtain the answer. We conduct our experiments on the validation set. A-OKVQA benchmark encompasses both multiple-choice settings and anrwer-answer settings without answer options.
Implementation Details. In the experiments, implement the task scheduler using gpt-3.5-turbe For the vision executor, we implement the spatial functions using Grounding DINO [47). which an open-set object detector. The attribute functions implemented by vision-language pre-training models BL.IP2 (33 and X-VLM [48.. The representative in-context examples are selected using agglomerative clustering
49 with cosine metrics, and the feature embeddings are extracted by CLIP (41). We prompt the scheduler with 16 in-context examples in the experiment.
Evaluation Metrics. The commonly used evaluation metrics for both OK-VQA and A- OKVQA direct-answer settings are the soft accuracy proposed VQAv2 [3), where accuracy = min (#humans that provided that answer/3, 1). The predicted answer is deemed 100% accurate if at least humans provided the exact answer. However, with the rapid development tic generative models, the conventional exact matching evaluation of open-ended question answering. Since our answer is directly predicted by LLM without prior knowledge of the answer vocabulary, it may generate answers that are semantically equivalent to the ground truth but use different expressions. Thus, we evaluate our results with a combination of
probabilis- not meet the requirements

---

acquire more Table 1: Comparison to the state-of-the-as methods on OK-VQA dataset. 4: the result is based on reimplementation.
Method
Knowledge Source
Image Representation
Accuracy
Methods with explicit knowledge sources
ConceptBERTy
ConceptNct
Feature
33.7
KRISP 5T
Wikipedia=CConceptNNe
Feature
38.9
Vis-DPR
Google Scarch
Feature
39.2
MAVEX
Tiikipddii-ConneeptteetCCCoo Images
Feature
39.4
KAT 5
Wikidata+GPTT3
Caption+Tags+Feaalle
54.4
REVIVE 6
Wikidata+GPI-33
Caption+Feature
58.0
Methods using only implicit knowledge
IPVR 112
GPT-3
Caption
44.6
GPT-3
Caption
43.4
PICa-Full
GPT-3
Caption
48.0
PromptCAPt 13
GPT-3
Caption
58.8
TOA (ours)
ChatGPT
Caption+VVsuall evidence
60.6
Table 2: The results A-OKVQA dataset. DA refers to direct answer settings and MC refers to multiple choices settings.
Table 3: Ablation study on important compo-
nents of the proposed method.
Model
Accuracy
Method KRSIP 51
DA
MC
TOA-full
60.6
33.7
51.9
TOA (w/o hypothesis)
52.5
IPVR
12
46.4
TOA (w/o multi-round)
57.8
PromptCAP 13
56.3
73.2
Visual Programming 40.9
TOA (ours)
61.2
63.1
keyword matching and nearest-neighhor projection. Specifically, first match the keywords of the predicted answer with the answer candidates. For the synonym paraphrasing without overlap of keywords, project the prediction into the vocabulary using cosine similarity in the space of GloVe (50] word embeddings, which is similar to the multiple-choice evaluation of A-OKVQA. After this post-processing,
follow the conventional soft accuracy report our results. It is worthwhile to note that the evaluation of open-ended answers is still an open problem and we believe that more efforts should be dedicated addressing this problem given its increasing demand in the future.
Results on OK-VQA and A-OKVQA
We compare our proposed method with the state-of- ooftthe-rrt methods on OK-VQA dataset in Table I
The compared methods are categorized into two classes. Methods in the first category 9 [1 52
4
5 6. [0] leverage explicit knowledge sources other than the implicit knowledge of LLM. The
commonly used external knowledge source include ConceptNet (9 51 4. Wikipeida [51 4 5
and Google 4 (2). Note that KAT [5) and REVIVE [6) use both explicit knowledge sources and
LLM as implicit knowledge. Methods in the second category
[3) only use LLM as the implicit implicit
knowledge source obtain the answer. The results show that the introduction of LLM knowledge source brings significant improvement compared to explicit knowledge retrieval methods. PromptCap 13 trains task-aware captioning model with synthesized data. is the primary baseline of our proposed method since the caption given to our task scheduler at the initial step is generated by PromptCap. The improvement in performance
method indicates that useful visual information other than the caption through multi-round dialogue.
We also report the results on the A-OKVQA dataset in Table 2 where we outperform the compared methods on direct answer settings while achieving the second-best result on multiple choices settings. This implies that proposed method is better at open-ended questions than multiple choices questions, since generates open-ended answers free-form natural language.
4.3 Ablation Study
We conduct extensive ablation studies to have an in-depth view of each component of our proposed method. Due to the expensive price of calling the OpenAl API and the limited visit frequency allowed to the public, we conducted our ablation studies randomly selected subset of OK-VOK OK-VQA.
7

---

Question: Which furnishing here is of type reminiscent of what is seen in some classroomns?
Question: What media used to the tigers face?
Caption: group of people sitting on sidewalk next two ligers.
Caption: person walking messy room.
Reusoning: Since captioning does not provide any information them first.
Reasoning: Since the question asks about the tigers face, we first image. Hypothesis: None Verification: locateftiger [ace)
about furnishings, Hypon pothesis: None Verification: (ocate((miishing) Answer: None
Answer: None
Response: Located f'rmishing..
Response: Located face.
Reusoning: use attributes compare
Reasoning: The media create tigers face could puint,
fumishings to those in classoomss, Desks and chairs often seen in classrooms. We can verily there exists desk or chair. Hypothesss: [desk, Verification: cxist(desk) Answer: Nouc
peacil, digital, We verify if the tigers face is painted.
Hypothesis: [pain! Verification: filtertigger face, painted) Answer: None
Response: True
Response: True
Reusoming: Since the vision model verifies the hypothesis, we can decide answer.
Reasoning: Since the verification agrees with the hypothosis, we can decide answer
Hypothesás: None Verification: None Answer: desk
Hypothesis: None Verification: None Answer: paint
Question:
What the white screen on the back for?
Question: What design is on the headboard of the bed?
Caption: classroom with laptops on desks and white
Caption: with a design headboard.
the back.
screen projection We verify there exists projector in the image. Hypothesis: [projection] Verification: proje(pro)cctee) Answer: Node
Reasoning: infer captioning
Reasoning:
question
bed, we first locate it in the locate(headhoard) Answer: None
Hypothesis: None Verification:
Response: Located headboard.
Response: Filse
Reasoning: Since there is no projector in the image, we assume the screen writing drawing markers.
Reasoning: We can use vision model to detect the design on the headbourd. Hypothesi::
Hypothesis: [writing, drawing]
Verification: query attrbbue(headboard design) Answer: None
Verification: filter(white sereen writing) Auswer: None
Response: gold patterned headboard.
Response: False
Reasoning:
Since the verifieation confliets with the hypothesis,
Reasoning: Since the result shows that the headboard has pattern.
we call the simple query function to obtain the answer.
we can funther query about the puttern. Hy Hypothesis: [gold pattern)
Hypothesis: None Verificution: simple (pery white screen on the back for?) Answer: None
Verification: query. attribatehheubhorr, patter)) Answer: None
Response: floral pattern.
Response: projector
Reasoning: auswer obtained
simple query fnnction projection screen, which implies that the white screen is used for projecting images or videos from projector. Hypothesis: None Verification: None Answer: projector
Reasoning: The model specifies pattern the headboard, so we can decide the answer. Hypothesis: None
Verification: None Answer: flower
Figure 3: Examples of multi-round dialogue between task scheduler and vision executor.

---

Table 4: Results of OK-VQA dataset with different number of in-context examples and selection strategy.
examples
16 examples
24 examples
Random Similarity-base 55.3
54.9
58.8
60.2
60.6
60.6
There are two important components in our proposed method, i.e., the multi-round dialogue strategy. and the hypothesis-vert veriiisss-viioiitiionn process. To quantify the benefit of these two main strategies, remove each of them from our full model and compare the results. To evaluate the impact of the eypothesis-verifccan
strategy, we compare an ablative model which removes the REASONING and HYPOTHESIS parts from the instruction of the task scheduler. At each step, the scheduler directly specifies the vision function without explicit reasoning. To evaluate the benefit of multi-round dialogue, we let the task scheduler generate the whole program according to the initial hypothesis. We also compare the results of Visual Programming [19] since it is similar to our proposed method but generates the whole programming at one time. The results are shown in Table 3 and they demonstrate that the hypothesis- hypothesis-veriffiattonn process with multi-round dialogue is essential the performance. The clear output format based on the idea of hypothesis- verifiiatiiiiiii better activates the reasoning ability of large language models. Multi-round dialogue allows the model to decide the next step dynamically based on the previous results, hence having better efficiency of visual information collection and tolerance of the misleading results of the vision model.
Another factor that influences the performance is the in-context example selection. Previous works 45 M have discussed the influence of in-context example selection strategies and the number of examples. We show the results of random selection and the heuristic selection described in Section [34] each with respect to a different number of examples in Table 4 We can observe from the results that more in-context examples often bring better results, while the marginal impact is diminishing. The similarity- bbased in-context selection strategy can achieve better results than random selection since can pair the input question with the most similar examples. However, as the number of examples increases, the advantage is not as significant. This is probably because the model already has enough similar examples selected from the fixed available example pool.
4.4 Qualitative Results
We show a few qualitative results in Figure to further illustrate our proposed task-oriented active VQA approach. In the upper left example, the captioning does not provide relevant information to answer the question. For the upper right example, the captioning fails to distinguish between real tigers and tiger-shaped statues. However, the scheduler makes a reasonable hypothesis based on common-sense knowledge, and the hypothesis is verified by the vision model. For the bottom left example, the vision model fails detect the projection screen, and the responses conflict with the initial hypothesis. In this scenario, the scheduler makes a new reasonable hypothesis and actively acquires more information. Despite the failure of the vision model, it makes the correct decision considering both the common-sens. knowledge and the visual information. Finally, the bottom right example shows how the scheduler progressively acquires the essential visual information based on the previous dialogue to give a more accurate answer.
5 Conclusion
We propose task-oriented active VQA (TOA), which uses LLM as an implicit knowledge source and answers the question through a sequential iypothesis-verificattio process. In TOA, the LLM as the task scheduler actively acquires visual evidence from a set of vision models in a task-oriented manner, and hence can attend the essential information in the image more accurately and efficiently. Our proposed method dynamically decides the next step through multi-round dialogue. It can better deal with situations when the vision models are imperfect, though still limited by the performance of them. A potential limitation of our method is that the design of the prompting examples is subjective, which may bring uncertainty to the reasoning process. Besides, addressing the evaluation of open-ended question answering remains open problem for future works.

---

Acknowledgments
This work was supported in part by National Science Foundation grant 181-1815561 and IIS 11S-2007613.
References
[1] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/evf conference on computer vision and patiern recognition, pages 3195-3204, 2019.
[2] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqu: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162 Springer, 2022.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[4] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge-hased vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2712 2721, 2022.
[5] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: A knowledge augmented transformer for vision-and-language. arXiv preprint arXiv:21 2021.
Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. Revive: Re- gional visual representation matters knowledge-based visual question answering. arXiv preprint arXiv:2206.01200 2022.
[7] Feng Gao, Qing Ping, Govind Thattai, Aishwarya Reganti, Ying Nian Wu, and Prem Natarajan. Transform- retrieve-gemerate: Natural language-centrie outside-knowledd visual question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5067-5077, 2022.
[8] Hugo Liu and Push Singh. Conceptnet-a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211 1-226, 2004.
[9] François Gardères, Maryam Ziacefard, Baptiste Abcloos, and Freddy Lecue. Conceptbert: Concept-awart representation for visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489-498, 2020.
[10] Zhenwei Shao, Zhou Yu, Meng Wang. and Jun Yu. Prompting large language models with answer heuristics for knowledge-based visual question answering. Computer Vision and Pattern Recognition (CVPR), 2023.
[11] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-basce vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081 108999 2022.
[12] Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-basee based visual reasoning. arXiv preprint arXiv:2301 05226, 2023.
[13] Yushi Hu, Hang Hua, Zhengyuan Yang. Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt- guided task-aware image captioning. arXiv preprint arXiv:2211. 1.021.,,9999,.9 2022.
[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. Advances in neural information processing systems, 13:1877-1901 2020.
[15] Chenguang Wang. Xiao Liu, and Dawn Song. Language models open knowledge graphs. arXiv preprint arXiv:2010. 19977 2020.
[16] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter. Xia, H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
10

---

[17] Woojeong Jin. Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth mil- lions of parameters? low-resource prompt-based learning for vision-language models. arXiv preprint arXiv:21 0844444, 2021.
[18] Maria Tsimpoukelli, Jacob I. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-213 2021.
[19] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint 1rXiv:2211.1155, 2022.
[20] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08 2023.
[21] Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony Dick. Explicit knowledge-based reasoning for visual question answering. arXiv preprint arXiv: 1.02570, 2015.
[22] Peng Wang. Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions pattern analysis and machine intelligence, 40(10):2413 2427, 2017.
[23] Medhini Narasimhan and Alexander G Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question answering. In Proceedings of the European conference on computer vision (ECCV), pages 451-468, 2018
[24] Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. Advances neural information processing systems, 31, 2018.
[25] Denny Vrandečié and Markus Krötzsch. Wikidata: a free collaborative knowledgehase. Communications of the ACM, 57(10)78-85 85, 2014.
[26] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[27] Tushar Khot, Kyle Richardson, Daniel Khashabi. and Ashish Sabharwal. Learning solve complex tasks by talking agents. arXiv preprint arXiv:21 10.08542, 2021.
[28] Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /9166, 2022.
[29] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quae Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 220.::2203..171,, 2022.
[30] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprini urXiv:2210.0349: 2022.
[31] Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: modular approach for solving complex tasks. arXiv preprint 10Xiv22210.024066 2022.
[32] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.088 056008. 2022.
[33] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open- aXXi:2230.001560, 2023.
open-world world multi-task agents. arXiv preprint
[34] Drew A Hudson and Christopher D Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6700-6709, 2019.
[35] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang. Houdong Hu, Li Dong. Furu Wei, et al. Oscar: Object-semantic aligned pre- pre-training for vision-language tasks. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XXX 16, pages 121-137. Springer. 2020.
11

---

[36] Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistie representations. arXiv preprint urXiv:1908.08330) 2019.
[37] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for rision-language representation learning. arXiv preprint arXiv:2111. 10023, 2021.
[38] Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in lisnun-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588 5588, 2021.
[39] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. urXiv preprint arXiv:1908.07490 07490, 2019.
[40] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert:
Pretraining task- agnostic visiolinguistic representations for vision-and- ann-uagg tasks. Advances in neural information processing systems, 32, 2019.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[42] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Le, Hun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation lcarning with noisy text supervision. In International Conference on Machine Learning, pages 1904-4916. PMLR, 2021.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language -mage pre- pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[44] Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. Chatgpt asks, 2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2333.06594, 2023.
[45] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.066 006800444444... 2021.
[46] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Swirzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[47] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.
2023.
[48] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-graine vision language pre-training: Aligning texts with visual concepts. International Conference on Machine Learning, pages 25994-26009, PMLR, 2022.
[49] Marie Lisandra Pepeda-Mendoze and Oshaldo Resendis-Antonio. Hierarchical Agglomerative Clustering, pages 886-887. Springer New York, New York, NY, 2013.
[50] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[51] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14111-14121. 2021.
[52] Man Luo, Yankai Zeng, Pratyay Banerjee, and Chitta Baral, Weakly-supervised visual-ettriever-reader for knowledge-haset based question answering. arXiv preprint arXiv: 2:X11:2109.04014, 2021.
12

---

Appendix
A Demonstration of our prompt instructions and in-context examples
We illustrate the prompt instructions and in-context examples we used in our method in Figure
B Human evaluation and Using GPT for evaluation
Since the final answers are generated open-ended and unaware of the answer vocabulary in our proposed method, it is improper to evaluate the results by conventional exact matching. To handle the semantically equivalent answers, we proposed method combining keyword matching and nearest-neighhor projection. To further analyze our evaluation strategy, conduct human evaluation on the predictions of OKVQA dataset and also leverage ChatGPT to judge whether the predicted answer and the ground truth can be considered consistent, given the original question. The results of different evaluation strategies are shown in Table
These results further demonstrate that although our initial attempt on open-ended answer evaluation may not be perfect, it did not over-claim the performance of our proposed method. Thus we can affir the effectiveness of our method. Moreover, this analysis further validates the necessity of addressing the evaluation of open-ended question answering.
Table 5: Comparison of results OKVQA dataset by different evaluation methods.
Matching and projection
Human
ChatGPT
Accuracy
60.6
65.0
64.7
C Ablation study on the design choice of our prompting strategy
To demonstrate the effectiveness of our design principles, we conduct ablation study on different components of the prompting. We removed the task description, vision tools description and hypothesis guidance in the prompt instruction and in-context examples, respectively. The results are shown in T
The results show that these concise yet essential design principles have proven to be effective to guide the LLM for knowledge- visual reasoning. However, we also recognize that our design might not be optimal. Further refinement in prompt design might yield improvements, which will be explored in our future work.
Table 6: Ablation study on prompting strategy.
Model
Accuracy
TOA-full
60.6
w/o task description
57.2
w/o vision tools description
58.6
w/o hypothesis guidance
55.1
13

---

Prompting Instructions
I have question about image. Fach time you need to make hypothesis and verify by collecting visual information. You can use the following visual functions gather information.
Task description
Constraints
You can only call one vision function at time. The answer should be very concise.
Vision tools description
existoojeet_pmmeesst>>>booe
Returns True if the object specified by object_name is found the image, and False otherwise.
attribute(object naterestr, visual attribuue:strr>>>tt
Returns the attribute of the object specified by object name. Presupposes the nominal class
existence of the object. The parameter such color, shape, material.
filteroojectt_anne nateeestr properly:str) >booll
Returns True if the object possesses the property, and False otherwise. Presupposes the existence of the object.
In-context examples
Question: Which of these appliances would you heat your noodles in?
Caption: A kitchen with refrigerator and dishwusher
Assistant: Reasoning: We know that there refrigerator and dishwasher in the kitchen.
None them can heat. We often use microwave for heating. We can verify there exists a microwave. Hypothesis: [microwave]
Verification: exist(microwave) Answer: None
User:
True
Assistant: Reasoning: Since the hypothesis is verified by the vision model, decide the answer. Hypothesis: None Verification: None Answer: microwave
Question: Name the material used make seat shown this picture?
Caption:
cat sitting on the back a car seut
Assistant: Reasoning: Since the question asks about the car seat, we first locate in the image. Hypothesis: None Verification: locate(car seat) Answer: None
User:
Located scat
Assistant: Reasoning: Car seats are often made leather, fabric or vinyl. can directly verify each them. Hypothesis: [leather, fabric, vinyl] Verification: filter(car seat, leather) Answer: None
User:
False
Assistant:
The seat is made leather. Next verify if it fabric. Hypothesis: [fabric, viny]] Verification: filter(car seat, fabric) Answer: None
Reasoning:
User:
Truc
Assistant:
Since the vision model verifies the hypothesis, the answer is fabric. Hypothesis: None Verification: None Answer: fabric
Reasoning:
Figure 4: The prompting instructions and in in-context examples which guide the large language model to follow the sypothesis-verificune reasoning stream. The instructions consist of task description, format constraints and vision tools description.
14


Reviews:
[{"review_id": "MDCrlYVnsm", "reviewer": "Reviewer_BgCz", "summary": "This paper tries to solve the knowledge-based visual question answering (VQA) by proposing a new approach that utilizes LLMs for calling visual modules in a task-oriented manner. The method employs a reasoning-hypothesis-verification process in multiple rounds to progressively find the answer. Evaluations are conducted on OK-VQA and A-OKVQA to demonstrate the effectiveness of this method.", "questions": "See weaknesses.", "limitations": "The authors didn't discuss limitations in the paper. However, it might happen that the LLM (used in paper) generates some toxic or biased response. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Multi-round interactions and reasoning-hypothesis prompting are introduced in this paper. Compared with previous one-time program generation, those two improvements are reasonable intuitively and can benefit on reported benchmarks.", "weaknesses": "1. In the title, abstract and introduction, `task-oriented` seems to be emphasized as the main advantage of this paper over others. However, ViperGPT and Visual Programming are also task-oriented. The main difference between this paper and the previous two are rationale and multi-rounds. The corresponding part should be rewritten.\n2. I don't understand the necessity of the hypothesis. From the given examples, it seems without an assumed hypothesis, the LLM should still be able to call the right verification function. In Fig3's examples, imagine you remove the hypothesis, the workflow of LLM seems still smooth and reasonable, and many hypotheses are actually None. In Tab3's ablation, both reasoning and hypothesis are removed which is not indicative enough. Can you do an ablation to remove the hypothesis while preserving reasoning and verification and answer?\n3. Why don't you compare with ViperGPT in OK-VQA? Is that because ViperGPT is in a zero-shot manner, while yours is in a few-shot manner? Then why your method must be reported in a few-shot manner and what performance it can achieve in the same manner as ViperGPT?\n4. Only two benchmarks are compared. In ViperGPT/Visual Programming, 4/3 datasets are benchmarked. To demonstrate the generalization ability of the proposed method, at least one or two more typical datasets are needed.\n5. Similar ideas of multi-rounds and reasoning in each step have been used in [1]. Please compare the difference.\n6. Many visual models are employed in this paper however the technical details as well as related prompts are not included in the paper. It makes reproduction difficult.\n\nRef:\n[1] See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable and constructive comments. We have made the necessary revisions or clarifications to improve the quality of our manuscript as follows. We have attached a one-page pdf in our global response to demonstrate our prompting instruction and in-context learning examples. Here are our responses to each of your questions.\n\n**[Reviewer BgCz Weakness-1: The term of `task-oriented`]**: Thank you for your suggestions! We want to clarify that the term `task-oriented` refers to a dynamic, iterative process where the Large Language Model (LLM) actively engages with both the question and image. The LLM formulates hypotheses and conducts active vision verifications, adaptively reasoning based on the results until the LLM itself determines the final answer. This whole process is intimately connected to the task at hand, continually interacting with and adapting to the information gathered, which justifies our use of the term `task-oriented`.\n\nIn contrast, existing methods such as Visual Programming and ViperGPT use the LLM merely as a tool for task planning or code generation. They generate an immutable program based solely on the question **without deeply or actively** engaging with the image, and they obtain the answer directly from the execution of vision models. Thus they are **not** `task-oriented` under our definition. In these methods, the LLM can not make adjustments for errors in either program planning or vision models, and their application on questions requiring open-world knowledge is limited.\n\nWe appreciate your concern, and we will revise the relevant sections to make this distinction clearer and underline the unique aspects of our method.\n\n**[Reviewer BgCz Weakness-2: Necessity of Hypothesis]**: Please refer to the **Global Response-3**.\n\n**[Reviewer BgCz Weakness-3: Zero-shot setting]**: Our approach and all other compared methods need to employ few-shot instructions. This is because the in-context examples enable the LLM to follow our expected sequential hypothesis-verification reasoning process and give the output as formatted. The output format facilitates the distillation of the essential information from free-form text generated by LLM into executable visual functions. In our proposed method, the LLM acts as the reasoning agent like a human brain which conducts reasoning and task planning by actively utilizing its common-sense knowledge and finally outputs the answer by the LLM itself. While for the ViperGPT, it uses CodeX to directly generate Python code based on the question and derive answers from the execution of the generated code. Since the CodeX is pretrained specifically for code generation, ViperGPT does not require additional instructions to guarantee the output format and can work in a zero-shot setting.\n\nGiven these fundamental differences between the two types of methods, we compare our method with the methods that also employ the LLM as a reasoning agent in a few-shot manner in our experiment section. Such comparison is fair, and our experiments have sufficiently demonstrated the effectiveness of our method.\n\n**[Reviewer BgCz Weakness-4: More datasets]**: Please refer to the **Global Response-2**.\n\n**[Reviewer BgCz Weakness-5: Comparison to IPVR[12]]**: Thank you for bringing the IPVR work to our attention, where we have already cited it and discussed it as IPVR [12] in our main paper with explicit comparison in both **Table 1 and 2** in the main paper. In IPVR, the process consists of three key modules: a 'see' module for object-level detection, a 'think' module for selecting attended objects and transforming them into captions, and a 'confirm' module for rationale verification. While the superficial similarities exist, we would like to highlight the fundamental differences between IPVR and our method:\n\n1. **Object-level detection vs. Active image understanding**: In IPVR, they focus on object-level detection and attending, which may either bring irrelevant objects to the question or drop essential information in the caption. However, our method is not constrained by object-level image understanding and captioning, as we utilize various vision functions to actively acquire visual information.\n\n2. **Confirmation Process**: While IPVR also involves a confirmation process, they merely use the LLM to generate a **textual** rationale to support the answer. They confirm the answer when the generated rationale matches the predicted answer. Contrarily, our method's verification part allows the LLM to acquire **new visual evidence** by vision executor based on its hypothesis, and facilitates subsequent rounds of reasoning. The LLM confirms the answer when it has obtained sufficient information to reason about the answer using its open-world knowledge.\n\nBy elucidating these differences, we establish that our method, although superficially similar to IPVR, is **essentially different** in its approach and underlying principles. These differences contribute to the uniqueness and innovation of our proposed method. Moreover, these underlying disparities are manifest in our performance results. Specifically, our method's scores of **60.6** and **61.2** on the OK-VQA and A-OKVQA benchmarks, respectively, are significantly superior to IPVR's scores of **44.6** and **46.4** on the same benchmarks, as shown in Tables 1 and 2 in our main paper.\n\n**[Reviewer BgCz Weakness-6: Technical details]**: In designing and implementing our vision executor, we considered commonly used vision models in line with the methods compared in Table 1 and 2 with public code. This makes experimental comparisons fairer, and the re-implementation will not be difficult, since all the models we used have user-friendly API to call. This design choice allows us to highlight the improvements and advantages of our new method over existing methods by eliminating differences in vision models. We will publicly release our full implementation after acceptance and offer help for reproduction."}, {"review_id": "4MOuMbkxgm", "reviewer": "Reviewer_ckp5", "summary": "Early methods for Knowledge-based visual question answering (VQA) explicitly retrieve knowledge from external knowledge bases, often introducing noisy information. Current large language models like GPT-3 as implicit knowledge sources cannot effectively understand image inputs. Thus, extracting the image information and inputting it into large language models remains an open problem. Using image captioning and object descriptions to represent the image may either drop the essential visual information to answer the question correctly or involve irrelevant objects to the task of interest. To address this problem, the authors propose to let large language models make an initial hypothesis according to their knowledge, then actively collect the visual evidence required to verify the hypothesis. In this way, the model can attend to the essential visual information in a task-oriented manner. The authors leverage several vision modules from the perspectives of spatial attention (i.e., Where to look) and attribute attention (i.e.,16 What to look), which is like human cognition. ", "questions": "Please refer to the questions in the weakness.", "limitations": "I cannot find a potential negative societal impact in this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The idea to exploit ChatGPT to solve knowledge-based VQA interactively seems novel.\n- Compared to the recent similar method, Visual Programming, the proposed method shows favorable performance, and the authors provided an ablation study and hyper-parameter analysis to verify the characteristics of each component of the proposed method.", "weaknesses": "- It would be more helpful to add an experiment to verify whether the proposed method can be generally applied to other tasks. For example, can the proposed ChatGPT-based interactive framework also help improve the performance of other datasets, such as the VQAv2 dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for your insightful comments, especially your appreciation of the novelty of our proposed approach and our experimental results. For the concern you expressed in the Weakness section, we have addressed it in our **Global Response-2**. We paste the response here for your convenience.\n\n**[Global Response-2: About experiments on more datasets]** We greatly value your suggestions regarding the expansion of our experiments to more datasets and different vision-language tasks. We acknowledge the importance of demonstrating generalizability, but here's why we have opted for a specific focus on Knowledge-based VQA:\n\n- **Our specific focus: Knowledge-based VQA**: First, as explicitly demonstrated in our abstract and introduction, our method targets exploiting LLM for the Knowledge-based VQA task. This is because Knowledge-based VQA requires open-world knowledge to answer the question about an image and is more challenging than traditional VQA tasks [3] since the model needs to extract relevant external knowledge and then perform joint reasoning on the question, image, and the knowledge. Such a challenging requirement essentially demands the LLM to be actively engaging in collecting the essential information from both the question and image, and motivates us to propose our task-oriented active VQA method for Knowledge-based VQA in this paper. Extending our method to more general vision-language tasks is not our current priority in this paper, because we believe that it is more beneficial and important for us to comprehensively demonstrate the effectiveness of our proposed method on the challenging Knowledge-based VQA.\n\n- **Existing datasets for Knowledge-based VQA**: For the task of knowledge-based VQA, OKVQA and A-OKVQA datasets are the standard benchmarks in the existing literature [9, 51, 52, 4, 5, 6, 12, 11, 13]. The comparison of these two datasets with state-of-the-art methods in Table 1 and 2 in our main paper sufficiently demonstrates the efficacy of our proposed method for the knowledge-based VQA.\n\n- **More results on other VQA task**: As complementary, we have conducted additional experiments on the VQAv2 dataset to demonstrate the broader applicability of our method.  We achieved **74.8%** (**16 shots**) on a randomly selected subset of VQAv2 val set, which is a competitive few-shot result, compared with the powerful vision-language model Flamingo[a] (**68.4%, 32 shots**). We consider these exploratory results as a stepping stone for future research.\n\n[a] Flamingo: a visual language model for few-shot learning. NeurIPS 2022."}, {"review_id": "zlj4MCZu0q", "reviewer": "Reviewer_TKN8", "summary": "This paper addresses knowledge-based visual question answering by leveraging a large language model (LLM) as a knowledge source. To mitigate the limited capability of vision models, the paper suggests letting the LLM predict hypothesis and actively gather visual evidence. Experimental results show that the proposed method outperforms baseline approaches in open-ended knowledge-based VQA.", "questions": "- About the design choice of heuristic prompting, investigating the effects of different prompts would be interesting.\n- It would be helpful to see in-context examples for specific image-question pairs.\n- A table reference error (Table 4.1) should be corrected.", "limitations": "The paper discusses two limitations.\n- Evaluating open-ended VQA remains an unresolved challenge.\n- The design of prompting examples may not be optimal.\nThey are important issues in this area.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, clearly structured, and easy to follow.\n- The motivation behind using an LLM for VQA to assist imperfect vision models is reasonable, and qualitative results demonstrate that the method behaves as intended.\n- Ablation studies reveal that the multi-round dialogue strategy and hypothesis-verification process positively contribute to the approach.\n- The experiments are thorough, offering insights into the impact of design choices.", "weaknesses": "- The method's reliance on heuristic prompting may limit its generalizability to other LLMs. \n- The validity of post-processing, which paraphrases predictions using a predefined vocabulary, is unclear. A small analysis of the post-processing would make the paper more convincing. For instance, providing examples of raw predictions and post-processed ones would help readers better understand the post-processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our writing, motivation, and experimental results! We have considered all your suggestions and have made the necessary revisions or clarifications to improve the quality of our manuscript as follows. We have attached a one-page pdf in our global response to demonstrate our prompting instruction and in-context learning examples.\n\n**[Reviewer TKN8 Weakness-1: Prompting strategy]**: We would like to apologize if naming Section 3 as 'heuristic prompting' in our paper gave the reviewer the impression that the design of the instruction prompting template was heuristic or without basis. Actually, it is designed to guide LLM reasoning following a thoughtful and deliberate process that includes several key elements:\n\n1. **Defining the Task**: We begin by defining the task to inform the LLM about the specific reasoning task at hand.\n2. **Introducing Available Vision Tools**: We then present the available vision tools within the prompt, enabling the LLM to recognize the executable functions to verify the hypotheses.\n3. **Specifying Output Format Constraints**: By delineating the expected output format, we ensure consistent parsing of the LLM's output to automatically query our designed vision functions.\n\nYou can also check the one-page pdf in global response for more concrete examples of our prompting strategy.\n\n**Ablation study on the design choice of our prompting strategy.** To demonstrate the effectiveness of our design principles, we have added a new ablation study. We removed the task description, vision tools description and hypothesis guidance in the prompt instruction and in-context examples, respectively. The results are shown in the table below:\n| Model                        | Accuracy (%) |\n|------------------------------|--------------|\n| TOA-full                     | 60.6         |\n| w/o task description         | 57.2         |\n| w/o vision tools description | 58.6         |\n| w/o hypothesis guidance      | 55.1         |\n\nThe results show that these concise yet essential design principles have proven to be effective to guide the LLM for knowledge-based visual reasoning. However, we also recognize that our design might not be optimal. Further refinement in prompt design might yield improvements, which will be explored in our future work.\n\n**Applicability to other LLMs.** Last but not least, we change the GPT to the free-to-use Claude-2 model by the Anthropic and conducted the same experiment on OKVQA, where we can achieve comparable accuracy **60.1%** to GPT. This further demonstrating the applicability of our framework to different LLMs. \n\n\n**[Reviewer TKN8 Weakness-2: Analysis of the evaluation process]**: Thank you very much for the suggestion. Here we provide some examples of raw predictions and mapping to the answer vocabulary: \n\n(1) Prediction: to advertise; Ground Truth: advertising \n\n(2) Prediction: to tell directions; Ground Truth: navigation \n\n(3) Prediction: one-way traffic; Ground Truth: one way\n\n(4) Prediction: bike; Ground Truth: bicycle\n\nFrom the above examples, we can see that these matchings are reasonable and semantically highly relevant. \n\nMoreover, to analyze our evaluation strategy more comprehensively, we have further conducted two different types of evaluation:\n\n- **Human Evaluation**: We conducted a human evaluation on the predictions of OKVQA dataset by our proposed method, and the accuracy is **65.0%**, which is even higher than the result of 60.6% obtained by our original evaluation strategy in Table 1 of our main paper. \n\n- **Leverage GPT for evaluation**: After we obtain the predicted results, we separately call the OpenAI API to initialize a new GPT to judge whether the predicted answer and the ground truth can be considered consistent, given the original question. The resulting accuracy is **64.7%** for the prediction from our proposed method, which is similar to the human evaluation.\n\nThese results further demonstrate that although our initial attempt on open-ended answer evaluation may not be completely perfect, it **did not over-claim** the performance of our proposed method. Thus we can affirm the effectiveness of our method. Moreover, this analysis further validates the necessity of addressing the evaluation of open-ended question answering.\n\n **[Reviewer TKN8 Question-1]**: Please refer to our reply to **[Reviewer TKN8 Weakness-1]** for the **Ablation study on the design choice of our prompting strategy**.\n\n **[Reviewer TKN8 Question-2: Demonstration of in-context examples]**: Please refer to the one-page pdf we attached in our global response for the demonstration of the in-context examples.\n\n **[Reviewer TKN8 Question-3: Typo]**: Thank you for pointing this typo out; we have revised it accordingly!\n"}, {"review_id": "bKiLbElQPt", "reviewer": "Reviewer_8uLs", "summary": "This paper proposes task-oriented active VQA (TOA), which uses LLM as an implicit knowledge source and answers the question through a sequential hypothesis-verification process. This method can more accurately attend to the essential information in the images and reduce the introduction of irrelevant information. And they develop a multi-round dialogue approach to solve the problem progressively and decide the next step dynamically, which has a clear answering process and better tolerance of mistakes in previous steps. The experiments show the method outperforms the baselines and presents clear reasoning procedures.", "questions": "Will this method be limited by the high cost of ChatGPT? What is the impact on the generation effect if other open-source LLMs are used?", "limitations": "The authors say that subjective example design brings uncertainty to the reasoning process, and evaluation methods for open-ended question answering are incomplete. I think it may be necessary to introduce some human evaluation as support in both aspects.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of this paper is very novel, and the key visual content is obtained through multiple rounds of dialogue.\n\n2. The method takes full advantage of the rich knowledge and application flexibility of LLM.\n\n3. The process design of reasoning-hypothesis-verification can reflect clear reasoning procedures and has better interpretability.\n\n4. Experiments show that the performance of the method is very good, and verify the validity of the design idea.\n", "weaknesses": "1. Most of the sub-modules are somehow similar to existing models, and the design of the reasoning-verification process may be ad hoc and has great limitations.\n\n2. In the  experiment, other methods use GPT-3 and this method uses ChatGPT, which is a bit unfair.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript, for providing such valuable feedback and your appreciation of our novelty regarding the knowledge-based VQA and LLM, the better interpretability of our approach, and our experimental results! Here are our response for each of your questions. \n\n**[Reviewer 8uLs Weakness-1: Choices of vision models and paradigm design]**: We appreciate the reviewer's concern. First, we want to emphasize that: our core contribution lies in creating a novel computational paradigm where the LLM can iteratively and sequentially combine question and image information, imitating human cognitive processes and using its knowledge for reasoning. Although we utilize existing sub-modules, it is our unique integration and the underlying philosophy that sets our approach apart. This new approach allows the LLM to actively form and verify hypotheses by collecting visual information, representing a significant innovation in the field.\n\n- **Regarding using existing vision models in our vision executor**: We utilized common vision models (e.g., vision models used in Visual Programming) to ensure a **fair and meaningful comparison** with existing methods. This is because only by doing so can we readily show that the improvement brought by our method is not because we are using different or more advanced vision models, but from the novelty of our proposed task-oriented active VQA approach for Knowledge-based VQA. This can be seen from the comparison between our proposed method and the Visual Programming in our main paper Table 3. Moreover, our flexible prompting design can allow the vision modules to be easily added or changed for different task requirements in our proposed method.\n\n- **Regarding \"the design of the reasoning-verification process\"**: Our hypothesis-verification reasoning process is inspired by the human cognitive process, where assumptions and inferences are made based on their knowledge and then verified actively. Translating this into a computational form is non-trivial, and our work is the first to tackle this challenging problem. We acknowledge that our implementation can be enhanced, but we believe our current work has effectively demonstrated the potential and feasibility of this new paradigm. \n\n**[Reviewer 8uLs Weakness-2: Using GPT-3 for our approach]**: Thank you for highlighting this concern. We observe that both GPT-3 and ChatGPT possess their unique strengths, and it's not a matter of one definitively being better than the other. GPT-3 tends to have stronger imitation abilities through in-context learning, whereas ChatGPT may have better reasoning capabilities and creativity, especially in multi-round dialogues. However, this might also make maintaining format consistency across multiple dialog rounds more challenging with ChatGPT. \n\nOur method involves using a multi-round dialogue format, which initially made ChatGPT a more convenient choice for us. To address your concern, we conducted additional experiments by switching from ChatGPT to GPT-3, the same model used in our compared baselines. Interestingly, the results showed that our method performed even better with GPT-3 than with ChatGPT (improved from **60.6%** to **61.5%**). This more even-handed comparison further accentuates the advantages of our proposed method over compared methods. Our conjecture for this interesting outcome is that GPT-3 already has sufficient reasoning ability and knowledge for current knowledge-based VQA tasks, and it may better imitate the in-context examples and follow the instructions.\n\n**[Reviewer 8uLs Question-1: Usage of other LLMs]**: Thank you for this question. We would like to point out that all the methods we have compared, based on LLMs, are dependent on GPT. Thus, in general, all related methods would be somewhat constrained by the cost of reasoning with GPT. Compared to existing methods such as Visual Programming, our method has significantly reduced the average reasoning cost for a single question from approximately 0.049 to 0.0085 dollars. \n\nRegarding open-source LLMs, the current open-source models still have some gaps in reasoning ability compared to the GPT series, which may somewhat impact the final answer's effectiveness. We conducted experiments with LLaMa and the free-to-use Claude-2 model by Anthropic. The accuracy of OKVQA using LLaMa is **51.8%**, which is still at an acceptable level. We believe that as open-source LLMs continue to advance, this issue will gradually be resolved. Our experiments on Claude-2 yielded an accuracy of **60.1%**, which is comparable to GPT, further demonstrating the applicability of our framework to different LLMs."}]

Author rebuttal:
We sincerely thank all reviewers for your insightful and constructive reviews! In this global response, we want to address the common questions inquired by different reviewers, and also demonstrate the necessity of making hypotheses in our proposed method. 

**[Global Response-1: Demonstration of our prompt instructions and in-context examples]** In the one-page pdf allowed by the rebuttal, we provide figures to illustrate the prompt instructions and in-context examples we used in our method. We hope this can help reviewers better comprehend the simple yet effective design principles of our prompts.

**[Global Response-2: About experiments on more datasets]** We greatly value the suggestions from Reviewers **BgCz** and **ckp5** regarding the expansion of our experiments to more datasets and different vision-language tasks. We acknowledge the importance of demonstrating generalizability, but here's why we have opted for a specific focus on Knowledge-based VQA:

- **Our specific focus: Knowledge-based VQA**: First, as explicitly demonstrated in our abstract and introduction, our method targets exploiting LLM for the Knowledge-based VQA task. This is because Knowledge-based VQA requires open-world knowledge to answer the question about an image and is more challenging than traditional VQA tasks [3] since the model needs to extract relevant external knowledge and then perform joint reasoning on the question, image, and the knowledge. Such a challenging requirement essentially demands the LLM to be actively engaging in collecting the essential information from both the question and image, and motivates us to propose our task-oriented active VQA method for Knowledge-based VQA in this paper. Extending our method to more general vision-language tasks is not our current priority in this paper, because we believe that it is more beneficial and important for us to comprehensively demonstrate the effectiveness of our proposed method on the challenging Knowledge-based VQA.

- **Existing datasets for Knowledge-based VQA**: For the task of knowledge-based VQA, OKVQA and A-OKVQA datasets are the standard benchmarks in the existing literature [9, 51, 52, 4, 5, 6, 12, 11, 13]. The comparison of these two datasets with state-of-the-art methods in Table 1 and 2 in our main paper sufficiently demonstrates the efficacy of our proposed method for the knowledge-based VQA.

- **More results on other VQA task**: As complementary, we have conducted additional experiments on the VQAv2 dataset to demonstrate the broader applicability of our method.  We achieved **74.8%** (**16 shots**) on a randomly selected subset of VQAv2 val set, which is a competitive few-shot result, compared with the powerful vision-language model Flamingo[a] (**68.4%, 32 shots**). We consider these exploratory results as a stepping stone for future research.

**[Global Response-3: the necessity of the hypothesis-making]** We thank reviewer **BgCz** for inquiring about this question. We believe it is better to treat it as a global response to provide a comprehensive answer. Here is our response:

- **Necessity of Hypothesis**: In our proposed method, the hypothesis is derived from the knowledge of LLM and the current information that it processes. With a clear hypothesis-verification process, LLM will be prompted to actively incorporate and exploit its open-world knowledge for its reasoning, instead of completely relying on the output of the vision models. For instance, when the output of the vision models contradicts common-sense knowledge from the LLM, the LLM instructed by our hypothesis-verification mechanism will tend to collect more information from the image, e.g., make another hypothesis or change to a different verification tool, such that the LLM itself can integrate more comprehensive information to give the answer. Such merit has also been demonstrated in our main paper’s **lines 299-303**, with an example in the bottom-left subfigure of **Figure 3**. 

- **The reason why the workflow is smooth in Figure 3**: When we design our instruction and in-context examples, we explicitly instruct the LLM to make hypotheses and verify them in the 'Reasoning' section in our prompting. **Thus the reasoning process of LLM has intrinsically included the process of hypothesis-making, and they are interconnected.** We distinguish ‘Hypothesis’ to keep the output format consistent and clear.

- **Regarding 'many hypotheses are actually None’**: This is due to the multi-round dialogue nature of our demonstration in Figure 3. It appears that many hypotheses in Figure 3 are ‘None’ in single steps, but that does not mean the model did not formulate them in the whole dialogue. In every single round, the ‘Hypothesis’ may be ‘None’ in three cases: 
1. LLM lacks sufficient information to make any hypothesis and needs to gather more visual information, often at the beginning of the dialogue. 
2. The previous hypothesis is overturned, and a new hypothesis is not made.  
3. The answer can be decided by the collected information, and no more hypotheses are required. This is the final round of the dialogue, and the ending is determined adaptively by the LLM agent. 

- **Ablation of removing the hypothesis**: Finally, we conducted two supplementary experiments. We first removed only the ‘Hypothesis’ from the output format. It does not obviously influence the overall accuracy, since the LLM is still following the hypothesis-verification stream. Then we rewrote the ‘Reasoning’ in our prompts to **remove the guidance for LLM to make hypotheses** during the ‘Reasoning’, and the performance on the OKVQA dataset dropped significantly from **60.6%** to **55.12%**, which emphasizes the **indispensability of the hypothesis** in our proposed method.

[a] Flamingo: a visual language model for few-shot learning. NeurIPS 2022.

Meta-review: All reviewers are quite positive about this submission and no negative scores are given. All reviewers confirmed the contributions and novelty of this submission. And the authors' rebuttal addressed the reviewers' concerns very well. After the discussion and final update from reviewers, we all voted to accept this submission.
Decision: Accept (poster)