SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoying Xing Mingfu Liang Ying Wu
{xiaoyingxing2026，mingfuliang2020}@u.northwestern.edu
yingwu@northwestern.edu
Knowledge-based visual question answering (VQA) requires external knowledge to answer the question about an image. Early methods explicitly retrieve knowledge from external knowledge bases, which often introduce noisy information. Recently large language models like GPT-3 have shown encouraging performance as implicit knowledge source and revealed planning abilities. However, current large language models can not effectively understand image inputs, thus it remains an open problem to extract the image information and input to large language models. Prior works have used image captioning and object descriptions to represent the image. However, they may either drop the essential visual information to answer the question correctly or involve irrelevant objects to the task-of-interest. To address this problem, we propose to let large language models make an initial hypothesis according to their knowledge, then actively collect the visual evidence required to verify the hypothesis. In this way, the model can attend to the essential visual information in a task-oriented manner. We leverage several vision modules from the perspectives of spatial attention (i.e., Where to look) and attribute attention (i.e., What to look), which is similar to human cognition. The experiments show that our proposed method outperforms the baselines on open-ended knowledge-based VQA datasets and presents clear reasoning procedure with better interpretability.
The problem of knowledge-based visual question answer (VQA) [1, 2] requires open-world knowledge to answer the question about an image, which is more challenging than traditional VQA tasks [3] since the model needs to extract relevant external knowledge and then perform joint reasoning on the question, image, and the knowledge. To this end, most existing solutions explicitly retrieve external knowledge from various sources like Wikipedia [4, 5, 6, 7] and ConceptNet [8, 9, 4]. However, their performances are mainly limited by the explicit knowledge-retrieval stage [10]. First, the required knowledge may not be successfully retrieved, leading to insufficient information to answer the question. Second, the retrieved knowledge may be noisy and introduce irrelevant information, hence degrading the performance of the joint reasoning stage.
Recent works [11, 10, 12, 13] explore to use the large language models (LLMs) like GPT-3 [14] as implicit knowledge base, since they have shown powerful abilities in knowledge retrieval [15] and reasoning [16]. However, it remains a challenge to enable a pretrained LLM to further exploit the visual information, as this requires large computation resources to fine-tune the LLM on vision-language tasks [17, 18]. To sidestep this challenge, captioning-based methods translate the image into a caption such that the LLM can understand image information like a textual prompt. Despite the significant improvement compared to the explicit retrieval-based methods, their performances are still limited because the caption may usually drop essential visual information required to answer the question. On the other hand, the detection-based method leverages object detectors to extract all
the candidate visual concepts in the image and provide the LLM with textual object tags [5, 6] or brief description [12]. In this way, the LLM is provided with more exhaustive visual information. Nevertheless, many of the detected objects are irrelevant to the question and introduce noisy visual information to the language model. Another category of works utilizes the in-context learning ability of LLM to decompose the input questions into modular programs and execute on a set of pre-defined vision functions [19, 20]. However, they directly generate the answers through the execution of the program with simple symbolic logic, which limits their application to knowledge-based VQA.
To help the LLM better understand the image, we propose a task-oriented active (TOA) VQA method to actively and progressively acquire the visual information according to the task, as depicted in Figure 1. In general, our TOA method imitates the human cognitive process of reasoning, hypothesis, and verification. It consists of a LLM as the scheduler to make task planning and decision, and visual models to execute the order from the scheduler. Based on the understanding of the input question and the open-world knowledge obtained during pretraining, the LLM makes hypotheses on the answer and actively acquires relevant visual evidence from the visual model to verify the hypothesis. As a consequence, the LLM in our TOA method can integrate open-world knowledge and the collected visual evidence to answer the question, in contrast to previous works that directly obtain the answer only by a generated program [20, 19]. On the other hand, our method extracts the visual information in a task-oriented manner instead of looking at the whole image aimlessly, hence attending to the essential visual information more accurately and reducing the irrelevant information. Moreover, differing from the existing methods that generate a complete program at once and solely based on the question [20, 19], the LLM in our TOA interacts with the vision modality progressively through multi-round dialogue and decides the next instruction based on the previous interaction experience. When the visual evidence conflicts with the hypothesis, the LLM can make a new hypothesis or ask for other visual evidence to make the final decision. Therefore, even when the vision model makes a mistake, our method can still verify the hypothesis from a different angle.
To verify the effectiveness of the proposed method, we evaluate our methods and other competitive methods on OK-VQA [2] and A-OKVQA [2] datasets, where both datasets ask questions that require open-world knowledge beyond the image to obtain the correct answer. The empirical results show that our method outperforms the comparison methods on open-ended question answering, indicating our capability to better leverage implicit knowledge and capture essential visual information.
- We propose a new method to actively collect visual information in a task-oriented manner, which can more accurately attend to the essential information in the images and reduce the introduction of irrelevant information.
- We design a human-like cognitive process, i.e., reasoning, hypothesis, and verification, to better activate the reasoning abilities of the LLM through clear problem-solving procedures and high-quality visual information.
- We develop a multi-round dialogue approach to solve the problem progressively and decide the next step dynamically, which has a clear answering process and better tolerance of mistakes in previous steps.
Knowledge-based Visual Question Answering. Early works [21, 22, 23, 24] retrieve supporting knowledge from fixed knowledge bases annotated with the required facts to answer the questions. Subsequent open-knowledge VQA tasks [2, 1] need to acquire open-world knowledge instead of fixed knowledge bases. One category of solutions to open-knowledge VQA [4, 7, 8, 9] explicitly retrieves external knowledge from various sources, such as Wikipedia [25], ConceptNet [26] and Google Images [4]. The performances of the explicit retrieval-based methods are mainly limited by the knowledge retrieval stage, since it may either fail to retrieve the required knowledge or introduce noisy information. Another line of works [11, 10, 12, 13] leverages LLM as an implicit knowledge source. They often translate the visual information into captions [11, 13] or object-level descriptions [12]. Although the pioneering caption-based method PICa [11] significantly outperforms explicit retrieval-based methods, it is still limited since the generic caption may drop some essential visual information required to answer the question. To remedy this issue, PromptCAP [13] generates
Figure 1: The overview of our task-oriented active (TOA) VQA method. TOA consists of a task scheduler and vision executor which communicates through multi-round dialogue. The task scheduler actively analyzes the existing information. Then it either requires specific visual information to help it make a further decision, or makes hypotheses on the answer and asks for visual evidence to verify its hypothesis. On the other hand, the vision executor follows the instruction from the scheduler to perform different kinds of visual verification and then returns the execution results to help the scheduler come up with the next decision. The gray arrows indicate the interaction process.
more relevant image captions conditioned on the input question. However, how to efficiently extract image information and translate it into textual prompts for LLMs remains an open problem. Different from the previous works that prompt LLMs with passively extracted visual information, we propose to let the LLM actively acquire the essential visual information in a task-oriented manner.
Prompting Large Language Model for Reasoning. Recent works have found that LLMs emerge reasoning abilities [27, 16, 28, 29]. When prompted with in-context examples of chain-of-thought rationales, LLMs have shown impressive performance on tasks that require reasoning abilities [16, 29, 30]. Subsequent work [28] further explores the zero-shot reasoning ability of language models by simply prompting with 'Let's think step by step'. Another family of works excavate the task-planning ability of large language models [19, 20, 31, 32, 33]. They prompt the LLM to generate a programming over sub-tasks which are executed by pre-defined sub-modules. Among them Visual Programming [19] achieves impressive performance with good interpretability on GQA dataset [34], which is designed to examine the compositional reasoning ability. However, since the answer is generated through the program execution, it has limited application on open-knowledge VQA.
Vision-language Model. Vision-language pre-training models have made significant progress in recent years [35, 36, 37, 38, 39, 40]. They first pretrain the model on large-scale image-text data to learn a good image and language representation, then fine-tune on various down-streaming tasks. They exhibit strong transfer ability and achieve state-of-the-art on various vision-language tasks. Another line of works train the model by contrastive learning, aiming to align visual concepts and language representations [41, 42]. Besides, BLIP-2 [43] proposes a pre-training strategy to first learn vision-language representation learning from a frozen image encoder, then learn vision-to-language generative learning from a frozen language model. However, these vision-language pre-training models require large vision-language datasets and computation resource to fine-tune. Our method uses some pre-trained vision-language models to attend to the relevant region on the image and further collect the required visual information.
In this section, we introduce our task-oriented active (TOA) VQA which simulates the cognitive process of humans involving three essential parts: reasoning, hypothesis, and verification. To this end, we design two agents, the task scheduler and the visual executor. These two agents engage in multi-round dialogues to communicate with each other. The scheduler parses the question, analyzes the required visual information, makes proper hypotheses to the visual executor for verification, and decides on the final answer with comprehensive consideration. The visual executor consists of a set of vision functions and collects visual information according to the requests from the scheduler. The illustration of the proposed method is shown in Figure 1.
3.1 Task Scheduler
The task scheduler leverages the natural language understanding and chain-of-thought [16] reasoning abilities of LLM, which plays the role of task analysis, planning, and decision-making. Given the input question and a brief description of the image generated by a captioning model [13], the LLM first parses the question and the information provided by the caption. Then it either makes an initial hypothesis and collects visual evidence to verify the hypothesis, or acquires more visual information when there is insufficient information to make any hypothesis. In the subsequent steps, the LLM makes further analysis considering the collected visual information. It can adjust the current hypothesis according to the previous conversation and ask for further verification. When the hypothesis is confirmed by the visual evidence, it decides the final answer and ends the dialogue.
Since the language model generates natural language based on a probabilistic model, they tend to generate diverse contents. To facilitate the interaction between the language modality and the vision models, we restrict the output format to a combination of free-form natural language and symbolic representation. Specifically, the output format consists of four parts: REASONING, HYPOTHESIS, VERIFICATION, ANSWER as the following template:
REASONING:  $r$  HYPOTHESIS:  $[h_1, h_2, \ldots]$  VERIFICATION: function(args) ANSWER: a
At each step, the instructions generated by the scheduler contain all four parts above to guarantee format consistency. Each of them can be None when inapplicable. REASONING is the analysis basis of the subsequent parts. It contains a chain-of-though reasoning process that illustrates the hypothesis and requires visual verification. HYPOTHESIS is a list of possible answers to the question. There can be multiple hypotheses at each time, while when the current information is insufficient to make any hypothesis, then HYPOTHESIS would be marked as None. VERIFICATION specifies the instruction to call the vision functions. The instruction is constructed in Python similar to [20, 19], which consists of the name of the function and the specified args. ANSWER is the prediction result to the question. It is only specified when the scheduler has decided on the final answer, otherwise marked as None.
3.2 Visual Executor
Despite the strong reasoning and language understanding abilities of LLMs, they have limited capability for image understanding as fine-tuning on vision-language data requires large computation resource. To alleviate this issue, we leverage several vision models to execute the requests from the scheduler and collect the required visual evidence to verify the hypothesis. By doing so, the LLM can actively acquiring specific visual information that it believes to be crucial for enriching its understanding of the image to make an appropriate decision. Moreover, our proposed method extracts the visual information in a task-oriented manner, i.e., tailored to the question, which can more efficiently attend to the useful information and reduce the noisy information than existing methods.
Motivated by human cognition, we correspondingly design our visual executor to have the task-oriented visual understanding module from two aspects, spatial attention (i.e., Where to look) and attribute attention (i.e., What to look). Instead of looking at the whole image aimlessly, our task-oriented image understanding module first locates the region of interest that is most relevant to the question. Then it extracts different image textures according to the query, including attribute-level features and semantics-level features. The functions of the vision executor are shown in Figure 2. We leverage several pre-trained vision-language models to implement the visual executor module, which will be elaborated in Section 4.1.
locate(object)
relate(object, relation)
crop的方向
Returns an image patch containing the object
Locates the region-of-interest according to the relationship with the specified object
Crops the image to the specified direction
attribute(object, attribute)
filter(object, property)
exist(object)
query(property)
count(object)
Returns the queried attribute of the object
Classifies if the object possesses the property
Detects if the target object exists
Answers simple queries about the image region
Returns the number of the target object
locate(elephant) relate(elephant, left)
crop(above)
Figure 2: Description of the spatial and attribute vision modules (left) and execution examples of spatial function (right).
3.3 Multi-rounds Interaction
In our proposed method, the task scheduler and the visual executor can interact with each other through multi-round dialogue. In this way, the scheduler in our method can acquire the essential information progressively, and it can also dynamically decide the next step based on the previous conversation. When there are multiple hypotheses that can be the possible answers to the question, the scheduler will verify each of them sequentially. If the visual verification conflicts with one of the hypotheses, then this hypothesis is excluded from all the hypotheses. However, sometimes all the hypotheses may be excluded by the visual verification. This may be caused by misleading results of the vision model or because all the hypotheses made by the scheduler are wrong. In such cases, the scheduler needs to jointly consider the common sense knowledge and the collected visual information. Thus we let the scheduler consider all the information at hand to either ask for more visual information to make a new hypothesis, or maintain the initial hypothesis and give the answer with the highest probability. Compared to the methods that directly generate the answer in the program execution [19, 20] and fully rely on the results of the vision models, our proposed method encourages the LLM to actively seek essential visual evidence oriented to the question in a sequential hypothesis-verification procedure. This may reduce the impact of the inferior results of vision models especially when asked to perform challenging vision tasks, since the LLM can make an alternative request which is more feasible to vision models.
When the scheduler confirms the answer through visual verification, it stops the dialogue and outputs the answer. Since the scheduler decides the cessation of the dialogue accordingly, in contrast to the previous works that end the dialogue after fixed rounds [44], our hypothesis-verification process through the dialogue can be more efficient and reduces redundant interactions.
3.4 Prompting Strategy
Now we introduce our prompting strategy to activate the reasoning ability of LLM. It consists of a brief description of the task requirements, instructions to leverage the available vision models, and a few in-context examples to enforce the LLM to generate the output in a consistent format. Specifically, the task description emphasizes the idea of hypothesis-verification, and prompts the task scheduler to leverage the vision functions to gather information. It is designed as follows:
I have a question about an image. Each time you need to make a hypothesis and verify by collecting visual information. You can use the following visual functions to gather information. You can only call one vision function at a time. The answer should be very concise.
The instruction for leveraging the available vision models is Python-like pseudo code that describes the vision function, inspired by the API design in ViperGPT [20]. In this way, the scheduler can
better understand the function of the vision models, and it facilitates the scheduler to call the required vision functions. An example is shown as follows:
def filter(_object:str,_property:str)->bool:  
    ['']  
presupposes the existence of _object.  
['']  
return True if _object possesses the _property else False.
We also provide the large language model with a few in-context examples of the multi-round dialogue to guarantee the format consistency of the textual output. Previous works [45, 11] have indicated that in-context example selection is essential to the result. To find the most representative examples, we extract the image and question embedding using CLIP model [41]. Then we cluster the image-question pairs with the feature embedding and select the examples that are closest to the cluster centers as the representative examples. In the inference stage, we follow the in-context example selection approach in [11]. For each input data, we compute its cosine similarities with the available examples in the embedding space and select the top  $k$  examples with the highest similarities. Specifically, for the training data  $X = \{v_{i}, q_{i}\}_{n}$ , we denote the fused image and question embeddings as  $z_{i} = (v_{emb}^{i}, q_{emb}^{i})$ , and the clustering centers as  $C = \{c_{1}, c_{2}, \dots, c_{m}\}$ . The representative examples of each cluster  $c_{j}$  are selected as follows:
$$
r _ {j} = \arg \max  _ {z _ {i}} \frac {c _ {j} ^ {\top} z _ {i}}{\| c _ {j} \| _ {2} \| z _ {i} \| _ {2}}. \tag {1}
$$
Then given a testing input image-question pair  $(v,q)$  and  $z = (v_{emb},q_{emb})$ , the top  $k$  in-context examples with maximum similarities are:
$$
\mathcal {I} = \underset {i \in \{1, 2, \dots , m \}} {\arg \operatorname {T o p K}} \frac {z ^ {\top} r _ {i}}{\| z \| _ {2} \| r _ {i} \| _ {2}}. \tag {2}
$$
Datasets. We mainly evaluate our proposed method on knowledge-based VQA dataset OK-VQA [1] and conduct several experiments on A-OKVQA [2] as supplementary. OK-VQA is a commonly used knowledge-based VQA dataset that contains 14,055 image-question pairs associated with 14,031 images from MSCOCO dataset [46]. The questions are manually filtered to ensure all questions require outside knowledge to answer. Each question is annotated with ten open-ended answers. We evaluate our method on the test set. A-OKVQA is an augmented knowledge-related dataset containing 25K image-question pairs. It also provides supporting facts and rationales to obtain the answer. We conduct our experiments on the validation set. A-OKVQA benchmark encompasses both multiple-choice settings and direct-answer settings without answer options.
Implementation Details. In the experiments, we implement the task scheduler using gpt-3.5-turbo<sup>1</sup>. For the vision executor, we implement the spatial functions using Grounding DINO [47], which is an open-set object detector. The attribute functions are implemented by vision-language pre-training models BLIP2 [43] and X-VLM [48]. The representative in-context examples are selected using agglomerative clustering [49] with cosine metrics, and the feature embeddings are extracted by CLIP [41]. We prompt the scheduler with 16 in-context examples in the experiment.
Evaluation Metrics. The commonly used evaluation metrics for both OK-VQA and A-OKVQA direct-answer settings are the soft accuracy proposed in VQAv2 [3], where accuracy = min (#humans that provided that answer/3, 1). The predicted answer is deemed  $100\%$  accurate if at least 3 humans provided the exact answer. However, with the rapid development of probabilistic generative models, the conventional exact matching evaluation can not meet the requirements of open-ended question answering. Since our answer is directly predicted by LLM without prior knowledge of the answer vocabulary, it may generate answers that are semantically equivalent to the ground truth but use different expressions. Thus, we evaluate our results with a combination of
<sup>1</sup>https://platform.openai.com/docs/models/gpt-3-5
Table 1: Comparison to the state-of-the-art methods on OK-VQA dataset. †: the result is based on reimplementation.
Method	Knowledge Source	Image Representation	Accuracy
Methods with explicit knowledge sources
ConceptBERT[9]	ConceptNet	Feature	33.7
KRISP [51]	Wikipedia+ConceptNet	Feature	38.9
Vis-DPR [52]	Google Search	Feature	39.2
MAVEx [4]	Wikipedia+ConceptNet+Google Images	Feature	39.4
KAT [5]	Wikidata+GPT-3	Caption+Tags+Feature	54.4
REVIVE [6]	Wikidata+GPT-3	Caption+Feature	58.0
Methods using only implicit knowledge
IPVR [12]	GPT-3	Caption	44.6
PICa-Base [11]	GPT-3	Caption	43.4
PICa-Full [11]	GPT-3	Caption	48.0
PromptCAP† [13]	GPT-3	Caption	58.8
TOA (ours)	ChatGPT	Caption+Visual evidence	60.6
Table 2: The results on A-OKVQA dataset. DA refers to direct answer settings and MC refers to multiple choices settings.
Method	DA	MC
KRSIP [51]	33.7	51.9
IPVR [12]	46.4	-
PromptCAP [13]	56.3	73.2
TOA (ours)	61.2	63.1