SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Box 5
Show caption of images similar to box 3
Show related products to the object in box 4
Show caption of images similar to box 4
Show caption of images similar to box 5
(b) Tools shown to user
Outputs of "show entity of box2"
Harley-Davidson
- Harley-Davidson, Inc. is an American motorcycle manufacturers to survive the Great Depression a arrangements, periods of poor economic health brand widely known for its loyal following. There
- Harley-Davidson is noted for a style of customiz motorcycles with engine displacements greater
- Harley-Davidson manufactures its motorcycles and Despatch rider
- A despatch rider is a military messenger, mount
- In the UK 'despatch rider' is also a term used for
- Despatch riders were used by armed forces to d telecommunications were limited and insecure.
to access diverse information pertaining to the image crop of the box. This includes details such as corresponding knowledge graph entities, captions of similar images, titles of associated related products, and captions of identical images. An example set of tools and APIs are shown in Figure 4(b).
When the user initiates an action, such as clicking on a button or submitting a query to web search, PALM, or PALI VQA, the corresponding tool is invoked, and the resulting output is displayed to the user. We record the sequence of actions taken by the user and the outputs that they receive at each step. For instance, in Figure 4, we show an example of how a user needs to perform four actions to answer the question:  $i$  ) display entities in box 2, ii) show the caption of similar images to box 2, iii) conduct a search for "In what year was Harley-Davidson XA built?", and iv) utilize PALM using the combination of the search output and the question "In what year was Harley-Davidson XA built?" When the user is prepared to proceed to the next question, they click on either of the two buttons: "Success! Found the Answer!" or "Couldn't Find the Answer." Subsequently, a new visual question is presented to them.
The collected user behavior serves as a guide for our system in two key ways. Firstly, we construct a transition graph by analyzing the sequence of decisions made by users. This graph defines distinct states and restricts the available set of actions at each state. For example, at the START state, the system can take only one of these three actions: PALI caption, PALI VQA, or object detection. Figure 3 illustrates the transition graph that has been constructed based on the decision-making process of the users. Secondly, we utilize the examples of user decision-making to guide our planner and reasoner with relevant contextual instances. These in-context examples aid in enhancing the performance and effectiveness of our system.
We conducted a user study involving 10 participants who collectively answered a total of 644 visual questions. During the study, we presented users with visual questions that were randomly selected from both the Infoseek [7] and OK-VQA [26] datasets. This approach allowed us to provide the participants with a varied and diverse set of visual questions to assess and respond to. We show the details for this study as well as example prompts in the Appendix.
We evaluate AVIS on two visual question answering datasets:  $i$  ) OK-VQA [26], which requires common-sense knowledge not observed in given image; and  $ii$  ) Infoseekwikidata [7], which further necessitates more fine-grained information that cannot be covered by common sense knowledge.
Experimental Setup. We follow the decision-making workflow in Alg. 2 to implement AVIS to solve visual questions. For the Planner, we write the basic instructions for describing each tool, and keep a pool of real user behavior when they select each tool, which we collected in the user study. At each
Model	Unseen Entity	Unseen Question
PALM [9] (Q-only, few-shot)	3.7	5,1
OFA [22] (fine-tune)	9.7	14.8
PALI [6] (VQA, zero-shot)	1.8	2.2
PALI [6] (fine-tune)	16.0	20.7
PALM [9] w/ CLIP [32] (few-shot + external knowledge)	21.9	18.6
FiD [45] w/ CLIP [32] (fine-tune + external knowledge)	20.7	18.1
(baselines without dynamic decision making, sequentially execute the tools—)
baseline-PALM w/ (PALI*, few-shot)	12.8	14.9
baseline-PALM w/ (PALI* + Object, few-shot)	31.3	36.1
baseline-PALM w/ (PALI* + Object + Search, few-shot)	36.1	38.2
AVIS (ours, few-shot)	50.7	56.4
w/o PALI*	47.9	54.2
w/o Object	41.2	48.4
w/o Search	42.5	49.6
Table 1: Visual Question Answering results (accuracy) on InfoseekWikidata. The first four rows are results from their paper that do not use external knowledge, and the next two are from their paper that use CLIP as knowledge source. The tool PALI* denotes the frozen multi-task PALI-17B model for both visual question answering and image captioning. Object means object detection, and search means image and text search.
step  $s$ , we prepare the prompt based on the feasible action lists  $\mathcal{A}_s$ . For the Reasoner, we write the prompt for all APIs that return a long list of results, including Object Detection, Product Detection, Web Image Search and Web Text Search, that guides reasoner to extract the relevant information. Note that we design the reasoner in a way such that the "uninformative" answers can be detected. In order to support this, we manually prepare several bad examples that do not provide any useful information, pass it to the reasoner as a part of the prompt. We show the detailed prompts for these two modules in the Appendix.
We use the frozen PALM 540B language model [9] for both the planner and the reasoner, with deterministic generation ensured by setting the temperature parameter to zero. We use 10 examples as in-context prompts for each dataset, and report the VQA accuracy [11] as the evaluation metric.
Baselines. A significant novelty of AVIS is the ability to dynamically determine the relevant tools according to different states. To show that this design choice is useful, we add a number of baselines that do not contain a LLM-planner for dynamic decision making. Instead, they follow a pre-determined sequence to call a list of tools. We propose the following baselines:
- baseline-PALM w/ PALI*, which integrates the captions generated by PALI and the visual answers from PALI VQA. PALI* denotes the combination of both VQA and captioning tool.
- baseline-PALM w/  $(\mathbf{PALI}^{*} + \mathbf{Object})$ , which in addition calls the object detection tool, and then integrates all object data, including products and text detected by OCR.
- baseline-PALM w/ (PALI* + Object + Search), a model which first selects a relevant object with the help of PALM, then sequentially executes the image search and Google search with the object name. It then calls PALM again to answer the question.
For each of the three baselines, we prepare a few-shot Chain-Of-Thought (COT) prompting [39], in which the COT prompt guides the model to explain why predictions are made based on the provided information. Note that these baselines utilize a set of tools in a fixed order, without the capacity for dynamic decision making.
We also evaluate the usefulness of each tool group (i.e., PALI*, Object, and Search) through an ablation study. This involves removing each tool group from our framework individually, and assessing the impact on performance.
Experimental Results. Table 5 presents the results of AVIS and other baselines on the Infoseekwikidata dataset. Infoseekwikidata is a challenging dataset that requires identifying highly specific entities. Even robust visual-language models, such as OFA [22] and PALI [6], fail to yield
Model	Accuracy (%)
Supervised	KRISP [25]	38.4
KAT [12]	54.4
ReVIVE [20]	58.0
REVEAL [15]	59.1
PALI [6] (OK-VQA, finetune)	64.5
Zero-shot	PALI [6] (VQA, zero-shot)	41.6
PICa-Full [41]	48.0
Flamingo (zero-shot) [4]	50.6
BLIP-2 [18]	45.9
Few-shot	ViperGPT (one-shot) [36]	51.9
Flamingo (few-shot) [4]	57.8
(baseline without dynamic decision making, sequentially executing the tools)
baseline-PALM w/ (PALI*)	44.3
baseline-PALM w/ (PALI*+Object)	38.2
baseline-PALM w/ (PALI*+Object + Search)	47.9
AVIS (ours)	60.2
w/o PALI*	47.1
w/o Object	58.3
w/o Search	55.0
Table 2: Visual Question Answering results (accuracy) on OK-VQA. The tool PALI* denotes the frozen multi-task PALI-17B model for both visual question answering and image captioning. Object means object detection, and search means image and text search.
Figure 5: Examples of AVIS's dynamic planning and reasoning procedure for solving visual questions.
high accuracy when fine-tuned on this dataset. However, our AVIS, without fine-tuning and by leveraging a complete set of tools guided by 10 in-context examples, achieves the accuracy of 50.7 and 56.4 on the unseen entity and question splits, respectively. This significantly outperforms the fine-tuned results of PALI-17B, which are 16.0 and 20.7, as well as the PALM model augmented with CLIP knowledge, which are 21.9 and 18.6, respectively.
Table 5 also illustrates that our improvements are not solely due to the additional information provided by the external tools, but due to our dynamic decision-making pipeline. We compare the results of AVIS with the three baselines that conduct sequential execution. While these baselines do improve the performance, our AVIS framework outperforms the best baseline model by up to 17.3 accuracy. Note that AVIS and the baselines use exactly the same set of tools. This considerable performance gap clearly shows the clear advantage of our dynamic decision-making design. Furthermore, we show the importance of each tool in the last block of Table 5. Removal of any of the tools degrades the overall accuracy. Among the three tool groups, Object and Search are more important than PALI, as they provide more fine-grained information crucial for the Infoseek dataset.
We report the OK-VQA experiments in Table 2. AVIS with few-shot in-context examples achieves an accuracy of 60.2, higher than most of the existing methods tailored for this dataset, including KAT [12], ReVIVE [20] and REVEAL [15]. AVIS achieves lower but comparable performance compared to PALI model fine-tuned on OK-VQA. This difference, compared to Infoseek, may be attributed to the fact that most QA examples in OK-VQA rely more on commonsense knowledge than on fine-grained knowledge. Therefore, it is feasible to encode such generic knowledge in the model parameters and requires less external knowledge. Note that PALI zero-shot VQA model itself achieves 41.6 accuracy, which is significantly higher than in Infoseek, which supports this hypothesis. Table 2 also shows that the object detection is less crucial as a tool on this data set, compared to PALI captioning and VQA. Nonetheless, AVIS equipped with all tools achieves the best performance.
Case studies for dynamic decision making. One of the key features of AVIS is its ability to dynamically make decisions instead of executing a fixed sequence. Figure 5 presents three examples of AVIS's dynamic planning and reasoning process. They demonstrate the flexibility of AVIS to use different tools at various stages. It is also worth noting that our reasoner design enables AVIS to identify irrelevant information, backtrack to a previous state, and repeat the search. For instance, in the second example concerning the taxonomy of fungi, AVIS initially makes an incorrect decision by selecting a leaf object. However, the reasoner identifies that this is not relevant to the question, prompting AVIS to plan again. This time, it successfully selects the object related to false turkey-tail fungi, leading to the correct answer, Stereum. Some detailed error analysis is shown in Appendix F.
In this paper, we propose a novel approach that equips the Large Language Models (LLM) with the tree-search to use a variety of tools for answering knowledge-intensive visual questions. Our methodology, anchored in human decision-making data collected from a user study, employs a structured framework that uses an LLM-powered planner to dynamically decide on tool selection and query formation. An LLM-powered reasoner is tasked with processing and extracting key information from the output of the selected tool. Our method iteratively employs the planner and reasoner to leverage different tools until all necessary information required to answer the visual question is amassed.
**Limitation Statement:** Currently AVIS is specifically designed for visual question answering. We aim to extend our LLM-powered dynamic decision-making framework to address other reasoning tasks. Additionally, our current framework depends on a computationally intensive LLM, namely, the PALM model. We are interested in investigating whether this decision-making framework can also be performed by lighter weight language models.
[1] Google lens. Web interface available at https://images.google.com.
[2] Google search. Web interface available at https://www.google.com.
[3] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[4] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
[7] Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M.-W. Chang. Can pre-trained vision and language models answer visual information-seeking questions? In arXiv preprint arXiv:2302.11713, 2023.
[8] F. Chern, B. Hechtman, A. Davis, R. Guo, D. Majnemer, and S. Kumar. TPU-KNN: K nearest neighbor search at peak flop/s. CoRR, abs/2206.14286, 2022.