AVIS: Autonomous Visual Information Seeking with Large Language Model Agent
Authors: Ziniu Hu
Ahmet Iscen
Chen Sun
Kai-Wei Chang
Yizhou Sun
David A Ross
Cordelia Schmid
Alireza Fathi
Venue: NeurIPS 2023 poster (2023)
Keywords: large language model
visual question answering
dynamic decision making
Tool augmented LLM
TL;DR: we use the Large Language Model as a planner to dynamically utilize APIs based on recorded human bahaviors to solve knowledge-based vision question answering

Abstract: In this paper, we propose an autonomous information seeking visual question answering framework, AVIS. Our method leverages a Large Language Model (LLM) to dynamically strategize the utilization of external tools and to investigate their outputs via tree search, thereby acquiring the indispensable knowledge needed to provide answers to the posed questions. Responding to visual questions that necessitate external knowledge, such as "What event is commemorated by the building depicted in this image?", is a complex task. This task presents a combinatorial search space that demands a sequence of actions, including invoking APIs, analyzing their responses, and making informed decisions. We conduct a user study to collect a variety of instances of human decision-making when faced with this task. This data is then used to design a system comprised of three components: an LLM-powered planner that dynamically determines which tool to use next, an LLM-powered reasoner that analyzes and extracts key information from the tool outputs, and a working memory component that retains the acquired information throughout the process. The collected user behavior serves as a guide for our system in two key ways. First, we create a transition graph by analyzing the sequence of decisions made by users. This graph delineates distinct states and confines the set of actions available at each state. Second, we use examples of user decision-making to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions. We show that AVIS achieves state-of-the-art results on knowledge-based visual question answering benchmarks such as Infoseek and OK-VQA.

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent
Ziniu Hu 12+
Ahmet Iscen?
Chen Sun?
Kai-Wei Chang
Yizhou Sun'
David Ross
Cordelia Schmid
Alireza Fathi²
University California, Los Angeles,
Google Research
Abstract
In this paper, propose an autonomous information seeking visual question swering framework, AVIS. Our method leverages a Large Language Model (LLM) dynamically strategize the utilization of external tools and to investigate their outputs via tree search, thereby acquiring the indispensable knowledge needed provide answers the posed questions. Responding visual questions that necessitate external knowledge, such as "What event is commemorated by the building depicted in this image?", is a complex task. This task presents a com- binatorial search space that demands sequence of actions, including invoking APIs, analyzing their responses, and making informed decisions. We conduct a user study to collect a variety of instances of human decision faced with this task. This data is then used to design a system comprised of three components: an LLM powered planner that dynamically determines which tool to use next, an L.M-powered reasoner that analyzes and extracts key information from the tool outputs, and working memory component that retains the acquired information throughout the process. The collected user behavior serves as a guide for our system in two key ways. First, create transition graph by analyzing the sequence of decisions made by users. This graph delineates distinct states and confines the set of actions available at each state. Second, we examples of user decision-male massionnmaking to provide our LLM-powered planner and reasoner with relevant contextual instances, enhancing their capacity to make informed decisions. We show that AVIS achieves state-of-te -ortte--rttttttt results on knowledge-intent -ntteeiiiee visual question answering benchmarks such as Infoseck [7] and OK- VQA [26].
when
Introduction
Large language models (LLMs), such as GPT3 [5), LaMDA [16], PALM [9], BLOOM [34] and LLaMA [37], have showcased the capacity to memorize and utilize significant amount of world knowledge. They demonstrate emerging abilities [38] like in-context learning [5). code genera- tion [19], and common sense reasoning [24]. Recently, there is growing focus towards adapting LLMs to handle multi- modal inputs and outputs involving both vision and language. Noteworthy examples of such visual language models (VLMs) include GPT4 [29], Flamingo [4] and PALI [6].
They set the state of the art for several tasks, including image captioning, visual question answering. and open vocabulary recognition.
While LLMs excel beyond human capabilities in tasks involving textual information retrieval, the current state of the VLMs perform inadequately on datasets designed for visual information
"This work was done when Ziniu was an intern at Google.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

START Question: When the boat first used for this event?
Step Informativel
Uninformative, backtrack
Uninformative, backtrack
Image
Select Object
Select Object #1 Big wooden bost with many people on R...
People wering
Stop 4
Informative!
Search Similar Images Object
Gin Matuur 19l6
Step S Informative!
Q: In the Image, what is the boat
A: tamaboka Hoaat; Gion Matsuri
Step Uninformative. backtrack
Stap Informative!
Answer Textual Question
Web Search
Hlabbyy The Gian Testral centr
Q: When Yamabokk Flost first used? A: Ancieen Japan
malicii the wrssyy Feets
Step Answerable!
Answer Textual Question
Starting from 14th Century,
Yamahoko for Festival every July 17. Answer: 14th century.
Figure 1: An example of AVIS's generated workflow for answering challenging visual question using LLM with tree search to use tools. The input image is taken from the Infoseek dataset.
seeking such as Infoseek [7] and VOA [26]. Many of the visual questions in these datasets are designed in such way that they pose challenge for humans, often requiring the assistance of various APIs and web search to obtain the answer. Examples such questions include "where is this church located?". "what species of butterfly is this?", "what is the brand of this dress?".
Current state-of mate-of-the-se the vision- Lasiua-language models (VLMs) find challenging to answer such questions for several reasons. Firstly, they are trained with objectives that encourage them discern fine-graine categories and details within images. Secondly, they utilize a relatively smaller language model compared state-of-the-an Large Language Models (LLMs), which constrains their reasoning capabilities. Lastly, they do not compare the query image against a substantial corpus of images associated with varying metadata, unlike systems that employ image search techniques.
To overcome these challenges, we introduce novel method in this paper that achieves state-of-the-an results visual information seeking tasks by enabling LLM Agent use tools via tree-search decision-making. We use three types tools: (i) computer vision tools such as object detection, OCR, image captioning models, and VQA models, which aid in extracting visual information from the image, (ii) a web search tool that assists in retrieving open world knowledge and facts, and (iii) an image search tool that enables to glean relevant information from metadata associated with visually similar images. Our approach utilizes an LLM LLM--ooered planner to dynamically determine which tool to use at each step and what query to send to it. Furthermore, that scrutinizes the output returned by the tools and extracts the crucial information from them. To retain the information throughout the process, we working memory component. Figure I shows an example information seeking process performed by our method.
employ LLM LLM-powered reasoner
Several recent studies [13, 23, 36, 40, 42] have enhanced LLMs with APIs to handle multi-modal vision-language inputs. These systems generally employ a two-stage strategy, namely plan and execute. Initially, the LLM breaks down question into plan, typically represented as a structured

---

program or sequence of instructions. Following this, the necessary APIs are activated to collect the required information. While this method has shown potential elementary visual-langun language tasks, frequently fails in more complex real-world situations. In such cases, comprehensive plan cannot be inferred merely from the initial question. Instead, necessitates dynamic modifications based on real-time feedback.
The primary innovation in our proposed method lies in its dynamic decision-making capability, Answering visual information seeking questions is a highly complex task, requiring the planner to take multiple steps. At each of these steps, the planner must determine which API to call and what query to send. It is unable to predict the output of complex APIs, such as image search, or to anticipate the usefulness of their responses prior calling them. Therefore, unlike previous methods that pre- pre-plan the steps and API calls at the beginning of the process, we opt for a dynamic approach. We make decisions at each step based on the information acquired from previous API calls, enhancing the adaptability and effectiveness of our method.
We conduct a user study to gather wide range of instances of human decision-making when using APIS to answer questions related to visual information seeking. From this data, we formulate a structured framework that directs the Large Language Model (LLM) to use these examples for making informed decisions regarding API selection and query formulation. The collected user behavior informs our system in two significant ways. First, by analyzing the sequence of user decisions, we construct a transition graph. This graph delineates distinct states and constrains the set of actions available at each state. Second, we use the examples of user decision maaiinn to guide our planner and reasoner with pertinent contextual instances. These contextual examples contribute to improving the performance and effectiveness of our system.
The primary contributions of this paper can be summarized as follows:
We propose a novel visual question answering framework that leverages a large language model (LLM) to dynamically strategize the utilization of external tools and to investigate their outputs, thereby acquiring the necessary knowledge needed to provide answers to the posed questions.
We leverage the human rrcision-making data collected from user study to develop structured framework. This framework guides the Large Language Model (LLM) to utilize examples of human decision in making informed choices concerning API selection and query construction.
Our method achieves state-of-the-an results on knowledge-based visual question answering benchmarks such as Infoseek [7] and OK-VQA [26]. Notably, We achieve an accuracy of 50.7% on the Infoseek (unseen entity split) dataset which is significantly higher than the results achieved by PALI [6] with accuracy of 16.0%.
2 Related Work
Augmenting LLMs with Tools. Large Language Models(LLMs) have shown impressive language understanding [33], and even reasoning capabilities [39]. Nevertheless, certain limitations of LLMs evident, due to their intrinsic characteristics. Such limitations include providing up-to-date answers based on external knowledge or performing mathematical reasoning. Consequently, a recent surge of techniques have integrated LLMs with various external tools [27]. For example, TALM [31] and ToolFormer [35] use in-context learning to teach the language model how to better leverage various tools on benchmarks such as question answering and mathematical reasoning.
In the computer vision domain, LLMs also show significant improvements when combined with external visual tools. For example, Visual ChatGPT [40] and MM-ReAc [42] enable LLMs to call various vision foundation models as tools to understand visual inputs, and even better control the image generation. VisProg [13] and ViperGPT [36] explore the decomposition of visual language tasks into programs, where each line corresponds to general code or a visual API. Chameleon [23] uses LLM natural language planner to infer the appropriate sequence of tools to utilize, and then executes these tools to generate the final response.
Most of these previous works follow plan-then-execute paradigm, i.e., i) they pre-plan the sequence of actions (API calls) that they will take (either hard coded or using code generation); and ii) they exccute the generated plan. One drawback of such approach is that it cannot update and improve
3

---

its plan based on the output of the tools it calls. This is not a trivial problem. as it requires to predict the oulput quality of each tools beforehand. In contrast, our proposed method allows the system to dynamically decide its next steps based on the output it receives from the tools at each step.
Decision Making with LLM as an Agent. There has also been surge of interest in applying Large Language Models (LLMs) as autonomous agents. These agents are capable interacting with external environments, making dynamic decisions based on real-time feedback, and consequently achieving specific goals. For example, WebGPT [28] enables an L.LM to access real-time information from the web search engines. ReAct [44] further improves external search engine usage via the self- reasoning of LLM in an interleaved manner. Similar ideas have also been adopted for robotic action planning. SayCan [3], for instance, uses LLMs to directly predict robot actions, and PALM-E [10] further tunes LLMs to make better decisions based on instructions and open web media.
When compared to works that follow plan- plan-then-exee then-execute paradigm, these Al agents exhibit increased flexibility, adjusting their actions based on the feedback that they receive. However, many of these methods do not restrict the potential tools that can be invoked at each stage, leading to an immense search space. This becomes particularly critical for web search APIs [1. 2] that return extensive result lists and span combinatorial search space of multiple tools. Consequently, even the most advanced LLMs today can fall into infinite loops or propagate errors. To alleviate this issue, we propose restricting and guiding LLMs to mimic human behavior when solving complex visual questions with APIs. This idea similar to the Al aligmment research [21, 30] that teaches LLMs to follow human instructions. The difference is that our model only uses the human prior the Decision-making stage via prompt guidance, instead of re- re-training the model.
One concurrent work Tree-Of Thought (ToT) [43] also utilize tree search guided by a self-critic reward model to find optimal path of problem solving. Compared with this concurrent work, our AVIS further constrains the tree search via human-defined transition graph, and guide the decision-make via a dynamic prompt manager. addition, though AVIS is designed for tool-use, the success of ToT shows that such idea can be generally improve many LLM Reasoning tasks.
Method
3.1 General Framework
Our approach employs dynamic decision-make making strategy designed to respond to visual information- seeking queries. Our system is comprised of three primary components. First, we have planner P. whose responsibility is determine the subsequent action, including the appropriate API call and the query it needs to process. Second, we have a working memory M that retains information about the results obtained from API executions. Lastly, have reasoner R, whose role is to process the outputs from the API calls. It determines whether the obtained information is sufficient to produce the final response, or if additional data retrieval is required.
Available Tools (actions) &
LLM
Cur. State
Algorithm Planner P(state,
Planrer
1: Ag + ó(saatt, (tstatt,,,MM G.M) list of feasible actions A, given the current state from transition graph and the information in the working mem-
D> Get the
Query
Working Memory
Executor
ory 2: E, + O(EE A.) in-context examples related to actions A.
Execution Results
Get list of
Transition Graph and In-Context Examoles
New State
S LUM Reasoner
3: + ().,.M) > Build prompt based on the in-context examples E, and the current working memory M ta + LLM & Decide the next tool t, to use and the query 9, to pass by feeding the prompt P. LLM
Vos, update state Informative transition graph...
No. Backtrack
Figure 2: AVIS employs dynamic decision -akiin to plan (find optimal tool and query), execute results, and then reason (estimate whether continue or backtrack).

---

START
Algorithm AVIS Decision Making Workflow
1: M <- {input), state <- START
2: tu. 9s + P(state, MM)
D Call the
Captioning
planner P to decide the next tool use L, and the query to pass to it
3: Or Execc(f, 9s) Call tool E, with query and get output Os
Select Cibject
4: o --,. MM)
Process the output and extract the key info ós using the reasoner R
Imagee Search
VQ4
5: M.add((g)
Update the working memory
6: switch Ô₂ do case Ô, is not informative got
Web Search
8:
Go to line 2 to make decision at the same state, excluding 1.s.
LLM shoet QA
9:
case Ôg has useful information
10: state ts
Update state
11:
goto((2 decision for the next state.
D Go to line to make
FINISH
Figure 3: Transition graph defines feasible ac- 12:
case Og is ready as final answer
tions the planner can take. This graph is induced by our user study introduced in Sec. 3.3.
ans Ô,
D Output answer
Considering the potential intricacy of the task, we conduct user study to gather a broad range of examples of human decision-making process. when using tools respond visual information- seeking queries (we introduce the details of data collection in Sec. 3.3). This helps us to establish a structured framework for decision- manonn. We utilize the data collected from this study to construct transition graph G shown in Figure 3, which outlines all the possible actions at each given state. Additionally, we employ real-life decision-making examples E, i.e., users choose which tool at different states, to guide the planner in choosing the appropriate action at each stage of the process.
The Algorithm presents the operations of the planner P. The planner undertakes series of steps each time decision is required regarding which tool to employ and what query to send to it. Firstly, based on the present state, the planner provides range of potential subsequent actions A,, The potential action space A, may be large, making the search space intractable. To address this issue, the planner refers to the human decisions from the transition graph G to eliminate irrelevant actions. The planner also excludes the actions that have already been taken before and are stored in the working memory M. Formally, this procedure is A, < ((state, 9.M).
Next, it collects set of relevant in-context examples E, that are assembled from the decisions previously made by humans during the user study relevant actions A.. that is &, A,). With the gathered in-context examples & and the working memory M that holds data collected from past tool interactions, the planner formulates prompt. denoted by Pa + MM). The prompt ps is then sent to the LLM which returns structured answer, determining the next tool t. to be activated and the query 9s to be dispatched it. We denote this action by t,, qs + LLM This design allows the planner to be invoked multiple times throughout the process, thereby facilitating dynamic decision-makingg that gradually leads to answering the input query.
The Algorithm 2 shows the overall decision-making workflow of AVIS. The entire process repeats until satisfactory answer is produced. Initially, the working memory is populated only with the input visual question I, and the initial state is set to START. At each iteration, we first invoke the planner P to determine the next tool and the query to employ, selected external tool executes and delivers its output o.. The output from the tools can be quite diverse, ranging from list of identified objects, collection of similar images with their captions, to snippets of search results or knowledge graph entities.
outlined in Algorithm 1. Subsequently, the
Therefore, we employ reasoner RR to analyze the output Os, extract the useful information and decide into which category the tool output falls: informative, uninformative, or final answer. Our method utilizes the L.M with appropriate prompting and in-context examples perform the reasoning. If the reasoner concludes that it's ready to provide an answer, it will output the final response, thus concluding the task. If it determines that the tool output is uninformative, it will revert back to the
5

---

planner to select another action based on the current state. If it finds the tool output to be useful, it will modify the state and transfer control back to the planner to make a new decision at the new state.
Our approach, which employs dynamic decision-making coupled with backtracking, differs from previous methods [23,36] that follow a plan-then-execute paradigm. Our system is structured to make decisions grounded to the results of current executions and to conduct iterative searches for tool combinations. This process eventually yields the most effective strategy to accomplish the task.
3.2 Tools and their APIs
To respond effectively to visual queries that necessitate in- in-depth information retrieval, it's important to equip AVIS with a comprchensive suite of tools. In this section, we describe these tools.
Image Captioning Model:
We employ the PALI 17B [8] captioning model, which obtains state-of- the-art results for image captioning. This tool has the capability to generate captions for either the entire image or for a cropped image corresponding to the bounding box of detected object.
Visual Question Answering Model: We utilize the PALI 17B [8] VQA model, which has been fine-tuned on the VQA-v2 [11] dataset. This tool takes an image and a question as inputs and provides text-based answer as the output.
We use an object detector trained on a super-set of Open Images dataset [17] categories that is provided by Google Lens API [1]. We use high confidence threshold to only keep the top-ranked detected boxes for the input image.
Object Detection:
We utilize Google Image Search to obtain broad range of information related to the image crop of detected box as provided in Google Lens API [1]. This information encompasses various details, such as knowledge graph entities, titles of associated products, and captions of analogous or identical images. The availability of these details can vary based on the image crop input provided to Google Image Search. When it comes to decision-making, our planner considers the utilization of each piece of information as a separate action. This due to the fact that each information could contain hundreds of tokens that necessitate complex processing and reasoning.
Image Search:
OCR: In some cases, images may include textual content such street names or logos. To detect and utilize this text, we take advantage of the Optical Character Recognition (OCR) feature available in the Google Lens API [1].
Web Search:
Web search enables our approach to acquire up-to-date world knowledge and retrieve relevant documents on any topic of interest. For this objective, we employ the Google Web Search API [2]. It accepts a text-based query as input and produces the following outputs: (i) related document links and snippets, (ii) in certain instances, knowledge panel providing direct answer to the query, and (iii) up to five questions that are related to the input query. If knowledge panel is available, we parse into a sentence or few sentences that summarize its information.
LLM short QA: We incorporate a Language Model (LLM) powered question-anss answering component as another tool. This tool accepts a query in text form and produces answer also in text form. It is important to note that the use the LLM here as a question-answering tool is distinct from its role in the planner or reasoner outlined in Alg. 1 and Alg. 2.
3.3 Gathering User Behavior to Inform LLM Decision Making
Many of the visual questions in datasets such as Infoseek [7], Oven [14] and OK-VQA [26) ask for fine-grained answers, which poses a challenge even for humans, often requiring the assistance of various APIs and web searches for answers. Figure 4(a) illustrates example visual question taken from the OK-V VQA [26] dataset. In order to gather insights into human decision-making process, we carried out user study. More specifically, our goal is to understand how humans utilize external tools to answer visual queries that involve seeking information.
The user is equipped with an identical set of tools as our method. They are presented with the input image and question, along with image crops for each detected object. Additionally, tools like PALI Caption, PALI VQA, PALM, and Web Search are made available to the user. Furthermore, based on the information obtained through image search for each cropped image, the user offered one multiple buttons associated with each box. These buttons provide the user with the ability

---

Outputs of "show entity of box2"
MAU Daprion
Query
PAU VGA
Harley-Davee Davidson
Hariey-Daviddoon Inc. is an Americcnn motoreycle manufactuners survive the Great Depression errangementa, perodd
Search
Bearch
economic health brand widely known for its loyal following. There Hariey-Daviddon Davidson Is noted for style of customiz motorcycles engine d'pplacemeets greater Harley Davidson manufactures nes its matorcycles. Despatch rider
NALM Query RRUM
Question: in what this motorcycle built?
Show enilyy ofe
Show cerdiiv imsppe simiar box
despatch rider military messenger, mount In the UK 'despetch rider' is also term used for Despatich riders
Show
Show relited products the mbject box
by armed forces telecommunicatioos were limited and Insecure.
BoX4
Show caption of images a smiar to box
Show caption idantioll images box
Boo
Show box
Userleso
Show caption of mages aiiiarr to box
Show related products to the object box
Sucoossl Anowsr
Show caption images dimilarto
Couldt End Arsmeii!
Brx 2
Box
Show caption of similar box
(a) Input visual question and detected objects
(b) Tools shown to user
(c) Tool Output
Figure 4: We conduct a user study to gather examples of user decision-maken when responding to visual information-seekinn series of tool calls using the available APIs shown in (b). Each tool call yields an output which the user reviews whether it is useful and determines the subsequent action, illustrated in (c).
questions. Given visual question as depicted in (a), the user makes
to access diverse information pertaining the image crop of the box. This includes details such as corresponding knowledge graph entities, captions of similar images, titles of associated related products, and captions of identical images. An example set of tools and APIs are shown in Figure 4(b).
When the user initiates an action, such clicking on a button or submitting query to web search, PALM, or PALI VQA, the corresponding tool is invoked, and the resulting output is displayed to the user. We record the sequence of actions taken by the user and the outputs that they receive each step. For instance, in Figure 4, we show example of how a user needs to perform four actions to answer the question: i) display entities in box 2, ii) show the caption of similar images to box 2, ii) conduct a search for "In what year was Harley-Davidson XA buil?", and iv) utilize PALM using the combination of the search output and the question "In what year was Harley-Davidson Davidson XA built?". When the user is prepared to proceed the next question, they click on either of the two buttons: "Success! Found the Answer!" or "Couldn't Find the Answer." Subsequently, new visual question is presented to them.
The collected user behavior serves as a guide for our system in two key ways. Firstly, we construct a transition graph by analyzing the sequence of decisions made by users. This graph defines distinct states and restricts the available set of actions each state. For example, at the START state, the system can take only one of these three actions: PALI caption, PALI VQA, or object detection. Figure 3 illustrates the transition graph that has been constructed based on the decision-making process of the users. Secondly, we utilize the examples of user decision-making to guide our planner and reasoner with relevant contextual instances. These in-context examples aid in enhancing the performance and effectiveness of our system.
We conducted a user study involving 10 participants who collectively answered total of 644 visual questions. During the study, we presented users with visual questions that were randomly selected from both the Infoseek [7] and OK-VQA [26] datasets. This approach allowed to provide the participants with varied and diverse set of visual questions to assess and respond to. We show the details for this study as well as example prompts in the Appendix.
4 Experiments
We evaluate AVIS on two visual question answering datasets: i) OK- [26], which requires common-sense knowledge not observed in given image; and ii) Infoseekwitidaa [7], which further necessitates more fine- ane-grained information that cannot be covered by common sense knowledge.
Experimental Setup. We follow the decision-making workflow in Alg. 2 to implement AVIS to solve visual questions. For the Planner, we write the basic instructions for describing each tool, and keep a pool of real user behavior when they select each tool, which collected in the user study. At each
7

---

Model
Unseen Entity
Unseen Question
PALM [9] (Q-only, few-shot)
3.7
5,1
OFA [22] (fine-tune)
9.7
14.8
PALI [6] (VQA, zero-shot)
1.8
2.2
PALI [6] (fine-tune)
16.0
20.7
PALM [9] w/ CLIP [32] (few-shot + external knowledge)
21.9
18.6
FiD [45] w/ CLIP [32] (fine-tune external knowledge)
20.7
18.1
(baselinee without dynamic decision making, sequentially execute the tools-) baseline-PAI PALM w/ (PALI", few-shot)
12.8
14.9
baseline-PAn PALM w/ (PALI* Object, few-shot)
31.3
36.1
baseline- PALM w/ (PALI* Object + Search, few-shot)
36.1
38.2
AVIS (ours, few-shot)
50.7
56.4
w/o PALI"
47.9
54.2
w/o Object
41.2
48.4
w/o Search
42.5
49.6
Table 1: Visual Question Answering results (accuracy) Infoseekwıidda: The first four results from their paper that do not use external knowledge, and the next two are from their paper that use CLIP knowledge source. The tool PALI" denotes the frozen multi-task PALI-17B model for both visual question answering and image captioning. Object means object detection, and search means image and text search.
step prepare the prompt based on the feasible action lists A.. For the Reasoner, write the prompt for all APIs that return long list of results, including Object Detection, Product Detection, Web Image Search and Web Text Search, that guides reasoner extract the relevant information. Note that we design the reasoner in a way such that the "uninformative" answers can be detected. In order support this, we manually prepare several bad examples that do not provide any useful information, pass it to the reasoner as part of the prompt. We show the detailed prompts for these two modules in the Appendix.
We use the frozen PALM 540B language model [9] for both the planner and the reasoner, with deterministic generation ensured by setting the temperature parameter to zero. We use 10 examples as in-context prompts for each dataset, and report the VQA accuracy [11] as the evaluation metric.
Baselines. A significant novelty of AVIS is the ability to dynamically determine the relevant tools according to different states. To show that this design choice is useful, we add a number of baselines that do not contain LLM-planner for dynamic decision making. Instead, they follow pre-determined sequence to call a list tools. We propose the following baselines:
baseline-PALMM w/ PALI", which integrates the captions generated by PALI and the visual answers from PALI VQA. PALI* denotes the combination of both VQA and captioning tool. baseline-PALM w/ (PALI* + Object). which in addition calls the object detection tool, and then integrates all object data, including products and text detected by OCR.
baseline-PALMM w/ (PALI* + Object Search), a model which first selects relevant object with the help of PALM, then sequentially executes the image search and Google search with the object name. It then calls PALM again to answer the question.
For each of the three baselines, we prepare a few-shot Chain-Of-Thought (COT) prompting [39], in which the COT prompt guides the model to explain why predictions are made based on the provided information. Note that these baselines utilize a of tools in fixed order, without the capacity for dynamic decision making.
We also evaluate the usefulness of each tool group (i.c., PALI*, Object, and Search) through an ablation study. This involves removing each tool group from our framework individually, and assessing the impact on performance.
Experimental Results. Table 5 presents the results of AVIS and other baselines on the Infoseekwittduus dataset. Infoseekwikiäm challenging dataset that requires identifying highly specific entities. Even robust visual-language models, such OFA [22] and PALI [6], fail to yield
8

---

Model
Accuracy (%)
KRISP [25]
38.4
Supervised
KAT [12]
54.4
ReVIVE [20]
58.0
REVEAL [15]
59.1
PALI [6] (OK-VQA, finetune)
64.5
Zero-shot
PALI [6] (VQA, zero-shot)
41.6
PICa-Full [41]
48.0
Flamingo (zero-shot) [4]
50.6
BLIP-2 [18]
45.9
ViperGPT (one-shot) [36]
51.9
Flamingo (few-shot) [4]
57.8
(baselines without dynamic decision making, sequentially executing the tools)
Few-shot
baseline-PArI PALMM w/ (PALI*)
44.3
baseline-PALi w/ (PALI" +Object)
38.2
baseline-PALM w/ (PALI"+ +Object + Search)
47.9
AVIS (ours)
60.2
w/o PALI*
47.1
w/o Object
58.3
w/o Search
55.0
Table 2:
Visual Question Answering results (accuracy) on OK-VQA. The tool PALI* denotes the frozen multi-task PALI-17B model for both visual question answering and image captioning. Object means object detection, and search means image and text search.
rlcors
nony
Object Select
Image Search
Reason
Birt digosattr Place
Web Search
Reason
does this
building
waterside building on feninnges fice and part readentaalnn function. The storey buildig
Bridgewater Place puts 'iing'' place
g: what is
Parent
Object Select
Reason
Not
Object Select
Reason
caxonony
or this
Zaiie Luckey-Lail ia Lungua in the genus Stereun
rungi?
o:
D: which body part ouid baby work out with chow toy?
body pact
could
Object
Product Search
Resson
Chow Toy
LLM Short QA
A: Zeo
baby
Select
Emily Star Shape Rubber Chew Toy for..
out
Figure 5: Examples of AVIS's dynamic planning and reasoning procedure for solving visual questions.
high accuracy when fine-tuned on this dataset. However, our AVIS, without tineituning and by leveraging complete set of tools guided by 10 in-context examples, achieves the accuracy of 50.7 and 56.4 on the unseen entity and question splits, respectively. This significantly outperforms fine-tuned results of PALI-17B, which are 16.0 and 20.7, as well as the PALM model augmented with CLIP knowledge, which are 21.9 and 18.6, respectively.
Table 5 also illustrates that our improvements are not solely due to the additional information provided by the external tools, but due our dynamic decision-making pipeline. We compare the results of AVIS with the three baselines that conduct sequential execution. While these baselines do improve the performance, our AVIS framework outperforms the best baseline model by up to 17.3 accuracy. Note that AVIS and the baselines use exactly the same set of tools. This considerable performance gap clearly shows the clear advantage of our dynamic decision-making design. Furthermore, we show the importance of each tool in the last block of Table 5. Removal of any of the tools degrades the overall accuracy. Among the three tool groups, Object and Search are more important than PAL.I, they provide more fine-graine information crucial for the Infoseek dataset.

---

experiments in Table 2. AVIS with few-shot in-context examples achieves an accuracy of 60.2, higher than most of the existing methods tailored for this dataset, including KAT [12], ReVIVE [20] and REVEAL [15] AVIS achieves lower but comparable performance compared PAIII model fine-tuned on OK-VQA. This difference, compared to Infoseek, may be attributed to the fact that most QA examples in OK-VQA rely more on commonsense knowledge than on fine-graing graiggr knowledge. Therefore, it is feasible to encode such generic knowledge in the model parameters and requires less external knowledge. Note that PALI zero-shot VQA model itself achieves 41.6 accuracy, which is significantly higher than in Infoseek, which supports this hypothesis. Table 2 also shows that the object detection is less crucial as tool this data set, compared to PALI captioning and VQA. Nonetheless, AVIS equipped with all tools achieves the best performance.
We report the
Case studies for dynamic decision making. One of the key features of AVIS is its ability to dynamically make decisions instead of executing a fixed sequence. Figure presents three examples of AVIS's dynamic planning and reasoning process. They demonstrate the flexibility of AVIS to use different tools at various stages. It is also worth noting that our reasoner design enables AVIS to identify irrelevant information, backtrack to previous state, and repeat the search. For instance, in the second example concerning the taxonomy of fungi, AVIS initially makes an incorrect decision by selecting leaf object. However, the reasoner identifies that this prompting AVIS to plan again. This time, it successfully selects the object related to false turkey-tail fungi, leading the correct answer, Stereum. Some detailed error analysis is shown in Appendix F.
relevant to the question,
5 Conclusion
In this paper, we propose a novel approach that equips the Large Language Models (LLM) with the tree-search to use a variety of tools for answering knowledge-intensive visual questions. Our methodology, anchored in human decision-making data collected from a user study, employs a structured framework that uses an LLM --M--odeeed planner to dynamically decide on tool selection and query formation. An LLL--powered reasoner is tasked with processing and extracting key information from the output of the selected tool. Our method iteratively employs the planner and reasoner to leverage different tools until all necessary information required to answer the visual question is amassed.
Limitation Statement: Currently AVIS is specifically designed for visual question answering. We aim extend our LLM-powered dynamic doeision-making framework to address other reasoning tasks, Additionally, our current framework depends on a computationally intensive LLM, namely, the PALM model. We are interested in investigating whether this decision making framework can also be performed by lighter weight language models.
References
[1] Google lens. Web interface available at https//images.goollc.com
[2] Google search. Web interface available at https:/wwww.googll.com
[3] M. Ahn, Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, Finn, Gopalakrishnan, K. Hausman,
A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166 2022.
[4] J.B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, Millican. M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems. 35:23716 1-2276666.776,,.. 2022.
[5] T. Brown, B. Mann, Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neclakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
[6] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener, B. Mustafa, I.. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint
D.Xiv:2209.06994 2022.
[7] Y. Chen, H. Hu, Y. Luan, H. Sun, S. Changpinyo, A. Ritter, and M.-W. Chang. Can pre-trained vision and language models answer visual information -ooeiiiiio--oniiigggg questions? In arXiv preprint arXiv:2302.11711 2023.
[8] F. Chern, B. Hechtman, A. Davis, R. Guo, D. Majnemer, and S. Kumar. TPU-KNN: K nearest neighbor search at peak fiop/s. CoRR, abs/2206. 14286, 2022.
10

---

[9] A. Chowdhery, S. Narang, J. Devlin, M. Bosma. G. Mishra. A. Roberts, P. Barham, H. W. Chung. C. Sutton.
S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:200.002311. 2022.
[10] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong. T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv: 03378, 2023.
[11] Y. Goyal, T. Khot, D. Summers-Stay D. Batra, and D. Parikh, Making the V in VQA matter: Flevating the role of image understanding in visual question answering. In 2017 IEFE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI. USA, July 21-26, 2017, pages 6325-6334, IEEE Computer Society, 2017.
[12] L. Gui, B. Wang, Q. Huang, A. Hauplmann, Y. Bisk, and J. Gao. Kat: A knowledge augmented transformer for iision-and-language arXiv preprint ww..v:2112.0066 2021.
[13] T. Gupta and A. Kembhavi, Visual programming: Compositional visual reasoning without training. In arXiv preprint arXiv:2211. 2022.
[14) H. Hu, Y. Luan, Y. Chen, U. Khandelwal, M. Joshi, K. Lee, K. Toutanova, and M.-W. Chang. Open- domain visual entity recognition: Towards recognizing millions of wikipedia entities. In arXiv preprint arXiv:2302. 114,, 2023.
[15] Z. Hu, A. Iscen, C. Sun, Z. Wang, K.-W. Chang. Y. Sun, C. Schmid, D. A. Ross, and A. Fathi. Reveal: Retrieval-augmented disual-languaee pre-training with multi-source multimodal knowledge memory. In CVPR, 2023.
[16] A. Kulshreshtha, D. D. F. Adiwardana, D. R. So, G. Nemade, J. Hall, N. Fiedel, Q. V. Le, R. Thop- pilan, T. Luong, Y. Lu, and Z. Yang. Towards human-like open-domain chatbot. In arXiv preprint 0rXiv:2001.0997 009777, 2020.
[17] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset. S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, T. Ducrig, and V. Ferrari. The open images dataset v4; Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[18] J. Li, D. Li, S. Savarese, and S. C. H. Hoi. BLIP-2: bootstrapping language-iman -mnage pre-traine pre-training with frozen image encoders and large language models. CoRR, abs/2301. 12597, 2023.
[19] Y. Li, D. H. Choi, J. Chung, N. Kushman, Schrittwieser, R. Lehlond, T. Eceles, J. Keeling, F. Gi- meno, A. D. Lago, T. Hubert, P. Choy, C. de Masson d' Autume, Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Fre- itas, K. Kavukcuoglu, and O. Vinyals. Competition-level code generation with alphacode. CoRR, abs/2203.07814, 2022.
[20] Y. Lin, Y. Xie, D. Chen, Y. Xu, C. Zhu, and L. Yuan. Revive: Regional visual representation matters in knowledge-baser based visual question answering. arXiv preprint arXiv:2206.012 02222 2022.
[21] H. Liu, C. Ti, Q. Wa, and Y. Lee. Visual instruction tuning. In arXiv preprint arXiv:2304.0843 2023.
[22] Lu, C. Clark, R. Zellers, Mottaghi, and A. Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. CoRR, abs//200.00816.. 2022.
[23] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao. Chameleon: Plug-and-pla play compositional reasoning with large language models. In arXiv preprint arXiv:2304.0980 09842, 2023.
[24] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig. Language models of code few-shot common- sense learners. Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1384-1403. Association for Computational Linguistics, 2022.
[25] K. Marino, X. Chen, D. Parikh, Gupta, and M. Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-bent based vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111-14121, 2021.
[26] K. Marino, M. Rastegari, Farhadi, and R. Mottaghi. OK-V OK-VQA: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195-3204. Computer Vision Foundation / IEEE, 2019.
[27] G. Mialon, R. Dessi, M. Lomeli, C. Nalmpantis, R. Pasunuru, R. Railcanu, B. Rozière, T. Schick, J. Dwivedi-YY Yu, A. Celikyilmaz, E. Grave, Y. LeCun, and T. Scialom. Augmented language models: a survey, In arxiv preprint wxXi:23302.07842 2023.
[28] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted guestion-answeingg answering with human feedback. arXiv preprint arXiv:2112.09332. 2021.
11

---

[29] OpenAL. Gpt-4 technical report. In arXiv preprini arXiv:2303. 087744 2023.
[30] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama,
A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 277444, 2022.
[31] A. Parisi, Y. Zhao, and N. Fiedel. Talm: Tool augmented language models. arXiv preprint
arXiv:2205. 12255, 2022.
[32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763.
PMLR, 2021.
[33] A. Radford, K. Narasimhan, Salimans, I. Sutskever, et al, Improving language understanding by generative pre-training. 2018.
[34] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilce, D. Hesslow, R. Castagné. A. S. Luccioni, F. Yvon, Gallé, J. Tow, A. M. Rush, Biderman, A. Webson, P. S. Ammanamanchi,
Wang, B. Sagot, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy,
H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jermite, J. Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji, A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. Adelani, and al. BLOOM: A 176b-parame pod-pparaetter open-access multilingual language model. CoRR, abs/221 1. 1.05100, 2022.
[35] T. Schick, J. Dwivedi-Ye Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. In arXiv preprint 302.v2230..04761, 2023.
[36] D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. In arXiv preprint arXiv:2303.08122 2023.
[37] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro,
F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 133971 2023.
[38] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals. P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. In arXiv preprint arXiv:2206.0768) 2022.
[39] J. Wei, X. Wang. D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
[40] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. In arviv preprint arXiv:2303.04671 2023.
[41] Z. Yang. Z. Gan, J. Wang. X. Hu, Y. Lu, Z. Liu, and L. Wang. An empirical study of GPT-3 for few-shot 2021.
knowledge-based VQA. ArXiv preprint, abs/21099055l
[42] 7. Yang, I.. Ii, J. Wang, K. Lin, F. Azarnasab, F. Ahmed, Z. Liu, C. Liu, M. Zeng, and L., Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. In arXiv preprint arXiv:2303. /131,, 2023.
[43] S. Yao, D. Yu, Zhao, Shafran, T. L. Griffiths, Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305. 10601, 2023.
[44] S. Yao, J. Zhao, D. Yu, N. Du, Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[45] D. Yu. C. Zhu, Y. Fang. W. Yu, S. Wang. Y. Xu. X. Ren, Y. Yang, and M. Zeng. KG-FiD; Infusing knowledge graph in fusion-in-decoder for open -omain question answering. In P Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pages 4961- 4974, Dublin, Ireland, 2022, Association for Computational Linguistics.
12


Reviews:
[{"review_id": "882zgMthUX", "reviewer": "Reviewer_mb9w", "summary": "This work aims to more general VQA task (often needs external knowledge) via LLM-based information seeking. First, the info-seeking system is built with three components: planner, reasoner, and memory, seeking useful information with external tools/APIs. Second, they build a dataset with human decision-action samples via user study. Then, they build the transition graph from the dataset, for depicting the user state and action, which can be used as contextual instances serving the info-seeking system.", "questions": "How to avoid potential infinite loop such like those emerge in AutoGPT? Considering, rather than powerful LLMs, the capacity bottleneck is often caused by external tools. ", "limitations": "None.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The info-seeking system is designed in the reasonable and mainstream VQA/QA fashion.", "weaknesses": "How to compare this work with the end-to-end frameworks? As LLMs growing gradually, along with the visual knowledges being grounded better and better with language knowledges, the external knowledges will become thinner and an inclusion of internal ones.\nIn general, this is a good paper with much groundwork. I am looking forward author feedback and better version to persuade me.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and remarks. \n\n**Q: Compare AVIS with end-to-end models? Will external knowledge still be useful?**\n\nA: The key difference between AVIS and a single end-to-end model is that AVIS separates knowledge memorization from reasoning. Within this architecture:\n\n- The external tools take the responsibility of memorizing diverse and flexible knowledge.\n\n- The neural network, freed from the necessity to store vast amounts of information, focuses its full capacity on planning and reasoning.\n\nBased on the modular nature of AVIS, adding or modifying knowledge doesn't necessitate finetuning any model parameters (which is costly especially for LLMs). Instead, one only needs to update the knowledge base in different tools (e.g., Google Search index). This provides distinct advantages in situations where end-to-end models might fail:\n\n- Handling addition of new knowledge (e.g., news & updates): While it's true that LLMs are able to memorize more and more facts within the model weights, it's essential to highlight that their knowledge is static and limited to the last training set. AVIS, through its dynamic utilization of APIs, can access real-time information and news updates that an LLM could not inherently possess. This ensures our model stays up-to-date with the latest information, which is crucial for many visual questions rooted in real-time contexts.\n\n- Domain-specific Knowledge: Despite the broad spectrum of information LLMs might encompass, specialized and long-tail domains often call for specific tools and databases. AVIS is primed for such integrations, making it a good candidate for domain-centric visual question-answering tasks.\n\n- Personal & private knowledge: Given privacy constraints, in many cases private data (such as photos & message history in personal phone) cannot be included into the training corpus of LLM, making it hard to become a personalized model. AVIS, on the other hand, can be customized to incorporate personal and private knowledge sources, should users consent, allowing for a more tailored and precise response mechanism.\n\nIn summary, while the capabilities of LLMs are indeed advancing, there remains a distinct advantage in combining their reasoning prowess with the dynamic, real-time, and specialized knowledge retrieval capabilities of external tools, such as Google Search Engine and LENS. \n\n\n**Q: Will AVIS be stuck into an infinite-loop as AutoGPT?**\n\nA: Unlike models such as AutoGPT, AVIS has been designed with specific safeguards against infinite-loop scenarios:\n\nTransition graph for defined action spaces: At the heart of our model lies a transition graph defining all possible actions for executing tools. This ensures that our search space remains finite (every action our planner of AVIS generates is a <tool_id, input_arg> pair). This is very different from some other autonomous agents such as AutoGPT that use LLM’s generated language as action, which have infinite search space.\nTraversal history & repetition removal: As AVIS navigates over a given transition graph with finite paths, we could easily record all previous traversed states (similar to standard DFS algorithm). In this way, everytime we call the planner to perform an action, we remove the traversed state and only ask the model to predict the actions that are not traversed before, which avoids redundancy & potential infinite-loop.\nDecision to stop: Based on the design, if AVIS tries to traverse all possible paths and still cannot find the answer, AVIS will terminate and output “we cannot find the answer”, instead of falling into an infinite-loop. \n\nThrough these design strategies, we've ensured that AVIS remains efficient in its operations without getting ensnared in unproductive loops.\n\n\n**Q: Is Tool the bottleneck or LLM?**\n\nA: In our general response, we show a detailed error analysis statistics (examples of each type shown in one-page pdf file). Among the three major error types, tool error consists of 23.3%, while the other two types of error caused by LLM planning and reasoning add up to 56.7%. We admit that these existing tools are definitely not perfect (otherwise we could directly use them for solving the task), but a good LLM as an agent shall be learnt to use only the important information from imperfect tools to make correct decision, which shares similar insights like the classical boosting idea, while we use LLM as controller to make the ensemble. \n\nBased on this and what we describe above, we believe that the key bottleneck for building up better AI agent is 1) the basic capability of LLM itself; 2) a better algorithm and framework that enable LLM to utilize external tools to provide required knowledge, and it could focus on reasoning. We believe the planning & reasoning framework of AVIS over a human-defined transition graph is one of a good starting point of such framework."}, {"review_id": "GKs4tZxvZC", "reviewer": "Reviewer_MSpf", "summary": "This paper introduces an autonomous framework for visual question answering framework named AVIS. AVIS utilizes a Large Language Model (LLM) as its core component to dynamically strategize the utilization of external tools. The framework comprises three key components: the planner, reasoner, and working memory. The planner determines the appropriate actions, such as selecting relevant APIs, in each step. The working memory stores the results obtained from executing the APIs, while the reasoner processes the outputs derived from the API calls. Experimental evaluations conducted on the OK-VQA and Infoseek datasets validate the effectiveness of the AVIS framework.", "questions": "Refer to Weakness", "limitations": "N/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors present a novel framework that leverages external tools to overcome the challenge of external knowledge dependency in Visual Question Answering (VQA) tasks, thereby enhancing the practicality of VQA in real-world scenarios.\n\n2. The authors introduce a dynamic decision process for selecting the most suitable external APIs to address specific sub-problems at each step.\n\n3. The authors utilize human decision-making data to construct a transition graph, which is helpful for narrowing down the action spaces in the decision-making process. \n\n4. Extensive experiments on the OK-VQA and Infoseek datasets demonstrate the effectiveness of AVIS.\n", "weaknesses": "1. AVIS necessitates the utilization of costly human-annotated data to guide the (LLM). To further substantiate the generalizability and practicality of AVIS, it would be advantageous to conduct experiments on the A-OKVQA[1] dataset, which incorporates the OK-VQA and Infoseek human-annotated data as in-context samples.\n\n2. As indicated in Line 5 of Algorithm 2, the authors directly incorporate the outputs into the working memory. However, the inclusion of uninformative outputs may potentially impact the obtained results.\n\n3. For the three baseline methods, it would be better to demonstrate the few-shot COT prompts.\n\n4. In Line 250, some human-annotated samples may be “Could not find the answer”. How do the authors leverage this type of sample? When using this type of sample as an in-context sample, does it lead to the models rejecting the question when generating a response?\n\n5. It would be advantageous to provide an average step count that AVIS typically requires to address a single VQA sample.\n\n6. In Line 300, “Table 4” should be “Table 1”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Q: it would be advantageous to conduct experiments on the A-OKVQA[1] dataset, which incorporates the OK-VQA and Infoseek human-annotated data as in-context samples.**\n\nA: We appreciate the emphasis on the generalizability and practicality of our model. To address this concern, we've conducted experiments with the A-OKVQA dataset, which, share similar question types as OK-VQA and Infoseek but have unique properties (as discussed in their paper). Importantly, we adopted the same prompts used in Infoseek and OKVQA for our AVIS evaluation on A-OKVQA, without the need for additional human annotations.\n\n| Model Configuration                      | Result (%) |\n|------------------------------------------|------------|\n| AVIS                                     | 56.7       |\n| PALM + PALI*                             | 41.6       |\n| PALM + PALI* + Object                    | 47.6       |\n| PALM + PALI* + Object + Search           | 50.8       |\n| GPV-2                                    | 48.6       |\n| KRISP                                    | 33.7       |\n\nThis indicates the generalizability of our framework with the great reasoning capability of existing LLM, which shall be able to transfer the decision capability across datasets, and reduce the requirement to annotate additional prompts for some new VQA datasets.\n\n\n**Q. The inclusion of uninformative outputs in working memory may potentially impact the obtained results**\n\nA. Our approach to handling failure cases within the working memory is two-pronged:\n\nDifferentiation between working memory and LLM prompt: \nFirstly, it's essential to clarify that the working memory of AVIS is distinct from the prompt provided to the LLM. Within the working memory, we do maintain a record of both successful and unsuccessful queries. However, when interfacing with the LLM, only the most pertinent or top-ranked information is presented, ensuring that the LLM isn't unduly influenced by extraneous or unproductive details.\n\nPurpose of retaining uninformative records: \nOur goal of recording the uninformative path (failure) is to avoid repetition of previously traversed paths. When we make the planning, we only provide the model the actions that it has not traversed before, and remove all uninformative actions stored in working memory. This helps AVIS avoid going to the same uninformative path again and solves the repetitive and even infinite loops.\n\n\n**Q: How are the examples where human indicates “cannot find answer” are used to guide AVIS?**\n\nA: The inclusion of \"cannot find answer\" human annotations serves a strategic purpose in AVIS's decision-making mechanism:\n\nIncorporation into the prompt: We integrate these specific annotations into the model's prompt, ensuring that the model is aware of scenarios where human annotators couldn't discern a clear answer. This awareness is pivotal in providing a holistic context to the LLM during its reasoning process.\n\nAvoid hallucination: Should AVIS exhaust all action spaces and still be unable to generate a result with high confidence, these \"cannot find answer\" annotations guide it to echo a similar response. In practice, the frequency of such outcomes is within an acceptable range. By mirroring human annotator behavior in these challenging scenarios, we aim to provide a realistic and candid response to users, rather than forcing a potentially inaccurate answer (i.e., hallucination).\n\n\n**Q: Average step count**\n\nA: The average step of AVIS is 5.2, and the full distribution is shown in Fig 8 of Appendix.\n\n**Q: In Line 300, “Table 4” should be “Table 1”.**\n\nA: Thank you for pointing out this typo. We will fix it.\n\n\n**Q: Show COT prompts for other baselines**\n\nA: Sure, we will definitely include all the details in appendix as well as open-sourced codes. Below we show a few examples:\n\n\n\npali_prompt = \"\"\"\n\nPlease based on the \"Caption\" to answer the question:\n\n\nQuestion: What type of fruit are they holding?\nCaption: a couple of men standing next to each other holding oranges . There are two persons standing and holding oranges in their hands and there are few people beside them and there is a building in the background.\nreason: from caption, it says the men are holding oranges.\nAnswer: orange\n\n\n......\n\n\"\"\"\n\n\npali_object_prompt = \"\"\"\n\nPlease based on the \"Caption\" and \"Entity\" to answer the question:\n\n\nQuestion: What does the train carry?\nCaption: a train traveling down train tracks next to a forest . There are four trains on the railway track. In the background there are trees,poles and sky.\nEntity: [\nBNSF Railway: BNSF Railway is one of the largest freight railroads in North America (score=89.3)\nExtracted Text: BNSF (score=100.0)\n]\nReason: from caption, there are no enough information about what the train is carrying. From entity, it says the train is BNSF railway that is freight ralroads. So the train carry freight, which is good.\nAnswer: good\n\n......\n\n\n\"\"\"\n\n\nFor the PALI+Object+Search baseline, we adopt two-level procedure: first use pali_object_prompt to get visual answer, and then feed it into search API to get documents, with the same prompt shown in appendix to get final answer.\n\n"}, {"review_id": "tf0vJejPEe", "reviewer": "Reviewer_hDYr", "summary": "This paper proposes a VQA system that mimics the human decision-making process and leverages LLM and web searches to perform multimodal reasoning. The system consists of three main components: a transition graph, an LLM planner, and an LLM Reasoner. The transition graph is manually designed based on the human decision-making process for knowledge-intensive VQA. It guides the system to infer the answer from different states of information. The transition graph enhances the interpretability of the inference process and allows the system to focus on the current state only. The transition graph also introduces uncertainty to the decision procedure, making it more dynamic than previous methods that use static plans for tool usage. The LLM planner decides the next action (API calls and query content) based on the current state. The LLM Reasoner produces the answers from the collected information. Experiments demonstrate the effectiveness of this method on Visual-Question answering tasks OK-VQA and Infoseek.", "questions": "How does the system handle the situation when the output is not informative and the pipeline gets stuck in an infinite loop?\n\nRegarding the weakness: The model improved the performance on the dataset, but it is still far from perfect. It would be helpful to analyze the sources of the errors (Prompts? LLM? transition graph?) which will benefit the community.", "limitations": "This paper acknowledges the limitations of the model: 1. it is computationally intensive, and 2. it is currently only suitable for the VQA task.\n\nnegative societal impact: not found", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Leveraging the user study to build a state transition graph is a reasonable way to construct LLM applications, especially for well-understand applications such as VQA or specific expert domains.\n\nThe motivation is clear and the writing is easy to follow.\n\nExperiments are well-designed to show the effectiveness of the method.", "weaknesses": "Lack of error analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Error Analysis**\n\nA: We look through the error samples by AVIS and categorize them  with three major types of error:\n\n- LLM Planning Module Errors: Cases where the LLM planning component failed to discern crucial information, leading to inaccurate decision-making.\n\n- LLM Reasoning (QA) Module Errors: Instances where the LLM's reasoning and question-answering capabilities identified incorrect evidence or derived flawed inferences.\n\n- Tool-Induced Errors: Situations where the external tools, invoked by the AVIS framework, furnished misleading or incorrect information.\n\nTo provide a more tangible understanding of these errors, we've detailed specific instances of each type in an attached one-page PDF file, offering a granular view of the issues encountered.\n\nTo further contextualize, we carried out an analysis on 30 randomly chosen misclassified samples from the Infoseek dataset. Our findings are summarized as follows:\n\n- Reasoning & QA Errors: 11 instances (36.7%)\n- Tool-Induced Errors: 7 instances (23.3%)\n- LLM Planning Module Errors: 6 instances (20%)\n- Ambiguous or Imperfect Matches: For the remaining 6 instances (20%), our assessment suggests that AVIS's prediction was accurate. The classification as erroneous likely arises due to slight mismatches or semantic nuances.\n\n\nBy far we haven't found examples that are caused by incompleteness of transition graph, as we think it already cover most of the possible and promising actions, and the mistakes are mainly caused by LLM didn't make correct decision. Probably when the number of tools increase, the error caused by transition graph will be more prominent, which we leave such analysis for future work.\n\n\n**Q.  Will AVIS be stuck into an infinite-loop as AutoGPT?**\n\nA: Unlike models such as AutoGPT, AVIS has been designed with specific safeguards against infinite-loop scenarios:\n\nTransition graph: At the heart of our model lies a transition graph defining all possible actions for executing tools. This ensures that our search space remains finite (every action our planner of AVIS generates is a <tool_id, input_arg> pair). This is very different from some other autonomous agents such as AutoGPT that use LLM’s generated language as action, which have infinite search space.\nTraversal history & repetition removal: As AVIS navigates over a given transition graph with finite paths, we could easily record all previous traversed states (similar to standard DFS algorithm). In this way, everytime we make the planning, we will remove those traversed states and only ask the model to predict the actions that have not traversed before, which avoids redundancy & potential infinite-loop.\nTerminating Decision: Based on the design, if AVIS tries to traverse all possible paths and still cannot find the answer, AVIS will terminate and output “we cannot find the answer”, instead of falling into an infinite-loop. \n\nThrough these design strategies, we've ensured that AVIS remains efficient in its operations without getting ensnared in unproductive loops.\n\n"}, {"review_id": "uDem4FK0BG", "reviewer": "Reviewer_M3n5", "summary": "1. This paper proposes a novel VQA framework that uses LLMs to select external tools in multiple stages to extract the necessary information to answer the question. \n2. The authors gather human decision data to develop a decision graph and construct in-context examples to guide the LLM to perform API selection and query construction. \n3. The experiments validate the effectiveness of the proposed method on two knowledge-based VQA benchmarks. ", "questions": "1. It will be interesting to see how the framework performs on zero-shot setting and related discussions, especially the related work ViperGPT is zero-shot. \n\n[1] ViperGPT: Visual Inference via Python Execution for Reasoning", "limitations": "The limitation and potential solutions are discussed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experiment result is solid and shows the advantage of the framework and the improvement of each sub-module. \n2. The guidance for tool selection from human demonstration is novel, and the huge improvement also gives us some insight into the weakness of LLMs. ", "weaknesses": "1. The LLM used is PALM, while the compared methods used GPT series models. It's better to have some results based on GPT models to demonstrate the effectiveness of the proposed framework. \n2. The proposed human guidance requires a certain amount of human labor and may be hard and costly to generalize to other tasks using LLMs (manipulation, navigation, etc.). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q. Use GPT**\n\nWe change the backbone LLM as GPT4\n\n| Model Configuration                   | Result (%) |\n|--------------------------------------|------------|\n| AVIS w/ GPT-4                        | 61.9       |\n| GPT-4 w/ PALI*                       | 13.1       |\n| GPT-4 w/ PALI* + Object              | 36.4       |\n| GPT-4 w/ PALI* + Object + Search     | 43.8       |\n\nAs it shows, AVIS’s decision framework can still benefit a powerful LLM as GPT-4, showing its generalizability to different LLM.\n\n\n**Q: The proposed human guidance requires a certain amount of human labor and may be hard and costly to generalize to other tasks using LLMs**\n \nA: For AVIS, we carried out a user study that engaged 10 participants, who collaboratively responded to a cumulative count of 644 visual inquiries. On average, an individual took approximately 1 minute to address a visual question utilizing our user interface. As a result, we expended roughly 10 hours of human effort on AVIS. The planner and reasoner, grounded in LLM technology, demonstrated adeptness in generalization with minimal exposure to contextual instances. This characteristic substantially diminishes the demand for human labor.\n\nA future work is to combine human supervision with reinforcement learning to further reduce the need for human annotations.\n\n**Q: Could AVIS with same prompt generalize to other tasks?**\n\nWe appreciate the emphasis on the generalizability and practicality of our model. To address this concern, we've conducted experiments with the A-OKVQA dataset, which, share similar question types as OK-VQA and Infoseek but have unique properties (as discussed in their paper). Importantly, we adopted the same prompts used in Infoseek and OKVQA for our AVIS evaluation on A-OKVQA, without the need for additional human annotations.\n\n| Model Configuration                      | Result (%) |\n|------------------------------------------|------------|\n| AVIS                                     | 56.7       |\n| PALM + PALI*                             | 41.6       |\n| PALM + PALI* + Object                    | 47.6       |\n| PALM + PALI* + Object + Search           | 50.8       |\n| GPV-2                                    | 48.6       |\n| KRISP                                    | 33.7       |\n\nThis indicates the generalizability of our framework with the great reasoning capability of existing LLM, which shall be able to transfer the decision capability across datasets, and reduce the requirement to annotate additional prompts for some new VQA datasets.\n\nFor other types of tasks reviewer mention (e.g, manipulation, navigation), we definitely need another set of prompts as their underlying logic is very different from VQA. But similar as what we observe in VQA, we believe that AVIS with a designed transition graph of a certain task shall have some sorts of transferability and generalization across different environments and distributions.\n\n\n**Q: Zero(one)-shot setup**\n\nA: We'd like to emphasize that ViperGPT is not zero-shot, it provides one example in its documentation per API. To make a fair comparison, we also keep one prompt for each decision action, and don't use any in-context example (zero-shot) for reasoning, and the result on OK-VQA is 53.2, slightly higher than ViperGPT which is 51.9. Note that the two framework doesn't use the same set of API, so it's not directly comparable. But it shows that our framework AVIS could also work for cases without many prompts.\n"}]

Author rebuttal:
# Response to Reviewers

We thank the reviewers for their valuable comments and remarks.

In the rebuttal, we mainly add:

**1. Experimental results with GPT4 on Infoseek dataset**

| Model Configuration                   | Result (%) |
|--------------------------------------|------------|
| AVIS w/ GPT-4                        | 61.9       |
| GPT-4 w/ PALI*                       | 13.1       |
| GPT-4 w/ PALI* + Object              | 36.4       |
| GPT-4 w/ PALI* + Object + Search     | 43.8       |

**2. Generalization Analysis**
   - Showing that our model with the same prompts written on infoseek and okvqa generalize also to a-okvqa without re-human annotation.

| Model Configuration                      | Result (%) |
|------------------------------------------|------------|
| AVIS                                     | 56.7       |
| PALM + PALI*                             | 41.6       |
| PALM + PALI* + Object                    | 47.6       |
| PALM + PALI* + Object + Search           | 50.8       |
| GPV-2                                    | 48.6       |
| KRISP                                    | 33.7       |

**3. Compare with ViperGPT over one-shot setup**
   - We'd like to emphasize that ViperGPT is not zero-shot, it provides one example in its documentation per API. To make a fair comparison, we also keep one prompt for each decision action, and don't use any in-context example (zero-shot) for reasoning, and the result on OK-VQA is 53.2, slightly higher than ViperGPT which is 51.9. Note that the two framework doesn't use the same set of API, so it's not directly comparable. But it shows that our framework AVIS could also work for cases without many prompts.

**4. Error Analysis**
   - Conduct thorough error analysis with three major types of error: 1) LLM planning module miss important information to make mistake decision; 2) LLM reasoning (QA) module extracts wrong evidence; 3) Tool provide incorrect information
   - **Note:** We put the detailed examples of error in the one-page PDF file, please check it.
   - We randomly select 30 wrong classified samples of infoseek, and find that reasoning & QA part consists of 11 instances (36.7%); tool error consists of 7 instances (23.3%); and LLM planning module takes 6 instances (20%). There are other 6 instances we think the model make correct prediction, and it's just because of not perfectly matched


In the following, we will respond each authors' concerns.


Meta-review: Three of the four reviewers voted for acceptance of this submission and confirmed that the rebuttal has addressed their major concerns. Only one reviewer still remains negative after the rebuttal. The reviewer mb9w's major concern is how to compare this work with the end-to-end frameworks. The AC has checked the authors' response and believes the rebuttal is strong and sufficient. However, the reviewer mb9w has no responses in the discussion period so not sure whether he/she checked the authors' responses. The AC made the final acceptance recommendation and encouraged the authors to include all the discussion and responses in the final version.
Decision: Accept (poster)