AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning
Authors: Shirley Wu
Shiyu Zhao
Qian Huang
Kexin Huang
Michihiro Yasunaga
Kaidi Cao
Vassilis N. Ioannidis
Karthik Subbian
Jure Leskovec
James Zou
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM agents
Tool utilization
Automatic prompt optimization
Complex retrieval
Question-Answering tasks
Primary area: natural_language_processing
TL;DR: We introduce AvaTaR, a novel framework that automates the optimization of LLM agents for enhanced tool utilization and generalization in multi-step problems

Abstract: Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AvaTaR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demon- strate AvaTaR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AvaTaR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar.

AVATAR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning
Shirley Wu Shiyu Zhao Qian Huang Kexin Huang", Michihiro Yasunaga Kaidi Cao
Vassilis N. loannidis Karthik Subbian'. Jure Leskovec* James Zou
"Equal senior authorship.
Department of Computer Science, Stanford University +Amazon
Abstract
Large language model (LLM) agents have demonstrated impressive capabilittes utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AVATAR, novel and automated framework that optimizes an L.LM agent to effectively leverage provided tools, improving performance a given task. During optimization, we design comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. demon- strate AVATAR four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answerrig (QA) datasets. We find AvATAR consistently outperforms state-of-the- ooffthe-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving average relative improvement of 14% on the Hit @ metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are gwrggrrruuaaarr... avatar.
available https::/ // thub. com/ /zuu/zooeee coou-grruppaee
Introduction
Autonomous agents powered by large language models (LLMs) offer substantial promise for complex problem-solving [6, 39, 41, 55, 65]. These agents demonstrate remarkable capabilities in reason- ing [46, 47, 54, 55) and planning [8, 13, 14, 62). Additionally, their functionality extended through the of external tools that provide access to external private data and specialized operations, such as APIs for interacting with knowledge bases and search engines. These tools enable agents to perform complex tasks like multi-step problem-solvine and retrieving diverse information, which is essential for complex retrieval and question-answering (QA) [13, 21, 26, 33, 38, 40, 48).
Despite the promising capabilities of LLM agents, it remains challenging to engineer effective prompts that guide these agents through multi-stage process for real- real-world problem-solving This process involves (1) decomposing complex question into actionable plan with simpler steps, (2) strategically using provided tools to gather relevant information, and, finally, (3) synthesizing intermediate results to produce coherent and accurate response. Each step requires extensive manual effort and numerous iterations trial and error to refine the prompts.
Current approaches have primarily focused on directly deploying agents using complex human- designed "mega-prompts" [18, 24, 55), which require lots manual trial and error. Nevertheless, such hand-engineered mega-prompts may also result in brittle implementations with suboptimal accuracy (see Figure 2 (a)), where the ReAct agent [55] easily produces trivial and misleading answers to
Correspondence: (shirwu, jure, jamesz)0cs. stanford.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompt
Updated instructions tool usage and problem--oluingg strategies
(question, ansuer) pairs
Optimization phrase
Sampled Current actions
Contrastur reasoning
Actor
Comparator
LATE
Actions
10015
Initial Instructions
Postlue negalio queries
Deployment phrase
Query
Utilization
Answer
User
Optimized Actor Actions
Tools
Figure 1: Overview of AVATAR. AVATAR consists of a actor LLM and a comparator LLM. (a)
During optimization, the actor generates actions answer queries by leveraging the provided tools. Then, the comparator contrasts set of well well-performing (positive) and poorly-performing (negative) queries, automatically generating holistic prompts teach the actor more effective retrieval strategies and tool usage (ef. Section 4). (b) At deployment, the actor with optimized prompts or actions can be effectively used to answer new queries.
customers' queries about specific products. Furthermore, existing research [4, 5, 45, 50, 56, 60, 64] on employing LLMs as optimizers often fails to adequately refine the complex strategies for enhancing tool integration and usage. This lack of strategic optimization can lead to less effective, non- generalizable agent applications in complex real-world scenarios.
Present work: AVATAR. To address these challenges, we introduce AVATAR, an automated frame- work that optimizes agents for effective tool utilization and excellent task performance. Specifically, we leverage key insights from contrastive reasoning and build comparator module ("trainer"") to generate holistic instructions and prompts (i.e., computing robust "gradient") to optimize an actor L.M.. We demonstrate our framework on challenging tasks of knowledge base retrieval, which involve complex multi-stage procedures and extensive tool usage, and general QA tasks. Specifically, AvATAR includes two phases:
Optimization phase. The core our optimization framework (Figure 1) is a comparator LLM that automatically generates holistic prompts to teach a actor LLM to differentiate between effective and ineffective tool usage. The comparator takes positive and negative data samples, where the current agent performs well and poorly, respectively, to identify overall gaps and systematic errors exhibited by the agent. Unlike per-sample instructions, which can easily lead to overfitting on individual data points, by constructing multiple samples as "batch," " the comparator can extract more robust "gradient" to "backpropagate" to the actor. In other words, the comparator can provide more effective and adaptive prompts through batch-wise contrastive reasoning. helping the agent identify flaws in solving challenging multi-stage problems. Following previous methods [30, 41, 56, 63], we also maintain a memory bank with selected past instructions prevent the actor LLM from repeating previous mistakes.
Deployment phase. After the optimization phase, the actor with best rest-performing prompts can be selected for the testing instances. Moreover, in complex retrieval tasks, the iterative optimization through our AVATAR framework updates the actor for more effective and generalizable action sequences, enabling direct generalization to novel user inquiries at deployment. In Figure 2 (b), the optimized actor creates three novel strategies: 1) precise decomposition of problems by extracting multifaceted attributes, 2) effective tool usage through a sophisticated and robust scoring system, and 3) the strategic combination of different scores, determined by learned coefficients, ensuring accurate and comprehensive retrieval.
Experimental evaluation. We conduct extensive experiments on four retrieval datasets and three QA datasets. The retrieval tasks are highly complex, involving multimodal data, including textual, visual, and relational information. AVATAR consistently outperforms state-of-the-ar ohethe-art methods, showing

---

substantial 14% improvement in the @it@1 I metric. Impressively, with only 25 iterations, AVATAR boosts the @it@| I metric from an initial 5.1% 28.6% on FLICKR30K-ENTTTIE [35] and the Recall 20 metric from 30.3% to 39.3% on STARK. M-AAR--PRIME [49]. For general QA datasets, AVATAR outperforms state-of-the-art methods by 13% on average. These improvements, achieved through iterative updates to the prompts, underscore AVATAR's ability to optimize agents for complex tasks and effective tool usage. Our key contributions are:
We introduce AVATAR, a novel framework that optimizes an actor for effective tool utilization through comparator module that automatically generates holistic prompts.
We demonstrate AVATAR on four complex retrieval tasks and three QA tasks, where it significantly outperforms existing agent methods in terms of task performance and generalization ability.
We provide a comprehensive analysis of the actor's evolution during optimization, highlighting how comparator automatically provides targeted instructions that improve and generalize the actor.
2 Related Work
LLM Agents. Recent research has leveraged the remarkable language understanding and reasoning abilities of LLMs [1, 41, 47, 54, 55] to complete downstream tasks. For complex tasks that require enhanced capabilities, previous works have positioned LLMs as agents that can interact with environ- ments [4, 6, 13, 18, 21, 26, 27, 40, 48, 55], leverage external tools [6, 28, 31, 33, 36, 38, 39, 66, 68], and gather experiences 17, 61]. For example, ReAct [55] conducts reasoning and action in an interleaved way, retrieving information from Wikipedia to support reasoning.
LLM Agents for Retrieval. Previous research has applied L.M agents to Information Retrieval (IR)
systems through pretraining [2, 9, 16, 57], reranking [12, 42], and prompting techniques [11, 18). In IR systems, the retriever module directly influences the performance of downstream tasks, such as retrieval-augmennte 52]. For example. Eget [40] is designed for EHR cvesstion-answeingg capable of retrieving relevant clinical knowledge through a structured tool-use planning process and an interactive coding mechanism. However, these LLM agents usually employ heuristic (zero-shot) prompts or rely on few-shot examples [18. 25, 40, 55] for downstream tasks, which lack more informed guidance on generating effective retrieval strategies and tool-assisted actions.
generation [20, 29, 30] and mnowledge-intensive question answering [34,
Agent Optimization. In the field of optimizing LLM agents, previous works have modified the parameters of LLM backbones through fine-tuning or instruction tuning to enhance agent capability [3, 15, 19, 23, 32, 33, 37, 43, 51, 58, 59] generated better prompts through iterative prompt tuning [11, 18, 45, 50, 56]. Recently, Zhang et al. [60] conducted agent training by iteratively updating the agents' functions according to the execution history. However, these methods do not explicitly consider targeted optimization for tool usage or the impact on complex multi-stage tasks. Additionally, enhancing agents' generalization abilities [10, 31, 44], essential for real-world applications, has received less attention. In our work, we focus on automatically generating holistic instructions via a novel contrastive reasoning mechanism, targeting effective tool usage and agents' generalization ability. Compared to fine-tuning approaches, Ava TaR only require a small subset of training data and tool descriptions, making it more adaptable and less computationally intensive.
3 Problem Formulation
Definition 1: Tools. We define tools or APIs as set of implemented functions with specified input and output variables. We denote the abstract tool space as T = {fx In, + Or k = .,?....). where fo maps the input Ifz to the output Oj For example, the tools can be APIs used for accessing external knowledge via a search index, an encoder model that generates vector representations from text image data, task-specific classifier that outputs probabilities over a list of classes.
Definition Agents. An LLM agent, defined as A: P - a, is controlled by verbal prompts to generate flow of actions needed to complete task. Here denotes the action sequence [a...... azl, where each action defined by tuple (ʃ E T,i E I, o E Or), consisting of a tool function, specified input(s), and designated variable that receives the output(s). Each action in the sequence can leverage the outputs generated by previous actions, with the final action ar rendering the results for the task.
3

---

(a) ReAct: Unoptimized Agent
Input: Q "can you suggest any 705s swin fins that has a split fin design for better propulsion?"
Azsien space (GetBatitylypes, Get2ntityboccments, StringNatching Output: (sswer(s)
Thought 1: Compute similarity sccres based the product description
Action 1: Compute@mbodddnggiimlaaa
Maaim rina", tyceunttyDoomnntts
Result/Obs 1: al similarity scoses ignore
information "TUSA"
Thought 2: Check the functionality requiremens
Action 2: StringMatching ("spis Ein design batter prepulsion" Getantitybocument Result/Obs 2: =2 string matching scores
X All zeros due to no perfect match
Thought 3: Syntheaize the final result
Action 3: Multiplylsi, =2]
Result/Obs 3:
scores
X seros which
trivial solution
Final Result: answers GetTopkBntities s k-5] Pouz Lask per Cornanue
Small-scale Q&A Pairs
Testing Queries
[Q'1: Looking for durable 15-inch wide NFL car flag that car hold in windy condicions", A1 (Ground truth entity IDs): [17, 105, 2517]]
Q1: Can you auggest TUSA swim fing that has a split fin design for better propul sfon? W2:What's a high-quality fishiin sinker from Sportaman Supply Tno. that'a designed avoid snags rocks and weeds?
(Qa, An)
Optimization
Deploy
(b) AvATAR: Optimized Agent
Input: Any query
[demor [demorstration stration example: Q1): Actton on space IGetRntityTypes, ...
Output: Anawer (a)
Accurately decomnose the query into multiple sepects Pareekttributterronuueee//
(brand, type, naterial, festures);
Result 1: subquery brand: "TUSA", type: "swim fins", materfal: NA, fealures:
"split fin design for belter propuleivn"
Use unbediiing Looi Lo filler entilies
Action Computeeneeeddinggiiiillrrr (subquesy.tyyy type, GetEntityTypes
similarity scores Action 3: [at k-20)
Result 2: sl
Result candidates top-20 entities with
type similarity
Use taken manch tool flexible conron taken matching Action Getäntityyrand [sadddeees]]
Result 4: brands e brands of the top-23 entities TokenMatchScore (subquery. brand, branda]
Result 5: 52 - brond natching scores
reasoning
validate required functionality
Action 6: cetsatiaffictiossorreyyll
y. features GetBattityboccmentaal [1]
faature scores by ressening
Synthesize final sccres with optimized peramaters
7: WeightedSum (a1, 63, coef"lclerta- 0.37, 0.20))
Result 7: combined scores
Final Result:
answers Gattopkentitiesls. k=5)
Excellent task performance
Figure 2: Comparison between AVATAR and ReAct. (a) The ReAct agent exhibits incomplete task decomposition and employs suboptimal tool combinations, such as lengthy string matching, leading to poor task performance. (b) AVATAR decomposes the task into multiple steps, such as type filtering and flexible token matching. Moreover, implements robust tool usage and precise synthesis with learned parameters from the optimization phase to achieve excellent performance new queries.
Multi-step problem-solving. Real-world problems are inherently complex and cannot be effectively addressed through straightforward solutions simple tool usage alone. Solving real-world problems with LLM agents can be structured into a multi-stage procedure:
Decomposition of the problem: The procedure begins by breaking down complex question into an actionable plan characterized by simpler steps. This decomposition is crucial for setting clear objectives and facilitating focused mooblem-solving
Tosl-assisted subproblem solving: In the subsequent phase, agents strategically utilize tools from the established tool space T to gather solutions for each step. This stage is essential for acquiring

---

Table Key differences between AVATAR and prevailing agent methods. AVATAR demonstrates on specific tasks, 2) retain memory throughout the optimization process, the ability to: 1) generalization capability, and 4) autonomously generate holistic, high-quality 3) enhance the agent's prompts for better tool usage. Please refer to Section 4 for details.
Holistic Prompt Generation (on Tool Usage)
Self- Self-Improvement
Memory
Generalization
ReAct [55]
x x
Self-refine [27]
Reflexion [41]
x
AVATAR (Ours)
the necessary information required to effectively address each subproblem of the decomposed problem.
Synthesis and response formulation: The final stage involves synthesizing the intermediate results to construct a precise response. This synthesis not only combines the data but may also refine the response through trials and adjustments, ensuring the solution's 's accuracy and relevance.
For example, retrieval tasks are inherently complex and demanding. Given a user query g, retrieval tasks aim identify generate ranked list of relevant entities E from the entity space of knowledge base. Each query is associated with a set of ground truth answers, denoted as Y. which used to compute the quality of the prediction. Specifically, the LLM agent is required to I) comprehend user's request, 2) utilize the provided tools to identify and analyze relevant information in the large knowledge space, which may contain multimodal data sources, and finally, 3) integrate all gathered information to reason and generate an accurate response.
4 Our Method: Optimizing Agents for Tool-Assisted Multi-Step Tasks
Each step in the multi-stage problem-solving process (described in Section 3) requires effective prompts to identify key flaws and improve task performance. However, refining the agents' prompts demands extensive manual effort and numerous iterations of trial and error.
To address this, we introduce automated and novel optimization framework, AVATAR, which generates prompts to improve agents' tool usage and task performance. In Table 1, we highlight four critical aspects of our approach compared with prevailing agent frameworks [27, 41, 55]. Here, introduce the two main LLM components in AVATAR: a actor LLM (Section 4.1) and a comparator LLM (Section 4.2).
4.1 Actor Construction and Challenges
Actor. The actor agent, as defined in Section 3, is responsible for generating initial actions based on the initial instructionsspromppl and adjusting actions according to updated instructions. Specifically, the initial instructions provide details about the task and available tools, where tools can be introduced in programming languages such as Python. During optimization, the prompts further incorporate the previous action sequence and updated instructions adjust these actions. The actor then generates revised actions, which could include a combination of tool usage through programming language (code generation) along with natural language explanations of how the tools are employed.
Challenges in multi-step complex tasks. A common approach to updating instructions utilizes exccution results or performance data from a specific instance, often through techniques like self- explanation [4, 27] or self-reflection [41, 56]. However, this approach may not be suitable for complex tasks involving tool usage. Complex multi-step tasks include multiple interacting factors that influence overall performance, such as problem decomposition and tool selection. Consequently, instructions generated for failed/negative query instance tend be narrow in scope and may fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary across different scenarios, potentially leading to decreased performance when applied to varied cases.
5

---

Iteration
Action
actolluta
Pacesttrribbtaaccsoooo [brand]]
Zooking Sonto Our skotebaand decks I'm entereste the cradiiiona ciicle proferably white red, or..arree yewloo color schemseni
Actions
Action
brand ECDCE
Cosputatabe wegpuutaazeeeliaaiiiiiaae..
Input
Task for Comparator
capa from Tap featuring the SEC logn the side?
"Whot some adjustable
clask Tool descriptions, slctioncs
echibited sarving performances.
litt
performing oregethee query
"Whot offordoble and quality car ersilens from the Footbol Fonebks brond?"
back:
Comparaterr
Contrast groupe
2] Eaminee socusing tey query chamatertedics. Identily systematic dicrepancies
"Where /find Amiren wlicor rubbet prosslets tott will match wisdbodi??.
Generate Updated
Instructions
Updated Instructions (Partial)
<Contrastive Reasoning
sugges <Suggested improvement>
Prompt
yie gand result, those specitcc product descriptions more impict brand mentiod underae foomedd
with straighifrroed
quety altriuttes
including type, any mantioned products.
Moreaver, actioss should employ better toos accurate
Thes su ggests that the actions may not effectively capture and utiluw decais selated to speci's fextume.
brand matching and more sophisticatdd scoring that leverage both bettuel and relational nfo.
Action 1:
ataribute Russedttrrbbbtaerrrroaaayyy cuasy, (bzund, features, type, related selatee_products)) products))
Actions
Action 5: brand accoee Chaessaeepiinneettttttaaaanennne
Iteration (t 1)
Action 6:
candi - wet.eeeeeeeeeeeeeeeeeeeeeeeeeeee aattaolstterrroottttttttttttooee related products]
Figure 3: Demonstration example during optimization. Best viewed in color. The task of the comparator is to automatically generate instructions based on sampled positive and negative queries. Then comparator provides holistic instructions that guide the actor improve query decomposition. utilize better tools, and incorporate more comprehensive information.
4.2 Automate Holistic Instruction Generation with Comparator
To address these challenges, we construct a comparator LLM to update the instructions for the actor. Instead of optimizing on a sampled instance, comparator aims to identify systematic flaws throughout the structured actions/solutions.
Step 1: Constructing positive and negative queries. To achieve this goal, as shown in Figure 1, the comparator samples a set of data (question-answ)) ansser pairs), evaluates the current action sequence on the queries, and categorizes them into well- perl-peerooming (positive) and poorly-pen porrly-performing (negative) groups 1),
based on their performance. Specifically, we define two thresholds, and h (where which serve as the upper and lower bounds for constructing positive and negative queries, respectively. Queries with an evaluation metric (e.g., Recall) value above are classified as positive, while those below l are classified as negative. Based on the training dynamics, one could consider adapting the lower bound ensure sufficient number of negative samples for selection. After classification, we random sampling to create a mini-batch of queries, with equal split of positive and negative queries (b/2 each) for contrastive reasoning.
Step 2: Generating instructions through contrastive reasoning. After this, the comparator is tasked with contrasting the two groups of queries based on their key characteristics, attributing the performance gap to specific tool usage within the complex solution, and finally suggesting general modifications that can improve overall task performance. The instructions generated by the comparator are then appended to the initial prompts to update the actor.
Insights// Justfficatio for the comparator. To illustrate the insights, we draw an analogy from deep neural network training, where extremely small batch sizes can introduce significant noise in gradient estimates and high variance in model updates. By adopting batched training strategy and sampling positive and negative queries as two "mini-batches," comparator can extract a robust "gradient" update the actor. This approach encourages comparator to generate more general and comprehensive instructions on the complex action sequence, including problem decomposition, solutions to subproblems, and the final synthesis. Moreover, as contrastive reasoning directly targets disentangling the performance gap related to input patterns and how they are handled differently by the tools, is particularly effective helping comparator differentiate and select tools for use. Finally, by identifying systemic flaws across a wide array of negative queries, comparator generates modifications that are not only tailored to individual samples but also to diverse data samples, enhancing generalization to novel cases.

---

Demonstration example. Figure 3 illustrates an example where comparator contrasts the patterns of positive and negative queries, identifying discrepancies in tool usage within the action sequence. reveals that, compared to positive queries, negative queries feature more complex product descriptions, more subtle brand mentions, and additional relevant product mentions. These observations suggest: 1) an incomplete problem decomposition involving query attributes like detailed product features, 2) a potentially imprecise brand match using embedding similarity, and 3) a lack of consideration for related products in the results. Informed by these insights, actor updates its action sequence to address these subproblems and use the tools more effectively for the task, such as replacing the embedding tool with an LLM verification tool.
4.3 Logistic Instructions and Memory Construction
Logistic instructions. While instructions from the comparator are designed to improve task perfor- mance, we incorporate two types of orthogonal instructions to ensure the actions are valid and can be executed efficiently.
Validity check: This instruction is triggered internally during the execution of each action. It ensures the validity of the actor's 's actions, such as verifying the correct use of function calls. Timeout error: To prevent inefficient action sequences that may stall the actor, we implement a timeout mechanism that triggers an error if processing exceeds specified threshold. This error prompts the actor to adopt more efficient strategies, such as climinating redundant operations.
Memory Bank. During optimization, we utilize a memory bank inspired by human decision-making processes, following Shinn al. [41], where humans typically address current problems by analyzing the current situation and referencing past experiences. The memory bank stores tuples of action sequences, instructions from comparator, and the performance of these action sequences on a small training set (sampled from positive and negative queries). To manage the context size input to actor, we retain only the top-5 action sequences with the best performance. This memory bank enables actor to learn from both immediate instructions and historical results.
apply the optimized instructions or, shown in Figure 1, the optimized actor /action sequence, which includes effective tool utilization and groleem-solvigg strategies, to answer queries retrieve entities. In the experiments, demonstrate AVATAR's flexibility by applying different deployment strategies.
Deployment. At deployment,
Experiments
Tasks and Evaluation. We conduct experiments on the following datasets:
Four challenging retrieval datasets from STARK [49] and FLICKR30K- ENTITIES [35] to demonstrate AVATAR in handling complex real-world tasks (cf. details in Appendix A). For each query in the retrieval datasets, the task is retrieve relevant entities, such as nodes in a knowledge graph or images in knowledge bases. During deployment, we directly apply the optimized action sequence the test queries. We assess task performance by comparing the consistency of the results with the ground truth answers in the datasets, using Hit 1, @it@5,, 5, Recall @ 20, and Mean Reciprocal Rank (MRR) as the metrics.
Three question-answering (QA) benchmarks: HotpotQA [53], ArxivQA [22], ToolQA [67]. where the task is to provide natural language answers the questions. We sample 100, 100, and 40 training queries, and 100. 100, and 60 testing queries for the three benchmarks, respectively. During deployment, the actor LLM uses optimized instructions to generate the action sequence for obtaining the answer. We use exact match (EM) score on HotporQA, following previous methods. For ArxivQA and ToolQA, we use the LLM judge score for more reliable evaluation.
Baselines. For the knowledge retrieval tasks, we employ several embedding based retriever models for our evaluation, following Wu et al. [49]: Dense Passage Retriever (DPR) Karpukhin et al. [17]; Vector Similarity Search methods ada-002 and multi-ada-002 using anbee-eeedddig-aaaa from OpenAI; and relation-aware model, QAGNN [57], for the STARK benchmark. Additionally, we include four prevailing agent frameworks to further enrich our evaluation:
ReAct [55] conducts reasoning and action in an in-context and interleaved manner to enable L.LMs to interactively analyze observed information and perform actions.
7

---

Table 2: Retrieval performance (%) on STARK benchmark. Last row shows the relative improvements over the best metric value in each column.
AMAZON
MAG
PRIME
Hit@1
H
R@ 20
MRR
Hit@1
Hit@5
R@ 20
MRR
Hit
Hit@5
R@
MRR
DPR
15.29
47.93
44.49
30.20
10.51
35.2
42.11
21.34
4.46
21.85
30.13
12.38
QAGNN
26.56
50.01
52.05
37.75
12.88
39.01
46.97
29.12
21.35
29.63
14.73
ada-002
39.16
62.73
53.29
50.35
29.08
49.61
48.36
38.62
12.63
31.49
36.00
21.41
multi-ada-000
40.07
64.98
55.12
51.55
25. 92
50.43
50.80
36.94
15.10
33.56
38.05
23.49
ReAct
50.81
$2.30
47.03
31.95
Reflexion
42.79
65.05
54.70
52.91
40.71
54.44
49.55
47.06
14.28
34.99
38.52
24.82
AVATAR-C
40.92
63.63
53.68
51.73
33.25
52.17
47.88
41.34
8.82
23.82
30.32
16.20
AVATAR
60.57
58.70
50.63
51.15
36.73
Relative
16.6%
6.3%
9.9%
12.2%
9.6%
2.1%
-0.3%
8.7%
20.7%
5.0%
2.1%
7.7%
Improvement
Reflexion [41] uses self-reflection on the current task completion and stores these reflections in an episodic memory buffer to enhance decision-making in subsequent trials.
Expel. [61] extracts insights from successful and failed action sequences, retrieving and including them the context during inference. We apply Expel. on the QA datasets and, due to its high cost on large-scale retrieval tasks, compare it with AVATAR on sampled STARK-MAG test set.
Retroformer [56] reinforces LLM agents and automatically tunes their prompts by learning a retrospective model through policy gradient. We compare the performance of AVATAR with the reported result by Retroformer on HotpotQA due to the additional training involved.
We include ablation model, AVATAR-C, which removes the comparator from our optimization pipeline. This comparison aims to validate the effectiveness of the comparator. The LLM version information provided in Appendix B.
Function library. For the knowledge retrieval tasks, our function library consists of tight functions that facilitate access to, operation on, and reasoning over the knowledge information by LLM agents. For the QA tasks, we provide web search tools such as Google and Arxiv search APIs. See Appendix E for details. We used the same function library across all agent methods.
General pipeline. For AVATAR, we optimize the agent for a fixed number of epochs and select the action sequence or instruction with the highest performance. We use the same initial prompt structure, the metric Recall @0 or Accuracy for constructing positive and negative queries, and hyperparameters (L h 0.5, b 20) for all datasets.
5.1 Textual and Relational Retrieval Tasks
We employ the AMAZON, MAG, and PRIME datasets from the STARK benchmark [49], large-scale semi-structured retrieval benchmark that integrates textual and relational knowledge (cf. detailed description in Appendix A). Here, the entities to be retrieved are defined nodes in graph structure, with knowledge associated with each entity including both textual deseriptions and relational data. We the official splits from the STARK benchmark.
Takeaway 1: AVATAR outperforms state-of-the-art models. Table 3 shows that AVATAR substan- tially outperforms leading models such as Reflexion across all metrics on the STARK benchmark. Notably, the average improvement of AVATAR is 15.6% on Hit@ and 9.5% on MRR. ReAct agents, however, cannot optimize based on instructions for improved tool usage and tend to select tools based on the LLM's prior knowledge, which may not be optimal for the given task. We observe that ReAct agents apply similar tools across various queries and struggle to explore alternative tool usage with extensive in in-context reasoning. Results for agent methods using GPT-4 Turbo are provided in Appendix B, showing similar conclusions. For comparison with ExpeL, the results in Table 6 show that it performs similarly to ReAct, underperforming AvATARby large margin.
Takeaway 2: Comparator greatly impacts the actor's performance. The comparison of AVATAR with its ablation variant, AVATAR-C. highlights the significant advantages of the comparator module. Although AVATAR-C conducts validity and timeout checks, integrating Comparator into AVATAR adds a comprehensive instruction mechanism crucial for identifying clear directions to improve the agents, underlining comparator's key role in optimizing actor.
8

---

Amazzn
MAG
Prims
Ondtrtion Matic
Hourg roraga
Moving Awarys
0.1
C43
3/0
035
Figure 4: Optimization dynamics of AVATAR agents on STARK. The figures show validation performance (solid line) and its moving average (dashed line) during the optimization of AvATAR.
Hit@1
H
R@20
MRR
Sokket Maric
37.2
72.8
46.3
ReAct (claude3)
38.8
54.8
71.6
46.1
Reflexion (claude3)
28.4
53.2
75.2
41.2
AVATAR-C (claude3)
28.8
53.2
78.4
40.0
AvATAR (claude3)
42.4
63.0
79.2
52.3
Relative Improvement
9.2%
11.7%
5.3%
13.0%
Figure 5: Performance (left) and AVATAR's optimization dynamics (right) on FLICKR30K SEEKRRRRK-ENTTEEE.. ENTITIES.
Takeaway 3: AVATAR effectively improves agents during optimization. Figure 4 illustrates the agents' performance the validation set during optimization. Impressively, AVATAR agents show significant performance improvements, e.g.. from 35% to 75% on AMAZON and from 20% to 78% on MAG. This evidence strongly supports the effectiveness of the instructions generated by our comparator. Additionally, our memory bank, which stores past best-performing actions, encourages AVATAR agents to gradually converge by the end of the optimization process.
Takeaway 4: AvATAR can generalize to real-world tasks. Comparator generates instructions tailored to groups of retrieval queries, promoting generalizable modifications for novel queries. We validate this capability by applying optimized actions to human human-generated leave-out queries from the STARK benchmark, which differ notably from the training data used optimize agents. Results in Table (Appendix B) show that AVATAR significantly outperforms other models, achieving an average improvement of 20.9% Hit@ Further, in another study of Appendix B, assess AVATAR's robustness to hyperparameters and f, showing that maintains stable performance and generalization across different parameter values.
5.2 Image Retrieval Task
We further experiment on FLICKR30K ENTITIES [35], an image retrieval dataset 30k images with annotated bounding boxes and descriptive phrases (Appendix A). In Table 2, AvATAR again shows significant improvements. In contrast, Reflexion agents struggle with "overfitting," where they are easily misled by specific image data, leading impppropriaee actions (e.g., trying "extract the color of a hat" from images without hats). AVATAR effectively avoids such pitfalls through batch-wise contrastive reasoning, which provides broader perspective.
Takeaway AVATAR generates impressive and generalizable actions. Figure (left), final actions of the AVATAR agent detailed in Figure 8 (Appendix B), achieve advanced performance. Notably, AVATAR skillfully manages input queries and leverages Inverse Document Frequency (IDF) scores to refine phrase matching. ultimately synthesizing accurate answers. Beyond using existing tools, AVATAR agents can develop high-level tools, such as IDF-based reweighting, suggesting a promising direction for dynamic tool libraries and enhanced tool generation.
Takeaway Emerging behaviors during optimization. Figure 6, we present concrete cases illustrating key interactions between actor and comparator. In each instance, comparator identifies critical flaws, including information omission, ineffective tool usage, and suboptimal synthesis of varying scores. The instructions subsequently prompt actor to enhance retrieval strategies, tool selection, and precise score combinations. Furthermore, frequent references to tool usage underscore comparator's focused examination of tool utilization during optimization.

---

Improved Divide and Conquer v4/z)
Differentiation corras
Learning
Eur PAOAAA LiALLaee Hincharr ulad, pirnsetenc
incetton orean consupts for saveent yeaer of otherr cancaptsc dug, acticns trids, cook), and altnitities lifee light)
Rearl's ansti Long Joce zoerch teeteaee
requecing pnaged froee spe iscations
such as animalia people in dig, man (aonar))
componnd houra uero ptrauee tor ouediac more conorpiiiffe negatiinnggnne purtic
"uquare" "Shoet" "itadars.
facuaed que Rug pepplendating graan, person)
pamplng
This enrating
peod to wegfte cutputt www.cccccscccoooooooo coaty Sint
wandl weggt This pepptt Beewwrrd soone
HP Batic bour Puuntit
Cusrenpe Wiiieeee
Frri cor
CEEEMACCIO Jocti Call Bustocatoo
Figure 6: Representative instruction types from the comparator. We provide three cases where the comparator guides the actor towards (1) better divide-and-come -nn--ue strategies for multi-step problem- solving, (2) more sensible differentiation between good and bad tool usage/combinations, and (3) adjustments in the weights to generate the final answers. We record the number of occurrences X under each instruction type over 25 iterations on BACKKROK-ENTITTE indicated by (X/25).
Table 3: Performance (%) on three QA benchmarks. Last row shows the relative improvements over the best metric value in each column.
TooLQA
HOTPOTQA
ARXIVQA
SCIREX-FASY
SCIREX-HARD
AGENDA-EASY
AGENDA-HARD
Col
28.0%
58.0%
1.7%
0.0%
0.0%
0.0%
ReAct
40.0%
72.0%
31.7%
17.5%
38.3%
3.33%
Reflexion
46.0%
77.0%
28.3%
13.3%
30.0%
3.33%
ExpeL 39.0%
73.0%
36.7%
14.5%
56.6%
1.67%
Retroformer (#retrr==1
51.0%
AVATAR-C
41.0%
73.0%
31.7%
13.3%
31.7%
1.67%
AVATAR
53.0%
84.0%
37.5%
23.3%
60.0%
4.17%
Relative Improvement
3.92%
9.09%
2.18%
33.1%
5.82%
25.0%
5.3
Question Answering Tasks
Finally, we applied AvATAR to three widely used QA benchmarks. For ToolQA, we tested AVATAR and the baselines on two different domains: SciREX, which focuses on extracting information from full-length machine learning papers, and Agenda, which involves personal agenda-related questions. Both datasets have easy and hard versions.
Takeaway 7: AVATAR outperforms on QA tasks by offering better context understanding, Table 3 shows that AVATAR consistently outperforms state-of-the-as methods across all three QA datasets, with especially strong results on TooLQA. In SCIREX-HAME complex information from long scientific papers, AVATAR shows a 33.1% 1% improvement, while in AGENDA-HAN HARD, it achieves a 25.0% relative gain. These improvements are attributed to AVATAR's ability to generate optimized prompts that help the agent better understand the broader patterns and contexts of the questions, leading to more accurate answers and improved generalization across question types, from simple to complex.
which focuses extracting
6 Conclusion and Future Work
In this study, we introduce AVATAR, a novel framework that automates the optimization of L.MM agents for enhanced tool utilization multi-step problems, focusing on complex retrieval and QA tasks. AvATAR demonstrates remarkable improvements across seven diverse datasets. This success can largely be attributed to the comparator module, which effectively refines agent performance through the iterative generation of holistic and strategic prompts. A key innovation of comparator its use of contrastive reasoning with batch-wiss sampling, enabling it to identify systemic flaws and extract robust "gradients" for comprehensive agent improvement across diverse scenarios. While we observe substantial progress from AVATAR, we discuss its limitations in Appendix D regarding its scalability etc. Future work can explore extending this methodology to other challenging agent tasks, visual reasoning tasks, and more dynamic environments, or designing better memory banks for dynamically storing knowledge and experience from past training.
10

---

Acknowledgement
We thank lab members in Zou and Leskovec's s labs for discussions and for providing feedback on our manuscript. We also gratefully acknowledge the support of DARPA under Nos. N66001 1924033 (MCS); NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), (HHBEM); Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for Human-Centered AI, Chan Zuckerberg Initiative, Amazon, Genentech, GSK, Hitachi, SAP, and UCB. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
MMMS22327709
REFERENCES
[1] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. [n. d.j. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. ([n. d.D. arXiv:2308.00 09687
[2] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving Language Models by Retrieving from Trillions of Tokens. In ICML.
[3] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. FireAct: Toward Language Agent Fine- tine-tuning. arXiv:2310.05915
[4] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self- Self-Debug. (2023).
[5] Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang, Han Guo. Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022, RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In EMNLP. ACL.
[6] Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. [n. d.J. Agent AI: Surveying the Horizons of Multimodal Interaction. ([n. d.J).
arXiv:2401.03.. 033568
[7] Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. 2024. AutoGuide: Automated Generation and Selection of State- Aware Guidelines for Large Language Model Agents. CoRR abs/2403.08978 (2024). arXiv:2403.08978
[8] Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. [n.d.]. MindAgent: Emergent Gaming Interaction. ([n. d.j). arXiv:2309.099) 19997
[9] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML. PMLR.
[10] Simon Jerome Han, Keith J. Ransom, Andrew Perfors, and Charles Kemp. 2024. Inductive reasoning in humans and large language models. Cogn. Syst. Res, (2024).
[11] Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-Retriever: Retrieval-Augmentee Generation for Textual Graph Understanding and Question Answering. (2024). arXiv:2402.07630
[12] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu La, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers recommender systems. arXiv:2305. 08845 (2023).

---

[13] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML.
[14] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models. CoRL..
[15] Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiinng Gong, and Lichao Sun. 2024. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. ICLR (2024).
[16] Vassilis N loannidis, Xiang Song, Da Zheng, Houyu Zhang, Jun Ma, Yi Xu, Belinda Zeng, Trishul Chilimbi, and George Karypis. 2022. Efficient and effective training of language and graph neural network models. arXiv preprint arXiv:2206. 10781 (2022).
[17] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In EMNLP.
(18) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Perey Liang, Christopher Potts, and Matei Zaharia. 2023. Semonstrate-Searchh Predict: Composing retrieval and language models for knowledge-intensive NLP. #rXiv:2212.14024 [cs.CL]
[19] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurIPS.
[20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensie NLP Tasks. arXiv:2005.1 1401 [cs.CL]
[21] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. abs/2303.17760 (2023). arX1v:2303.17760
[22] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang. Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision- Language Models. ACL.
[23] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for -ugmentetd LLMs. In EMNLP. Association for Computational Linguistics.
[24] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. AgentBench: Evaluating LLMs Agents. (2023). arXiv:2308.08 036688
[25] Robert Lo, Abishek Sridhar, Frank Xu, Hao Zhu, and Shuyan Zhou. 2023. Hierarchical Prompting Assists Large Language Model on Web Navigation. In Findings of the Association for Computational Linguistics: EMNLP 2023.
[26] Pan Lu, Baolin Peng. Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In NeurIPS.
[27] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In NeurIPS.
12

---

[28] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schul- man. 2021. WebGPT: Browser-assisted question-answering with human feedback. ArXiv (2021).
[29] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schul- man. [n. d.j. WebGPT: Browser-assisted question-answering with human feedback. ([n. d.J). arXiv:2112.09332
[30] Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. 2310.08560 (2023).
[31] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.0001 (2023).
[32] Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: Tool Augmented Language Models. arXiv:2205. 12255
[33] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. 2023, Gorilla: Large Language Model Connected with Massive APIs. CoRR (2023). arXiv:2305.15334
[34] Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arxiv 2302. (2023).
[35] Bryan A. Plummer, Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k Entities: Collecting Region-to- Misoontto-Praan io-Phrase Correspondences for Richer Image-to-Sentence Models. Int. J. Comput. Vis. (2017).
[36] Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. In Proceedings of ACI 2023. Association for Computational Linguistics.
[37] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. TooILLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arxiv 2307.16777 16789 (2023).
[38] Timo Schick, Jane Dwivedi-Ya Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In NeurIPS.
[39] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. In NeurIPS.
[40] Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, and May D Wang. 2024. EHRAgent: Code Empowers Large Language Models for Complex Tabular Reasoning Electronic Health Records. arXiv:2401.0rF .07128 (2024).
[41] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS.
[42] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. [n. d.j. Is ChatGPT Good at Search? Investigating Large Language Models as Re- Re-Ranking Agents. In EMNLP, year 2023.
13

---

[43] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. ToolAl- paca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. (2023).
arXiv:2306.05301
[44] Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. 2024. Hypothesis Search: Inductive Reasoning with Language Models. ICLR (2024).
[45] Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang. Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. 2023. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. (2023). atXiv:2310.1642
[46] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR.
[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thougee Prompting Elicits Reasoning in Large Language Models. In NeurIPS.
[48] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. (2023). arXiv:2308.088 08155
[49] Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. loannidis, Karthik Subbian, James Zou, and Jure Leskovec. 2024. STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases. (2024). arXiv:2404.1320)
[50] Chengrun Yang, Xuezhi Wang. Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. (2024).
[51] Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. In NeurIPS, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).
[52] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdi- nov, and Christopher Manning. 2018. HotpotQA: dataset for diverse, explainable multi-hop question answering. EMNLP (2018).
[53] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. 2018 HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In EMNLP.
[54] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS.
[55] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.
[56] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. 2024. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. (2024).
[57] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Perey Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering.
[58] Junjie Ye, Guanyu Li. Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. ToolEyes: Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios. arXiv:2401.0013 (2024).
[59] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. [n. d. AgentTuning: Enabling Generalized Agent Abilities for LLMs. ([n. d.J). arXiv:2310.12877
14

---

[60] Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. 2024. Training Language Model Agents without Modifying Language Models. (2024).
arXiv:2402.11359
[61] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM Agents Are Experiential Learners. In AAAI
[62] Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023. Outline, then details: Syntactically guided coarse-to-fine code genera- tion. ICML (2023).
[63] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. MemoryBank Enhancing Large Language Models with Long-Term Memory. In AAAI.
[64] Yongehao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models are Human-Level Prompt Engineers. In ICLR.
[65] Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: survey. arXiv:2308.0710 10008..77100 107 (2023).
[66] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. In NeurIPS.
[67] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. arXiv:2306.1330 13304
[68] Chang Zong, Yuchen Yan, Weiming Lu, Eliot Huang. Jian Shao, and Yueting Zhuang. [n. d. Triad: A Framework Leveraging a Multi-Role LLM--bsed Agent to Solve Knowledge Base Question Answering. ([n. d.)). arXiv:2402.14320
15

---

A Retrieval Tasks
given relation-text knowledge base, based knowledge graph G = (V, E) and a collection of free-text documents D. We represent the relation- text knowledge base of size as = {(v, di, gi i=z, where v; V represents node the knowledge graph, d, E D is the text document related to the node, and gi is the connected component of G containing V4-
STARK. On the STARK benchmark,
The query set Q in STARK is derived from both G and D, where each qi E Q contains requirements based on d; and 9;- The answer set A,, which includes U; is a set of nodes satisfying both relational and textual requirements. The task on STARK is defined as follows: Given the knowledge base & consisting of relational and textual information, and text query g.. the output is of nodes A; such that Va; E A, a; satisfies the relational requirements the knowledge graph and textual requirements in its text documents.
FLICKR30K ENTITIES. On the FLICKR30K ENTITIES dataset, we are given an image-text knowledge base. We denote image-text knowledge base of size as E {(v,, 9s T:))=-- Sample consists of an image 84 its descriptive caption qi, and entity bounding box information T, Specifically, T, = {(Ctg. } j=1 where b; represents the number bounding boxes annotated in image Gj is the coordinate of the j-th bounding box, and Pij describes the entity in the corresponding bounding box.
In our task, the image captions serve as the text query; therefore, all qi in the dataset are not accessible to the agent to prevent information leakage. However, the agent can access Y and T; to fully utilize the vision and language information. The task on FLICKR30K ENTITIES is defined follows: Given the knowledge base & with images and bounding box information, and text query di the output is an image v, that satisfies the visual requirements in the image and textual requirements in the corresponding bounding boxes Ti.
Rainbow flags
Wedding attire
Orango hat
Aldd
Wedding cake
Glasees
City street
orange hal
group of youth
A table
Figure 7: Example data on FL.CKK30KK ENTITIES. Each entity is an image along with its image patches and associated phrases with the image patches.
B Experiment Details and Additional Results
B.1 Experiment Setup
LLM versions for agent methods.
For the knowledge retrieval tasks, we use by default, and report results using gpt-4-turbo Appendix B due space limitations.
the backbone LLM in the main paper
For the QA tasks, we use gpt-4 for HotpotQA for fair comparison with previous methods and gpt-4o for the other two QA datasets.
16

---

Table 4: Retrieval performance (%) on STARK benchmark. Last row shows the relative improvements over the best metric value among the baselines.
AMAZON
MAG
PRIME
Hit@1
@@
MRR
Hit@1
Hit@5
R@
MRR
Hit
Hit@5
R@
MRR
DPR (roberta)
15.29
47.93
44.49
30.20
10.51
35.23
42.11
21.34
4.46
21.85
30.13
12.38
QAGNN (roberta)
26.36
50.01
52.05
37.75
12.88
39.01
46.97
29.12
8.85
21.35
29.63
14.73
ada-002
39.16
62.73
53.29
50.35
49.61
48.36
38.62
12.63
31.49
21.41
multi-ada-002
40.07
64.98
55.12
51.55
25 25.92
50.43
50.80
36.94
15.10
33.56
38.05
23.49
ReAct (gpt4)
38.83
62.50
50.39
49.16
23.50
46.50
43.11
33.91
10.83
30.83
32.16
19.39
Reflexion (gpt4)
41.45
64.83
53.98
52.22
33.44
51.35
49.14
41.34
14.27
35.11
39.29
23.61
Reranker (gpt4)
44.79
71.17
AVATAR-C (gpt4)
32.03
58.46
$4.03
44.00
25.97
45.62
46.68
35.12
9.52
26.04
32.62
17.58
(gpt4)
48.82
59.32
Relative Improvement
-2.2%
(over Best Baseline)
B.2 Additional Experimental Results
(1) AvATAR results on STARK using GPT-4 Turbo (0125) as LLM backbone. In Table 4, we provide the results on STARK using GPT-4 Turbo (0125) as the backbone LLM.
Table 5: Retrieval performance (%) on the leave-out sets of human-generated queries in STARK.
AMAZON
MAG
PRIME
Hit@1
Hit@5
R@20
MRR
Hit@1
Hit@5
R@20
MRR
@it@1 1
Hit@5
R@
MRR
DPR (roberta)
16.05
39.51
15.23
27.21
4.72
9.52
25.00
7.90
2.04
9.18
10.69
ada-000
39.50
64.19
35.46
52.65
28.57
41.67
35.95
35.81
17.35
34,69
41.09
26.35
multi-ada-O00 002
46.91
72.84
40.22
58.74
23.81
41.67
39.85
31.43
24.49
39.80
47.21
32.98
ReAct
45.65
71.73
35,95
58.81
27.27
40.00
35.95
33.94
21.73
33.33
41.09
28.20
Reflexion
49.38
64.19
35.95
58.96
28.57
39.29
35.95
36.53
16.52
33.03
41.09
23,99
AVATAR
58.32
76.54
42.43
65.91
33.33
42,86
35.94
38.62
33.03
51.37
53.34
41,00
Rel. Impr.
17.5%
5.1%
11.8%
16.7%
2.9%
5.7%
28.7%
27.3%
21.4%
(2) AVATAR results on STARK's human-generated splits. In Table 5, demonstrate AVATAR's ability to generalize to test queries with distributions different from the question-answering pairs used optimize the actor agents.
Table 6: Performance metrics for different models on the subset of the STARK-MAG dataset.
MAG (#Test=50)
Hit@ 1
Hit@5
Recall
MRR
DPR
16.00
40.00
51.84
27.39
QAGNN
20.00
52.00
49.71
36.39
ada-002
40.00
58.00
55.93
47.76
multi-ada-002
32.00
58.00
58.81
43.58
ReAct
46.00
60.00
54.67
50.92
ExpeL
40.00
58.00
$5.94
47.43
Reflexion
48.00
64.00
$7.43
52.31
AvaTaR-C
44.00
60.00
52.49
50.16
AvaTaR 52.00
64.00
53.86
56.74
Relative Improvement
8.33%
0.00%
-8.42%
8.48%
(3) AVATAR R results and comparison with Expel. on STARK-MAG subset. Table 6, AvATAR demonstrates consistently higher performance than ExpeL across most metrics, notably achieving the highest Hit@ and MRR scores. While ExpeL performs well in Recall @ 20, AVATAR 's overall improvements highlight its superior capability in precise retrieval tasks and tool-assisted knowledge retrieval.
(4) Final action sequence by AVATAR FLICKR30K-ENTIN final actions optimized by AVATAR on FLICKR3OK ENTITIES.
In Figure 8, present the
(5) Sensitivity of AVATAR to upper and lower bounds. We evaluated various combinations of e and h, focusing on the STARK-AMAZON dataset due to computational constraints. Table presents the Hit@ I results for different & and h values.
17

---

Any query (example: *A mar. with pierced eare ie wearing glaeses and orange hat 1"1 Space: [GotInagas, GatsmecdtarEiniiaattyy Gatlaaoikttribbnccyyyyx ...] Octput: Retrieved Inage IDs
Input:
Removc eпpty spaces non-aiphazetic chassetars
Action 1: cleasquryyaxt(oeer))
Result 1:
nornelizen izze query + "s nan with pierced ears is searing glasses and an orange hat"
phrases
knosledge base
GetSagoEPhsasent) Result phrases__it +
Action 2:
"graes", "sky"), "ost", olll
Congute IDP for phicase ingostance
Flatten rphresss lixt]] 1df pcc_es ("gicrsed") 0.3, "Maa": 9.0012, ....
Get visuel att-ibutes for the cendidate images Action GeViausiAtrrbbteeayyl (ettaaae ["ealoe", "object", "action" *epunt"]] Result 4: visual attributes e (nodo 1d_1: (*colos": "red", ooll code 14 Io.ly ...
textual
Action 5: (", "ooiniiittt for liet ir phrases Hetl
Result 5: phaaae ventences [fa
sky", "s, playground",...
Rvs loate textual and viauel releeence
Action 6: Conputakneediigiiniiee
Lxed plaass sentanees)
text sccore
Action 7: Cogputetmbeddilgiiimiiaa. htt.m//iaaatttttooooiii.... (natmalizes_ query,, vieusl_attriootes]
Result 7: siausl
Hatch query pbzases with node actributes using IDP secres
Action B; masetquaryhhraeee phrase_matee wwt.emaatttceeeeeeeeeeeeeeeeeeee scores
spery- visual siselleccrrbbteen
Reweight the plaaaee match
with 1DP idt_acores] saceeeenntth_eeeeeeeeeeeeeeeeeee
Action BeweightaytDraooeelphccc
Result 9: xcwcichtoc mstccecssssss___oece mstch
sewcccttoe_mttca_cccre.. scares
Aggregste scores veighted paraneters
WeightedSun (tent, scores, visusi_ soxcightad aatch__cozze, woights-(D.3,, 0.2, 0.213 Result 10: aggregated acozes
Nocmalian scures for final ranking
Action 11: NormalisaScorea iammliisetorre/seggraaede [asgrecated scores]
Result 11: nornal ised scores = (rode_ Id: normsl soreslized, iize__soore, oo.
Tinal Reseit: answers GetPophantstten inornalized scores, k-5]
Exce lent task performanee
Figure 8: Optimized Action Sequence by AVATAR on FLICKR30K-ENTITIES..
Table 7: Hit@ results for different combinations of and values on the STARK-AMAZON dataset.
h
0.3
h = 0.4
h 0.5
f 0.5
48.32
50.01
49.87
e= 0.6
47.89
49.56
50.45
f= 0.7
47.75
48.56
49.34
Key Observations
The framework exhibits robustness to variations and h, with Hit I fluctuations limited to range of 2.7%.
A performance decline is observed when the gap between & and h becomes too large, potentially due to the exclusion of certain training queries that fall within the (h, E) interval.
moderate gap between & and h leads to slight performance improvements, suggesting that balanced separation between positive and negative queries can enhance pattern differentiation without compromising the number of training queries.
The results indicate that setting 0.6 and = 0.5 yields an improved Hit@ score compared to the baseline reported in the original paper. Overall, this analysis underscores the robustness of the framework, which relies on minimal set of hyperparameters, including h, batch size b, and training epochs.
18

---

C
Prompts
We keep only two prompt templates for our framework on all tasks: (1) The prompt template given actor initially instructions, and (2) the prompt template given the comparator conduct contrastive reasoning and generate the instructions for the actor. Below are the complete templates:
This is the prompt given to actor initially instructions:
You are an expert user of knowledge base, and your task is to answer a set of + queries. I will provide your with the schema of this knowledge base: <knowledge_ achema>
You have access to several APIs that are pre-implemented for interaction with the + knowledge base:
chunc_call_eescription
Information of queries: Below are several query examples that you need to carefully
4 read through:
<example queries?
<examele_uueree>>
Task:
Given an input query, you should write the actions in Python code to calculate 'node_ score_dicts for cnit_candidates> node IDs, which input
are subset of node IDs from the knowledge base, and the nodes belong to the type(s) <
list. These node IDs, referred to as 'candidate_ids"
candidate_typest In mode__coree_dict: Dict[lint float] ' each key should be
node ID, and each value should be the corresponding node score. This should indicate the likelihood of the node being the correct
the query.
Output format:
should establish connection between the given queries and the query patterns to the schema of the knowledge base. Secondly, generate an outline for the code that will compute scores for all
Firstly,
candidate nodes provided in the query examples. Finally, develop the main wwwoeeoooeeeeeeeeeeeeeeooeeeeeee
function named
which takes two required parameters: query' and 'candidate_ide" and optional parameters declared in parameter_dice dict' Note that 'parameter_dice
is dictionary of parameters and their default values where you can declare any parameters or weights used during computing the node scores. If no optional parameters are needed, leave 'parameter_diee as an empty dictionary. Overall, your output should follow the structure:
"' "python
# <code outlines> import <package1>
parameter_dict
pttrrmmeternnamel> <default_valle
prramaternname2: deefault_value2:
_et_node_score___e cecccoreeeitttooeeeeeeeee (quuur, (query, candidate_ids, *xparameter_dict): mode_score_diie = O
# your code return mode_score_diee dict
Hints:
Observe the example queries carefully and consider the key attributes to extract. Use "pyytoon and ett wrap the complete code, and do not any other delimiters.
You can use any of the pre-implemented APIs but should avoid modifying them. You can include other functions besides "get_node_score chode_score_dict ' but ensure they are fully implemented.
19

---

The code should be complete without placeholders and dummy functions. Optimize the integrity of the code, e.g., corner
Minimize computational expenses by early elimination candidate nodes that don't
+ meet relational requirement (if any).
Avoid conducting unnecessary and redundant computations, especially when using
loops.
Make of 'parameter_dict
avoid hard-coding parameters and weighta. Use the functions that end with 'by_llm' wisely for more accurate searches.
Use 'debug-print" smartly to print out any informative intermediate results for
+ debugging.
Exclude comment any example of "get_node__d _iiced' in the output code.
Your output:
This is the prompt given to comparator generate the instructions for the actor:
<initial.prompt>
sprevious_actions.
After executing the above actions
queries, some queries have yielded good results, while others have not. Below are the queries along with their
corresponding evaluation metrics:
Nell-performing queries:
seriitvve_queriee andde_a______aeeeee aal_aandd______ metrii> _nd_mttrri>>>>>>>>
Poorly-performing queries:
aaatee__andmmmtrriieee metric> songative_querieee
Task:
(1) Firstly, identify and contrast the patterns of queries that have achieved good results with those that have
(2) Then, review the computational logic for any inconsistencies in the previous
+ actions.
(3) Lastly, specify the modification that can lead to improved performance on the negative queries. You should focus capturing the high-level pattern of the queries relevant to the knowledge base schema.
D Limitations
We identify several potential limitations of our work:
Scalability: AvaTaR is designed to scale with large language models (LLMs) that support extended context lengths (up to 128k tokens), enabling it to handle numerous tools and complex tasks. However, increased latency and other practical limitations may hinder performance in scenarios requiring hundreds of tools or high complexity. Future research could focus on incorporating specialized, tool-augmented LLMs as auxiliary agents to facilitate smoother scaling.
Computation Requirements: Managing longer contexts and multiple tool interactions within AvaTaR increases computational demands, which significantly raise operational costs. These requirements necessitate substantial resources to maintain efficient performance, particularly when scaling larger datasets or more intricate tasks.
Potential Failure Modes: Although AvaTaR performs well on known queries, performance may diminish when faced with queries that require new unfamiliar combinations of tools. This limitation could be mitigated by integrating adaptive learning techniques and continuous monitoring, which would allow AvaTaR to better handle novel tool requirements.
20

---

E
Function library
E.1
Complex Retrieval Tasks
Please refer Table and Table 9 for the detailed functions.
E.2
General QA Tasks
For general QA tasks, we use the following tools:
WEB_SEARCH: A
tool that performss web searches answer questions. Useful for retrieving up-to-date information from the internet when other sources are unavailable.
ARXIV_SEARCH: This tool retrieves information about academic papers from Arxiv using a paper's unique ID. This function call provide metadata and other details for academic references.
If you have question name to lookup, this tool uses Wikipedia search to retrieve relevant information.
RETRIEVE_ This tool is used to retrieve relevant information from database. This is only available on TooIQA
21

---

Function Name PurseAttrrbuleFrromuury
Input
Output
query: string to be parsed,
This function parses "query" into a dictionary based on the imput list "attributes'
attributes: The list of attributes to be extracted from the query string: The array of list to he
GeeTeexxEmbedding
Embeds strings in list into
embedded
N tensors
GietRelevamtChunk
query: The input query string,
Get the relevant chunk of infor- mation for the node based on the query
node_ie id: The ID of the node
GetFullInfo
node_id id: The ID the node
Get the full information of the node with the specified ID
DetumentyDocuments
node id: The ID of the node
Get the text information of the node with the specified ID Get the relation information of the node with the specified ID Get the relation dictionary for the node with the specified ID, where the keys are relation type and values neighbor nodes.
GetRelationInfo
node id: The ID of the node
GetRelltioonDict
node_id: The ID of the node
GetRelatedEntities
node_id: id: The ID the node
Get nodes related the spec- ified node
GetEntityldsByTyyp
type:
The type of node to re-
Get the IDs of nodes with the specified type
trieve
GelEntityTyyps
node id: The ID of node
Get type the node with the specified ID
GelEmnityEmmedddin
node ids: An array of candidate node ids be embedded embedding_
Get the embedding indices of
nodes with ID mode_ids*
CompuiingEmiccddinnSSllaarr
emhedding_?
The cosine similarity score two embeddings
CompuucQQueryEnittyyiiiiaay
query: The input query string. node ids: An array candidate node to he compared with the
Compute embedding similarity between 'query' (str) and the nodes' 'node_ids iid'idds (list)
query
ComputeExactMatchhccrre
string: The string to be matched, node_ide jid_ The list of candidate node to be compared with the
For each node in 'node ids' compute the match score based on whether "string" is in- cluded in the information of the node
string
TokenMatchScore
The string to be matched, node_ids The list of candidate node id to be compared with the string
string:
For each node in mode_ids; computes recall scores between 'string' and the full information of the node
Summarize LoM
texts: The list texts be sum marized
Use LLM to summarize the pro-
vided texts
ClassttyEnttieeBBLLLMM
node_ide _d_: The array of candi-
Use LLM to classify each node
date node ids be classified,
specified by 'node_ide ids' into the given 'classes" 'NA'
classes: The list classes classified into
ClassifyByLLM
texts: The list of texts to be classified, classes: The list of classes classified into texts: The list of texts to ex- tract info from, extract term: the terms to identify relevant in- formation
Use LLM to classify each text into one of the given 'classes' 'NA'
ExtracikelevnnnnooByLLLM
Use LLM to extract relevant in- formation from the texts based extract term, return 'NA'
CheckRequiieneent:BBLLLMM
node ids: The array of candi- date node ids to be checked, re- quirement: The requirement to be checked
Use LLM to check if node(s)
with 'node ids' satisfies to 're- quirement*
GesSauisficionSSooeeyLLLLLLL
node ids: The array of can- didate node ids to be scored, query: The input query from user
Use LLM to score the node with 'node ids' based on the given 'query'
FINISH
final reranked answer_ list:
This function is used to indicate the end of the task
The final answer
Table 8: Function library on STARK 22

---

string to t____t____aaooo match: The key word to be matched, strings: Function Name ParseAltrrbuteerromuuey
Input
Output
query; The string to be parsed, attributes: The list of attributes to be extracted from the query
This function parses a "query" into a dictionary based on the input list "attributes"
GetiBaOOPPhrasss
image_ide __ds: The image id ar- ray to get the phrases from
Returns list of phrase list for each image the im-
age ids list
GerEntityDocuments
image_ iid:: The image id ar- ray get the text informa- tion from
Returns a list of text informa- tion for each image in the im- age_ age_ids list
GelClipTexEEmbeedding
string: The list of strings to be embedded
Embed a string or list of N strings into N embeddings
GetPachhidIoPIrrseDDie
image ids: The image list to get the patch id to phrase list dictionary from
Returns a list of patch id to phrase list dictionary for each image
Getlmages
image id ist: The list im- age ids
Return a list of images with corresponding ids
GelCliplmageEübedddnn
image_lst: The list of images to be embedded
Embed the images of list of image_ie ids into N tensors Return the patch image for the given image_id id and patch_ia id
GellmagePacchyyPPrassll
image 1d: the id of an image,
patch_id: id: the patch on the image
ComputingEmbecddiggiiiaaaty
embedding_1 I and embed- ding_
The cosine similarity score of two embeddings
ComputeF1
string_to to_mathhhtth:: The key word be matched, strings: The list of strings to be cal- culated fl score with the key word
Compute the FI score based on the similarity between 'string to match and each string in 'strings'
TokenMatchScore
Compute the recall score based on the similarity be- tween 'string to match and each string in 'strings'
The list of strings to cal- culated recall score with the
key word
ComputeExxccMMatchhsoor
string_to so__tt___m__ match: ton_gttt__tthhh The key word to be matched, strings: The list of strings to be exact matched with the key word
Compute the exact match score based on whether 'string match' exactly the same as each string in 'strings'
VqaByLLM
question:
The question to be answered, image_lst The list of images
Use LLM to answer the given 'question' based on the image(s)
ExtraatViuulAAttruutteBBLLLLL
attribute_ The list of at- tributes to be extracted, im- age_lst: The list of images
Use LLM to extract at- tributes about the given 'at- tribute_ from each image This function used indi- cate the end of the task
FINISH
final mnal_reranked, baankkedd_aawwrr..em answer__iss list: The final answer
Table 9: Function library on Flickr30K Entities
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We design novel and automatic framework that optimizes an LLM agent to effectively use the provided tools and make comprehensive analysis on the evolution of our key modules.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We did extensive survey related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
We elaborate the experiment details in the Experiment section including datasets, baselines, function libraries etc. We also release all the prompts we are using in the experiments for reproducibility.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Our code and data are accessible at https: haapa:://nnonnya /anonyoooomuuuuu 4opey science/ r //Ava /AvaTaR-FFCC//
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cic//ublic/guu hablliccgiideesSddSSuuiiiie wwweeesumisssioooooeo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include dataset information and training details in the Experiment part and Appendix. We also clearly describe the knowledge base and formally introduce the task settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We run our experiments on a single NVIDIA A100-SXM4-80GB GPU and 32-core CPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //nerrips///
Answer: [Yes]
Justification: We do induce any potential research harm mentioned in NeurIPS Code of Ethics in our paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the impact in the introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
27

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method provides framework to better use LM but not releasing LM.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Creators or original owners of assets mentioned in the paper are properly cited and the license and terms of use are respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "LieoZHUWbf", "reviewer": "Reviewer_vMWD", "summary": "This paper proposes a new framework AVATAR for LLM agent that operates in two stages:\n-The first stage is optimization during the training process, which integrates the LLM comparator component into the AVATAR. The comparator summarizes holistic prompts from positive and negative queries and iteratively optimizes LLM actor using the prompts.\n-The second stage is deployment during the inference process, where participants respond without the involvement of the comparator.", "questions": "Is \"AIR (gpt4)\" in Table 3 a typo?", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Developing prompts for LLM agents is heuristic and laborious, but this paper proposes a new method called AVATAR, which automatically iteratively generates holistic prompts from positive and negative queries. Experimental results show that AVATAR outperforms SOTA methods.\nand some points summary as following:\n1. It is a novel framework, comparing with the current SOTA approach such as ReAct, the Comparator component design is reasonable that it generates prompts from positive and negative queries.\n2. It is new SOTA of LLM Agent. Experiments are conducted against current SOTA approaches as baseline, the framework outperform the current SOTA\n3. It is a common framework, and it is effective with both Claude and GPT4.", "weaknesses": "1. The optimization is data-driven, so comparator works effectively when the positive and negative queries are well sampled and balanced during iteration; when to stop iteration and how to sample the queries; it is not clearly clarified.\n2. Regarding the construction of positive and negative queries, it is necessary to conduct experiments on the lower and upper bounds(l&h) to select appropriate values for optimal performance.\n3. Regarding the memory bank, it is a important component to swap in and out according to data quality or balanced distribution,\nit is not well designed and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and insightful comments! Here are our point-to-point responses:\n\n---\n\n## **Comment 1: Clarification about when to stop iteration and how to sample the queries**\n\nThanks! We apologize for the brief information there. We added the following details in our revision:\n\n- **Early stopping for iteration control**: In our implementation, we train AvaTaR for a fixed number of epochs and select the action sequence with the highest validation performance. Similar to training deep neural networks, early stopping is efficient when performance on a hold-out subset doesn’t improve for a set number of epochs. This approach can further save time and cost for our framework. \n\n=> ***We included more details in the experiment section and made early stopping available in our implementation.***\n\n- **Positive & Negative Query Sampling**: We employed random sampling, a simple yet effective method. Meanwhile, to ensure a balanced set of positive and negative queries (totaling $b$ queries) for contrastive reasoning, our training framework uses a larger sampling batch size, such as $1.5b$. The sampled queries are classified as positive or negative based on performance and then balanced. For example, if there are $0.5b$ positive and $b$ negative queries in the **sampling batch**, we further randomly sample $0.5b$ from the negative queries, resulting in a **training batch** of $0.5b$ positive and $0.5b$ negative queries.\n\n=> ***We added the detail to Section 4.2, Step 1: Constructing positive and negative queries.***\n\nWe hope the above discussion can address your concern about the clarity of our work!\n\n---\n\n## **Comment 2: Hyperparameter search on  lower and upper bounds ($\\ell / h$)**\n\nGood question! Yes it is important to search these two hyperparameters.\n\nIn our previous experiments, we found these two hyperparameters are pretty robust *w.r.t.* small adjustments. Therefore, we set $\\ell=h=0.5$ for all datasets. But now we draw more clear conclusions with additional experiments. To recap, queries where the Recall@20 metric is higher than $\\ell$ are grouped into positive and queries where the Recall@20 metric is lower than $h$ are grouped into negative groups. Thus we have $\\ell \\geq h$. \n\nSpecifically, we tested multiple combinations of $\\ell$ and $h$. We conducted experiments on STaRK-Amazon only due to the time and computational cost involved. And here are the Hit@1 results:\n\n|  | $h=0.3$ | $h=0.4$ | $h=0.5$ |\n|:---:|:---:|:---:|:---:|\n| $\\ell=0.5$  | $48.32$ | $50.01$ | $49.87$ |\n| $\\ell=0.6$  | $47.89$ | $49.56$ | $50.45$ |\n| $\\ell=0.7$  | $47.75$ | $48.56$ | $49.34$ |\n\nInterestingly, we obtained **a better Hit@1 result** than the one reported in the paper when $\\ell = 0.6$ and $h = 0.5$. Results for other metrics can be found in the uploaded PDF.\n\n**Key Observations:**\n\n- The performance is **robust** to the large adjustments, with variations of 2.7% in Hit@1.\n- We observe some **decline when the gap between $\\ell$ and $h$ is too large**. This is likely due to omitting part of the training queries (the performance on which are within $(h, \\ell)$.\n- We see **slight improvement when there is a moderate gap** between $\\ell$ and $h$, which could help establish a clearer pattern difference between positive and negative queries without “sacrificing” too many training queries.\n\nWe add the above results and discussion to Appendix B. We hope our study can address your concern about hyperparameter selection. In general, our framework relies on a very small number of hyperparameters, *i.e.*, $\\ell$, $h$, batch size $b$, and training epochs. We believe the study on $\\ell$ and $h$ demonstrate the robustness of our framework. \n\n---\n\n## **Comment 3: Design of memory bank and its influence on the optimization process**\n\nThanks a lot for this insightful comment!\n\nFor memory bank construction, we follow Reflexion [1], which also exhibits long-term memory. Alternatively, our framework allows for the direct plug-in of other kinds of memory banks. For example, a dynamic and multi-layered memory structure proposed by [2] can be used for storing knowledge and experience dynamically from past training. We believe a more advanced memory component is a great addition to our optimization framework, which has been less explored in previous studies.\n\nWe also want to point out that our main contribution is orthogonal to the design of memory banks. Specifically, our novelty and success largely attribute to the comparator in the optimization module, which is validated in the ablation study between AvaTaR and AvaTaR-C.\n\nFollowing your suggestions, in our revision, 1) we added more discussion about the memory bank, and 2) we pointed out that enhancing the memory bank could be a key innovation in the future. Note that while revision is not allowed during rebuttal, we are happy to copy the updated paragraphs here if you would like to see them!\n\nWe hope the memory module is well-discussed now. We thank you for these insightful suggestions again, which inspired us for future extensions.\n\n**Typo**: Good catch! Yes it should be \"AvaTaR,\" and we've fixed it in the revision. Thanks!\n\n---\n\n# **Summary**\n\nWe thank you for your time and insights! We hope our 1) clarification on the training process, 2) experiments on hyperparameters, and 3) discussions on memory banks address your concerns well. \n\nAt last, we would greatly appreciate your support and reconsideration given our response.  We would like to emphasize that our main contribution is the development of an optimization framework featuring a novel comparator module. This module enhances contrastive reasoning and improves generalization, allowing our approach to significantly outperform baseline methods. Thank you in advance!\n\n---\n\n**Reference**\n\n[1] Shinn et al. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS.\n\n[2] Zhong et al. MemoryBank: Enhancing Large Language Models with Long-Term Memory.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sccMVb1A8A", "reviewer": "Reviewer_CkbF", "summary": "This paper introduces AVATAR, a novel framework for optimizing large language model (LLM) agents to effectively use provided tools and improve performance on complex multi-step tasks, with a focus on retrieval tasks. The key innovation is a comparator module that generates holistic instructions to improve the actor (main agent) through contrastive reasoning on batches of well-performing and poorly-performing queries.\nThe authors demonstrate AVATAR's effectiveness on four challenging retrieval datasets from the STARK and FLICKR30K-ENTITIES benchmarks. Results show significant improvements over state-of-the-art baselines like ReAct and Reflexion.", "questions": "Regarding the use of the memory bank as in-context learning examples, \nHow does AVATAR compare to directly finetuning the actor model in terms of performance, efficiency, and data requirements? What are the key advantages of the instruction-based approach?\nHow does the method handle scenarios where the action model improves significantly, making it difficult to find negative examples for the contrastive learning process? Is there a strategy for addressing this?\nHave you explored applying AVATAR to other types of tool-use tasks beyond retrieval? What challenges do you anticipate in extending to more diverse task types?", "limitations": "The authors have not adequately addressed the limitations of their work. I recommend adding a dedicated limitations and broader impacts section to address these points more thoroughly. \n\nThis should include: Discussion of computational requirements and scalability limitations, potential failure modes or scenarios where the method may struggle", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel approach: The comparator module using contrastive reasoning on query batches is an innovative way to generate holistic instructions for improving agent performance. This addresses limitations of per-sample instruction approaches.\n\nClear motivation and explanation: The paper draws insightful analogies to concepts like batched training and gradient computation in neural networks to explain the intuition behind the approach. This makes the core ideas easy to understand.\n\nStrong empirical results: AVATAR consistently outperforms strong baselines across multiple datasets, with significant improvements on key metrics. The ablation study clearly demonstrates the value of the comparator module.", "weaknesses": "Limited scope of evaluation: While the retrieval tasks are complex, it would be valuable to see AVATAR applied to a broader range of tool-use benchmarks to demonstrate generality.\n\nLack of comparison to finetuning: The paper does not discuss or compare to alternative approaches like directly finetuning the actor model via rejection sampling. It's unclear how AVATAR compares to such methods in terms of performance and efficiency.\n\nScalability considerations: The paper does not thoroughly address how the method scales with increasing numbers of tools or more complex task structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive feedback! We provide point-to-point responses:\n\n---\n\n## **Comment 1: Applying AvaTaR to more tool-use benchmarks**\n\nThanks! We've conducted initial experiments on ArxivQA and are currently running tests on ToolQA. Please stay tuned! For ArxivQA, we randomly sampled 500/200 training/testing queries, providing three web search tools and five LLM tools such as summarization.\n\n| | CoT| ReAct | AvaTaR |\n|:---|:---|:--|:--|\n|Correctness by LLM Judge|$58.0$\\%|$73.5$\\%|$85.5$\\%|\n\nWe are happy to provide more details. Notably, AvaTaR achieved a substantial improvement (12%) over ReAct. For our other evaluation datasets, we highlight that they are more comprehensive than some existing tool-use datasets. For example, ToolEye [1] offers a fine-grained system but only has 54 retrieval queries, while MetaTool [2] involves only one or two tools per query. In contrast, our datasets involve action sequences with over ten tools (Figure 2) and span multiple modalities, totaling over 4k testing queries. \n\n---\n\n## **Comment 2 & Question 1,2: Comparison between finetuning methods and AvaTaR (with memory module)**\n\nThanks for the insights! We agree that finetuning is a potential way to improve the actor LLM. We give an overview first and then conduct the comparison.\n\n- **Overview**: Several benchmarks and methods focus on fine-tuning tool-augmented LLMs. ToolBench (ICLR’24) [3] and GPT4Tools [4] create instruction-tuning datasets to enhance tool-use capabilities. Similarly, API-Bank [5] provides a dataset to improve planning, retrieval, and tool-calling skills. ToolAlpaca [6] generates tool-use corpus to develop generalized tool-use abilities in smaller LMs through fine-tuning.\n\nHowever, we believe **finetuning approaches may be challenging to adapt to our tasks** and AvaTaR has better advantages in the following aspects:\n\n- **Data Requirements**: Previous works [4,5,6,7] require large-scale datasets involving thousands of tools for LLM finetuning, while some tasks use fewer than a hundred tools, making data generation challenging. Moreover, creating instructions with ground truth tool use from ChatGPT [3,4] or multi-agent frameworks [5,6] can be difficult, especially for retrieval tasks with external knowledge bases. In contrast, AvaTaR needs only a subset of training QA and tool descriptions, making its data requirements less demanding.\n\n- **Efficiency:**\n  - **Inference:** AvaTaR applies a fixed solution to testing queries without additional memory modules or comparators, making its inference efficiency comparable to or better than finetuning methods.\n  - **Training:** AvaTaR reduces human effort by eliminating the need to design GPT prompts or multi-agent frameworks for annotating tool usage. For time cost, AvaTaR is observed to be efficient (c.f. Figure 4), typically converges within 50 iterations. At each iteration, we track only the top-5 action sequences with the best performance, minimizing token costs from the memory bank. \n\n- **Performance:** The reasoning ability of the actor LLM impacts task performance, motivating the use of state-of-the-art LLMs like Claude3, which are closed-source and mostly unavailable for finetuning. Besides more flexibility to use these models, AvaTaR can use any finetuned model with enhanced tool usage abilities as the actor LLM, which can be further trained by the comparator.  \n\nWe added this discussion in our revision. Thank you for highlighting finetuning methods to help us better position our work. \n\n---\n\n## **Question 3: Strategy for selecting negative samples**\n\nGreat question! This inspires us to implement an adaptive threshold strategy. Specifically, we track evaluation results on a subset of training queries from the past several epochs and use the median performance as the threshold for the current epoch. This helps ensure a sufficient number of negative samples for selection.\n\n---\n\n## **Question 4 and Comment 3.2: Challenges of extending AvaTaR**\n\nWe have extended AvaTaR to general QA tasks with newly added datasets. With recent support for AvaTaR in the DSPy [7] library, we expect its application to more tasks, such as coding problems, where positive/negative sampling can be determined by unit test results.\n\nA key challenge for complex tasks, like visual reasoning [8,9], is that the input space can make extracting insights difficult. These tasks often require stronger reasoning to identify patterns compared to simpler formats like natural language queries. We added this to the future work.\n\n---\n\n## **Comment 3.1: Scalability and limitations**\nOur paper used around 25 tools per dataset, offering a decent level of complexity. Further, we discuss the limitations:\n- **Scalability**: With LLM context lengths increasing (up to 128k), AvaTaR can scale to handle hundreds of tools and complex tasks, but practical limitations like increased latency may impact performance. Future efforts could integrate finetuned tool-augmented LLMs as actors or comparators to facilitate smooth scaling.\n- **Computation Requirements**: Scaling AvaTaR can increase computational costs due to the need to manage longer contexts and multiple tool interactions, leading to higher expenses.\n- **Potential Failure Modes**: While AvaTaR generalizes well to testing queries, performance may degrade if queries require novel tool combinations that AvaTaR hasn't trained for. Robust monitoring and adaptive learning strategies may help mitigate these risks.\n\nWe added a limitation section with extended discussions.\n\n---\n\n# **Summary**\n\nWe appreciate your approval on our novelty, motivation, and effectiveness. We hope we address your concerns with 1) more experiments on two QA datasets (ongoing), 2) extensive comparison with finetuning methods, 3) a limitation section. We are more than happy to follow up. \n\nWe also kindly ask if you could reevaluate our work if some concerns are addressed. Thank you for your insights and support again!\n\n*Reference is in the uploaded PDF*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BFjtW8S184", "reviewer": "Reviewer_JqLP", "summary": "This paper introduces a new framework, AVATAR, that allows LLM agents to optimize the performance of Knowledge Retrieval. In the framework, a Comparator agent is adopted to extract insight from the positive and negative samples. The experiments on four retrieval datasets show the effectiveness of the method.", "questions": "In line 160, \"the comparator samples a group of data samples (question-answer pairs), executes the current actions for each question,\" but how can the comparator execute the current actions? What are the current actions for the comparator?\n\nThe text in Figure 3 is too small to read and the meaning of the marked text is difficult to understand.", "limitations": "There is no separate Limitation or Broader Impacts section in the paper. In addition, the authors said, \"We did an extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations,\" but the discussion of the limitations in this paper is required.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The schematics in the paper are well drawn and demonstrate the proposed method well.\n2. The experimental settings and prompts are detailed.\n3. The experiments are performed on STARK and Flickr30K Entities benchmarks to show the effectiveness of the methods.", "weaknesses": "The main weakness is the lack of innovation and comparison compared with cutting-edge works.\n\n1, This proposed AVATAR is very similar to ExpeL [1] in terms of method and has no obvious innovation in comparison. ExpeL adopts a Reflexion agent (actor) to gather success and failure experiences of multi-step tasks, which is the same as the actor's role in this paper. ExpeL adopts another agent to compare a failed trajectory with a successful trajectory for the same task and extract insights from the comparison, which is the same as the instruction generation process of the comparator in this paper. ExpeL adopts the experience pool to retrieve successful trajectories, which is the same as the Memory Bank in this paper. Similarly, Autoguide [2] also extracts insights from experiences. Therefore, the method of auto-optimizing the instruction by comparing the successful and failed trajectories has no innovation compared with Expel [1] and Autoguide [2]. The author should conduct more surveys and read relevant cutting-edge works.\n\n2, The paper claims their method achieves SOTA on STARK and FLICKR30K-ENTITIES benchmarks, but they only compared with relatively basic methods. The author employs several outdated embedding-based retrievers as the comparison. However, text-to-image retrieval on FLICKR30K-ENTITIES has long been dominated by the vision-language model [3] [4]. Internvl [3] and Beit [4] achieve high recall@1 (from 80 to 90) on this benchmark. Therefore, it is unreasonable to claim that the proposed method achieves the retrieval SOTA on these two benchmarks.\n\n3, The results of the comparison on FLICKR30K-ENTITIES are missing. Quantitative results on FLICKR30K-ENTITIES are only shown in Figure 5 (right), but the results of all methods on this benchmark are not found in the paper. Considering that STaRK is a relatively new benchmark, not many methods have experimented on it, so results and analysis on recognized benchmarks are important.\n\n4, Many sentences in the paper are too obscure to understand and lack clear explanations. For example, \"these per-sample instructions tend to be narrow and fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary with others, leading to decreased performance when applied across different scenarios.\" in lines 152-154, what do \"per-sample instructions\" mean, and why will these tool combinations \"lead to decreased performance\"? Providing some simple examples can make it easier to read.\n\n[1] Zhao, Andrew, et al. \"Expel: Llm agents are experiential learners.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.\n\n[2] Fu, Yao, et al. \"Autoguide: Automated generation and selection of state-aware guidelines for large language model agents.\" arXiv preprint arXiv:2403.08978 (2024).\n\n[3] Chen, Zhe, et al. \"Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wang, Wenhui, et al. \"Image as a foreign language: Beit pretraining for vision and vision-language tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments! We carefully justify our novelty and clarify the misunderstandings. We will be grateful for your patience of reading our response:\n\n---\n\n## **Comment 1: Comparison with Expel [1] and Autoguide [2]**\nWe apologize for missing these important and relevant papers. Thank you for helping us do right by the existing work. While agree that they are relevant, we summarize the key differences:\n\n||ExpeL|AutoGuide|AvaTaR|\n|:--|:--|:--|:--|\n|**Contrast Over**| Successful and failed trajectories (action sequences)|Trajectories|Positive & negative data samples|\n|**Training Phase**|Two phases: sample experiences, then extract insights|Two phases, similar to ExpeL|One phase training with iterative instruction generation and action refinement|\n|**Inference Method**|In-context generation with extracted insights|Iterative generation with state guidelines|**Direct generalization with optimized action sequence|\n\n---\n\nSpecifically, ExpeL is an innovative method that leverages experiences across multiple training tasks, and AutoGuide introduces state-aware guidelines for agents. All three methods learn from past experience.\n\nHowever, the high-level ideas are different. Specifically, AvaTaR **contrasts positive and negative queries for the action sequence generated by the actor at current step**, while both ExpeL and AutoGuide **contrast failed and success action sequences given an instance**. As an analogy with RL, the instructions from AvaTaR comparator is **on-policy** (target at current action sequence), while ExpeL and AutoGuide requires **off-policy** approximation with data pool. Moreover, AvaTaR conducts end-to-end training and direct inference on up to 4k testing queries in total, which achieves higher scalability compared to in-context inference methods.\n\nWe added discussions in our revision. Once again thank you very much for helping us position our work better.\n\n---\n\n## **Comment 2.1: “The paper claims their method achieves SOTA on the benchmarks”**\nOur statement in the abstract - “We find AvaTaR consistently outperforms SOTA  approaches” - refers to SOTA in the scope of LLM agents. We apologize for the confusion and have clarified the statement.\n\n---\n\n## **Comment 2.2: “they only compared with relatively basic methods”**\n\n- ### **a) Recap: Baselines used**\n  - For STaRK, we compared **all available methods** reported by the benchmark and two agent methods: ReAct and Reflexion.\n  - While image retrieval tasks are less explored by previous agent works including Expel and Autoguide, we made **non-trivial adaptations** to apply ReAct and Reflexion.\n\n- ### **b) Justification: Are the baselines sufficient?**\n  We believe AvaTaR is compared against essential baselines. For reference, Expel compared with ReAct and Act. While ReAct generally outperforms Act, we use Reflexion as an alternative. Similarly, Retroformer [3] used ReAct and Reflexion as baselines.\n\n---\n\n## **Comment 2.3: “outdated embedding-based retrievers ... dominated by the VLMs (Internvl and Beit)”**\n- ### **a) Recap: What are the embedding-based retrievers we used?**\n  - For STaRK, we used `text-embedding-ada-002`, a competitive model on the Mteb leaderboard during the time of this work. \n  - For Flickr30k-entities, we used `clip-vit-large-patch14`, a commonly used VLM.\n- ### **b) Clarification: The embedding models are not only baselines, but also tools for agents**\n  Table 5 and 6 listed a set of tools for **all agent methods**. E.g., `GetClipImageEmbedding` uses the same CLIP model as the baselines for fair evaluation.With the **given tools**, our goal is to improve agents' tool-use ability, which leads to improved performance.\n-  ### **c) Justification: AvaTaR v.s. VLMs or AvaTaR+VLMs?**\n   In the above setting, AvaTaR achieve proof-of-concept results which outperforms the baselines **with the same tools or embedding model**. \n\n   Differently, `Internvl (6b)` and `Beit (1.6b)` pretrain large VLMs, which are not fairly comparable with AvaTaR since it used a much smaller clip model with 427m parameters. More importantly, *the goal of AvaTaR (improving agents) and the pretraining methods (training powerful VLMs) are different*.\n   \n    In fact, **these two method classes are not conflicting. Both VLMs can be tools leveraged by AvaTaR!** This inspires a practice future direction to construct powerful tools from open world.\n\n---\n\n## **Comment 3: Missing comparison on FLICKR30K-ENTITIES**\nThe results include all available methods. Compared with Table 2, Dense Retriever is a finetuned text encoder from STaRK; QAGNN is for knowledge graphs; Multi-VSS chunks documents and is not applicable to images. We added explanations for clarity.\n\n---\n\n## **Comment 4: Statements in L152-154**\n - **“Per-sample instructions”**: Instructions generated for a failure/negative query instance.\n- **Example**: When answering query `What are some recommended traditional setup fishing rigs from the Thill brand?` Reflexion tend to rely on specific details like \"Thill\". E.g., It (1) computes a token match score, and if there is a match (2) returns all fishing rigs items, which fails to generalize when user requests other Thill items\"\n\n---\n\n## **Comment 5: Limitations**\nWe added a limitation section. Due to character limit, please refer to our response to `reviewer CkbF`. Thanks!\n\n---\n\n**Question 1: Clarification of comparator**\n- “Current actions” refers to the action sequence generated by the actor in the current training step. \n- “Execute the current actions” means evaluating the current action sequence on sampled queries. Actions are executed by actor, we clarify it in the revision. \n\n**Question 2: Figure 3.** \nThe light orange and blue highlight features of positive and negative queries, respectively. We will adjust the caption size for readability.\n\n---\n\n# **Summary**\nWe sincerely hope we address your concerns. We would very much appreciate it if you could reconsider your evaluation if some concerns are addressed. Thank you very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General response**

We truly appreciate the reviewers' efforts and valuable suggestions in reviewing our paper. We are glad that all/most reviewers reached a positive consensus on our work's presentation, motivation, novelty, and experimental effectiveness. Here is a summary of the reviewers’ major feedback and our corresponding actions: 

-----

| | Reviewer JqLP | Reviewer CkbF | Reviewer vMWD | Action/Summary | 
|:---|:---|:---|:---|:---|
| **Presentation** | “Presentation: good” | “Clear motivation and explanation” | “demonstrate the proposed method well” | `NA`|
| **Novelty** | “novel framework, …, the Comparator component design is reasonable” | “The comparator module … is an innovative way” | “the method…has no innovation compared with Expel [1] and Autoguide [2]” | RE reviewer vMWD: `We carefully read the related works shared by reviewer vMWD. While highly relevant, we  found essential differences between the mentioned works and AvaTaR in terms of novelty and insights. Please see our response to reviewer vMWD for a detailed explanation.` |
| **Empirical Improvements** | “effective with both Claude and GPT4” | “AvaTaR consistently outperforms strong baselines across multiple datasets, with significant improvements on key metrics” | “The experiments are performed … to show the effectiveness of the methods.” | `NA`|
| **Evaluation Setting** | NA| Suggestions on a broader application of AvaTaR on tool-use benchmarks | “The paper.. (1) only compared with relatively basic methods, … (2) employs several outdated embedding-based retrievers…(underperform) Internvl and Beit VLMs. ” | RE reviewer CkbF: `We add additional experiments on two QA and tool-use benchmarks - ArxivQA and ToolQA (ongoing).` RE reviewer vMWD: (1) `We justified the baselines included, where ReAct and Reflexion are two prevailing agent methods.` (2) `We emphasize that our goal is to optimize agents and improve their tool-use abilities. We use clip-vit-large-patch14 (427 million parameters) embedding model as a tool for all of the agent methods. In contrast, Internvl (6b parameters) and Beit (1.6b parameters) study pretraining methods for large-scale VLMs, which have different goals and settings than AvaTaR.` |
| **More Study and Discussion on AvaTaR** | Pos & neg query selection and the contribution of memory bank | (1) Scaling up with more tools and complex task structures. (2) Comparison between AvaTaR and finetuning methods | NA| RE reviewer JqLP: `We added experimental studies on pos & neg query selection.` RE reviewer CkbF: (1) `We highlight the level of complexity that our tasks reached and discuss future directions on scaling up.` (2) `We added extensive discussion between AvaTaR and finetuning methods  in terms of performance, efficiency, and data requirements` |


Moreover, following the suggestions from Reviewers CkbF and vMWD, we added a dedicated `limitation section` (see response to the reviewers) in our final draft. 

------

# **Summary**

We thank the reviewers for their suggestions which make our work more solid. We have improved our manuscript accordingly. We hope our responses can clarify any confusion and alleviate the remaining concerns above. 

We would be thrilled if you could let us know whether your concerns have been addressed or if you have any follow-up questions! 

— Best,

  Authors of Paper 5514

Meta-review: The paper initially received 2 Weak Accepts and 1 Reject ratings. While reviewers praised the presentation, effectiveness and interesting ideas, there were concerns on lack of details, lack of novelty and missing experimental comparisons. The rebuttal and discussions helped significantly and assuaged most concerns, leading to substantial discussion between the reviewers and resulting in a mostly positive score. The AC believes that the paper has merit and can be published at NeurIPS. We require that the clarifications, rewritings and additional information provided in the rebuttal be incorporated in the camera-ready version.
Decision: Accept (poster)