Title: AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM agents

Tool utilization

Automatic prompt optimization

Complex retrieval

Question-Answering tasks
Abstract: Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AvaTaR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demon- strate AvaTaR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AvaTaR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar.

AVATAR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning

Abstract

Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AVATAR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demonstrate AVATAR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AVATAR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar .

1 Introduction

Autonomous agents powered by large language models (LLMs) offer substantial promise for complex problem-solving [6, 39, 41, 55, 65]. These agents demonstrate remarkable capabilities in reasoning [46, 47, 54, 55] and planning [8, 13, 14, 62]. Additionally, their functionality is extended through the use of external tools that provide access to external or private data and specialized operations, such as APIs for interacting with knowledge bases and search engines. These tools enable agents to perform complex tasks like multi-step problem-solving and retrieving diverse information, which is essential for complex retrieval and question-answering (QA) [13, 21, 26, 33, 38, 40, 48].

Despite the promising capabilities of LLM agents, it remains challenging to engineer effective prompts that guide these agents through a multi-stage process for real-world problem-solving. This process involves (1) decomposing a complex question into an actionable plan with simpler steps, (2) strategically using provided tools to gather relevant information, and, finally, (3) synthesizing intermediate results to produce a coherent and accurate response. Each step requires extensive manual effort and numerous iterations of trial and error to refine the prompts.

Current approaches have primarily focused on directly deploying agents using complex humandesigned 'mega-prompts' [18, 24, 55], which require lots of manual trial and error. Nevertheless, such hand-engineered mega-prompts may also result in brittle implementations with suboptimal accuracy (see Figure 2 (a)), where the ReAct agent [55] easily produces trivial and misleading answers to

OptimizedActor & Actions

Tools

Positive & negative queries

(question, answer)

pairs

Optimization

phrase

Current

actions

Comparator

Sampled data

Contrastive

reasoning

Initial instructions

Actions

User

Deployment

phrase

Answer

Query

Utilization

Actor

Tools

Updated instructions ontoolusage

andproblem-solvingstrategies

Generate

Prompt

A

b

A

.T.

customers' queries about specific products. Furthermore, existing research [4, 5, 45, 50, 56, 60, 64] on employing LLMs as optimizers often fails to adequately refine the complex strategies for enhancing tool integration and usage. This lack of strategic optimization can lead to less effective, nongeneralizable agent applications in complex real-world scenarios.

Present work: AVATAR . To address these challenges, we introduce AVATAR, an automated framework that optimizes agents for effective tool utilization and excellent task performance. Specifically, we leverage key insights from contrastive reasoning and build a comparator module ('trainer') to generate holistic instructions and prompts ( i.e., , computing a robust 'gradient') to optimize an actor LLM. We demonstrate our framework on challenging tasks of knowledge base retrieval, which involve complex multi-stage procedures and extensive tool usage, and general QA tasks. Specifically, AVATAR includes two phases:

Optimization phase . The core of our optimization framework (Figure 1) is a comparator LLM that automatically generates holistic prompts to teach a actor LLM to differentiate between effective and ineffective tool usage. The comparator takes positive and negative data samples, where the current agent performs well and poorly, respectively, to identify overall gaps and systematic errors exhibited by the agent. Unlike per-sample instructions, which can easily lead to overfitting on individual data points, by constructing multiple samples as a 'batch,' the comparator can extract a more robust 'gradient' to 'backpropagate' to the actor. In other words, the comparator can provide more effective and adaptive prompts through batch-wise contrastive reasoning, helping the agent identify flaws in solving challenging multi-stage problems. Following previous methods [30, 41, 56, 63], we also maintain a memory bank with selected past instructions to prevent the actor LLM from repeating previous mistakes.

Deployment phase . After the optimization phase, the actor with best-performing prompts can be selected for the testing instances. Moreover, in complex retrieval tasks, the iterative optimization through our AVATAR framework updates the actor for more effective and generalizable action sequences, enabling direct generalization to novel user inquiries at deployment. In Figure 2 (b), the optimized actor creates three novel strategies: 1) precise decomposition of problems by extracting multifaceted attributes, 2) effective tool usage through a sophisticated and robust scoring system, and 3) the strategic combination of different scores, determined by learned coefficients, ensuring accurate and comprehensive retrieval.

Experimental evaluation . We conduct extensive experiments on four retrieval datasets and three QA datasets. The retrieval tasks are highly complex, involving multimodal data, including textual, visual, and relational information. AVATAR consistently outperforms state-of-the-art methods, showing a

substantial 14% improvement in the Hit@1 metric. Impressively, with only 25 iterations, AVATAR boosts the Hit@1 metric from an initial 5.1% to 28.6% on FLICKR30K-ENTITIES [35] and the Recall@20 metric from 30.3% to 39.3% on STARK-PRIME [49]. For general QA datasets, AVATAR outperforms state-of-the-art methods by 13% on average. These improvements, achieved through iterative updates to the prompts, underscore AVATAR's ability to optimize agents for complex tasks and effective tool usage. Our key contributions are:

We introduce AVATAR, a novel framework that optimizes an actor for effective tool utilization through a comparator module that automatically generates holistic prompts.

We demonstrate AVATAR on four complex retrieval tasks and three QA tasks, where it significantly outperforms existing agent methods in terms of task performance and generalization ability.

We provide a comprehensive analysis of the actor's evolution during optimization, highlighting how comparator automatically provides targeted instructions that improve and generalize the actor.

2 Related Work

LLM Agents . Recent research has leveraged the remarkable language understanding and reasoning abilities of LLMs [1, 41, 47, 54, 55] to complete downstream tasks. For complex tasks that require enhanced capabilities, previous works have positioned LLMs as agents that can interact with environments [4, 6, 13, 18, 21, 26, 27, 40, 48, 55], leverage external tools [6, 28, 31, 33, 36, 38, 39, 66, 68], and gather experiences [7, 61]. For example, ReAct [55] conducts reasoning and action in an interleaved way, retrieving information from Wikipedia to support reasoning.

LLM Agents for Retrieval . Previous research has applied LLM agents to Information Retrieval (IR) systems through pretraining [2, 9, 16, 57], reranking [12, 42], and prompting techniques [11, 18]. In IR systems, the retriever module directly influences the performance of downstream tasks, such as retrieval-augmented generation [20, 29, 30] and knowledge-intensive question answering [34, 52]. For example, EHRAgent [40] is designed for EHR question-answering, capable of retrieving relevant clinical knowledge through a structured tool-use planning process and an interactive coding mechanism. However, these LLM agents usually employ heuristic (zero-shot) prompts or rely on few-shot examples [18, 25, 40, 55] for downstream tasks, which lack more informed guidance on generating effective retrieval strategies and tool-assisted actions.

Agent Optimization . In the field of optimizing LLM agents, previous works have modified the parameters of LLM backbones through fine-tuning or instruction tuning to enhance agent capability [3, 15, 19, 23, 32, 33, 37, 43, 51, 58, 59] or generated better prompts through iterative prompt tuning [11, 18, 45, 50, 56]. Recently, Zhang et al. [60] conducted agent training by iteratively updating the agents' functions according to the execution history. However, these methods do not explicitly consider targeted optimization for tool usage or the impact on complex multi-stage tasks. Additionally, enhancing agents' generalization abilities [10, 31, 44], essential for real-world applications, has received less attention. In our work, we focus on automatically generating holistic instructions via a novel contrastive reasoning mechanism, targeting effective tool usage and agents' generalization ability. Compared to fine-tuning approaches, AvaTaR only require a small subset of training data and tool descriptions, making it more adaptable and less computationally intensive.

3 Problem Formulation

Definition 1: Tools . We define tools or APIs as a set of implemented functions with specified input and output variables. We denote the abstract tool space as T = { f k : I f k →O f k | k = 1 , 2 , . . . } , where f k maps the input I f k to the output O f k . For example, the tools can be APIs used for accessing external knowledge via a search index, an encoder model that generates vector representations from text or image data, or a task-specific classifier that outputs probabilities over a list of classes.

Definition 2: Agents . An LLM agent, defined as A : P → α , is controlled by verbal prompts to generate a flow of actions needed to complete a task. Here α denotes the action sequence [ α 1 , . . . , α L ] , where each action is defined by a tuple ( f ∈ T , i ∈ I f , o ∈ O f ) , consisting of a tool function, specified input(s), and a designated variable that receives the output(s). Each action in the sequence can leverage the outputs generated by previous actions, with the final action α L rendering the results for the task.

Input:

Q1

:

'

Can you suggest any TUSA swim fins that has a split fin design for better propulsion?'

Action

space

(GetEntityTypes,

GetEntityDocuments

,

StringMatching

,

…)

Output:

Answer(s)

Thought

1:

Compute

similarity scores based

on

the

product description

Action

1: ComputeEmbeddingSimilarity

['swim fins',

GetEntityDocuments

()]

Result/Obs 1:

s1

similarity scores

Ignore the brand information 'TUSA'

Thought

2:

Check the

functionality requirement

Action

2: StringMatching

['split fin design for better propulsion',

GetEntityDocuments

()]

Result/Obs 2:

s2

string matching scores

All zeros due to no perfect match

Thought

3:

Synthesize the

final result

Action

3: Multiply

[s1, s2]

Result/Obs 3:

s

final scores

All zeros which lead to a trivial solution

...

Final Result:

answers

GetTopkEntities

[s, k=5]

Poor task performance

Input:

Any query

(demonstration

example:

Q1

); Action space (GetEntityTypes

,

…)

Output:

Answer(s)

Accurately decompose the query into multiple aspects

Action

1: ParseAttributeFromQuery

[query,

(brand, type,

material, features)]

Result

1:

subquery

{

brand: 'TUSA',

type: 'swim

fins', material:

NA,

features:

'split

fin design

for better

propulsion'

}

Use embedding tool to filter entities

Action

2: ComputeEmbeddingSimilarity

[subquery.type,

GetEntityTypes

()]

Result

2:

s1

type

similarity scores

Action

3: GetTopk

Result

3:

candidates

top-20 entities with

the

highest

type

similarity

Use token match tool for flexible common token matching

Action

4

:

GetEntityBrand

[candidates]

Result

4:

brands

brands

of

the

top-20

entities

Action

5: TokenMatchScore

[subquery.brand,

brands]

Result

5:

s2

brand matching scores

Use LLM reasoning API to validate the required functionality

Action

6: GetSatisfictionScoreByLLM

[subquery.features,

GetEntityDocuments

()]

Result

6:

s3

feature scores by

LLM reasoning

...

Synthesize final scores with optimized parameters

Action

7: WeightedSum

[s1,

s2, s3,

coefficients=(0.43, 0.37, 0.20)]

Result

7:

s

combined scores

...

Final Result:

answers

GetTopkEntities

[s, k=5]

Excellent task performance

c

Q1

:Can you suggest any TUSA swim fins that

has

a split fin design for better

propulsion?

[Q'1

：

Looking for a durable 15-inch wide NFL

car flag that can hold up in windy conditions',

A1

(Ground truth entity IDs)

：

[17, 105, 2517]

]

...

(Qn, An)

Q2

:What's a high-quality fishing sinker from

Sportsman Supply Inc. that's designed to

avoid snags on rocks and weeds?

Small-scale Q&A Pairs

Testing Queries

Optimization

Deploy

(a) ReAct: Unoptimized Agent

(b) AVATAR: Optimized Agent

X

41

√

Multi-step problem-solving . Real-world problems are inherently complex and cannot be effectively addressed through straightforward solutions or simple tool usage alone. Solving real-world problems with LLM agents can be structured into a multi-stage procedure:

Decomposition of the problem : The procedure begins by breaking down a complex question into an actionable plan characterized by simpler steps. This decomposition is crucial for setting clear objectives and facilitating focused problem-solving.

Tool-assisted subproblem solving : In the subsequent phase, agents strategically utilize tools from the established tool space T to gather solutions for each step. This stage is essential for acquiring

the necessary information required to effectively address each subproblem of the decomposed problem.

Synthesis and response formulation : The final stage involves synthesizing the intermediate results to construct a precise response. This synthesis not only combines the data but may also refine the response through trials and adjustments, ensuring the solution's accuracy and relevance.

For example, retrieval tasks are inherently complex and demanding. Given a user query q , retrieval tasks aim to identify or generate a ranked list of relevant entities E from the entity space of a knowledge base. Each query is associated with a set of ground truth answers, denoted as Y , which are used to compute the quality of the prediction. Specifically, the LLM agent is required to 1) comprehend a user's request, 2) utilize the provided tools to identify and analyze relevant information in the large knowledge space, which may contain multimodal data sources, and finally, 3) integrate all gathered information to reason and generate an accurate response.

4 Our Method: Optimizing Agents for Tool-Assisted Multi-Step Tasks

Each step in the multi-stage problem-solving process (described in Section 3) requires effective prompts to identify key flaws and improve task performance. However, refining the agents' prompts demands extensive manual effort and numerous iterations of trial and error.

To address this, we introduce an automated and novel optimization framework, AVATAR, which generates prompts to improve agents' tool usage and task performance. In Table 1, we highlight four critical aspects of our approach compared with prevailing agent frameworks [27, 41, 55]. Here, we introduce the two main LLM components in AVATAR: a actor LLM (Section 4.1) and a comparator LLM (Section 4.2).

4.1 Actor Construction and Challenges

Actor . The actor agent, as defined in Section 3, is responsible for generating initial actions based on the initial instructions/prompts and adjusting actions according to updated instructions. Specifically, the initial instructions provide details about the task and available tools, where tools can be introduced in programming languages such as Python. During optimization, the prompts further incorporate the previous action sequence and updated instructions to adjust these actions. The actor then generates revised actions, which could include a combination of tool usage through programming language (code generation) along with natural language explanations of how the tools are employed.

Challenges in multi-step complex tasks . A common approach to updating instructions utilizes execution results or performance data from a specific instance, often through techniques like selfexplanation [4, 27] or self-reflection [41, 56]. However, this approach may not be suitable for complex tasks involving tool usage. Complex multi-step tasks include multiple interacting factors that influence overall performance, such as problem decomposition and tool selection. Consequently, instructions generated for a failed/negative query instance tend to be narrow in scope and may fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary across different scenarios, potentially leading to decreased performance when applied to varied cases.

'Looking for Santa Cruz skateboard decks.

I'm interested in the traditional circle motif, preferably

in white, red, or yellow color schemes'

'What are some adjustable

caps from Top of the World

featuring the SEC logo on the

side? '

Actions

'What are the affordable and

quality car emblems from the

Football Fanatics brand? '

'Where can I find Aminco silicone rubber

bracelets that will match my NFL Tampa

Bay Buccaneers wristbands?. '

Action

1:

attribute

ParseAttributeFromQuery

[query,

(brand)]

.

.

.

Action

3:

brand_score

ComputeEmbeddingSimilarity

[

attribute.brand,

GetEntityDocuments

()]

<Task & Tool description>, <Actions>

Post-action execution, the queries exhibited varying performances.

Well-performing queries: <positive query list>

Poorly performing queries: <negative query list>

Your task:

1) Contrast

the two groups of queries.

2) Examine

the actions focusing on key query characteristics.

3) Identify systematic discrepancies

in the actions.

Some queries with straightforward product features yielded good

results, while those with specific product descriptions and more

implicit brand mention underperformed.

This suggests that the actions may not effectively capture and

utilize details related to specific features.

<Suggested Improvement>

I suggest to better parse and utilize query attributes,

including type, and any mentioned products.

Moreover, actions should employ better tools for accurate

brand matching and more sophisticated scoring that

leverages both textual and relational info.

Action 1:

attribute

ParseAttributeFromQuery

[query, (brand,

features, type, related_products

)]

.

.

.

Action

5:

brand_score

CheckRequirementByLLM

[attribute.brand,

GetEntityDocuments

()]

Action

6:

candidates

GetRelatedProducts

[attribute.related_products]

Updated Instructions (Partial)

Task for Comparator

Actor

Comparator

Actions

Iteration

𝒕

Iteration

(𝒕 + 𝟏)

Generate

Updated

Instructions

Prompt

Generate

Input

4.2 Automate Holistic Instruction Generation with Comparator

To address these challenges, we construct a comparator LLM to update the instructions for the actor. Instead of optimizing on a sampled instance, comparator aims to identify systematic flaws throughout the structured actions/solutions.

Step 1: Constructing positive and negative queries . To achieve this goal, as shown in Figure 1, the comparator samples a set of data (question-answer pairs), evaluates the current action sequence on the queries, and categorizes them into well-performing (positive) and poorly-performing (negative) groups based on their performance. Specifically, we define two thresholds, ℓ and h (where 0 < h ≤ ℓ < 1 ), which serve as the upper and lower bounds for constructing positive and negative queries, respectively. Queries with an evaluation metric (e.g., Recall) value above ℓ are classified as positive, while those below h are classified as negative. Based on the training dynamics, one could consider adapting the lower bound to ensure a sufficient number of negative samples for selection. After classification, we use random sampling to create a mini-batch of b queries, with an equal split of positive and negative queries ( b/ 2 each) for contrastive reasoning.

Step 2: Generating instructions through contrastive reasoning . After this, the comparator is tasked with contrasting the two groups of queries based on their key characteristics, attributing the performance gap to specific tool usage within the complex solution, and finally suggesting general modifications that can improve overall task performance. The instructions generated by the comparator are then appended to the initial prompts to update the actor.

Insights/Justification for the comparator . To illustrate the insights, we draw an analogy from deep neural network training, where extremely small batch sizes can introduce significant noise in gradient estimates and high variance in model updates. By adopting a batched training strategy and sampling positive and negative queries as two 'mini-batches,' comparator can extract a robust 'gradient' to update the actor. This approach encourages comparator to generate more general and comprehensive instructions on the complex action sequence, including problem decomposition, solutions to subproblems, and the final synthesis. Moreover, as contrastive reasoning directly targets disentangling the performance gap related to input patterns and how they are handled differently by the tools, it is particularly effective in helping comparator differentiate and select tools for use. Finally, by identifying systemic flaws across a wide array of negative queries, comparator generates modifications that are not only tailored to individual samples but also to diverse data samples, enhancing generalization to novel cases.

Demonstration example . Figure 3 illustrates an example where comparator contrasts the patterns of positive and negative queries, identifying discrepancies in tool usage within the action sequence. It reveals that, compared to positive queries, negative queries feature more complex product descriptions, more subtle brand mentions, and additional relevant product mentions. These observations suggest: 1) an incomplete problem decomposition involving query attributes like detailed product features, 2) a potentially imprecise brand match using embedding similarity, and 3) a lack of consideration for related products in the results. Informed by these insights, actor updates its action sequence to address these subproblems and use the tools more effectively for the task, such as replacing the embedding tool with an LLM verification tool.

4.3 Logistic Instructions and Memory Construction

Logistic instructions . While instructions from the comparator are designed to improve task performance, we incorporate two types of orthogonal instructions to ensure the actions are valid and can be executed efficiently.

Validity check : This instruction is triggered internally during the execution of each action. It ensures the validity of the actor's actions, such as verifying the correct use of function calls.

Timeout error : To prevent inefficient action sequences that may stall the actor, we implement a timeout mechanism that triggers an error if processing exceeds a specified threshold. This error prompts the actor to adopt more efficient strategies, such as eliminating redundant operations.

Memory Bank . During optimization, we utilize a memory bank inspired by human decision-making processes, following Shinn et al. [41], where humans typically address current problems by analyzing the current situation and referencing past experiences. The memory bank stores tuples of action sequences, instructions from comparator, and the performance of these action sequences on a small training set (sampled from positive and negative queries). To manage the context size input to actor, we retain only the top5 action sequences with the best performance. This memory bank enables actor to learn from both immediate instructions and historical results.

Deployment . At deployment, we can apply the optimized instructions or, as shown in Figure 1, the optimized actor /action sequence, which includes effective tool utilization and problem-solving strategies, to answer queries or retrieve entities. In the experiments, we demonstrate AVATAR's flexibility by applying different deployment strategies.

5 Experiments

Tasks and Evaluation . We conduct experiments on the following datasets:

Four challenging retrieval datasets from STARK [49] and FLICKR30K-ENTITIES [35] to demonstrate AVATAR in handling complex real-world tasks ( cf. details in Appendix A). For each query in the retrieval datasets, the task is to retrieve relevant entities, such as nodes in a knowledge graph or images in knowledge bases. During deployment, we directly apply the optimized action sequence to the test queries. We assess task performance by comparing the consistency of the results with the ground truth answers in the datasets, using Hit@1, Hit@5, Recall@20, and Mean Reciprocal Rank (MRR) as the metrics.

Three question-answering (QA) benchmarks: HotpotQA [53], ArxivQA [22], ToolQA [67] , where the task is to provide natural language answers to the questions. We sample 100, 100, and 40 training queries, and 100, 100, and 60 testing queries for the three benchmarks, respectively. During deployment, the actor LLM uses optimized instructions to generate the action sequence for obtaining the answer. We use exact match (EM) score on HotpotQA, following previous methods. For ArxivQA and ToolQA, we use the LLM judge score for more reliable evaluation.

Baselines . For the knowledge retrieval tasks, we employ several embedding-based retriever models for our evaluation, following Wu et al. [49]: Dense Passage Retriever (DPR) Karpukhin et al. [17]; Vector Similarity Search methods ada-002 and multi-ada-002 using text-embedding-ada-002 from OpenAI; and a relation-aware model, QAGNN [57], for the STARK benchmark. Additionally, we include four prevailing agent frameworks to further enrich our evaluation:

ReAct [55] conducts reasoning and action in an in-context and interleaved manner to enable LLMs to interactively analyze observed information and perform actions.

Reflexion [41] uses self-reflection on the current task completion and stores these reflections in an episodic memory buffer to enhance decision-making in subsequent trials.

ExpeL [61] extracts insights from successful and failed action sequences, retrieving and including them in the context during inference. We apply ExpeL on the QA datasets and, due to its high cost on large-scale retrieval tasks, compare it with AVATAR on a sampled STARK-MAG test set.

Retroformer [56] reinforces LLM agents and automatically tunes their prompts by learning a retrospective model through policy gradient. We compare the performance of AVATAR with the reported result by Retroformer on HotpotQA due to the additional training involved.

We include an ablation model, AVATAR-C, which removes the comparator from our optimization pipeline. This comparison aims to validate the effectiveness of the comparator. The LLM version information is provided in Appendix B.

Function library . For the knowledge retrieval tasks, our function library consists of twenty-eight functions that facilitate access to, operation on, and reasoning over the knowledge information by LLM agents. For the QA tasks, we provide web search tools such as Google and Arxiv search APIs. See Appendix E for details. We used the same function library across all agent methods.

General pipeline . For AVATAR, we optimize the agent for a fixed number of epochs and select the action sequence or instruction with the highest performance. We use the same initial prompt structure, the metric Recall@20 or Accuracy for constructing positive and negative queries, and hyperparameters ( ℓ = h = 0 . 5 , b = 20 ) for all datasets.

5.1 Textual and Relational Retrieval Tasks

We employ the AMAZON, MAG, and PRIME datasets from the STARK benchmark [49], a large-scale semi-structured retrieval benchmark that integrates textual and relational knowledge ( cf. detailed description in Appendix A). Here, the entities to be retrieved are defined as nodes in a graph structure, with knowledge associated with each entity including both textual descriptions and relational data. We use the official splits from the STARK benchmark.

Takeaway 1: AVATAR outperforms state-of-the-art models . Table 3 shows that AVATAR substantially outperforms leading models such as Reflexion across all metrics on the STARK benchmark. Notably, the average improvement of AVATAR is 15.6% on Hit@1 and 9.5% on MRR. ReAct agents, however, cannot optimize based on instructions for improved tool usage and tend to select tools based on the LLM's prior knowledge, which may not be optimal for the given task. We observe that ReAct agents apply similar tools across various queries and struggle to explore alternative tool usage even with extensive in-context reasoning. Results for agent methods using GPT-4 Turbo are provided in Appendix B, showing similar conclusions. For comparison with ExpeL, the results in Table 6 show that it performs similarly to ReAct, underperforming AVATARby a large margin.

Takeaway 2: Comparator greatly impacts the actor's performance . The comparison of AVATAR with its ablation variant, AVATAR-C, highlights the significant advantages of the comparator module. Although AVATAR-C conducts validity and timeout checks, integrating Comparator into AVATAR adds a comprehensive instruction mechanism crucial for identifying clear directions to improve the agents, underlining comparator's key role in optimizing actor.

Amazon

MAG

Prime

0.75

Evaluation Metric

0.8

Evaluation Metric

0.40

0.70

Moving Average

0.7

Moving Average

0.35

Mean Reciprocal Rank

0.65

Mean Reciprocal Rank

0.6

Reciprocal Rank

0.30

0.60

0.5

0.25

0.55

0.50

0.4

Mean

0.20

0.45

0.3

0.15

0.40

0.2

0.10

Evaluation Metric

0.35

Moving Average

0

10

20

Iteration

30

40

50

0

10

20

Iteration

30

40

50

0

5

10

15

Iteration

20

25

30

35

0.6

Evaluation Metric

Moving Average

MeanReciprocal Rank

0.5

0.4

0.3

0.2

0

10

15

20

Iteration

Takeaway 3: AVATAR effectively improves agents during optimization . Figure 4 illustrates the agents' performance on the validation set during optimization. Impressively, AVATAR agents show significant performance improvements, e.g., from 35% to 75% on AMAZON and from 20% to 78% on MAG. This evidence strongly supports the effectiveness of the instructions generated by our comparator. Additionally, our memory bank, which stores past best-performing actions, encourages AVATAR agents to gradually converge by the end of the optimization process.

Takeaway 4: AVATAR can generalize to real-world tasks . Comparator generates instructions tailored to groups of retrieval queries, promoting generalizable modifications for novel queries. We validate this capability by applying optimized actions to human-generated leave-out queries from the STARK benchmark, which differ notably from the training data used to optimize our agents. Results in Table 5 (Appendix B) show that AVATAR significantly outperforms other models, achieving an average improvement of 20.9% on Hit@1. Further, in another study of Appendix B, we assess AVATAR's robustness to hyperparameters h and ℓ , showing that it maintains stable performance and generalization across different parameter values.

5.2 Image Retrieval Task

We further experiment on FLICKR30K ENTITIES [35], an image retrieval dataset of 30k images with annotated bounding boxes and descriptive phrases (Appendix A). In Table 2, AVATAR again shows significant improvements. In contrast, Reflexion agents struggle with 'overfitting,' where they are easily misled by specific image data, leading to inappropriate actions (e.g., trying to 'extract the color of a hat' from images without hats). AVATAR effectively avoids such pitfalls through batch-wise contrastive reasoning, which provides a broader perspective.

Takeaway 5: AVATAR generates impressive and generalizable actions . In Figure 5 (left), the final actions of the AVATAR agent detailed in Figure 8 (Appendix B), achieve advanced performance. Notably, AVATAR skillfully manages input queries and leverages Inverse Document Frequency (IDF) scores to refine phrase matching, ultimately synthesizing accurate answers. Beyond using existing tools, AVATAR agents can develop high-level tools, such as IDF-based reweighting, suggesting a promising direction for dynamic tool libraries and enhanced tool generation.

Takeaway 6: Emerging behaviors during optimization . In Figure 6, we present concrete cases illustrating key interactions between actor and comparator. In each instance, comparator identifies critical flaws, including information omission, ineffective tool usage, and suboptimal synthesis of varying scores. The instructions subsequently prompt actor to enhance retrieval strategies, tool selection, and precise score combinations. Furthermore, frequent references to tool usage underscore comparator's focused examination of tool utilization during optimization.

Improved Divide and Conquer

(4/25)

Sensible Tool Differentiation

(16/25)

Numerical Parameter Learning

(5/25)

Actions performs well on

queries mentioning objects,

such as animals and people (e.g. dog, man, woman)

But they struggles with

more abstract or scenery

focused queries (e.g. people relaxing on grass, person

jumping near a car).

Emphasize on the visual attributes by increasing the

visual_weight

. This will help capture the overall scene

elements better.

parameters = {phrase_weight: 0.5, visual_weight: 0.5}

WeightedSum

(phrase_scores, visual_scores, parameters)

parameters = {phrase_weight:

0.4,

visual_weight:

0.6

}

WeightedSum

(phrase_scores, visual_scores, parameters)

Actions perform well on

queries that more often

mention noun concepts like 'woman', 'man'.

The

queries with poorer results often

cover broader set

of other concepts, e.g., actions (ride, cook), and

attributes (blue, light).

Consider separation extraction on adjectives /

compound nouns / verb phrases to extract more

informative concepts for negative queries.

Added actions (summary)

1) Identify the following query patterns separately

NP (Basic Noun Phrase) : {<DT|PP\$>?<JJ>*<NN.*>+}

CP (Comparative Phrase) : {<JJ>?<NN.*>+<IN><JJ>?}

2) Correspond them with the image  attributes precisely

Action 1:

VqaByLLM

(GetImages(),

'

What is the location?

'

)

Result 1:

location

Long description about the location

Action 2:

StringMatch

(location, query_attribute.location)

Action 1: GetVisualAttributesByLLM

(GetImages(), 'location')

Result 1:

location

Concise location attribute

Action 2:

TokenMatchScore

(location, query_attribute.location)

Actions performs well on

queries without

requesting images from specific locations,

while fail on

queries that ask for locations such as

'square', 'street', 'stadium'.

This suggests issues with extracting image locations,

possibly due to VqaByLLM's lengthy outputs and

StringMatch's overly strict criteria.

5.3 Question Answering Tasks

Finally, we applied AVATAR to three widely used QA benchmarks. For ToolQA, we tested AVATAR and the baselines on two different domains: SciREX, which focuses on extracting information from full-length machine learning papers, and Agenda, which involves personal agenda-related questions. Both datasets have easy and hard versions.

Takeaway 7: AVATAR outperforms on QA tasks by offering better context understanding . Table 3 shows that AVATAR consistently outperforms state-of-the-art methods across all three QA datasets, with especially strong results on TOOLQA. In SCIREX-HARD, which focuses on extracting complex information from long scientific papers, AVATAR shows a 33.1% improvement, while in AGENDA-HARD, it achieves a 25.0% relative gain. These improvements are attributed to AVATAR's ability to generate optimized prompts that help the agent better understand the broader patterns and contexts of the questions, leading to more accurate answers and improved generalization across question types, from simple to complex.

6 Conclusion and Future Work

In this study, we introduce AVATAR, a novel framework that automates the optimization of LLM agents for enhanced tool utilization in multi-step problems, focusing on complex retrieval and QA tasks. AVATAR demonstrates remarkable improvements across seven diverse datasets. This success can largely be attributed to the comparator module, which effectively refines agent performance through the iterative generation of holistic and strategic prompts. A key innovation of comparator is its use of contrastive reasoning with batch-wise sampling, enabling it to identify systemic flaws and extract robust 'gradients' for comprehensive agent improvement across diverse scenarios. While we observe substantial progress from AVATAR, we discuss its limitations in Appendix D regarding its scalability etc. Future work can explore extending this methodology to other challenging agent tasks, visual reasoning tasks, and more dynamic environments, or designing better memory banks for dynamically storing knowledge and experience from past training.

REFERENCES

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. [n. d.]. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. ([n. d.]). arXiv:2308.09687

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving Language Models by Retrieving from Trillions of Tokens. In ICML.

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. FireAct: Toward Language Agent Fine-tuning. arXiv:2310.05915

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. (2023).

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In EMNLP. ACL.

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. [n. d.]. Agent AI: Surveying the Horizons of Multimodal Interaction. ([n. d.]). arXiv:2401.03568

Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. 2024. AutoGuide: Automated Generation and Selection of StateAware Guidelines for Large Language Model Agents. CoRR abs/2403.08978 (2024). arXiv:2403.08978

Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, and Jianfeng Gao. [n. d.]. MindAgent: Emergent Gaming Interaction. ([n. d.]). arXiv:2309.09971

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In ICML. PMLR.

Simon Jerome Han, Keith J. Ransom, Andrew Perfors, and Charles Kemp. 2024. Inductive reasoning in humans and large language models. Cogn. Syst. Res. (2024).

Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. (2024). arXiv:2402.07630

Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023. Large language models are zero-shot rankers for recommender systems. arXiv:2305.08845 (2023).

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In ICML.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2022. Inner Monologue: Embodied Reasoning through Planning with Language Models. In CoRL.

Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. ICLR (2024).

Vassilis N Ioannidis, Xiang Song, Da Zheng, Houyu Zhang, Jun Ma, Yi Xu, Belinda Zeng, Trishul Chilimbi, and George Karypis. 2022. Efficient and effective training of language and graph neural network models. arXiv preprint arXiv:2206.10781 (2022).

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In EMNLP.

Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2023. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv:2212.14024 [cs.CL]

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In NeurIPS.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 [cs.CL]

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. abs/2303.17760 (2023). arXiv:2303.17760

Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large VisionLanguage Models. In ACL.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. In EMNLP. Association for Computational Linguistics.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. AgentBench: Evaluating LLMs as Agents. (2023). arXiv:2308.03688

Robert Lo, Abishek Sridhar, Frank Xu, Hao Zhu, and Shuyan Zhou. 2023. Hierarchical Prompting Assists Large Language Model on Web Navigation. In Findings of the Association for Computational Linguistics: EMNLP 2023.

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. In NeurIPS.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. In NeurIPS.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. ArXiv (2021).

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. [n. d.]. WebGPT: Browser-assisted question-answering with human feedback. ([n. d.]). arXiv:2112.09332

Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. 2023. MemGPT: Towards LLMs as Operating Systems. 2310.08560 (2023).

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014 (2023).

Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: Tool Augmented Language Models. arXiv:2205.12255

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. CoRR (2023). arXiv:2305.15334

Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arxiv 2302.12813 (2023).

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. Int. J. Comput. Vis. (2017).

Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. In Proceedings of ACL 2023. Association for Computational Linguistics.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arxiv 2307.16789 (2023).

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. In NeurIPS.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. In NeurIPS.

Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, and May D Wang. 2024. EHRAgent: Code Empowers Large Language Models for Complex Tabular Reasoning on Electronic Health Records. arXiv:2401.07128 (2024).

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS.

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. [n. d.]. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In EMNLP, year = 2023.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. (2023). arXiv:2306.05301

Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. 2024. Hypothesis Search: Inductive Reasoning with Language Models. ICLR (2024).

Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P. Xing, and Zhiting Hu. 2023. PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization. (2023). arXiv:2310.16427

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. (2023). arXiv:2308.08155

Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. 2024. STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases. (2024). arXiv:2404.13207

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. (2024).

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction. In NeurIPS, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. EMNLP (2018).

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In EMNLP.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NeurIPS.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR.

Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. 2024. Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization. (2024).

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering.

Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios. arXiv:2401.00741 (2024).

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. [n. d.]. AgentTuning: Enabling Generalized Agent Abilities for LLMs. ([n. d.]). arXiv:2310.12823

Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. 2024. Training Language Model Agents without Modifying Language Models. (2024). arXiv:2402.11359

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. ExpeL: LLM Agents Are Experiential Learners. In AAAI.

Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023. Outline, then details: Syntactically guided coarse-to-fine code generation. ICML (2023).

Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. MemoryBank: Enhancing Large Language Models with Long-Term Memory. In AAAI.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models are Human-Level Prompt Engineers. In ICLR.

Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv:2308.07107 (2023).

Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. In NeurIPS.

Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. arXiv:2306.13304

Chang Zong, Yuchen Yan, Weiming Lu, Eliot Huang, Jian Shao, and Yueting Zhuang. [n. d.]. Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering. ([n. d.]). arXiv:2402.14320

A Retrieval Tasks

STARK . On the STARK benchmark, we are given a relation-text knowledge base, based on a knowledge graph G = ( V, E ) and a collection of free-text documents D . We represent the relationtext knowledge base of size n as E = { ( v i , d i , g i ) } n i =1 , where v i ∈ V represents a node on the knowledge graph, d i ∈ D is the text document related to the node, and g i is the connected component of G containing v i .

The query set Q in STARK is derived from both G and D , where each q i ∈ Q contains requirements based on d i and g i . The answer set A i , which includes v i , is a set of nodes satisfying both relational and textual requirements. The task on STARK is defined as follows: Given the knowledge base E consisting of relational and textual information, and a text query q i , the output is a set of nodes A i such that ∀ a i ∈ A i , a i satisfies the relational requirements in the knowledge graph and textual requirements in its text documents.

FLICKR30K ENTITIES . On the FLICKR30K ENTITIES dataset, we are given an image-text knowledge base. We denote an image-text knowledge base of size n as E = { ( v i , q i , T i ) } n i =1 . Sample i consists of an image v i , its descriptive caption q i , and entity bounding box information T i . Specifically, T i = { ( c ij , p ij ) } b i j =1 , where b i represents the number of bounding boxes annotated in image i , c ij is the coordinate of the j -th bounding box, and p ij describes the entity in the corresponding bounding box.

In our task, the image captions serve as the text query; therefore, all q i in the dataset are not accessible to the agent to prevent information leakage. However, the agent can access v i and T i to fully utilize the vision and language information. The task on FLICKR30K ENTITIES is defined as follows: Given the knowledge base E with images and bounding box information, and a text query q i , the output is an image v i that satisfies the visual requirements in the image and textual requirements in the corresponding bounding boxes T i .

Orange hat

Glasses

N/A

(masked)

An orange hat

Rainbow flags

A group of youth

City

street

Wedding attire

A bride

Wedding

cake

A table

N/A

(masked)

B Experiment Details and Additional Results

B.1 Experiment Setup

LLMversions for agent methods .

For the knowledge retrieval tasks, we use claude-3-opus as the backbone LLM in the main paper by default, and report results using gpt-4-turbo in Appendix B due to space limitations.

For the QA tasks, we use gpt-4 for HotpotQA for fair comparison with previous methods and gpt-4o for the other two QA datasets.

B.2 Additional Experimental Results

(1) AVATAR results on STARK using GPT-4 Turbo (0125) as LLM backbone . In Table 4, we provide the results on STARK using GPT-4 Turbo (0125) as the backbone LLM.

(2) AVATAR results on STARK's human-generated splits . In Table 5, we demonstrate AVATAR's ability to generalize to test queries with distributions different from the question-answering pairs used to optimize the actor agents.

(3) AVATAR results and comparison with ExpeL on STARK-MAG subset . In Table 6, AVATAR demonstrates consistently higher performance than ExpeL across most metrics, notably achieving the highest Hit@1 and MRR scores. While ExpeL performs well in Recall@20, AVATAR 's overall improvements highlight its superior capability in precise retrieval tasks and tool-assisted knowledge retrieval.

(4) Final action sequence by AVATAR on FLICKR30K-ENTITIES . In Figure 8, we present the final actions optimized by AVATAR on FLICKR30K-ENTITIES.

(5) Sensitivity of AVATAR to upper and lower bounds . We evaluated various combinations of ℓ and h , focusing on the STARK-AMAZON dataset due to computational constraints. Table 7 presents the Hit@1 results for different ℓ and h values.

Table 7: Hit@1 results for different combinations of ℓ and h values on the STARK-AMAZON dataset.

Key Observations

The framework exhibits robustness to variations in ℓ and h , with Hit@1 fluctuations limited to a range of 2.7%.

A performance decline is observed when the gap between ℓ and h becomes too large, potentially due to the exclusion of certain training queries that fall within the ( h, ℓ ) interval.

A moderate gap between ℓ and h leads to slight performance improvements, suggesting that a balanced separation between positive and negative queries can enhance pattern differentiation without compromising the number of training queries.

The results indicate that setting ℓ = 0 . 6 and h = 0 . 5 yields an improved Hit@1 score compared to the baseline reported in the original paper. Overall, this analysis underscores the robustness of the framework, which relies on a minimal set of hyperparameters, including ℓ , h , batch size b , and training epochs.

C Prompts

We keep only two prompt templates for our framework on all tasks: (1) The prompt template given to actor as initially instructions, and (2) the prompt template given to the comparator to conduct contrastive reasoning and generate the instructions for the actor. Below are the complete templates:

This is the prompt given to actor as initially instructions:

This is the prompt given to comparator to generate the instructions for the actor:

D Limitations

We identify several potential limitations of our work:

Scalability : AvaTaR is designed to scale with large language models (LLMs) that support extended context lengths (up to 128k tokens), enabling it to handle numerous tools and complex tasks. However, increased latency and other practical limitations may hinder performance in scenarios requiring hundreds of tools or high complexity. Future research could focus on incorporating specialized, tool-augmented LLMs as auxiliary agents to facilitate smoother scaling.

Computation Requirements : Managing longer contexts and multiple tool interactions within AvaTaR increases computational demands, which can significantly raise operational costs. These requirements necessitate substantial resources to maintain efficient performance, particularly when scaling to larger datasets or more intricate tasks.

Potential Failure Modes : Although AvaTaR performs well on known queries, its performance may diminish when faced with queries that require new or unfamiliar combinations of tools. This limitation could be mitigated by integrating adaptive learning techniques and continuous monitoring, which would allow AvaTaR to better handle novel tool requirements.

E Function library

E.1 Complex Retrieval Tasks

Please refer to Table 8 and Table 9 for the detailed functions.

E.2 General QA Tasks

For general QA tasks, we use the following tools:

WEB_SEARCH : A general-purpose tool that performs web searches to answer questions. Useful for retrieving up-to-date information from the internet when other sources are unavailable.

ARXIV_SEARCH : This tool retrieves information about academic papers from Arxiv using a paper's unique ID. This function call can provide metadata and other details for academic references.

Wiki_SEARCH : If you have a question or name to lookup, this tool uses a Wikipedia search to retrieve relevant information.

RETRIEVE_FROM_DB : This tool is used to retrieve relevant information from a database. This is only available on ToolQA.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We design a novel and automatic framework that optimizes an LLM agent to effectively use the provided tools and make comprehensive analysis on the evolution of our key modules.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We did extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We elaborate the experiment details in the Experiment section including datasets, baselines, function libraries etc. We also release all the prompts we are using in the experiments for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code and data are accessible at https://anonymous.4open.science/ r/AvaTaR-FBC4/ .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include dataset information and training details in the Experiment part and Appendix. We also clearly describe the knowledge base and formally introduce the task settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We run our experiments on a single NVIDIA A100-SXM4-80GB GPU and 32-core CPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not induce any potential research harm mentioned in NeurIPS Code of Ethics in our paper.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the impact in the introduction section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method provides a framework to better use LM but not releasing a LM.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Creators or original owners of assets mentioned in the paper are properly cited and the license and terms of use are respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework, AVATAR, that allows LLM agents to optimize the performance of Knowledge Retrieval. In the framework, a Comparator agent is adopted to extract insight from the positive and negative samples. The experiments on four retrieval datasets show the effectiveness of the method.\n\nSTRENGTHS:\n1. The schematics in the paper are well drawn and demonstrate the proposed method well.\n2. The experimental settings and prompts are detailed.\n3. The experiments are performed on STARK and Flickr30K Entities benchmarks to show the effectiveness of the methods.\n\nWEAKNESSES:\nThe main weakness is the lack of innovation and comparison compared with cutting-edge works.\n\n1, This proposed AVATAR is very similar to ExpeL [1] in terms of method and has no obvious innovation in comparison. ExpeL adopts a Reflexion agent (actor) to gather success and failure experiences of multi-step tasks, which is the same as the actor's role in this paper. ExpeL adopts another agent to compare a failed trajectory with a successful trajectory for the same task and extract insights from the comparison, which is the same as the instruction generation process of the comparator in this paper. ExpeL adopts the experience pool to retrieve successful trajectories, which is the same as the Memory Bank in this paper. Similarly, Autoguide [2] also extracts insights from experiences. Therefore, the method of auto-optimizing the instruction by comparing the successful and failed trajectories has no innovation compared with Expel [1] and Autoguide [2]. The author should conduct more surveys and read relevant cutting-edge works.\n\n2, The paper claims their method achieves SOTA on STARK and FLICKR30K-ENTITIES benchmarks, but they only compared with relatively basic methods. The author employs several outdated embedding-based retrievers as the comparison. However, text-to-image retrieval on FLICKR30K-ENTITIES has long been dominated by the vision-language model [3] [4]. Internvl [3] and Beit [4] achieve high recall@1 (from 80 to 90) on this benchmark. Therefore, it is unreasonable to claim that the proposed method achieves the retrieval SOTA on these two benchmarks.\n\n3, The results of the comparison on FLICKR30K-ENTITIES are missing. Quantitative results on FLICKR30K-ENTITIES are only shown in Figure 5 (right), but the results of all methods on this benchmark are not found in the paper. Considering that STaRK is a relatively new benchmark, not many methods have experimented on it, so results and analysis on recognized benchmarks are important.\n\n4, Many sentences in the paper are too obscure to understand and lack clear explanations. For example, \"these per-sample instructions tend to be narrow and fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary with others, leading to decreased performance when applied across different scenarios.\" in lines 152-154, what do \"per-sample instructions\" mean, and why will these tool combinations \"lead to decreased performance\"? Providing some simple examples can make it easier to read.\n\n[1] Zhao, Andrew, et al. \"Expel: Llm agents are experiential learners.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.\n\n[2] Fu, Yao, et al. \"Autoguide: Automated generation and selection of state-aware guidelines for large language model agents.\" arXiv preprint arXiv:2403.08978 (2024).\n\n[3] Chen, Zhe, et al. \"Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wang, Wenhui, et al. \"Image as a foreign language: Beit pretraining for vision and vision-language tasks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\nIn line 160, \"the comparator samples a group of data samples (question-answer pairs), executes the current actions for each question,\" but how can the comparator execute the current actions? What are the current actions for the comparator?\n\nThe text in Figure 3 is too small to read and the meaning of the marked text is difficult to understand.\n\nLIMITATIONS:\nThere is no separate Limitation or Broader Impacts section in the paper. In addition, the authors said, \"We did an extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations,\" but the discussion of the limitations in this paper is required.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AVATAR, a novel framework for optimizing large language model (LLM) agents to effectively use provided tools and improve performance on complex multi-step tasks, with a focus on retrieval tasks. The key innovation is a comparator module that generates holistic instructions to improve the actor (main agent) through contrastive reasoning on batches of well-performing and poorly-performing queries.\nThe authors demonstrate AVATAR's effectiveness on four challenging retrieval datasets from the STARK and FLICKR30K-ENTITIES benchmarks. Results show significant improvements over state-of-the-art baselines like ReAct and Reflexion.\n\nSTRENGTHS:\nNovel approach: The comparator module using contrastive reasoning on query batches is an innovative way to generate holistic instructions for improving agent performance. This addresses limitations of per-sample instruction approaches.\n\nClear motivation and explanation: The paper draws insightful analogies to concepts like batched training and gradient computation in neural networks to explain the intuition behind the approach. This makes the core ideas easy to understand.\n\nStrong empirical results: AVATAR consistently outperforms strong baselines across multiple datasets, with significant improvements on key metrics. The ablation study clearly demonstrates the value of the comparator module.\n\nWEAKNESSES:\nLimited scope of evaluation: While the retrieval tasks are complex, it would be valuable to see AVATAR applied to a broader range of tool-use benchmarks to demonstrate generality.\n\nLack of comparison to finetuning: The paper does not discuss or compare to alternative approaches like directly finetuning the actor model via rejection sampling. It's unclear how AVATAR compares to such methods in terms of performance and efficiency.\n\nScalability considerations: The paper does not thoroughly address how the method scales with increasing numbers of tools or more complex task structures.\n\nQUESTIONS:\nRegarding the use of the memory bank as in-context learning examples, \nHow does AVATAR compare to directly finetuning the actor model in terms of performance, efficiency, and data requirements? What are the key advantages of the instruction-based approach?\nHow does the method handle scenarios where the action model improves significantly, making it difficult to find negative examples for the contrastive learning process? Is there a strategy for addressing this?\nHave you explored applying AVATAR to other types of tool-use tasks beyond retrieval? What challenges do you anticipate in extending to more diverse task types?\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations of their work. I recommend adding a dedicated limitations and broader impacts section to address these points more thoroughly. \n\nThis should include: Discussion of computational requirements and scalability limitations, potential failure modes or scenarios where the method may struggle", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework AVATAR for LLM agent that operates in two stages:\n-The first stage is optimization during the training process, which integrates the LLM comparator component into the AVATAR. The comparator summarizes holistic prompts from positive and negative queries and iteratively optimizes LLM actor using the prompts.\n-The second stage is deployment during the inference process, where participants respond without the involvement of the comparator.\n\nSTRENGTHS:\nDeveloping prompts for LLM agents is heuristic and laborious, but this paper proposes a new method called AVATAR, which automatically iteratively generates holistic prompts from positive and negative queries. Experimental results show that AVATAR outperforms SOTA methods.\nand some points summary as following:\n1. It is a novel framework, comparing with the current SOTA approach such as ReAct, the Comparator component design is reasonable that it generates prompts from positive and negative queries.\n2. It is new SOTA of LLM Agent. Experiments are conducted against current SOTA approaches as baseline, the framework outperform the current SOTA\n3. It is a common framework, and it is effective with both Claude and GPT4.\n\nWEAKNESSES:\n1. The optimization is data-driven, so comparator works effectively when the positive and negative queries are well sampled and balanced during iteration; when to stop iteration and how to sample the queries; it is not clearly clarified.\n2. Regarding the construction of positive and negative queries, it is necessary to conduct experiments on the lower and upper bounds(l&h) to select appropriate values for optimal performance.\n3. Regarding the memory bank, it is a important component to swap in and out according to data quality or balanced distribution,\nit is not well designed and discussed.\n\nQUESTIONS:\nIs \"AIR (gpt4)\" in Table 3 a typo?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}