AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning

Authors: shirley  wu
shiyu  zhao
qian  huang
kexin  huang
michihiro  yasunaga
kaidi  cao
vassilis n ioannidis
karthik  subbian
jure  leskovec
james  zou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:18:17

Abstract: Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. However, developing prompting techniques that enable LLM agents to effectively use these tools and knowledge remains a heuristic and labor-intensive task. Here, we introduce AvaTaR, a novel and automated framework that optimizes an LLM agent to effectively leverage provided tools, improving performance on a given task. During optimization, we design a comparator module to iteratively deliver insightful and comprehensive prompts to the LLM agent by contrastively reasoning between positive and negative examples sampled from training data. We demon- strate AvaTaR on four complex multimodal retrieval datasets featuring textual, visual, and relational information, and three general question-answering (QA) datasets. We find AvaTaR consistently outperforms state-of-the-art approaches across all seven tasks, exhibiting strong generalization ability when applied to novel cases and achieving an average relative improvement of 14% on the Hit@1 metric for the retrieval datasets and 13% for the QA datasets. Code and dataset are available at https://github.com/zou-group/avatar.

Main paper: 
 introduction: 
 Autonomous agents powered by large language models (LLMs) offer substantial promise for complex problem-solving [6, 39, 41, 55, 65] . These agents demonstrate remarkable capabilities in reasoning [46, 47, 54, 55] and planning [8, 13, 14, 62] . Additionally, their functionality is extended through the use of external tools that provide access to external or private data and specialized operations, such as APIs for interacting with knowledge bases and search engines. These tools enable agents to perform complex tasks like multi-step problem-solving and retrieving diverse information, which is essential for complex retrieval and question-answering (QA) [13, 21, 26, 33, 38, 40, 48] .Despite the promising capabilities of LLM agents, it remains challenging to engineer effective prompts that guide these agents through a multi-stage process for real-world problem-solving. This process involves (1) decomposing a complex question into an actionable plan with simpler steps, (2) strategically using provided tools to gather relevant information, and, finally, (3) synthesizing intermediate results to produce a coherent and accurate response. Each step requires extensive manual effort and numerous iterations of trial and error to refine the prompts.Current approaches have primarily focused on directly deploying agents using complex humandesigned "mega-prompts" [18, 24, 55] , which require lots of manual trial and error. Nevertheless, such hand-engineered mega-prompts may also result in brittle implementations with suboptimal accuracy (see Figure 2 (a) ), where the ReAct agent [55] easily produces trivial and misleading answers to customers' queries about specific products. Furthermore, existing research [4, 5, 45, 50, 56, 60, 64] on Correspondence: {shirwu, jure, jamesz}@cs.stanford.edu Then, the comparator contrasts a set of well-performing (positive) and poorly-performing (negative) queries, automatically generating holistic prompts to teach the actor more effective retrieval strategies and tool usage (cf. Section 4). (b) At deployment, the actor with optimized prompts or actions can be effectively used to answer new queries.employing LLMs as optimizers often fails to adequately refine the complex strategies for enhancing tool integration and usage. This lack of strategic optimization can lead to less effective, nongeneralizable agent applications in complex real-world scenarios.Present work: AVATAR. To address these challenges, we introduce AVATAR, an automated framework that optimizes agents for effective tool utilization and excellent task performance. Specifically, we leverage key insights from contrastive reasoning and build a comparator module ("trainer") to generate holistic instructions and prompts (i.e., , computing a robust "gradient") to optimize an actor LLM. We demonstrate our framework on challenging tasks of knowledge base retrieval, which involve complex multi-stage procedures and extensive tool usage, and general QA tasks. Specifically, AVATAR includes two phases:• Optimization phase. The core of our optimization framework (Figure 1 ) is a comparator LLM that automatically generates holistic prompts to teach a actor LLM to differentiate between effective and ineffective tool usage. The comparator takes positive and negative data samples, where the current agent performs well and poorly, respectively, to identify overall gaps and systematic errors exhibited by the agent. Unlike per-sample instructions, which can easily lead to overfitting on individual data points, by constructing multiple samples as a "batch," the comparator can extract a more robust "gradient" to "backpropagate" to the actor. In other words, the comparator can provide more effective and adaptive prompts through batch-wise contrastive reasoning, helping the agent identify flaws in solving challenging multi-stage problems. Following previous methods [30, 41, 56, 63] , we also maintain a memory bank with selected past instructions to prevent the actor LLM from repeating previous mistakes. • Deployment phase. After the optimization phase, the actor with best-performing prompts can be selected for the testing instances. Moreover, in complex retrieval tasks, the iterative optimization through our AVATAR framework updates the actor for more effective and generalizable action sequences, enabling direct generalization to novel user inquiries at deployment. In Figure 2 (b), the optimized actor creates three novel strategies: 1) precise decomposition of problems by extracting multifaceted attributes, 2) effective tool usage through a sophisticated and robust scoring system, and 3) the strategic combination of different scores, determined by learned coefficients, ensuring accurate and comprehensive retrieval.Experimental evaluation. We conduct extensive experiments on four retrieval datasets and three QA datasets. The retrieval tasks are highly complex, involving multimodal data, including textual, visual, and relational information. AVATAR consistently outperforms state-of-the-art methods, showing a substantial 14% improvement in the Hit@1 metric. Impressively, with only 25 iterations, AVATAR Agent Optimization. In the field of optimizing LLM agents, previous works have modified the parameters of LLM backbones through fine-tuning or instruction tuning to enhance agent capability [3, 15, 19, 23, 32, 33, 37, 43, 51, 58, 59] or generated better prompts through iterative prompt tuning [11, 18, 45, 50, 56] . Recently, Zhang et al. [60] conducted agent training by iteratively updating the agents' functions according to the execution history. However, these methods do not explicitly consider targeted optimization for tool usage or the impact on complex multi-stage tasks. Additionally, enhancing agents' generalization abilities [10, 31, 44] , essential for real-world applications, has received less attention. In our work, we focus on automatically generating holistic instructions via a novel contrastive reasoning mechanism, targeting effective tool usage and agents' generalization ability. Compared to fine-tuning approaches, AvaTaR offers advantages by requiring only a small subset of training data and tool descriptions, making it more adaptable and less computationally intensive. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We did extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
problem formulation: 
 Definition 1: Tools. We define tools or APIs as a set of implemented functions with specified input and output variables. We denote the abstract tool space as T = {f k :I f k → O f k | k = 1, 2, . . .},where f k maps the input I f k to the output O f k . For example, the tools can be APIs used for accessing external knowledge via a search index, an encoder model that generates vector representations from text or image data, or a task-specific classifier that outputs probabilities over a list of classes.Definition 2: Agents. An LLM agent, defined as A : P → α, is controlled by verbal prompts to generate a flow of actions needed to complete a task. Here α denotes the action sequence [α 1 , . . . , α L ],where each action is defined by a tuple (f ∈ T , i ∈ I f , o ∈ O f ), consisting of a tool function, specified input(s), and a designated variable that receives the output(s). Each action in the sequence can leverage the outputs generated by previous actions, with the final action α L rendering the results for the task. Multi-step problem-solving. Real-world problems are inherently complex and cannot be effectively addressed through straightforward solutions or simple tool usage alone. Solving real-world problems with LLM agents can be structured into a multi-stage procedure:• Decomposition of the problem: The procedure begins by breaking down a complex question into an actionable plan characterized by simpler steps. This decomposition is crucial for setting clear objectives and facilitating focused problem-solving. • Tool-assisted subproblem solving: In the subsequent phase, agents strategically utilize tools from the established tool space T to gather solutions for each step. This stage is essential for acquiring Table 1 : Key differences between AVATAR and prevailing agent methods. AVATAR demonstrates the ability to: 1) self-improve on specific tasks, 2) retain memory throughout the optimization process, 3) enhance the agent's generalization capability, and 4) autonomously generate holistic, high-quality prompts for better tool usage. Please refer to Section 4 for details.Self-Improvement Memory Generalization Holistic Prompt Generation (on Tool Usage)ReAct [55] ✗ ✗ ✗ ✗ Self-refine [27] ✔ ✗ ✗ ✗ Reflexion [41] ✔ ✔ ✗ ✗ AVATAR (Ours) ✔ ✔ ✔ ✔the necessary information required to effectively address each subproblem of the decomposed problem.• Synthesis and response formulation: The final stage involves synthesizing the intermediate results to construct a precise response. This synthesis not only combines the data but may also refine the response through trials and adjustments, ensuring the solution's accuracy and relevance.For example, retrieval tasks are inherently complex and demanding. Given a user query q, retrieval tasks aim to identify or generate a ranked list of relevant entities E from the entity space of a knowledge base. Each query is associated with a set of ground truth answers, denoted as Y , which are used to compute the quality of the prediction. Specifically, the LLM agent is required to 1) comprehend a user's request, 2) utilize the provided tools to identify and analyze relevant information in the large knowledge space, which may contain multimodal data sources, and finally, 3) integrate all gathered information to reason and generate an accurate response.4 Our Method: Optimizing Agents for Tool-Assisted Multi-Step TasksEach step in the multi-stage problem-solving process (described in Section 3) requires effective prompts to identify key flaws and improve task performance. However, refining the agents' prompts demands extensive manual effort and numerous iterations of trial and error.To address this, we introduce an automated and novel optimization framework, AVATAR, which generates prompts to improve agents' tool usage and task performance. In Table 1 , we highlight four critical aspects of our approach compared with prevailing agent frameworks [27, 41, 55] . Here, we introduce the two main LLM components in AVATAR: a actor LLM (Section 4.1) and a comparator LLM (Section 4.2). 
actor construction and challenges: 
 Actor. The actor agent, as defined in Section 3, is responsible for generating initial actions based on the initial instructions/prompts and adjusting actions according to updated instructions. Specifically, the initial instructions provide details about the task and available tools, where tools can be introduced in programming languages such as Python. During optimization, the prompts further incorporate the previous action sequence and updated instructions to adjust these actions. The actor then generates revised actions, which could include a combination of tool usage through programming language (code generation) along with natural language explanations of how the tools are employed.Challenges in multi-step complex tasks. A common approach to updating instructions utilizes execution results or performance data from a specific instance, often through techniques like selfexplanation [4, 27] or self-reflection [41, 56] . However, this approach may not be suitable for complex tasks involving tool usage. Complex multi-step tasks include multiple interacting factors that influence overall performance, such as problem decomposition and tool selection. Consequently, instructions generated for a failed/negative query instance tend to be narrow in scope and may fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary across different scenarios, potentially leading to decreased performance when applied to varied cases. Actions "What are the affordable and quality car emblems from the Football Fanatics brand? " "Where can I find Aminco silicone rubber bracelets that will match my NFL Tampa Bay Buccaneers wristbands?. " <Task & Tool description>, <Actions> Post-action execution, the queries exhibited varying performances.Well-performing queries: <positive query list> Poorly performing queries: <negative query list> Your task: 1) Contrast the two groups of queries.2) Examine the actions focusing on key query characteristics.3) Identify systematic discrepancies in the actions.<Contrastive Reasoning> Some queries with straightforward product features yielded good results, while those with specific product descriptions and more implicit brand mention underperformed. This suggests that the actions may not effectively capture and utilize details related to specific features.<Suggested Improvement> I suggest to better parse and utilize query attributes, including type, and any mentioned products. Moreover, actions should employ better tools for accurate brand matching and more sophisticated scoring that leverages both textual and relational info. Then comparator provides holistic instructions that guide the actor to improve query decomposition, utilize better tools, and incorporate more comprehensive information.To address these challenges, we construct a comparator LLM to update the instructions for the actor. Instead of optimizing on a sampled instance, comparator aims to identify systematic flaws throughout the structured actions/solutions.Step 1: Constructing positive and negative queries. To achieve this goal, as shown in Figure 1 , the comparator samples a set of data (question-answer pairs), evaluates the current action sequence on the queries, and categorizes them into well-performing (positive) and poorly-performing (negative) groups based on their performance. Specifically, we define two thresholds, ℓ and h (where 0 < h ≤ ℓ < 1), which serve as the upper and lower bounds for constructing positive and negative queries, respectively. Queries with an evaluation metric (e.g., Recall) value above ℓ are classified as positive, while those below h are classified as negative. Based on the training dynamics, one could consider adapting the lower bound to ensure a sufficient number of negative samples for selection. After classification, we use random sampling to create a mini-batch of b queries, with an equal split of positive and negative queries (b/2 each) for contrastive reasoning.Step 2: Generating instructions through contrastive reasoning. After this, the comparator is tasked with contrasting the two groups of queries based on their key characteristics, attributing the performance gap to specific tool usage within the complex solution, and finally suggesting general modifications that can improve overall task performance. The instructions generated by the comparator are then appended to the initial prompts to update the actor.Insights/Justification for the comparator. To illustrate the insights, we draw an analogy from deep neural network training, where extremely small batch sizes can introduce significant noise in gradient estimates and high variance in model updates. By adopting a batched training strategy and sampling positive and negative queries as two "mini-batches," comparator can extract a robust "gradient" to update the actor. This approach encourages comparator to generate more general and comprehensive instructions on the complex action sequence, including problem decomposition, solutions to subproblems, and the final synthesis. Moreover, as contrastive reasoning directly targets disentangling the performance gap related to input patterns and how they are handled differently by the tools, it is particularly effective in helping comparator differentiate and select tools for use. Finally, by identifying systemic flaws across a wide array of negative queries, comparator generates modifications that are not only tailored to individual samples but also to diverse data samples, enhancing generalization to novel cases.Demonstration example. Figure 3 illustrates an example where comparator contrasts the patterns of positive and negative queries, identifying discrepancies in tool usage within the action sequence. It reveals that, compared to positive queries, negative queries feature more complex product descriptions, more subtle brand mentions, and additional relevant product mentions. These observations suggest: 1) an incomplete problem decomposition involving query attributes like detailed product features, 2) a potentially imprecise brand match using embedding similarity, and 3) a lack of consideration for related products in the results. Informed by these insights, actor updates its action sequence to address these subproblems and use the tools more effectively for the task, such as replacing the embedding tool with an LLM verification tool.Logistic instructions. While instructions from the comparator are designed to improve task performance, we incorporate two types of orthogonal instructions to ensure the actions are valid and can be executed efficiently.• Validity check: This instruction is triggered internally during the execution of each action. It ensures the validity of the actor's actions, such as verifying the correct use of function calls. • Timeout error: To prevent inefficient action sequences that may stall the actor, we implement a timeout mechanism that triggers an error if processing exceeds a specified threshold. This error prompts the actor to adopt more efficient strategies, such as eliminating redundant operations.Memory Bank. During optimization, we utilize a memory bank inspired by human decision-making processes, following Shinn et al. [41] , where humans typically address current problems by analyzing the current situation and referencing past experiences. The memory bank stores tuples of action sequences, instructions from comparator, and the performance of these action sequences on a small training set (sampled from positive and negative queries). To manage the context size input to actor, we retain only the top-5 action sequences with the best performance. This memory bank enables actor to learn from both immediate instructions and historical results.Deployment. At deployment, we can apply the optimized instructions or, as shown in Figure 1 , the optimized actor /action sequence, which includes effective tool utilization and problem-solving strategies, to answer queries or retrieve entities. In the experiments, we demonstrate AVATAR's flexibility by applying different deployment strategies. 
experiments: 
 Tasks and Evaluation. We conduct experiments on the following datasets:• Four challenging retrieval datasets from STARK [49] and FLICKR30K-ENTITIES [35] to demonstrate AVATAR in handling complex real-world tasks (cf. details in Appendix A). For each query in the retrieval datasets, the task is to retrieve relevant entities, such as nodes in a knowledge graph or images in knowledge bases. During deployment, we directly apply the optimized action sequence to the test queries. We assess task performance by comparing the consistency of the results with the ground truth answers in the datasets, using Hit@1, Hit@5, Recall@20, and Mean Reciprocal Rank (MRR) as the metrics. • Three question-answering (QA) benchmarks: HotpotQA [53] , ArxivQA [22] , ToolQA [67] ,where the task is to provide natural language answers to the questions. We sample 100, 100, and 40 training queries, and 100, 100, and 60 testing queries for the three benchmarks, respectively. During deployment, the actor LLM uses optimized instructions to generate the action sequence for obtaining the answer. We use exact match (EM) score on HotpotQA, following previous methods. For ArxivQA and ToolQA, we use the LLM judge score for more reliable evaluation.Baselines. For the knowledge retrieval tasks, we employ several embedding-based retriever models for our evaluation, following Wu et al. [49] : Dense Passage Retriever (DPR) Karpukhin et al. [17] ; Vector Similarity Search methods ada-002 and multi-ada-002 using text-embedding-ada-002 from OpenAI; and a relation-aware model, QAGNN [57] , for the STARK benchmark. Additionally, we include four prevailing agent frameworks to further enrich our evaluation:• ReAct [55] conducts reasoning and action in an in-context and interleaved manner to enable LLMs to interactively analyze observed information and perform actions.Table 2 : Retrieval performance (%) on STARK benchmark. Last row shows the relative improvements over the best metric value in each column.Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@ • Reflexion [41] uses self-reflection on the current task completion and stores these reflections in an episodic memory buffer to enhance decision-making in subsequent trials.• ExpeL [61] extracts insights from successful and failed action sequences, retrieving and including them in the context during inference. We apply ExpeL on the QA datasets and, due to its high cost on large-scale retrieval tasks, compare it with AVATAR on a sampled STARK-MAG test set.• Retroformer [56] reinforces LLM agents and automatically tunes their prompts by learning a retrospective model through policy gradient. We compare the performance of AVATAR with the reported result by Retroformer on HotpotQA due to the additional training involved.We include an ablation model, AVATAR-C, which removes the comparator from our optimization pipeline. This comparison aims to validate the effectiveness of the comparator. The LLM version information is provided in Appendix B.Function library. For the knowledge retrieval tasks, our function library consists of twenty-eight functions that facilitate access to, operation on, and reasoning over the knowledge information by LLM agents. For the QA tasks, we provide web search tools such as Google and Arxiv search APIs.See Appendix E for details. We used the same function library across all agent methods.General pipeline. For AVATAR, we optimize the agent for a fixed number of epochs and select the action sequence or instruction with the highest performance. We use the same initial prompt structure, the metric Recall@20 or Accuracy for constructing positive and negative queries, and hyperparameters (ℓ = h = 0.5, b = 20) for all datasets.We employ the AMAZON, MAG, and PRIME datasets from the STARK benchmark [49] , a large-scale semi-structured retrieval benchmark that integrates textual and relational knowledge (cf. detailed description in Appendix A). Here, the entities to be retrieved are defined as nodes in a graph structure, with knowledge associated with each entity including both textual descriptions and relational data. We use the official splits from the STARK benchmark.Takeaway 1: AVATAR outperforms state-of-the-art models. Table 3 shows that AVATAR substantially outperforms leading models such as Reflexion across all metrics on the STARK benchmark. Notably, the average improvement of AVATAR is 15.6% on Hit@1 and 9.5% on MRR. ReAct agents, however, cannot optimize based on instructions for improved tool usage and tend to select tools based on the LLM's prior knowledge, which may not be optimal for the given task. We observe that ReAct agents apply similar tools across various queries and struggle to explore alternative tool usage even with extensive in-context reasoning. Results for agent methods using GPT-4 Turbo are provided in Appendix B, showing similar conclusions. For comparison with ExpeL, the results in Table 6 show that it performs similarly to ReAct, underperforming AVATARby a large margin.Takeaway 2: Comparator greatly impacts the actor's performance. The comparison of AVATAR with its ablation variant, AVATAR-C, highlights the significant advantages of the comparator module.Although AVATAR-C conducts validity and timeout checks, integrating Comparator into AVATAR adds a comprehensive instruction mechanism crucial for identifying clear directions to improve the agents, underlining comparator's key role in optimizing actor. Hit@1 Hit@5 R@20 MRR Takeaway 3: AVATAR effectively improves agents during optimization. Figure 4 illustrates the agents' performance on the validation set during optimization. Impressively, AVATAR agents show significant performance improvements, e.g., from 35% to 75% on AMAZON and from 20% to 78% on MAG. This evidence strongly supports the effectiveness of the instructions generated by our comparator. Additionally, our memory bank, which stores past best-performing actions, encourages AVATAR agents to gradually converge by the end of the optimization process.Takeaway 4: AVATAR can generalize to real-world tasks. Comparator generates instructions tailored to groups of retrieval queries, promoting generalizable modifications for novel queries. We validate this capability by applying optimized actions to human-generated leave-out queries from the STARK benchmark, which differ notably from the training data used to optimize our agents. Results in Table 5 (Appendix B) show that AVATAR significantly outperforms other models, achieving an average improvement of 20.9% on Hit@1. Further, in another study of Appendix B, we assess AVATAR's robustness to hyperparameters h and ℓ, showing that it maintains stable performance and generalization across different parameter values.We further experiment on FLICKR30K ENTITIES [35] , an image retrieval dataset of 30k images with annotated bounding boxes and descriptive phrases (Appendix A). In Table 2 , AVATAR again shows significant improvements. In contrast, Reflexion agents struggle with "overfitting," where they are easily misled by specific image data, leading to inappropriate actions (e.g., trying to "extract the color of a hat" from images without hats). AVATAR effectively avoids such pitfalls through batch-wise contrastive reasoning, which provides a broader perspective.Takeaway 5: AVATAR generates impressive and generalizable actions. The final actions of the AVATAR agent, shown in Figure ?? (left) and detailed in Figure 8 (Appendix B), achieve advanced performance. Notably, AVATAR skillfully manages input queries and leverages Inverse Document Frequency (IDF) scores to refine phrase matching, ultimately synthesizing accurate answers. Beyond using existing tools, AVATAR agents can develop high-level tools, such as IDF-based reweighting, suggesting a promising direction for dynamic tool libraries and enhanced tool generation.Takeaway 6: Emerging Behaviors during Optimization. In Figure 6 , we present concrete cases illustrating key interactions between actor and comparator. In each instance, comparator identifies critical flaws, including information omission, ineffective tool usage, and suboptimal synthesis of varying scores. The instructions subsequently prompt actor to enhance retrieval strategies, tool selection, and precise score combinations. Furthermore, frequent references to tool usage underscore comparator's focused examination of tool utilization during optimization. Emphasize on the visual attributes by increasing the visual_weight. This will help capture the overall scene elements better. Consider separation extraction on adjectives / compound nouns / verb phrases to extract more informative concepts for negative queries.Added actions (summary) Actions performs well on queries without requesting images from specific locations, while fail on queries that ask for locations such as "square", "street", "stadium".This suggests issues with extracting image locations, possibly due to VqaByLLM's lengthy outputs and StringMatch's overly strict criteria. Finally, we applied AVATAR to three widely used QA benchmarks. For ToolQA, we tested AVATAR and the baselines on two different domains: SciREX, which focuses on extracting information from full-length machine learning papers, and Agenda, which involves personal agenda-related questions.Both datasets have easy and hard versions.Takeaway 7: AVATAR outperforms on QA tasks by offering better context understanding.Table 3 shows that AVATAR consistently outperforms state-of-the-art methods across all three QA datasets, with especially strong results on TOOLQA. In SCIREX-HARD, which focuses on extracting complex information from long scientific papers, AVATAR shows a 33.1% improvement, while in AGENDA-HARD, it achieves a 25.0% relative gain. These improvements are attributed to AVATAR's ability to generate optimized prompts that help the agent better understand the broader patterns and contexts of the questions, leading to more accurate answers and improved generalization across question types, from simple to complex. 
conclusion and future work: 
 In this study, we introduce AVATAR, a novel framework that automates the optimization of LLM agents for enhanced tool utilization in multi-step problems, focusing on complex retrieval and QA tasks. AVATAR demonstrates remarkable improvements across seven diverse datasets. This success can largely be attributed to the comparator module, which effectively refines agent performance through the iterative generation of holistic and strategic prompts. A key innovation of comparator is its use of contrastive reasoning with batch-wise sampling, enabling it to identify systemic flaws and extract robust "gradients" for comprehensive agent improvement across diverse scenarios. While we observe substantial progress from AVATAR, we discuss its limitations in Appendix D regarding its scalability etc.Future work can explore extending this methodology to other challenging agent tasks, visual reasoning tasks, and more dynamic environments, or designing better memory banks for dynamically storing knowledge and experience from past training.A Retrieval Tasks STARK. On the STARK benchmark, we are given a relation-text knowledge base, based on a knowledge graph G = (V, E) and a collection of free-text documents D. We represent the relationtext knowledge base of size n as E = {(v i , d i , g i )} n i=1 , where v i ∈ V represents a node on the knowledge graph, d i ∈ D is the text document related to the node, and g i is the connected component of G containing v i .The query set Q in STARK is derived from both G and D, where each q i ∈ Q contains requirements based on d i and g i . The answer set A i , which includes v i , is a set of nodes satisfying both relational and textual requirements. The task on STARK is defined as follows: Given the knowledge base E consisting of relational and textual information, and a text query q i , the output is a set of nodes A i such that ∀a i ∈ A i , a i satisfies the relational requirements in the knowledge graph and textual requirements in its text documents. FLICKR30K ENTITIES. On the FLICKR30K ENTITIES dataset, we are given an image-text knowledge base. We denote an image-text knowledge base of size n as E = {(v i , q i , T i )} n i=1 . Sample i consists of an image v i , its descriptive caption q i , and entity bounding box information T i . Specifically,T i = {(c ij , p ij )} bi j=1, where b i represents the number of bounding boxes annotated in image i, c ij is the coordinate of the j-th bounding box, and p ij describes the entity in the corresponding bounding box.In our task, the image captions serve as the text query; therefore, all q i in the dataset are not accessible to the agent to prevent information leakage. However, the agent can access v i and T i to fully utilize the vision and language information. The task on FLICKR30K ENTITIES is defined as follows: Given the knowledge base E with images and bounding box information, and a text query q i , the output is an image v i that satisfies the visual requirements in the image and textual requirements in the corresponding bounding boxes T i . Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We include dataset information and training details in the Experiment part and Appendix. We also clearly describe the knowledge base and formally introduce the task settings.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discussed the impact in the introduction section. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our method provides a framework to better use LM but not releasing a LM.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Creators or original owners of assets mentioned in the paper are properly cited and the license and terms of use are respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: LLM versions for agent methods.• For the knowledge retrieval tasks, we use claude-3-opus as the backbone LLM in the main paper by default, and report results using gpt-4-turbo in Appendix B due to space limitations.• For the QA tasks, we use gpt-4 for HotpotQA for fair comparison with previous methods and gpt-4o for the other two QA datasets.Table 4 : Retrieval performance (%) on STARK benchmark. Last row shows the relative improvements over the best metric value among the baselines.Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@20 MRR (1) AVATAR results on STARK using GPT-4 Turbo (0125) as LLM backbone. In Table 4 , we provide the results on STARK using GPT-4 Turbo (0125) as the backbone LLM.Table 5 : Retrieval performance (%) on the leave-out sets of human-generated queries in STARK.Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@20 MRR Hit@1 Hit@5 R@20 MRR DPR (roberta) 16 (2) AVATAR results on STARK's human-generated splits. In Table 5 , we demonstrate AVATAR's ability to generalize to test queries with distributions different from the question-answering pairs used to optimize the actor agents. (3) AVATAR results and comparison with ExpeL on STARK-MAG subset. In Table 6 , AVATAR demonstrates consistently higher performance than ExpeL across most metrics, notably achieving the highest Hit@1 and MRR scores. While ExpeL performs well in Recall@20, AVATAR 's overall improvements highlight its superior capability in precise retrieval tasks and tool-assisted knowledge retrieval.(4) Final action sequence by AVATAR on FLICKR30K-ENTITIES. In Figure 8 , we present the final actions optimized by AVATAR on FLICKR30K-ENTITIES. (5) Sensitivity of AVATAR to upper and lower bounds. We evaluated various combinations of ℓ and h, focusing on the STARK-AMAZON dataset due to computational constraints. Table 7 presents the Hit@1 results for different ℓ and h values. • The framework exhibits robustness to variations in ℓ and h, with Hit@1 fluctuations limited to a range of 2.7%.• A performance decline is observed when the gap between ℓ and h becomes too large, potentially due to the exclusion of certain training queries that fall within the (h, ℓ) interval.• A moderate gap between ℓ and h leads to slight performance improvements, suggesting that a balanced separation between positive and negative queries can enhance pattern differentiation without compromising the number of training queries.The results indicate that setting ℓ = 0.6 h = 0.5 yields an improved Hit@1 score compared to the baseline reported in the original paper. Overall, this analysis underscores the robustness of the framework, which relies on a minimal set of hyperparameters, including ℓ, h, batch size b, and training epochs.We keep only two prompt templates for our framework on all tasks: (1) The prompt template given to actor as initially instructions, and (2) the prompt template given to the comparator to conduct contrastive reasoning and generate the instructions for the actor. Below are the complete templates:This is the prompt given to actor as initially instructions:You are an expert user of a knowledge base, and your task is to answer a set of → queries. I will provide your with the schema of this knowledge base: <knowledge_base_schema> You have access to several APIs that are pre-implemented for interaction with the → knowledge base: <func_call_description> -Optimize the integrity of the code, e.g., corner cases.-Minimize computational expenses by early elimination of candidate nodes that don't → meet relational requirement (if any). -Avoid conducting unnecessary and redundant computations, especially when using → loops. -Make use of 'parameter_dict' to avoid hard-coding parameters and weights.-Use the functions that end with 'by_llm' wisely for more accurate searches.-Use 'debug_print' smartly to print out any informative intermediate results for → debugging. -Exclude or comment out any example uses of 'get_node_score_dict' in the output → code.Your output:This is the prompt given to comparator to generate the instructions for the actor: <initial_prompt> <previous_actions> After executing the above actions on user queries, some queries have yielded good → results, while others have not. Below are the queries along with their → corresponding evaluation metrics: Well-performing queries: <positive_queries_and_metric> Poorly-performing queries: <negative_queries_and_metric> Task:(1) Firstly, identify and contrast the patterns of queries that have achieved good → results with those that have not. (2) Then, review the computational logic for any inconsistencies in the previous → actions. (3) Lastly, specify the modification that can lead to improved performance on the → negative queries. You should focus on capturing the high-level pattern of → the queries relevant to the knowledge base schema.We identify several potential limitations of our work:• Scalability: AvaTaR is designed to scale with large language models (LLMs) that support extended context lengths (up to 128k tokens), enabling it to handle numerous tools and complex tasks. However, increased latency and other practical limitations may hinder performance in scenarios requiring hundreds of tools or high complexity. Future research could focus on incorporating specialized, tool-augmented LLMs as auxiliary agents to facilitate smoother scaling. • Computation Requirements: Managing longer contexts and multiple tool interactions within AvaTaR increases computational demands, which can significantly raise operational costs. These requirements necessitate substantial resources to maintain efficient performance, particularly when scaling to larger datasets or more intricate tasks. • Potential Failure Modes: Although AvaTaR performs well on known queries, its performance may diminish when faced with queries that require new or unfamiliar combinations of tools. This limitation could be mitigated by integrating adaptive learning techniques and continuous monitoring, which would allow AvaTaR to better handle novel tool requirements.Please refer to Table 8 and Table 9 for the functions.For general QA tasks, we use the following tools:• WEB_SEARCH: A general-purpose tool that performs web searches to answer questions. Useful for retrieving up-to-date information from the internet when other sources are unavailable. • ARXIV_SEARCH: This tool retrieves information about academic papers from Arxiv using a paper's unique ID. This function call can provide metadata and other details for academic references. • Wiki_SEARCH: If you have a question or name to lookup, this tool uses a Wikipedia search to retrieve relevant information. • RETRIEVE_FROM_DB: This tool is used to retrieve relevant information from a database. This is only available on ToolQA. Justification: We design a novel and automatic framework that optimizes an LLM agent to effectively use the provided tools and make comprehensive analysis on the evolution of our key modules.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper initially received 2 Weak Accepts and 1 Reject ratings. While reviewers praised the presentation, effectiveness and interesting ideas, there were concerns on lack of details, lack of novelty and missing experimental comparisons. The rebuttal and discussions helped significantly and assuaged most concerns, leading to substantial discussion between the reviewers and resulting in a mostly positive score. The AC believes that the paper has merit and can be published at NeurIPS. We require that the clarifications, rewritings and additional information provided in the rebuttal be incorporated in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper proposes a new framework AVATAR for LLM agent that operates in two stages:
-The first stage is optimization during the training process, which integrates the LLM comparator component into the AVATAR. The comparator summarizes holistic prompts from positive and negative queries and iteratively optimizes LLM actor using the prompts.
-The second stage is deployment during the inference process, where participants respond without the involvement of the comparator.
strength_weakness: strength_weakness: 1. The optimization is data-driven, so comparator works effectively when the positive and negative queries are well sampled and balanced during iteration; when to stop iteration and how to sample the queries; it is not clearly clarified.
2. Regarding the construction of positive and negative queries, it is necessary to conduct experiments on the lower and upper bounds(l&h) to select appropriate values for optimal performance.
3. Regarding the memory bank, it is a important component to swap in and out according to data quality or balanced distribution,
it is not well designed and discussed.
questions: questions: Is "AIR (gpt4)" in Table 3 a typo?
limitations: limitations: n/a
paper_summary: paper_summary: This paper introduces AVATAR, a novel framework for optimizing large language model (LLM) agents to effectively use provided tools and improve performance on complex multi-step tasks, with a focus on retrieval tasks. The key innovation is a comparator module that generates holistic instructions to improve the actor (main agent) through contrastive reasoning on batches of well-performing and poorly-performing queries.
The authors demonstrate AVATAR's effectiveness on four challenging retrieval datasets from the STARK and FLICKR30K-ENTITIES benchmarks. Results show significant improvements over state-of-the-art baselines like ReAct and Reflexion.
strength_weakness: strength_weakness: Limited scope of evaluation: While the retrieval tasks are complex, it would be valuable to see AVATAR applied to a broader range of tool-use benchmarks to demonstrate generality.

Lack of comparison to finetuning: The paper does not discuss or compare to alternative approaches like directly finetuning the actor model via rejection sampling. It's unclear how AVATAR compares to such methods in terms of performance and efficiency.

Scalability considerations: The paper does not thoroughly address how the method scales with increasing numbers of tools or more complex task structures.
questions: questions: Regarding the use of the memory bank as in-context learning examples, 
How does AVATAR compare to directly finetuning the actor model in terms of performance, efficiency, and data requirements? What are the key advantages of the instruction-based approach?
How does the method handle scenarios where the action model improves significantly, making it difficult to find negative examples for the contrastive learning process? Is there a strategy for addressing this?
Have you explored applying AVATAR to other types of tool-use tasks beyond retrieval? What challenges do you anticipate in extending to more diverse task types?
limitations: limitations: The authors have not adequately addressed the limitations of their work. I recommend adding a dedicated limitations and broader impacts section to address these points more thoroughly. 

This should include: Discussion of computational requirements and scalability limitations, potential failure modes or scenarios where the method may struggle
paper_summary: paper_summary: This paper introduces a new framework, AVATAR, that allows LLM agents to optimize the performance of Knowledge Retrieval. In the framework, a Comparator agent is adopted to extract insight from the positive and negative samples. The experiments on four retrieval datasets show the effectiveness of the method.
strength_weakness: strength_weakness: The main weakness is the lack of innovation and comparison compared with cutting-edge works.

1, This proposed AVATAR is very similar to ExpeL [1] in terms of method and has no obvious innovation in comparison. ExpeL adopts a Reflexion agent (actor) to gather success and failure experiences of multi-step tasks, which is the same as the actor's role in this paper. ExpeL adopts another agent to compare a failed trajectory with a successful trajectory for the same task and extract insights from the comparison, which is the same as the instruction generation process of the comparator in this paper. ExpeL adopts the experience pool to retrieve successful trajectories, which is the same as the Memory Bank in this paper. Similarly, Autoguide [2] also extracts insights from experiences. Therefore, the method of auto-optimizing the instruction by comparing the successful and failed trajectories has no innovation compared with Expel [1] and Autoguide [2]. The author should conduct more surveys and read relevant cutting-edge works.

2, The paper claims their method achieves SOTA on STARK and FLICKR30K-ENTITIES benchmarks, but they only compared with relatively basic methods. The author employs several outdated embedding-based retrievers as the comparison. However, text-to-image retrieval on FLICKR30K-ENTITIES has long been dominated by the vision-language model [3] [4]. Internvl [3] and Beit [4] achieve high recall@1 (from 80 to 90) on this benchmark. Therefore, it is unreasonable to claim that the proposed method achieves the retrieval SOTA on these two benchmarks.

3, The results of the comparison on FLICKR30K-ENTITIES are missing. Quantitative results on FLICKR30K-ENTITIES are only shown in Figure 5 (right), but the results of all methods on this benchmark are not found in the paper. Considering that STaRK is a relatively new benchmark, not many methods have experimented on it, so results and analysis on recognized benchmarks are important.

4, Many sentences in the paper are too obscure to understand and lack clear explanations. For example, "these per-sample instructions tend to be narrow and fail to identify flaws across all components of a complex solution. Additionally, while certain tool combinations may be effective for one type of input, their effectiveness can vary with others, leading to decreased performance when applied across different scenarios." in lines 152-154, what do "per-sample instructions" mean, and why will these tool combinations "lead to decreased performance"? Providing some simple examples can make it easier to read.

[1] Zhao, Andrew, et al. "Expel: Llm agents are experiential learners." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.

[2] Fu, Yao, et al. "Autoguide: Automated generation and selection of state-aware guidelines for large language model agents." arXiv preprint arXiv:2403.08978 (2024).

[3] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Wang, Wenhui, et al. "Image as a foreign language: Beit pretraining for vision and vision-language tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
questions: questions: In line 160, "the comparator samples a group of data samples (question-answer pairs), executes the current actions for each question," but how can the comparator execute the current actions? What are the current actions for the comparator?

The text in Figure 3 is too small to read and the meaning of the marked text is difficult to understand.
limitations: limitations: There is no separate Limitation or Broader Impacts section in the paper. In addition, the authors said, "We did an extensive survey on related work in the area of LLM agents, agent optimization, LLM agent for retrieval, and further discuss their limitations," but the discussion of the limitations in this paper is required.

Comments: title: 2nd Batch response (Continue)
2nd Batch response
Thanks for your response, no more comments
(Continue) Experiments on more tool-use benchmarks
comment: Correction: In the summary table above, the reviewer labels 'JqLP' and 'vMWD' should be switched, which matches the sequence order on the webpage. Apologies.
**Comment 3: More benchmarks.** Our experiments on HotpotQA improve this aspect. Please also see our response to `Reviewer CkbF` for AvaTaR results on ArxivQA and ToolQA.

**Comment 4: Clarity on comparator.** Thanks! Yes, we made sure the statement is clear now.

**Comment 5: Content requirements and token overhead.** On STaRK-MAG, the context length for the comparator is approximately 4k tokens per step, including initial instructions and tool usage (with pos/neg queries taking around 0.8k). The actor's token cost, including memories, is around 8k. Running AvaTaR for 50 steps accumulates about 600k tokens, costing under $10 with gpt-4-turbo. There is no inference cost on STaRK dataset as we directly apply the action sequence. 

As a reference, we also compute the token cost for ExpeL on STaRK-MAG: Training token cost: 357k in total (3.57k per task); Testing token cost: 389k in total (7.8k per task). 

In this comparison, we believe AvaTaR has an advantage in scaling especially when the number of testing queries are large.

For HotpotQA, the comparator's context length is around 2k tokens per step (with pos/neg queries at 0.5k). The actor's token cost ranges from 2k to 8k, depending on the number of actions. Running 10 steps accumulates around 80k tokens.

*Additional related works.** Thanks! We have also added them to our related work.
We appreciate the reviewer for further communication. We hope our response this time could mostly address your concerns. 

### **TL;DR;**
- We clarify Avatar's novelty, especially on AvaTaR $\neq$ ExpeL + on-policy training.
- We show that AvaTaR outperforms ExpeL and Retroformer on HotpotQA. We also show the performance and scalability advantage of AvaTaR over ExpeL on STaRK-MAG; 
- We show that AvaTaR works well on new QA datasets (HotpotQA, ToolQA, ArxivQA)

---

Before started, we make the following terms consistent to avoid confusion:
- Queries = instances (used in AvaTaR) = tasks (used in ExpeL)
- Actions to answer queries = action sequences (AvaTaR) = trajectories (ExpeL)

---

**Comment 1: About novelty**

While we pointed out "on-policy" and "off-policy" as one difference between AvaTaR and ExpeL, the novelty of AvaTaR goes beyond that. Here are our reasons:

- **What to contrast**: This is relevant to the reviewer's question about batch size - *"Given that 20 trajectories of Actors can be quite long."* We believe this is a misunderstanding (please let us know if otherwise). Please note that AvaTaR contrasts positive and negative queries, unlike ExpeL, which contrasts trajectories. Therefore, the context length is less of an issue since the queries are mostly short, we will provide detailed statistics. 
- **Benefits of contrast queries**: 
  Following the last point, the reviewer actually pointed out one benefit of contrasting queries - it allows us to contrast among a batch of pos/neg queries `(batch_size=b)` rather than a well-performing and an under-performing trajectory (`batch_size=2`). This design offers better scalability and generalization ability. We provided intuitions in L173-184, and we repeat them here for your convenience:


   ```
   Moreover, as contrastive reasoning directly targets disentangling the performance gap related to input patterns and how they are handled differently by the tools, it is particularly effective in helping comparators differentiate and select tools for use. Finally, by identifying systemic flaws across a wide array of negative queries, comparator generates modifications that are not only tailored to individual samples but also to diverse data samples, offering benefits for better generalization to novel samples.
   ```

  This design enables AvaTaR to generate more holistic instructions from the insights on multiple queries and/or generalize the final action sequence to hundreds of testing queries without test inference, which are validated by the strong generalization in our existing experiments.
 - **AvaTaR $\neq$ ExpeL + on-policy training**: Due to the differences in design, AvaTaR is able to conduct on-policy training since it directly optimizes the action sequence for multiple queries. However, one can imagine it would be hard for ExpeL to conduct on-policy training with trajectory comparison.

---

**Comment 2: Comparison with more baselines**

We added ExpeL and Retroformer as our baselines in our paper. We firstly compare them with AvaTaR on HotpotQA on the dev subset (100 queries) in their repositories. Here are the results, where all the baseline results are reported by ExpeL and Retroformer):

||HotpotQA (EM)|
|:---|--:|
|Act|29%|
|ExpeL|39%|
|ReAct |40% |
|Reflexion|46%|
|Retroformer (#retry=1)|51%|
|AvaTaR|55%|

For AvaTaR, we take 100 training queries to optimize prompts for 10 steps and set the number of retries as 0, which achieves the best performance. Then, we compare AvaTaR with ExpeL on STaRK-MAG. STaRK datasets involve over 20 tools, leading to long action sequences and high token overhead for ExpeL to process and evaluate. Due to this, we randomly sampled 100 training queries and evaluated ExpeL on 50 testing queries, comparing with the other methods on the same set.


|||MAG|(#Test=50)|| 
|:---|:---|:---|:---|:---|
||Hit@1|Hit@5|Recall@20|MRR|
|Dense Retriever|16.00|40.00|51.84|27.39|
|QAGNN|20.00|52.00|49.71|36.39|
|VSS|40.00 |58.00|55.93|47.76|
|Multi-VSS|32.00|58.00|58.81|43.58|
|ReAct|46.00|60.00|54.67|50.92|
|ExpeL|40.00|58.00|55.94|47.43|
|Reflexion|48.00| 64.00| 57.43|52.31|
|AvaTaR-C|44.00|60.00|52.49|50.16|
|AvaTaR|52.00| 64.00|53.86 |56.74|

For ExpeL, we used `text-embedding-3-large` embedding model to retrieve insights. We found ExpeL’s performance similar to ReAct, which might be because the STaRK queries are diverse, therefore require more training data to gather enough experience. 

We will have ExpeL results on the other two STaRK datasets and include a table similar to Table 2 in our paper. Hopefully, these will address your concern about the comparison with previous LLM Agent methods and justify our claims.
See it above.
Hi Reviwer CkbF,

Thanks for waiting for the follow-up results on ToolQA! We introduce the setup and present the full results here:

- **Datasets**: SciREX (a dataset for document-level information extraction based on full-length ML scientific papers) and Agenda (personal agenda questions based on a private knowledge base) are the two (and only two) datasets based on external text corpora from ToolQA. Each dataset has `easy` and `hard` splits.
- **Configuration for AvaTaR:** We randomly split the questions by 40%:60% for training and testing, as official dataset splits are not provided. We set the maximum epoch to 5.
- **Tools:** We use the knowledge base tools provided by ToolQA and web search tools for all agent methods.
- **Agent Backbone:** We use `gpt-4o` for the LLMs of AvaTaR and the baselines, which are evaluated on our testing split for fair comparison.

|            | SciREX | SciREX | Agenda | Agenda |
|------------|--------|--------|--------|--------|
|            | Easy   | Hard   | Easy   | Hard   |
| CoT        | $0.0$%   | $0.0$%   | $0.0$%   | $0.0$%   |
| ReAct      | $8.3$%   | $18.3$%  | $31.6$%  | $11.7$%  |
| Reflexion  | $10.0$%  | $13.3$%  | $30.0$%  | $13.3$%  |
| AvaTaR     | $11.6$%  | $21.7$%  | $36.7$%  | $23.3$%  |

We observed consistent improvements across these two datasets, with a significant improvement on the hard queries of Agenda. 

We hope the experiments on ToolQA and ArxivQA can further validate the effectiveness and applicability of AvaTaR on more tool-use datasets. We are happy to provide any details (e.g., insights or example instructions) that are not covered here for conciseness.
Thanks for the detailed responses, but they have not fully addressed my concerns.

1- Even though there are differences in details, the existence of papers like ExpeL and AutoGuide clearly reduces this paper's innovation in the direction of self-optimization of LLM agents. Given ExpeL and AVATAR both utilize the comparator to optimize the prompt, the main innovation of this paper would be replacing 'off-policy' with 'on-policy' learning. However, as 'off-policy' and 'on-policy' are both useful for RL, the authors should perform more experiments and studies to show that their 'on-policy' methods are better than the 'off-policy' version.

2- Previous works (ExpeL and Retroformer were published in Aug 2023) used ReAct and Reflexion as baselines, which cannot be the reason that this paper still uses the same baseline. The field of LLM Agents is a rapidly developing field. In the year after reflexion (published in Mar 2023), a large number of high-quality methods have been proposed, e.g., [1][2][3]. However, in the experiment, the author ignored the comparison with these cutting-edge methods. In addition, in the introduction and related work of the paper, the author repeatedly pointed out that the previous LLM Agent methods via self-optimizing cannot solve complex problems. However, there is no comparison with these methods in the experiment to verify these claims.

Therefore, the authors should keep up with recent methods, and consider these approaches in their experiments, rather than just the basic methods.

3- The method proposed in this paper is not only applicable to retrieval tasks but also to general agent tasks. Therefore, it should also be experimented on some common and difficult benchmarks. As I said before, one of my concerns is that STARK is a very new benchmark (published in April 2024), and Flickr30k-entities is a very rarely used benchmark for LLM Agent. Achieving good results on LLM agent benchmarks that have been studied more, such as HotpotQA [4], AgentBench [5], and WebArena [6], the method can be more convincing.

4- For the question about the clarification of the comparator, your answer should be added to the revised version because the presentation of the original text is unclear and misleading.

5- One more new question: I found the authors implemented AVATAR with a batch size of 20: "the metric Recall@20 for constructing positive and negative queries, and hyperparameters (l = h = 0.5, b = 20)". Given that 20 trajectories of Actors can be quite long, can the authors provide the context length required by the comparator and the token overhead required for AVATAR optimization? 

[1] Zhu, Zhaocheng et al. “Large Language Models can Learn Rules.” ArXiv abs/2310.07064 (2023): n. pag.

[2] Majumder, Bodhisattwa Prasad et al. “CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization.” ArXiv abs/2310.10134 (2023): n. pag.

[3] Qian, Cheng et al. “Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution.” ArXiv abs/2401.13996 (2024): n. pag.

[4] Yang, Zhilin et al. “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.” Conference on Empirical Methods in Natural Language Processing (2018).

[5] Liu, Xiao et al. “AgentBench: Evaluating LLMs as Agents.” ArXiv abs/2308.03688 (2023): n. pag.

[6] Zhou, Shuyan et al. “WebArena: A Realistic Web Environment for Building Autonomous Agents.” ArXiv abs/2307.13854 (2023): n. pag.