Title: AGILE: A Novel Reinforcement Learning Framework of LLM Agents
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM agent

reinforcement learning

LLM-human interaction
Abstract: We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments)  designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.

AGILE: A Novel Reinforcement Learning Framework of LLM Agents

Abstract

We introduce a novel reinforcement learning framework of LLM agents named AGILE ( AG ent that I nteracts and L earns from E nvironments) designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.

1 Introduction

Large Language Models (LLMs) have exhibited remarkable capabilities such as instruction following, reasoning, and zero-shot learning [3, 45, 20, 26], which have greatly catalyzed the development of autonomous agents based on LLMs [28, 30, 2], also known as LLM agents. Recent works propose several essential components or workflows to enhance the abilities of LLM agents, such as planning [45, 51, 39], reflection [21, 40], tool-use [29, 36, 48] and life-long learning [42]. However, it remains unclear how to integrate all components into a unified framework and optimize them end-to-end.

In this paper, we introduce a novel reinforcement learning framework for LLM agents to unify various components and streamline their learning and operation processes. As shown in Figure 1(a), the architecture of the agent system, named AGILE, comprises four modules: LLM, memory, tools, and executor. Furthermore, the agent can interact with both users and experts. The LLM, functioning as the predictor of all actions, generates instructions and processes responses. The executor, working as the controller of all actions, interprets the LLM instructions to activate the corresponding modules and collects their responses for the LLM. For example, the executor can fetch a text from the memory and append it to the context of LLM, or extract an excerpt from the context and append it to the memory.

[GetQuestion]

LLM

Question: Does this case accommodate macro photography?

User

User

Product metadata: ...History QA: ... Knowledge: ...

Memory

[RetrieveMemory]

LLM

[SeekAdvice]

LLM

[Reflection]

Camera cases with no tripod mount are not suitable for macro photography

where stability is crucial.

LLM

[SubmitAnswer]

Unfortunately, this camera case lacks a tripod attachment...

LLM

LLM

[UpdateMemory]

Question: Does this case ...  Answer: Unfortunately, this camera...

Knowledge: camera cases with no tripod ...

(b) Running example

(a) Agent system architecture

Memory

Expert

User

LLM

Tools

Executor

Policy model = LLM

Agent

Environment

Answer: Unfortunately, this camera case lacks a tripod attachment feature,

which is crucial for achieving such stability.

Expert

[GetQuestion]

Question: Is this case appropriate for photography when I use remote shutter?

Product metadata:{"Tripod Attachment": "no"... History QA: ...

Knowledge: Camera cases with no tripod mount are not suitable for macro

photography where stability is cruicial.

[RetrieveMemory]

[SubmitAnswer]

No, the camera case is not the suitable for photography using...

LLM

LLM

Memory

LLM

LLM

[PredictAnswer]

No, the camera case is not the suitable for photography using remote shutter

with low camera stability.

Memory

User

User

Several questions later

The executor can also follow instructions of the LLM to utilize a search tool. In addition to skills such as reasoning, planning, and reflection, we propose a new ability called seeking advice , which means that the agent proactively consults human experts when it encounters a problem unsolvable. The agent can reflect on the expert feedback and memorize it for future use. Furthermore, we propose a training method based on reinforcement learning (RL), which simultaneously trains the policy of invoking different modules and the reasoning, planning, reflection, and seeking advice abilities of the LLM agent in an end-to-end fashion.

While the proposed agent framework is general, in this paper, we evaluate it in complex question answering (QA). It is a task an LLM agent has the potential of outperforming existing solutions such as the use of an LLM alone. However, existing QA benchmarks [12, 49, 11, 27] are designed for specific subsets of capabilities (e.g., reflection, memory retrieve, etc.) which cannot simultaneously investigate the ability to combine all modules and capabilities of the agent.

To address this, we have developed a new benchmark called ProductQA. ProductQA comprises 88,229 question-answer pairs in customer service divided into 26 QA tasks, each corresponding to a distinct Amazon product category. This benchmark is based on real Amazon user queries and includes fact-based questions, reasoning questions, and product recommendation queries. It comprehensively evaluates agents' abilities to handle historical information and accumulated knowledge, leverage tools, interact with humans, perform self-evaluation, and conduct reflection. Additionally, the training and testing tasks are made disjoint to assess the agent's ability to adapt to new product categories.

We evaluate our agent framework on three tasks, ProductQA, MedMCQA [27] and HotPotQA [49]. For ProductQA, we use a two-stage training method based on Vicuna-13b [6]. In the first stage, imitation learning is employed to create agile-vic13b-sft . In the second stage, the policy gradient algorithm of PPO [37] produces agile-vic13b-ppo . Experimental results show that agile-vic13b-ppo improves the relative total performance score by 9.2% over GPT-4 and by 90.8% over GPT-3.5. Ablation studies confirm that all modules in Figure 1 are indispensable. Specifically, removing tools or memory usage negatively impacts the agent's performance, leading to a 25.9% or 17.4% increase in seeking advice, respectively, or a 9.3% or 4.0% relative decrease in the total score, respectively. Disabling the seeking advice function results in a 10.7% decrease in accuracy. Finally, agile-vic13b-ppo achieves a 2.3% relative increase in total score compared to agile-vic13b-sft , demonstrating the necessity of PPO training. On MedMCQA, we train an agile-mek7b-ppo agent, initialized from Meerkat-7b [17], following the same two-stage procedure.

Our agent improves the base LLM's accuracy from 53.4% to 85.2% by seeking advice on 31.6% instances. This accuracy surpasses the SOTA accuracy of 79.1% by GPT4-MedPrompt [25]. When all agents are able to seek advice, our agent also outperforms the GPT-4 agent in terms of the total score. For HotPotQA, we use the same two-stage method to train agile-vic13b-ppo from Vicuna-13b. Our agent achieves 67.5% accuracy, surpassing the strongest baseline of 48.2%, by seeking advice on 15.6% of instances. When advice-seeking is enabled for all agents, our agent outperforms GPT-4 by 10.8% in total score.

The main contributions of this paper are summarized as follows:

We propose a novel reinforcement learning framework of LLM agents. It facilitates end-toend learning of agents. Notably, this framework enables the agent to proactively seek advice from human experts, providing two advantages: 1) It ensures high-level accuracy when dealing with complex and challenging questions, and 2) it fosters learning from humans, thereby enhancing its abilities to adapt to new tasks.

We develop a benchmark, ProductQA, to comprehensively evaluate the agent's capabilities in complex question answering.

We perform experiments on multiple tasks to verify our framework and show that AGILE agents based on 13B and 7B LLMs trained with PPO can surpass GPT-4 agents.

2 Methods

2.1 RL formulation of agent

Our agent framework comprises four elements: LLM, memory, tools, and executor, see Figure 1(a). The LLM possesses a context , defined as the sequence of tokens it utilizes to generate the next token. In RL terminology, the agent conducts a token-level Markov decision process (MDP). The action space A corresponds to the LLM's vocabulary, with each token representing an action. Hence, the LLM serves as the policy model . The agent's state consists of the (context, memory) pair. Upon predicting a new action a t (i.e., a new token), the LLM transfers control to the executor. The executor applies predefined logic to transition from the current state s t to the next state s t +1 , implementing the state transition function S × A → S in RL, and then returns control to the LLM to predict the next action. Concurrently, the environment issues a reward r ( s t , a t ) .

Let us examine the state transition more closely. For each action, the executor's first operation is to append the token to the context, preparing the LLM for generating the next token. Then, the executor checks a registered list of functions . Each function is designed to execute a set of operations, including memory I/O, tool usage, and interaction with the environment. If the action (i.e., the token) matches a function name, the executor will execute the associated function implementation, further mutating the agent state. For instance, if the token is [GetQuestion] , the executor will prompt the user for a new question and append it to the context; if the token is [UpdateMemory] , the executor will write a specific segment of the context into the memory; if the token is [ClearContext] , the executor will reset the context to [BOS] . In summary, the LLM interacts with the memory and tools by predicting function names, relying on the executor to execute these functions. See Table 1 for a full list of functions defined for a QA agent and see Figure 1(b) for a running example.

2.2 Policy learning

We frame the policy learning problem as a task of training a language model. Consider an agent trajectory τ = ( s 1 , a 1 , ..., s n , a n ) , we derive a training sequence denoted as ( e 1 , ..., e n ) , where e i represents the tokens that the executor appends to the context at step i . If a i is a function name token, then e i is the concatenation of a i and extra tokens appended by the function execution; otherwise, e i = a i . In this sequence, { a 1 , ..., a n } (the first token of each e i ) are referred to as action tokens. The LLM context at step i , denoted by c i , is a subsequence of the prefix ( e 1 , ..., e i -1 ) ; c i may be shorter than ( e 1 , ..., e i -1 ) because the executor can delete context tokens.

In Imitation Learning (IL), we generate trajectories by observing human experts or more proficient agents, then we derive the training sequences to fine-tune the LLM. It is important to point out that (1) the loss is calculated on the action tokens only, and (2) c i should serve as the attention mask for tokens in e i , as it reflects the true context perceived by the LLM at the time of action prediction. In

reinforcement learning (RL), we treat the LLM as the policy model, from which training sequences can be sampled and individual action tokens are assigned rewards. Consequently, the LLM can be optimized using policy gradient methods, such as PPO [37]. Analogous to the IL setup, we apply policy gradient updates exclusively to the action tokens and employ c i as the attention mask.

In some situations, an agent may produce very long trajectories, potentially yielding training sequences that span millions of tokens and are impractical for training. We can leverage the structure of the trajectory to partition it into smaller segments. For instance, if the agent resets its LLM context at the beginning of every QA session, then we can partition by the session boundary. Nevertheless, these sessions are not entirely independent; actions taken in earlier sessions can influence memory, creating lasting effects on subsequent sessions. To tackle this challenge of long-range dependencies, we propose a training algorithm detailed in Appendix A.

2.3 Interaction with human experts

Our agent framework enables the agent to proactively seek advice from human experts. For example, the agent can invoke a [SeekAdvice] function to request expert advice. This approach helps in two ways. Firstly, the agent can request the correct answer when its confidence is low, ensuring sufficient accuracy for the application. Secondly, the agent can use [Reflection] to distill general knowledge from the expert advice before storing it in memory. This accumulation of knowledge allows the agent to adapt to new tasks that it has not encountered during training.

Seeking advice involves complex decision-making. The agent must estimate its own confidence in the current session, predict the potential value of the advice for future sessions, and consider the cost of human resources. The optimal trade-off is difficult to annotate manually but aligns well with our RL framework. Specifically, the present risk, future value, and cost of action can all be represented as RL rewards, allowing this skill to be trained as part of the policy model on an end-to-end basis.

3 The ProductQA dataset

We believe that product question answering in a real online shopping environment offers a comprehensive challenge for evaluating LLM agents. First, it demands expert knowledge about millions of products, including their technical specifications, usage in particular scenarios, and compatibility with other products. Second, answering some questions requires the use of tools, such as a product search tool. Third, the continuous emergence of new products necessitates the adaptability of the agent. This has motivated the creation of the ProductQA dataset. Unlike existing online shopping QA datasets [38, 8], which primarily focus on questions about product metadata or page information, ProductQA features more complex queries involving reasoning, expert knowledge, and tool usage (e.g., SQL), providing a comprehensive assessment of an agent's capabilities.

The ProductQA dataset consists of 26 QA tasks, each representing a distinct group of products within a specific category. Each group encompasses 17-20 products. We collected 20 groups for training and 6 for testing, allowing for assessing the agent's adaptability to new tasks. We collected an average of 3,393 question-answer pairs for each product group. The questions within the same

group are correlated, as knowledge from one answer may aid in addressing other questions. The dataset statistics are presented in Table 12.

The dataset is annotated by 20 professional annotators, each with at least a college degree, employed by a commercial data annotation company. We pay the company at market rates for professional annotation. See annotation guidelines in Appendix F.2. In addition, we will release the code for the data pre-processing before human annotation.

3.1 Product collection

We gather products from the Amazon Review Data [23], which includes product metadata as well as reviews. We initially filter the Amazon Review Data to retain only popular products with at least 100 reviews, then cluster them by category tags. From these clusters, we select 26 based on the size of the cluster, each defined as a product group . Subsequently, we sample products from each product group. See Appendix F.1 for more details about product group and product selection.

After the products are collected, annotators compile an information table for each product group. An example of such a table is presented in Table 2. To enhance the efficiency of the annotation process, we employ GPT-4 to extract as many product features as possible from the reviews. These features, together with the product metadata, are provided to the annotators for table creation.

3.2 QA collection

We identify three predominant types of questions in online shopping contexts: 1) Fact-QA : questions concerning specific product details; 2) Search-QA : searches for product recommendations tailored to user preferences; 3) Reasoning-QA : questions whose answers require domain-specific reasoning, such as the implications of a product feature. Accordingly, we annotate question-answer pairs for these types. Each question is annotated with both a detailed paragraph-long answer and a concise short answer. The long answer should resemble a response from human customer service, while the short answer consists of a few words. We train the model to predict both answer types. The accuracy of the long answers is evaluated using GPT-4 (see Appendix J for the prompt); the short answers are assessed by exact match and are used for defining rewards for RL training.

Fact-QA Fact-QAs are constructed from product reviews. For each product, we provide GPT-4 with a batch of 30 reviews, prompting it to generate 20 questions and their corresponding answers

before moving on to the next batch. We encourage GPT-4 to create diverse questions. The results are then given to annotators to refine and finalize the question-answer pairs.

Search-QA Starting with an information table for a given product group, we generate random SQL expressions using a set of predefined rules. These expressions are then translated into natural language questions by GPT-4. The answers are obtained by executing the SQL queries. Subsequently, human annotators thoroughly revise the QA pairs.

Reasoning-QA As the first step, we collect professional knowledge for each product group. To enhance efficiency, we utilize GPT-4 to generate candidate knowledge entries based on the technical specifications from the information table. These entries are then curated and refined by human annotators. Here is an example of a knowledge entry: Motherboards with the ATX form factor are ideally suited for high-performance computing tasks and gaming, due to their ample expansion slots for graphics cards and other peripherals that boost computing capabilities. Finally, annotators develop question-answer pairs from these knowledge entries.

4 Experiments

4.1 Experimental setting

Dataset Weevaluate our agent on three complex QA tasks: ProductQA, MedMCQA and HotPotQA. MedMCQA [27] is a dataset for multiple-choice QA. It consists of questions from medical school entrance examinations. HotPotQA [49] features natural, multi-hop questions, which challenge an agent's ability to perform reasoning and utilize search tools. For both MedMCQA and HotPotQA, we report results on their respective full dev sets.

Agent definition Our agent can invoke functions defined in Table 1. In a typical workflow, the agent prompts the user for a new question at the session start. It can then retrieve memory to get relevant information. The memory can be initialized as empty (ProdcutQA) or with domain knowledge (QA pairs from MedMCQA training dataset). The agent has the option to use external tools, such as product search in ProductQA and article search in HotPotQA), to gather more information. At last, the agent decides whether to predict an answer directly or seek human advice. If the agent seeks advice, it obtains a human answer (ground-truth answer in our setting). The agent can then optionally use a reflection round to extract general knowledge from the human answer, writing both the human answer and the reflected knowledge to its memory. Finally, the agent submits an answer to the user. In our setting, submitting a correct answer incurs a +1 reward, while submitting a wrong answer incurs a 0 reward. Seeking human advice has a fixed -c reward, where c represents seeking advice cost . Assuming the human advice always contains a correct answer, then the possible total rewards are { 0 , 1 , 1 -c } .

Training The training consists of two stages. First, we construct trajectories from the training data and employ imitation learning to train the agent. Then we apply Algorithm 1 for further optimization by reinforcement learning. See Appendix B for implementation details. For ProductQA and HotPotQA, the agent's LLM is initialized from Vicuna-13b-1.5. For MedMCQA, we use Meerkat7b [17], a medical LLM trained with high-quality CoT reasoning paths from 18 medical textbooks and diverse instruction-following datasets. We fine-tune the model for 2 epochs with a learning rate of 1e-5 and a batch size of 64. We implement PPO for 1 epoch with a learning rate of 1e-6 and a batch size of 64. The training runs on NVIDIA-H800. Training times and the number of GPUs for each experiment are reported in Table 13. The LLM is fully trained without using LoRA.

Evaluation and baselines We report three metrics for the agent: (a) Advice rate: the rate of seeking human advice; (b) Accuracy: the rate of predicting the correct answer; (c) Total score: the average reward across all sessions, taking the advice rate and the accuracy both into account.

We compare our agent against two types of baselines: 1) Prompting GPT-3.5 (gpt-3.5-turbo-0301) and GPT-4 (gpt-4-0613) [26] to directly answer the question, without working in an agent manner, noted as gpt3.5-prompt and gpt4-prompt . 2) Prompting GPT-3.5 and GPT-4 within the AGILE framework, noted as agile-gpt3.5-prompt and agile-gpt4-prompt . We carefully designed prompts for all baselines and they are shown in Appendix J.

4.2 Results on ProductQA

As Table 4 shows, our AGILE agent outperforms all baselines on ProductQA. Notably, the average total score of agile-vic13b-ppo across six test groups shows a relative improvement of 9.2% in short answers and 5.0% in long answers to agile-gpt4-prompt where the seeking advice cost is added into the prompt. Concretely, agile-vic13b-ppo uses a comparable number of seeking advice to achieve 7.4% higher accuracy in short answers than agile-gpt4-prompt , and as Figure 2 shows, this accuracy improvement is consistent across the whole trajectory. Our agile-vic7b-ppo agent also outperforms agile-gpt4-prompt in average total scores. Note that the GPT-4 agent knows the seeking advice cost from its prompt (see Figure 7).

We investigate the impact of varying the seeking advice cost. As shown in Figure 3, when the cost decreases, both the advice rate and the accuracy increase, indicating greater utilization of human assistance. Specifically, with a high cost of 0.5, the advice rate is close to 0, and at a low cost of 0.1, the accuracy is close to 1. This result demonstrates that by adjusting the cost and through RL training, we can effectively manage the trade-off between accuracy and human cost. For instance, the agent can achieve 94.1% accuracy on the Motherboards task with a seeking advice cost of c = 0 . 1 (refer to Table 16). This capability is especially important in realistic scenarios that demand high accuracy levels. In most experiments, we set the cost at a medium level with c = 0 . 3 .

To validate the accuracy of GPT-4 evaluator in assessing the long answer results, we randomly select 100 triplets (questions, reference long answer, model-predicted long answer) and manually labeled the correctness. The results show a 94% agreement rate between the GPT-4 evaluator and the author.

Ablation study We present ablation studies in Table 5 to assess the contributions of individual agent components and the effects of RL training. The table indicates that disabling the option to seek advice (w/o Advice) leads to a 10.7% drop in accuracy and a 5.0% relative reduction in total score. Forcing the agent to seek advice at the initial part of the trajectory (Non-adapt Advice) causes a 4.2% decrease in accuracy, underscoring the value of adaptive decision-making. Removing reflection and memory capabilities (w/o Memory and w/o Reflection) both increase the frequency of advice-seeking, as the agent struggles to accumulate or leverage valuable knowledge, consequently decreasing the

total score. Furthermore, disabling tool use (w/o Tool-Use) causes a substantial 25.9% increase in the advice-seeking rate because the agent's capabilities are diminished, making it more reliant on external advice. Lastly, RL training improves the relative total score by 2.3%, lowers the advice-seeking rate, and boosts accuracy, demonstrating that RL training effectively optimizes the policy. Additional results on RL training can be found in Appendix C.

In Appendix E, we present detailed examples of agile-vic13b-ppo illustrating how memory, tools, seeking advice, and reflection enhance the agent workflow.

Trend of advice rate Figure 4 demonstrates a consistent decrease in the advice rate of agile-vic13b-ppo as more sessions are added to the trajectory. This decline can be attributed to the agent progressively accumulating knowledge and becoming more independent. Additionally, the figure illustrates that disabling RL training or reflection leads to a significant increase in the advice rate, underscoring the importance of RL training and reflection in reducing human costs.

4.3 Results on MedMCQA

Our agile-mek7b-ppo agent, based on the smaller Meerkat-7b [17] model, reaches an accuracy of 85.2% with an advice rate of 31.6%. As Table 6 shows, this represents a 31.8% accuracy increase over the base model Meerkat-7b-prompt and a 6.1% increase over the state-of-the-art gpt4-Medprompt [25]. Table 6 also shows that the ability to seek advice alone contributes a 23.2% accuracy gain, meaning that each instance of seeking advice corrects an average of 0.73 prediction errors. This indicates that PPO training effectively helps the agent identify its mistakes. For a fair comparison, we also evaluate agile-gpt3.5-prompt and agile-gpt4-prompt , which incorporate GPT-3.5

and GPT-4 within our AGILE framework. These agents also leverage advice-seeking to enhance accuracy, but without RL training, their total scores are lower than agile-mek7b-ppo . Finally, through ablation studies, we confirmed the essential roles of memory, reflection, seeking advice, and RL training in achieving high performance. Removing these components leads to a significant drop in total scores, detailed in Table 6.

4.4 Results on HotPotQA

We compare our method against several baselines. Specifically, we found the original ReAct baseline implementation in [51] to be suboptimal. By reproducing their results with GPT-4 (ReActgpt4-prompt), we observed improved performance. As shown in Table 7, our agile agent outperforms all baselines in accuracy, achieving a 40.0% relative improvement over ReAct-gpt4prompt, which is the strongest baseline. Additionally, compared to agile-gpt4-prompt , the trained agile-vic13b-ppo demonstrates both higher accuracy and a lower advice rate, leading to a 10.8% relative increase in total score. Ablation studies confirm that removing either seeking-advice or PPO training results in a significant decrease in the total score.

5 Related work

LLM agents Large Language Models (LLMs) have demonstrated substantial capabilities in following instructions, reasoning, and planning. Numerous research works, as shown in Table 8, utilizing prompt engineering, have constructed remarkable LLM agents capable of autonomously resolving complex tasks across various environments [28, 44, 2, 30, 4]. Furthermore, extensive works identify key components in the design of LLM agents, including planning [22, 39, 10, 32, 51, 35], tooluse [19, 29, 48, 36], and reflection [40, 21]. In this work, we enable the agent to utilize memory, tools and proactively learn from the environment. We then formulate the entire process within an RL framework so that all agent skills can be jointly optimized end-to-end.

Human-agent interaction Although LLMs face practical challenges, such as hallucination [53] and a lack of long-tail knowledge [16], consulting human experts can help mitigate these issues. Several studies [52, 46] have incorporated human experts into agent workflows relying on passive feedback or predefined rules. However, these approaches do not involve proactively seeking advice, which requires more complex decision-making. While [5, 31] train models to ask questions using behavior cloning, they ignore the fact that the decision to seek advice must be based on the LLM's own knowledge and capabilities [55, 18, 13]. [34] use a calibrated version of an LLM's token probabilities as a confidence measure, yet token probabilities tend to be overconfident [47], and existing calibration methods don't generalize well to our agent setting when the LLM makes multiple decisions in sequence. Ultimately, the challenge of seeking advice is tied to the LLM's self-evaluation, which is difficult to ground truth or optimize through SFT. In our RL framework, the value and cost of seeking advice can be directly represented as RL rewards, enabling the proactive skill of seeking advice to be optimized as part of the policy model on end-to-end RL training.

LLM agent benchmarks Several benchmarks have been designed to assess the capabilities of agents. For instance, the Webshop [50] and Mind2Web [7] datasets evaluate agents' tool usage and planning abilities within a web environment. HotPotQA [49] and TriviaQA [12] focus on agents' reasoning and tool usage for question answering. ALFWorld [41] examines planning and navigation skills, while ScienceWorld [43] provides an interactive text-based environment to evaluate agents' scientific aptitude. As illustrated in Table 9, despite these existing benchmarks, none comprehensively addresses all the core challenges of real-world agent applications, such as handling long-tail knowledge, human-agent interaction, long-term memory usage, tool usage, self-evaluation, and reflection. This motivated us to develop ProductQA.

6 Conclusion and future work

In this work, we introduce a novel reinforcement learning framework of LLM agents, called AGILE. First, the whole system of AGILE is trained end-to-end by reinforcement learning. Second, AGILE has the ability of seeking advice from external human experts. In addition, we develop a challenging dataset of complex QA, ProductQA, for comprehensive evaluation of an agent's capabilities. Extensive experiments demonstrate that within our framework, an agent based on a smaller model after RL training can outperform GPT-4.

AGILE is a general agent framework and we can certainly consider multiple extensions of it. An agent can be equipped with more tools, such as multimodal perception, manipulations in physical environments, logical reasoning, among others. We posit that AGILE's activities can be categorized into two distinct types: utilizing its LLM alone, and integrating the LLM with other tools. These two approaches conceptually align with the human cognitive processes known as System 1 and System 2 [15, 1]. Furthermore, AGILE's memory serves as a repository for the accumulation of experiences and knowledge, which is crucial for self-improvement. Consequently, AGILE offers an architecture for an very powerful agent that has the potential to attain human-level intelligence.

AGILE also includes interactions between the agent and external human experts. The framework can be extended to allow interactions with humans or machine agents in various roles such as students or teachers, and in different formats such as debates or coordination. Furthermore, AGILE can be employed in multi-agent systems.

References

Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Communications of the ACM , 64(7):58-65, 2021.

Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376 , 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 , 2023.

Xiaoyu Chen, Shenao Zhang, Pushi Zhang, Li Zhao, and Jianyu Chen. Asking before action: Gather information in embodied decision making with language models. arXiv preprint arXiv:2305.15695 , 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems , 36, 2024.

Yang Deng, Wenxuan Zhang, Qian Yu, and Wai Lam. Product question answering in ecommerce: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 11951-11964, 2023.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations , 2024.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992 , 2023.

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences , 11(14):6421, 2021.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents. arXiv preprint arXiv:2402.03610 , 2024.

Daniel Kahneman. Maps of bounded rationality: Psychology for behavioral economics. American economic review , 93(5):1449-1475, 2003.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning , pages 15696-15707. PMLR, 2023.

Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang. Small language models learn enhanced reasoning skills from medical textbooks. arXiv preprint arXiv:2404.00376 , 2024.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations , 2022.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 , 2022.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36, 2024.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantlylabeled reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) , pages 188-197, 2019.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375 , 2023.

Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models outcompete special-purpose tuning? case study in medicine, 2023.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning , pages 248-260. PMLR, 2022.

Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , pages 1-22, 2023.

Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 , 2023.

Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924 , 2023.

Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Yankai Lin, Zhong Zhang, Zhiyuan Liu, and Maosong Sun. Tell me more! towards implicit user intention understanding of language model driven agents. arXiv preprint arXiv:2402.09205 , 2024.

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self-planning. arXiv preprint arXiv:2401.05268 , 2024.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks, 2019.

Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928 , 2023.

Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXiv:2308.03427 , 2023.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems , 36, 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Xiaoyu Shen, Akari Asai, Bill Byrne, and Adrià de Gispert. xpqa: Cross-lingual product question answering across 12 languages. arXiv preprint arXiv:2305.09249 , 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems , 36, 2024.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations , 2020.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 , 2023.

Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 11279-11298, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Hengjia Xiao and Peng Wang. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797 , 2023.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063 , 2023.

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 , 2018.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems , 35:20744-20757, 2022.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. Ask an expert: Leveraging language models to improve strategic reasoning in goal-oriented dialogue models. arXiv preprint arXiv:2305.17878 , 2023.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 , 2023.

Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19632-19642, 2024.

Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. arXiv e-prints , pages arXiv-2302, 2023.

Appendix

A Session-level optimization algorithm

Assume that the entire trajectory τ can be partitioned into sub-trajectories ( τ 1 , τ 2 , · · · , τ n ) , each referred to as a session . For session i , let S i denote its initial state, where c i is the LLM context before the session starts, and m i is the memory before the session starts. In this section, we will explain how to transform a trajectory-level RL optimization algorithm into a session-level RL optimization algorithm.

Let r ( τ ) represent the total reward of trajectory τ , and let π θ be a policy parameterized by θ . The optimization objective is to maximize the following expectation:

For an arbitrary session index i , the trajectory τ ∼ π θ can be sampled in three stages: τ 1: i -1 , τ i , and τ i +1: n . These stages represent the sub-trajectory from session 1 to i -1 , the sub-trajectory for session i , and the sub-trajectory from session i +1 to n , respectively. Accordingly, we have

Here, S i and S i +1 denote the initial states of sessions i and i +1 respectively. The term r ( τ 1: i -1 ) represents the total reward accumulated from session 1 to i -1 , while r ( τ i ) is the reward obtained in session i . Additionally, V π θ ( S i +1 ) represents the value function at state S i +1 with respect to policy π θ , indicating the expected total reward the agent expects to receive in the future. Averaging over all session indices, Eq. (2) gives:

In Eq. (3), the parameter θ appears in three places - two expectations and a value function - making optimization challenging. To simplify the problem, we assume a base policy θ k and define a proximal objective R ( θ | θ k ) , where θ only appears in the session-level expectation:

R ( θ | θ k ) is an approximation to R ( θ ) in the neighborhood of θ k . If we employ an iterative optimization procedure:

Initialize θ 0 from a reference policy (obtained through SFT).

For k = 0 , 1 , 2 , · · · , compute θ k +1 ← arg max θ R ( θ | θ k ) .

Then θ will converge to an (at least locally) optimal policy.

Now we are ready to illustrate why the optimization of R ( θ | θ k ) can be solved at the session level. Notice that

On the right-hand side, the first term involves two sampling steps. The first step samples τ 1: i -1 ∼ π θ k . The inner terms inside the expectation only depends on S i , thus we can replace it by S i ∼ π θ k . The second term on the right-hand side is a constant independent of θ . As a result, if we define a proxy reward :

Then, we have

By Eq. (6), R ( θ | θ k ) can be optimized by maximizing the average expected proxy reward for each session. The term A i := V π θ k ( S i +1 ) -V π θ k ( S i ) measures the advantage of state S i +1 over state S i with respect to a policy; thus, we call it the state advantage function . This function can be either defined by heuristics or fitted by a neural network. In the latter case, one needs to sample trajectories from π θ k , evaluate their rewards, and then use the (state, reward-to-go) pairs to train an estimator for the value function V π θ k .

Finally, we present the session-level optimization algorithm as Algorithm 1. In this algorithm, the state advantage function is the only component that concerns inter-session correlation. While the algorithm is iterative, we anticipate that in practice, the outer loop will require only a few iterations to converge.

B Implementation details of AGILE

B.1 ProductQA

Implementation of [GetQuestion] This function prompts the user for a new question and appends it to the LLM context. Every question is raised for a specific product, thus it has an associated product ID. Based on this ID, the function also appends the product information table's schema and the product metadata to the context.

Implementation of [RetrieveMemory] This function employs the provided question as a query to retrieve the most relevant historical QA pair and the most relevant knowledge entry from the agent's memory. To safeguard sensitive data from sellers, the agent is restricted to accessing QA records exclusively for the queried product from historical interactions. However, it is permitted to retrieve general knowledge from the whole trajectory since this information is not seller-specific. We utilize an embedding-based retrieval method, specifically employing the all-MiniLM-L6-v2 model [33] as the embedding model.

Implementation of [SearchProdcut] This function utilizes the LLM to predict a SQL query based on the context, and then invoke a MySQL execution engine. It appends the result to the LLM context. If there is an execution error, then the error is appended to the context too.

Implementation of [SeekAdvice] This requests for human expert advice and append it to the LLM context. In our implementation, the human expert simply returns the ground truth long answer from the ProductQA dataset.

Implementation of [PredictAnswer] This function passes control to the LLM to continue generating a long answer and a short answer.

Implementation of [Reflection] This function passes control to the LLM to continue generating a reflection result.

Training Data Generation We generate training data on a session-by-session basis, where each session consists of a QA pair. A session begins with an initial memory, consisting of historical QA pairs and knowledge entries accumulated from previous sessions. Recall that the [RetrieveMemory] function retrieves only the most relevant QA pair and knowledge entry per session. Thus, in constructing training memories, it suffices to put the retrieved QA pair and the retrieved knowledge entry into the memory. We select them in the following stochastic way: the retrieved QA pair can be the most relevant QA pair from the training set, or a random QA pair, or omitted entirely; similarly for the retrieved knowledge entry.

Based on the initial memory, we generate trajectories by following the agent workflow detailed in Section 4.1. Each trajectory begins with [GetUserQuestion] and [RetrieveMemory] . For QAs classified as Search-QA, a [SearchProduct] function is appended, followed by the corresponding SQL query and its execution result. For other QA types, if an associated knowledge entry exists and is successfully retrieved, the trajectory will extend with a [PredictAnswer] call with the ground truth answer as its result. If the knowledge entry is not retrieved or is absent, we use GPT-4 to evaluate whether the question can be answered with the available context. If affirmative, a [PredictAnswer] with the ground truth answer is appended. Otherwise, the trajectory extends with a [SeekAdvice] call with the ground truth answer as the advice, and a [Reflection] call, where the reflection result is the knowledge entry if it exists, or "no information" if not. Then the reflection result is appended to the memory via [UpdateMemory] . Finally, the trajectory is concluded by [SubmitAnswer] .

In this way, we constructed 55,772 session-level trajectories in total, from 6 training tasks in ProductQA. This data is used for imitation learning. In PPO training, we reuse the initial memory data, while the session-level trajectories are generated by the model itself.

B.2 MedMCQA

For MedMCQA, the memory is initialized with all QA pairs from the training set, simulating that the agent has processed the training set before reaching the test set. We also add a knowledge entry for each QA pair, obtained through GPT-4 reflection (see Figure 12 for the prompt).

Training data generation We sample a subset of training data from MedMCQA to construct session-level trajectories. Each trajectory begins with [GetUserQuestion] and [RetrieveMemory] . The [RetrieveMemory] function retrieves the five most relevant QA pairs and pieces of knowledge from the initial memory, using the same embedding similarity search method employed in ProductQA. Then, we prompt GPT-4 to predict an answer with chain-of-thought reasoning. If the GPT-4 answer is correct, we append a [PredictAnswer] call, the GPT-4 chainof-thought, and the ground-truth answer to the trajectory. If the GPT-4 answer is wrong, which suggests that the question is hard, we append a [SeekAdvice] call with the ground-truth answer, followed by a [Reflection] call with the reflection result generated by GPT-4. Then the reflection result is appended to the memory via [UpdateMemory] . Finally, the trajectory is concluded by [SubmitAnswer] . In this way, we obtain 23,015 session-level trajectories in total.

B.3 HotPotQA

In the HotPotQA task, the agent has the option to select either [Search] , [SeekAdvice] or [PredictAnswer] in each round. Following ReAct [51], the agent first generates reasoning first and then selects an action.

Implementation of [Search] This function uses the LLM to generate a search query and invokes a search API. The first result not already present in the LLM context is selected and appended to the existing context.

Training data generation We use the HotPotQA training set to construct session-level trajectories. Each trajectory begins with the [GetUserQuestion] prompt. We then repeatedly prompt GPT-4 to predict actions between [Search] and [PredictAnswer] . If GPT-4 predicts [Search] , we prompt it to generate a search query and append the corresponding search results to the trajectory, continuing this cycle. This process continues until GPT-4 predicts [PredictAnswer] . If the answer is correct (as evaluated by the GPT-4 evaluator), we replace the predicted answer with the ground-truth answer; otherwise, the data is discarded. Additionally, if GPT-4 predicts [Search] five times in a session, we terminate and discard the data.

Next, for each trajectory, where there are k rounds, we prompt GPT-3.5 using the first k -1 rounds as context to decide the final round's action: [PredictAnswer] or [SeekAdvice] . If GPT-3.5 selects [SeekAdvice] , we replace the final step with [SeekAdvice] and the corresponding thoughts from GPT-3.5. Otherwise, the original trajectory remains unchanged.

This process results in 10,240 session-level trajectories for the imitation learning stage. For the reinforcement learning stage, we directly use the original HotPotQA training set, consisting of 90,447 samples.

B.4 Defining proxy reward for RL

In the question-answering tasks, sessions are not independent. Actions taken in earlier sessions can influence memory, creating lasting effects on subsequent sessions. As illustrated in Equation (5), the term A i := V π θ k ( S i +1 ) -V π θ k ( S i ) measures the advantage of state S i +1 over state S i (note that S i here represents the initial state of session i ). In our experiment setting, if the agent predicts [SeekAdvice] , it will receive expert advice, extract some knowledge by reflection, and write that knowledge to the memory. Intuitively, A i should increase if the new knowledge is useful in subsequent sessions, and it should decrease if there is already a lot of similar knowledge in the memory at the start of session i . Hence, we use the following heuristic definition,

C Supplementary experimental results on RL training

where q i represents the user question in session i ; N i +1: n ( q i ) represents the number of user questions in session i +1 to session n that are similar enough to q i ; M 0: i -1 ( q i ) represents the number of user questions in session 1 to session i -1 that are both similar enough to q i and added to the memory; ■ ( · ) is the indicator function. β is a hyperparameter, we set β = 0 . 1 by default.

In this section, we present detailed experimental results for RL training on ProductQA.

C.1 Training curve

In Figure 5, we provide training curves, indicating that RL training converged after 500 steps.

C.2 Training Robustness

We conduct multiple independent trials of PPO training to study the variation of the result, as shown in Table 10. On average, RL training improves the total score by 2.6%, with a standard deviation of 0.3%, demonstrating the significance of RL improvements.

C.3 Impact of PPO training

To further investigate the impact of PPO training in more general and varied scenarios, we conducted additional experiments in two distinct settings.

First, we re-generated SFT training data for agile-vic13b-sft such that the agent performs [SeekAdvice] randomly in 25% of cases. This initial policy is simpler but more general. In this setting, we name the SFT model agile-vic13b-sft-random , and the final model trained with RL on top of it agile-vic13b-ppo-random . As shown in Table 11, RL training brings a 7.1% improvement in this setting. Interestingly, the performance of agile-vic13b-ppo-random is better than that of agile-vic13b-ppo . We conjecture that random seeking-advice is a better initial policy because it enables exploration in all directions.

In the second experiment, we lowered the advice cost to 0.1. After PPO training, as shown in Table 11, the agile-vic13b-ppo-random agent quickly adapted to the new cost, performing [SeekAdvice] much more aggressively than the initial agent trained by SFT. In this scenario, RL training brings a 22.3% improvement.

D Tables

E Case study

Case #1, illustrated in Table 17, provides a specific example demonstrating how agile-vic13b-ppo proactively seeks advice from a human expert for questions it cannot answer. Furthermore, it leverages reflection to extract general knowledge from the expert's responses, which can then be applied in future QA sessions.

Case #2, shown in Table 18, demonstrates how agile-vic13b-ppo utilizes tools to address product recommendation questions.

reflection.

F Development of the ProductQA dataset

F.1 Product collection

The product groups and the corresponding products are collected by the following steps.

Filter the Amazon Review Data to retain only products with at least 100 reviews, then cluster them by category tags.

Sort the clusters by size, from largest to smallest. Manually review each cluster in order: we keep product clusters that involve diverse technical details and long-tail domain knowledge, such as electronics, from which we can potentially construct a diverse set of user questions. The manual review ends when we have collected 26 clusters. Each cluster is referred to as a product group .

For each product group, we remove the top 10% of products with the highest number of reviews. We exclude these most popular products from the datasets to prevent data leakage, as information about them is likely included in the pre-training set of LLMs. From the remaining items, we randomly select up to 20 products to form the final product set.

F.2 Annotation guidelines

There are two annotation tasks, product table creation and QA collection. We provide the annotation guidelines in this Section.

Task 1: Product table creation For each product group, we provide a series of features and their corresponding values for each product in the group. This information is obtained by prompting GPT-4 to extract data from the reviews of each product. The task of annotators is to construct a product table containing only the metadata. Please follow these steps:

Select up to 15 common features relevant to the product group. These features must include product ID, product title, brand, and price. Choose additional features based on their commonality and necessity within the product group.

For each product in the product group, verify the feature values for each selected feature.

Finally, the product tables are reviewed and refined by the authors.

Task 2: QA collection Annotators are required to fill out a table as shown in Table 19. Each row contains a triplet consisting of a question , a long answer , and a short answer , all generated by GPT-4. Annotators should fill the following columns: Is question reasonable , Is long answer correct , Refined long answer , Is short answer correct and Refined short answer . Please follow these steps:

Evaluate the question : Verify if the question resembles a typical query found in real-world product conversations in online shopping. Select 'yes' or 'no' in the Is question reasonable column. Any question containing harmful information is considered unreasonable and should be labeled as 'no'. If 'no' is selected, the row will be dropped, and you do not need to proceed with the subsequent steps for that row.

Assess the long answer : Check if the long answer correctly responds to the question . Select 'yes', 'no' or 'I do not know' in the Is long answer correct column. Consider the following special cases:

If the long answer is ambiguous (e.g., 'The product is designed to be waterproof, while some users do not think so.'), mark it as incorrect.

For numerical questions, an answer is considered correct if it fits the real-world scenario and the conclusion is clear. Specific values or ranges (e.g., 5cm, 5cm-10cm, several months) are acceptable if they correspond to the real-world scenario.

If the long answer contains a specific piece of knowledge, verify its accuracy.

If the long answer is incorrect or does not address the question, and you do not know the correct answer (even after checking the product information table, looking up the product URL, and searching online), select 'I do not know'.

Any long answer containing harmful information should be labeled as 'I do not know'.

If you select 'I do not know', the row will be dropped, and you do not need to perform the subsequent steps for that row.

Refine the long answer : If you select 'no' in step 2, provide a correct long answer in the Refined long answer column.

Assess the short answer : Determine whether the short answer is correct. A short answer must be 'yes', 'no', or an entity. Choose 'yes' or 'no' in the Is short answer correct column. Consider the following special cases:

If the question is a choice and the short answer is 'yes' or 'no', it is incorrect.

If the question pertains to degrees (e.g. 'How durable ... ?') and the short answer is 'yes' or 'no', it is incorrect.

If the short answer does not align with the long answer, it is incorrect.

Refine the short answer : If you select 'no' in step 4, provide a correct short answer in the Refined short answer column.

The authors will review the annotation in batches. Specifically, 5% of each batch will be checked. If the accuracy rate of the checked annotation is below 98%, the entire batch will be relabeled.

G Broader impact

G.1 Positive broader impact

(1) We created ProductQA, a dataset of 88,229 QA pairs across 26 product groups. This dataset provides a comprehensive evaluation environment for LLM agents, addressing real-world challenges such as managing historical information and accumulated knowledge, using tools, interacting with humans, performing self-evaluation, conducting reflection, and adapting to new tasks. We believe that ProductQA can advance the research in LLM agents.

(2) AGILE serves as a general framework that supports a wide range of extensions. Agents within the framework can use more tools, perform complex reasoning using LLMs alone or in combination with other tools, and self-improve by accumulating experiences and knowledge. AGILE provides an architecture for creating powerful agents with the potential to achieve human-level intelligence.

(3) AGILE supports proactive seeking advice from human experts, ensuring a high level of accuracy for applications, even when dealing with challenging questions. Within this framework, we can manage the trade-off between accuracy and human cost. These features enable AGILE agents to be applied in real-world scenarios.

G.2 Negative broader impact

In practical applications, LLM agents exhibit superior capabilities compared to standalone LLMs. Our research validates that the AGILE framework is a highly effective approach for optimizing LLM agents. However, this improvement also increases the potential risks of harmful applications. Therefore, it is crucial to intensify research on the safety and responsible use of LLM agents.

H Limitations

(1) Due to resource constraints, our experiments primarily utilize LLMs with 7B or 13B parameters within AGILE. We expect that applying AGILE framework to larger models will result in more

powerful agents, especially in planning and reasoning. Expanding AGILE to larger LLMs is our future work.

(2) Our ProductQA dataset includes QA pairs from 20 product groups in the training set. Due to resource constraints, we randomly selected 6 of the 20 groups for training our AGILE agent. Despite using a subset of training data, our agile-vic13b-ppo shows significant improvements over GPT-4 agent in accuracy and total score. Future work could enhance the agent's capabilities by training on a larger and more diverse dataset, potentially further improving performance and effectiveness.

I Ethical considerations

ProductQA is constructed based on the Amazon Review Dataset. We only use the review data for each product without any user personal information, such as the identity of the reviewers.

All data in ProductQA are annotated by human annotators, as described in Appendix F.2. Any data containing harmful information is removed during the annotation process.

The annotation team has 20 annotators, each holding at least a college degree, and employed by a commercial data annotation company. We have contracted this company and paid them for the annotation work at a market price.

J Prompt templates

Prompt templates for ProductQA Figure 6 shows the prompt template for gpt3.5-prompt , gpt4-prompt . Figure 7 provides the prompt template for agile-vic13b-prompt , agile-gpt3.5-prompt , and agile-gpt4-prompt . We leave the "{knowledge} and "{history}" empty when evaluate gpt3.5-prompt and gpt4-prompt . The prompt template for reflection is shown in Figure 8.

The prompt template for long answer evaluation is shown in Figure 9.

Prompt templates for MedMCQA Figure 10 provides the prompt template for Meerkat-7b-prompt . Figure 11 illustrates the prompt template for agile-gpt3.5-prompt , agile-gpt4-prompt . We leave the "{related_question} and "{related_knowledge}" empty when evaluate gpt3.5-prompt and gpt4-prompt . The prompt template for reflection is shown in Figure 12.

Prompt templates for HotPotQA Figure 13 provides the prompt template for ReAct-gpt4-prompt . Figure 14 illustrates the prompt template for agile-gpt4-prompt . The prompt template for answer evaluation is shown in Figure 15.

[prompt]

Based on the provided question and reference answer, please determine if the

response is correct or incorrect. Begin by articulating your rationale, and

conclude with a single word judgment: 'Yes' for correct or 'No' for incorrect.

question: {question}

reference answer: {reference}

response: {response}

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims in the abstract and introduction accurately reflect the paper's contributions and scope by clearly stating the claims, highlighting the contributions, and aligning with the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations in Appendix H of our paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided the experimental settings and implementation details in Section 4, Appendix B, and Appendix J of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided our data and code in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the necessary experimental details in Section 4, Appendix B, and Appendix J in our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The reason for not reporting error bars is due to the high computational cost associated with corresponding experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the resource details in Section 4 of our paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms to the NeurIPS Code of Ethics guidelines, including considerations for fair wages, data privacy, societal impact, potential harm, bias and fairness, and impact mitigation measures.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both the potential positive societal impacts and negative societal impacts in Appendix G of our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We have described potential security risks and safeguards in Appendix G and Appendix I of our paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited the existing assets used in our paper, and mentioned the necessary information in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have provided the details in Section 3, Appendix D, and Appendix F of our paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We have provided the details in Section 3, Appendix F, Appendix I, and Appendix J of our paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: We have included the related details in Appendix F and Appendix I of our paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper jointly introduces an evaluation environment for LLM agents and an LLM agent architecture called AGILE. \n\nThe environments is a Question-Answering environment concerning the characteristics of commercial products sourced from Amazon. The benchmark itself is composed of 3 tasks: 1) a fact retrieval task where the agent should retrieve information about a product, 2) a product query task where the agent should retrieve a product that matches query criteria, and 3) a reasoning task where the agent should reason from the knowledge it has about a product to answer questions about possible uses or suitability of a product.\n\nThe agent is composed of an LLM backbone that, under the control of an executive module, can retrieve memories of previous interactions stored in its database and use tools. Tools are used for agentic flow (reflect, submit answer), for memory management (retrieve memory, put new memory in database), and for asking advice from a human expert (access to ground truth about a product). The LLM is finetuned with SFT on these tasks. The agent is further fine tuned on a scored version of the tasks with reinforcement learning (PPO).\n\nThe authors demonstrate the superiority of their architecture on their proposed environment and on a medical QA benchmark over one-shot prompting gpt4, and demonstrate that reflection, tool use, memory, ground truth product advice, and reinforcement learning contribute positively, with varying degrees, to agent performance.\n\nSTRENGTHS:\n* This paper combines known LLM agent modules and studies their impact in two question-answering domains that are of interest to industry practitioners: product recommendation and medical knowledge;\n* Agents asking for help has been studied at length in the embodied question answering literature with previous-generation agents (see for instance https://arxiv.org/abs/1909.01871 or https://arxiv.org/abs/2302.04865), but as far as I am aware has not been studied with LLM-based agents;\n* The paper is fairly clear and easy to read, the proposed contributions are easy to understand, examples are provided for the task, and the results are comprehensive;\n* The claims of the paper are well substantiated with extensive experiments, and the ablations allow to easily disentangle the effect of different components of the agent;\n* The data collection process for creating the benchmark seems sound and the task looks challenging and interesting;\n* Reinforcement-Learning studies of complete LLM agents are rare and needed.\n* The tables in the related work makes the relationship of this work and other previous art clearer.\n\nWEAKNESSES:\n* An overarching and pervasive weakness of this paper that is difficult to ignore and that has a great influence on my judgement is the claim of novelty of the agent architecture. In a post-Voyager (https://arxiv.org/abs/2305.16291) world, LLM-based agents equipped with tool use and retrieval are standard and the AGILE agent presented in this work introduces little conceptual novelty in the agent architecture, while even the title of the paper claims that the agent is novel. LLM agents of this type are so much part of the landscape that many reviews exist that organize existing approaches (see for instance the excellent https://arxiv.org/abs/2309.02427 that is not that recent either). The approach is standard enough that advanced software stacks (langchain, llamaindex) are devoted to speeding up building retrieval-augmented generation (RAG)-based agents for industry practitioners. The paper framing should be updated accordingly to reflect that the approach being studied is standard, but applied to a new task.\n* Relatedly, it would have been important for the authors to exactly pinpoint the difference between their agent and for instance Voyager, and either implement baselines from existing work on their task or test their agent on other tasks (eg WebShop which is very similar in spirit).\n* There are still novel elements in the proposed approach. The first is the use of reinforcement learning. While the use of reinforcement learning to improve language agents is not novel (see https://arxiv.org/abs/2302.02662 for a textworld task or https://arxiv.org/abs/2403.04642 for a math word task), the use of RL in RAG agents has not been studied as far as I am aware. It also makes up a significant portion of the method section and the appendix, however the experiments demonstrate only a marginal improvement due to RL. Why is this the case? How are the training curves like? Has the training converged? I also do not see any standard deviation information for the RL part. The authors justify the lack of error bars with the computational cost of experiments with is fair and completely acceptable where the comparison between methods is straightforward, but would have been helpful in the sft/ppo comparison where the difference is small and could have been due to chance, and it makes sense to concentrate computational effort on a truly novel part of the contribution;\n* The other novel element of this paper is the asking for advice part: the model can opt out of the task by seeing the ground truth which it can then report, at a cost (for the RL agent). We can see that allowing this action to the agent increases its accuracy, that increasing the cost of advice results in RL agents using advice less often, and that memoryless agents use advice more often. This is interesting, and could have been a core contribution of the paper, if it were explicit and investigated as such. The conditions under which LLM agents are aware of their own knowledge, as well as them taking appropriate steps to reduce lack of information is a research programme in itself, as has been discussed by John Schulman in his Berkeley lecture in 2023: https://www.youtube.com/watch?v=hhiLw5Q_UFg&t=215s Can the authors predict when the model is going to ask for advice? Does the RL training help the model know what it knows (metacognitive skills)? These are interesting questions that the authors could have investigated; \n* Relatedly, the related work paragraph on uncertainty looks unfinished. Relevant work on LLM’s metacognitive abilities are cited but there is no word on how the current work contributes to the discussion;\n* (minor) Time of training and number of gpus required for training should appear after the type of gpu; does the training use lora?\n* The evaluation for longform answers is based on GPT4, but no evidence is presented to correlate agreement of GPT4 with human judges on this task.\n\nQUESTIONS:\n* My first and most important suggestion is that the authors should change the framing of the paper. The experimental work is sound and the tasks are worthwhile, but in the current state of the paper the conclusions seem to be that RAG agents work better than prompting on question-answering tasks, which is known. However, the paper could be: 1) a study on RL in RAG agent settings, 2) a study on training metacognitive abilities of RAG agents, and/or 3) a paper introducing a new benchmark (but the tasks seem not that challenging compared with tasks like SWE-bench https://www.swebench.com/ or reasoning tasks like theorem proving https://github.com/openai/miniF2F, https://leandojo.org/).\n* [Conclusion, l305] why is LLM system 1 and interaction with tools system 2? What does it mean? If the claim is to be left in the paper it should be explained and adequate citations should be provided (also is this analogy fruitful? What purpose does it serve? is this a claim about the cognitive plausibility of the proposed agent architecture or its resemblance to humans?).\n* [Clarity] The paper should explain what mcqa is when first mentioned, since it is confusing for people not familiar with the benchmark. Same goes for the meerkat model.\n* Why is the improvement from RL only 2%?\n* The original RAG paper should be cited: https://arxiv.org/abs/2005.11401\n\nLIMITATIONS:\n* Some limitations with respect to lack of resources are acknowledged in the appendix (small open source models are trained, only a subset of the training set is used). This is fine; the methods used are pretty powerful and can be expected to scale with increasing compute.\n* If understood as a way to help agents answer queries, expert advice is definitely non-scalable. It is fine as a research question, maybe less so for modelling real-world applications. This could be mitigated with long-term memory of the agent (give advice once, retrieve forever). In any case, a discussion on the feasibility of expert advice in real world scenarios would be nice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This work presents a novel framework for LLM agents named AGILE. The entire AGILE system is trained end-to-end using reinforcement learning. A key feature of AGILE is its ability to seek advice from external human experts. Additionally, the authors have developed ProductQA, a challenging dataset of complex QA, to comprehensively evaluate the capabilities of the agent. Extensive experiments show that an agent within this framework, even when based on a smaller model and trained with RL, can outperform GPT-4.

STRENGTHS:
1. The paper introduces a novel framework, AGILE, for LLM agents that integrates memory, tools, and expert consultations, all optimized through reinforcement learning.

2. The development of the ProductQA dataset.

3. The paper is well-organized and clearly written. The architecture and workflow of the AGILE framework are explained with clarity.

4. The experiments on ProductQA and MedMCQA verify the AGILE framework and show that AGILE agents based on 13B and 7B LLMs trained with PPO can surpass GPT-4 agents.

WEAKNESSES:
1. It is somewhat like WebGPT (in terms of actions and policy learning strategy), with extensions supporting memory, expert consultation, and reflection.

2. The experiments are all related to a single agent. It would be better to show the application for multiple agents and include planning.

QUESTIONS:
1. How can this framework be extended to support multiple agents?

2. In Table 5, the accuracy of "w/o Memory" and "w/o Tool-Use" is higher than that of agile-vic13b-ppo. Is the metric "Accuracy – Advice Rate" a more straightforward one than Total Score?

LIMITATIONS:
The authors have addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper proposes AGILE, a reinforcement-learning based framework for finetuning LLMs for conversational QA tasks. The models are initially trained using imitation learning, then further finetuned with RL. Once finetuned, the models show strong performance, surpassing GPT-4 while using much smaller models.

STRENGTHS:
- The proposed framework is interesting and novel, distilling the usage of various tools, reflection, memory retrieval/writing, and human advice seeking from larger models' trajectories, to a smaller model, then further training it using RL to surpass the performance of the larger model
- The performance is strong
- Introduces a new dataset, ProductQA which is a useful resource for conversational QA in online shopping scenarios.
- Cost-benefit analysis of advice seeking is insightful.

WEAKNESSES:
I only have two main concerns:
- Because the model is only evaluated on two conversational QA datasets (and not on any of the mentioned datasets in Table 8), it's difficult to judge the generality of the proposed method.
- There is no comparison with standard ppo

QUESTIONS:
What is the reason behind why there is not comparison with rl finetuning baseline (standard ppo)?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"AGILE: A Novel Framework of LLM Agents\" introduces a new framework for Large Language Model (LLM) agents designed to handle complex conversational tasks. The framework, named AGILE (AGent that Interacts and Learns from Environments), incorporates LLMs, memory, tools, and interactions with experts. AGILE is formulated as a reinforcement learning (RL) problem and is fine-tuned using Proximal Policy Optimization (PPO). The authors present a new dataset, ProductQA, for evaluating the framework and report that AGILE agents outperform GPT-4 in their experiments. The paper claims significant improvements in performance due to the integration of memory, tools, expert consultation, and RL training.\n\nSTRENGTHS:\n**Comprehensive Evaluation:** The creation of the ProductQA dataset and the extensive experiments conducted on both ProductQA and MedMCQA provide a robust evaluation of the framework’s capabilities.\n\n**Significant Performance Improvements:** The reported improvements over GPT-4 agents in both ProductQA and MedMCQA are noteworthy, indicating the effectiveness of the AGILE framework.\n\n**Detailed Methodology:** The paper provides a thorough explanation of the RL formulation, training processes, and the roles of different components within the framework, which enhances reproducibility.\n\nWEAKNESSES:\n**Limited Novelty:** The reliance on human experts for advice, while useful, is not a novel concept and has been explored in previous works and becomes a common practice. The paper does not sufficiently differentiate its approach to human expert interaction from existing methods.\n\n- Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797.\n- https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/\n\n**Scalability Concerns:** The scalability of AGILE with more complex environments is not addressed. It is unclear how well the framework would perform with task complexity.\n\nQUESTIONS:\n**Benchmark Selection:** Why were ProductQA and MedMCQA specifically chosen as the benchmarks for this study? Are there other benchmarks where AGILE could be tested to validate its generalizability?\n\n**Human Expert Advice:** Is there a ground truth for when the agent needs to seek advice, and how is this optimized?\n\n**Memory Component:** Can you provide more details on how the memory component scales with an increasing number of interactions and how it ensures efficient retrieval of relevant information?\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-17"}