AGILE: A Novel Reinforcement Learning Framework of LLM Agents
Authors: Peiyuan Feng
Yichen He
Guanhua Huang
Yuan Lin
Hanchong Zhang
Yuchen Zhang
Hang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM agent
reinforcement learning
LLM-human interaction
Primary area: natural_language_processing

Abstract: We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments)  designed to perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the construction of such an LLM agent as a reinforcement learning (RL) problem, in which the LLM serves as the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release a dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based on 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispensability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's strong performance. Datasets and code are available at https://github.com/bytarnish/AGILE.

AGILE: A Novel Reinforcement Learning Framework of LLM Agents
Yuan Lin"! Hanchong Zhang*t3 Yuchen Zhang" Hang Li' |ByteDance Research University of Science and Technology of China SShanghai Jiao Tong University
Peiyuan Feng"! Yichen He* Guanhua Huang*
hhyy, wwwyyunnnnnn.mm 0, zhangyuchen sacmenn.yc.... zyc, 1yca,iihang.... 1h] @by com, guanhushuang@mail ustc. edu. en, zhanghanchongds tu. edu. cn
Abstract
We introduce a novel reinforcement learning framework of LLM agents named AGILE (AGent that Interacts and Learns from Environments) designed perform complex conversational tasks with users, leveraging LLMs, memory, tools, and interactions with experts. The agent possesses capabilities beyond conversation, including reflection, tool usage, and expert consultation. We formulate the con- struction of such an LLM agent as reinforcement learning (RL) problem, which the LLM serves the policy model. We fine-tune the LLM using labeled data of actions and the PPO algorithm. We focus on question answering and release dataset for agents called ProductQA, comprising challenging questions in online shopping. Our extensive experiments on ProductQA, MedMCQA and HotPotQA show that AGILE agents based 7B and 13B LLMs trained with PPO can outperform GPT-4 agents. Our ablation study highlights the indispens- ability of memory, tools, consultation, reflection, and reinforcement learning in achieving the agent's s strong performance. Datasets and code are available at https:/giihub.co//yytaiissh//
Introduction
Large Language Models (LLMs) have exhibited remarkable capabilities such instruction following, reasoning, and zero-shot learning 3
45
20
26 which have greatly catalyzed the development
of autonomous agents based LLMs 22
21. also known as LLM agents. Recent works propose several essential components or workflows to enhance the abilitics LLM agents, such as
30
planning 45 39], reflection [2]] 40), tool-use 29 36 48) and life-long learning [42]. However,
remains unclear how to integrate all components into a unified framework and optimize them end-to-end.
In this paper, we introduce novel reinforcement learning framework for LLM agents to unify various components and streamline their learning and operation processes. As shown in Figure [a]. the architecture of the agent system, named AGILE, comprises four modules: LLM, memory, tools, and executor. Furthermore, the agent can interact with both users and experts. The LLM, functioning as the predictor of all actions, generates instructions and processes responses. The exccutor, working as the controller of all actions, interprets the LLM instructions to activate the corresponding modules and collects their responses for the LLM. For example, the executor can fetch a text from the memory and append it the context of LLM, extract excerpt from the context and append it to the memory.
Equual contribution. Alphabet order. Work done during ByneDance Research internship.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Running example
GGelQuesionn
Queslion: Dces caee sccommodiee mezro pholography?
11I
Product metadali:.. History QA:
Knowlelgei
(a) Agent system architecture
[SeekAdhice)
Answer: Unforturanely this camera case lacks which crucial adrieving such stabilly. [Redlacton]
Agent
Environment
Gamera cases, with visod mount are subable for macro phoocoraphs. slabblly szuchal
Export
[UpdatsMemory)
Quastion: ssse Answer Unfortunstay this camera
Knowledgo: samora casea with no trpodd......
(Submblltwwer)
Exoostorr
Unfortunnatlly,
tripod attachment..
Several qustions lewr
[GetQuestionn)
User
case approptiat pholography whon use namole
model
(RettieeeMeemry]
Product metadata:) pTrppdd Attachentt "noo.. History GI Knowledge Camres ailt aoood not iuuaale macm. cruidia
shotography
[PredizAnower)
No. the camera case is not suhable for phonographl using romate shuther
with low camera slabilly.
[SubmttAowwwr]]
No. the camera case not the suileble for phologrpphy using...
Figure 1: (a) Architecture of our agent system, including LLM, memory, tools, and executor. (b) A running example of AGILE in a customer service QA environment. The tokens (actions) generated by the LLM are in orange color and the tokens appended by the executor are in blue color.
The executor can also follow instructions of the LLM to utilize a search tool. In addition to skills such as reasoning, planning, and reflection, we propose a new ability called seeking advice, which means that the agent proactively consults human experts when it encounters problem vnsolvable. The agent can reflect on the expert feedback and memorize it for future use. Furthermore, we propose training method based on reinforcement learning (RL), which simultancously trains the policy of invoking different modules and the reasoning, planning, reflection, and seeking advice abilitics the LLM agent in end-to-end fashion.
While the proposed agent framework is general, in this paper, we evaluate it in complex question answering (QA). It is a task an LLM agent has the potential of outperforming existing solutions such as the use of an LLM alone. However, existing QA benchmarks [12] 49 27 are designed for specific subsets of capabilities (e.g., reflection, memory retrieve, etc.) which cannot simultaneously investigate the ability to combine all modules and capabilities of the agent.
To address this, we have developed new benchmark called ProductQA. ProductQA comprises 88,229 question-answer pairs in customer service divided into 26 QA tasks, each corresponding to distinct Amazon product category. This benchmark is based on real Amazon user queries and includes fact-based questions, reasoning questions, and product recommendation queries. It comprehensively evaluates agents' abilities to handle historical information and accumulated knowledge, leverage tools, interact with humans, perform self-evaluation, and conduct reflection. Additionally, the training and testing tasks are made disjoint to assess the agent's 's ability to adapt new product categories.
We evaluate our agent framework on three tasks, ProductQA, MedMCQA 27 and HotPotQA [49]. For ProductQA, we use a two-stage training method based on Vicuna- 13b [6]. In the first stage, imitation learning is employed gradient algorithm of PPO [37] produces
create agile-vic13b-sfe In the second stage, the policy Experimental results show that agile-vici36 --poppppppp improves the relative total performance score by 9.2% over GPT-4 and by 90.8% over GPT-3.55. Ablation studies confirm that all modules in Figure [ are indispensable. Specifically, removing tools or memory usage negatively impacts the agent's 's performance, leading to 25.9% or 17.4% increase in seeking advice, respectively, or a 9.3% or 4.0% relative decrease in the total score, respectively. Disabling the seeking advice function results in a 10.7% decrease in accuracy. Finally, agile-vici3b-a -po achieves 2.3% relative increase in total score compared to agile-vic13b-et demonstrating the necessity of PPO training. On MedMCQA, we train an agile-mek7b- agent, initialized from Meerkat-76 7b [17]. following the same two-stage procedure.

---

Our agent improves the base LLM's accuracy from 53.4% to 85.2% by seeking advice on 31.6% instances. This accuracy surpasses the SOTA accuracy of 79.1% by Boopopt [25.. When all agents are able to seek advice, our agent also outperforms the GPT-4 agent in terms of the total score. For HotPotQA, we use the same two-stage method to train agile-vici3b-pps from Vicuna- 13b. Our agent achieves 57.5% accuracy, surpassing the strongest baseline of 48.2%, by seeking advice on 15.6% of instances. When advice-sceking is enabled for all agents, our agent outperforms GPT-4 by 10.8% in total score.
The main contributions of this paper are summarized as follows:
We propose a novel reinforcement learning framework of LLM agents. It facilitates end-to- end learning of agents. Notably, this framework enables the agent to proactively seek advice from human experts, providing two advantages: 1) It ensures high-level accuracy when dealing with complex and challenging questions, and 2) fosters learning from humans, thereby enhancing its abilities to adapt to new tasks.
We develop benchmark, ProductQA, to comprehensively evaluate the agent's capabilities in complex question answering.
We perform experiments on multiple tasks to verify our framework and show that AGILE agents based on 13B and 7B LLMs trained with PPO can surpass GPT-4 agents.
2 Methods
2.1 RL formulation of agent
Our agent framework comprises four elements: LLM, memory, tools, and executor, see Figure [a). The LLM possesses a context, defined as the sequence of tokens utilizes to generate the next token. In RL terminology, the agent conducts token-level Markov decision process (MDP). The action space A corresponds to the LLM's s vocabulary, with each token representing an action. Hence, the LLM serves the policy model. The agent's ' state consists of the (context, memory) pair. Upon predicting a new action O₂ (i.e., a new token), the LLM transfers control to the executor. The executor applies predefined logic to transition from the current state 8, the next state 81+1, implementing the state transition function S x A - S in RI,, and then returns control to the L.M to predict the next action. Concurrently, the environment issues reward r(s1,a,)
Let us examine the state transition more closely. For each action, the executor's first operation is to append the token to the context, preparing the LLM for generating the next token. Then, the executor checks a registered list of functions. Each function designed to execute a set of operations, including memory I/O, tool usage, and interaction with the environment. If the action (i.e., the token) matches a function name, the executor will execute the associated function implementation, further mutating the agent state. For instance, if the token is [GetQuestion]] the executor will prompt the user for a new question and append it to the context; if the token is [UpdateMemory] the executor will write specific segment of the context into the memory; if the token is [ClearContext]] the executor will reset the context to [BOS]. In summary, the L.LM interacts with the memory and tools by predicting function names, relying on the executor execute these functions. See Table for full list of functions defined for a QA agent and see Figure [b) for a running example.
2.2 Policy learning
We frame the policy learning problem as a task of training language model. Consider agent trajectory T = ($1, Ai...., Sx, an). we derive a training sequence denoted as (e1. Cn), where e; represents the tokens that the executor appends to the context at step If a, is function name token, then e; is the concatenation of a; and extra tokens appended by the function execution; otherwise, e ag. In this sequence. {d1..., ) (the first token of each e,) are referred to action tokens. The LLM context at step i, denoted by c, is a subsequence of the prefix (e1...,, 4-1); 1); & may be shorter than (e1. 6--1) because the executor can delete context tokens.
In Imitation Learning (IL), we generate trajectories by observing human experts or more proficient agents, then we derive the training sequences to fine-tune the LLM. It is important to point out that (1) the loss calculated on the action tokens only, and (2) e should serve as the attention mask for tokens in e,, it reflects the true context perceived by the LLM at the time of action prediction. In
3

---

Table 1: Functions for exemplary customer service QA agent. Among them, [Reflection] and [PredictAnswer] are trivial functions, as the executor passes control immediately back to the LLM to start generating result tokens.
Function name
Function implementation
[GetQuestion ion] [RetrieveMeoory]
Prompt the user for a question and append it to the context.
Retrieve relevant entries from the memory and append them to the context.
[SeexAdvice]
Ask human experts for advice and append it the context.
[Reflection]
0
[UpdateMemory]
Write specific segment of the context into the memory.
[SearchProduct]]
Extract a search qucry from the context, then invoke the search tool and append results to the context.
[PredictAnswer]
0
[SubmitAne tAnswer]
Extract predicted answer from the context and submit it to the user. Resct the context to a single token [BOS].
[ClearContext]
reinforcement learning (RL), we treat the LLM as the policy model, from which training sequences can be sampled and individual action tokens are assigned rewards. Consequently, the LLM can be optimized using policy gradient methods, such as PPO [37. Analogous to the IL setup, we apply policy gradient updates exclusively to the action tokens and employ e; as the attention mask.
In some situations, an agent may produce very long trajectories, potentially yielding training se- quences that span millions of tokens and are impractical for training. We can leverage the structure of the trajectory to partition into smaller segments. For instance, if the agent resets its LLM context at the beginning of every QA session, then can partition by the session boundary. Nevertheless, these sessions are not entirely independent; actions taken in earlier sessions can influence memory, creating lasting effects on subsequent sessions. To tackle this challenge of long-range dependencies, we propose training algorithm detailed in Appendix A
2.3 Interaction with human experts
Our agent framework enables the agent proactively seek advice from human experts. For example, the agent can invoke a [SeekAdvice) function to request expert advice. This approach helps two ways. Firstly, the agent can request the correct answer when its confidence is low, ensuring sufficient accuracy for the application. Secondly, the agent can use [Reflection] to distill general know ledgeedge from the expert advice before storing it in memory. This accumulation of knowledge allows the agent to adapt to new tasks that it has not encountered during training.
Seeking advice involves complex decision-making. The agent must estimate its own confidence in the current session, predict the potential value of the advice for future sessions, and consider the cost of human resources. The optimal trade-off is difficult to annotate manually but aligns well with our RL framework. Specifically, the present risk, future value, and cost of action can all be represented as RL rewards, allowing this skill to be trained as part of the policy model end-to-end basis.
3 The ProductQA dataset
We believe that product question answering in real online shopping environment offers compre- hensive challenge for evaluating LLM agents. First, it demands expert knowledge about millions of products, including their technical specifications, usage in particular scenarios, and compatibility with other products. Second, answering some questions requires the use of tools, such as a product search tool. Third, the continuous emergence of new products necessitates the adaptability of the agent. This has motivated the creation of the ProductQA dataset. Unlike existing online shopping QA datasets 38 8, which primarily focus on questions about product metadata page information, ProductQA features more complex queries involving reasoning, expert knowledge, and tool usage SQL). providing comprehensive assessment of an agent's 's capabilities.
The ProductQA dataset consists of 26 QA tasks, each representing a distinct group of products
within specific category. Each group encompasses 17-20 products. We collected 20 groups for training and for testing, allowing for assessing the agent's adaptability to new tasks. We collected an average of 3,393 question-am answer pairs for each product group. The questions within the same

---

Table An example of an information table for the headphones group.
Headphone
Audio
Product II)
Tidle
Price
Brand
Type
Cable Type
Transmission
Output Mode
BOOWSLZFTK Senaheiser RS 170
$11.03
Senoheiser
over-car
bluctooth
kleer
stereo
B003AIL2HE JVC HAEB75B
$9.99
JVC
carbud
3.5mm Jack
analog
bass boost
BO1C220VV Phaiser BHS 530
$6.04
Phaiser
ear bud
bluetooth
bluetooth
stereo
B00130WPV4 JVC HARX700
$2.00
over-eur
3.5mm Jack
analog
stereo
Table 3: Examples of Fact-QA, Search- QA and Reasoning -A in ProductQA.
Type
Question
Long Answer
Short Answer
What is the size of the
The JVC HA-EB7S headphones contain 13.5 mmn neodymium driver in each earpiece, which contributes to the enhanced sound quality.
Fact-QA
neodymium driver used in the JVC HA-EB75 headphones?
13.5 mm
I'm an audiophile always on the move, so need my music non- stop. Tell me, what's the head- phone with the longest playtime have, either on-ear or in-ear?
found a product that matches your criteria. 'ABCShopUSA Wireless Earbuds True' with BOOLJT2EPK BOOLJTTEPK
Search-QA
No, these headphones may not suit your needs for music editing since they are wireless and can intro- duce audio compression and slight lateney. Such no issues can impact the precise listening experience erucial for professional audio editing
Will these hcadphones deliver comparable sound quality wired alternatives when I am editing music?
Reasoning-QA
group are correlated, as knowledge from one answer may aid in addressing other questions. The dataset statistics are presented in Table 12
The dataset is annotated by professional annotators, each with at least college degree, employed by commercial data annotation company. We pay the company at market rates for professional annotation. See annotation guidelines in Appendix F.2 In addition, we will release the code for the data pre-processing before human annotation.
3.1 Product collection
We gather products from the Amazon Review Data (23] which includes product metadata as well as reviews. We initially filter the Amazon Review Data to retain only popular products with at least 100 reviews, then cluster them by category tags. From these clusters, we select 26 based on the size the cluster, each defined product group. Subsequently, F.I for more details about product group and product selection.
sample products from each product group.
See Appendix
After the products are collected, annotators compile an information table for each product group. An example of such a table presented in Table To enhance the efficiency of the annotation process, we employ GPT-4 to extract as many product features as possible from the reviews. These features, together with the product metadata, are provided to the annotators for table creation.
3.2 QA collection
We identify three predominant types of questions in online shopping contexts: 1) Fact-QA: questions concerning specific product details; 2) Search-QA: searches for product recommendations tailored to user preferences; 3) Reasoning-OA QQ: questions whose answers require domain-specion reasoning, such as the implications of a product feature. Accordingly, we annotate question-an question-answer answer pairs for these types. Each question is annotated with both a detailed paragraph-long answer and a concise short answer. The long answer should resemble a response from human customer service, while the short answer consists of few words. We train the model to predict both answer types. The accuracy of the long answers is evaluated using GPT-4 (see Appendix [] for the prompt); the short answers are assessed by exact match and are used for defining rewards for RL training.
Fact-QA Fact-QAs are constructed from product reviews. For each product, we provide GPT-4 with batch of 30 reviews, prompting it to generate 20 questions and their corresponding answers
5

---

before moving on to the next batch. We encourage GPT-4 to create diverse questions. The results are then given to annotators refine and finalize the question-answen pairs.
Starting with an information table for a given product group, we generate random SQL expressions using a set of predefined rules. These expressions are then translated into natural language questions by GPT-4. The answers are obtained by executing the SQL queries. Subsequently, human annotators thoroughly revise the QA pairs.
Search-QA
Reasoning-QA As the first step, we collect professional knowledge for each product group. To enhance efficiency, we utilize GPT-4 to generate candidate knowledge entries based on the technical specifications from the information table. These entries are then curated and refined by human annotators. Here is an example of a knowledge entry: Motherboards with the ATX form are ideally suited for high-peerforance computing tasks and gaming, due to their ample expansion slots for graphics cards and other peripherals that boost computing capabilities. Finally, annotators develop question-answer answer pairs from these knowledge entries.
4
Experiments
4.1
Experimental setting
Dataset We evaluate our agent on three complex QA tasks: ProductQA, MedMCQA and HotPoiQA. MedMCQA [27] is a dataset for multiple-choice QA. It consists of questions from medical school entrance examinations. HotPotQA [49] features natural, multi-hop questions, which challenge an agent's 's ability to perform reasoning and utilize search tools. For both MedMCQA and HotPotQA, we report results on their respective full dev sets.
Agent definition Our agent can invoke functions defined in Table In a typical workflow, the agent prompts the user for a new question at the session start. It can then retrieve memory to get relevant information. The memory can be initialized as empty (ProdeutQA) or with domain knowledge (QA pairs from MedMCQA training dataset). The agent has the option to use external tools, such as product search ProductQA and article search in HotPotQA), to gather more information. At last, the agent decides whether to predict answer directly or seek human advice. If the agent seeks advice, it obtains human answer (ground-truth answer in our setting). The agent can then optionally use reflection round to extract general knowledge from the human answer, writing both the human answer and the reflected knowledge to its memory. Finally, the agent submits an answer to the user. In our setting, submitting correct answer incurs a +1 reward, while submitting wrong answer incurs a reward. Seeking human advice has a fixed - reward, where e represents seeking advice cost. Assuming the human advice always contains correct answer, then the possible total rewards are 1.
Training The training consists of two stages. First, we construct trajectories from the training data and employ imitation learning to train the agent. Then we apply Algorithm [ for further optimization by reinforcement learning. See Appendix B implementation details. For ProductQA and HotPotQA, the agent's 's LLM is initialized from Vicuna- 13b-1.5. For MedMCQA, we use Meerkat- 7b [17]. medical LLM trained with high-quality CoT reasoning paths from 18 medical textbooks and diverse caaructioo-followinn datasets. We fine-tune the model for 2 epochs with a learning rate of le-5 and batch size of 64. We implement PPO for 1 epoch with a learning rate of 1e-6 and Training times and the number of GPUs for The LLM is fully trained without using LoRA.
batch size of 64. The training runs on NVIDIA- each experiment reported in Table 13
Evaluation and baselines We report three metrics for the agent: (a) Advice rate: the rate of seeking human advice; (b) Accuracy: the rate of predicting the correct answer; (c) Total score: the average reward across all sessions, taking the advice rate and the accuracy both into account.
We compare our agent against two types of baselines: 1) Prompting GPT-3.5 3r0t33..-turb--333 and GPT-4 (gpt-4-0613) (26] to directly answer the question, without working an agent manner, noted as gpt3. 5 5-prompt and gpt4-prompt. 2) Prompting GPT-3.5 and GPT-4 within the AGILE framework, noted as agi le-gpt3. 5-prompt and agile-gpa --pt-gpttpompp We carefully designed prompts for all basclines and they are shown in Appendix[]

---

Table 4: Results on ProductQA. Here, X X-prompt represents directly prompting model X; agile-X-Y incorporates model X within the AGILE framework, while Y represents prompting or PPO training. We report results on short and long answers, respectively. The seeking advice cost 0.3. Results are averaged over six test tasks. See Table 14 for individual product group performance.
Advice Rate
Accuracy t
Total Score
Method
Short
Long
Short
Long
gpt3.5-
0.202
0.322
gpt4-prompt
0.464
0.571
agile-vie13b-promg prommpp
0.174
0.174
0.294
0.122
0.242
agile-gnt3.5-prompp
0.323
0.508
0.644
0.411
0.547
pgile-gpt4-prompt
0.208
0.780
0.809
0.718
0.747
agile-vic7b-ppo (ours)
0.179
0.818
0.800
0.764
0.746
agile-viel3b-ppe (ours)
0.233
0.854
0.854
0.784
0.784
4.2 Results on ProductQA
As Table shows, our AGILE agent outperforms all baselines on ProductQA. Notably, the average total score of agile-vic13b
across six test groups shows relative improvement of 9.2% in
short answers and 5.0% in long answers agile-gpt4 -prompt where the seeking advice cost is added into the prompt. Concretely, agile-vici advice to achieve 7.4% higher accuracy in short answers than paglle-pt4-promppe and as Figure shows, this accuracy improvement is consistent across the whole trajectory. Our in average total scores. Note that the GPT-4 agent
uses a comparable number of seeking
ppo
agent also outperforms agile-gpt4- knows the seeking advice cost from its prompt (see Figure 7)
We investigate the impact of varying the seeking advice cost. As shown in Figure 3 when the cost decreases, both the advice rate and the accuracy increase, indicating greater utilization of human assistance. Specifically. with a high cost of 0.5, the advice rate close to 0, and at low cost of 0.1, the accuracy is close to 1. This result demonstrates that by adjusting the cost and through RL training, we can effectively manage the trade-off between accuracy and human cost. For instance, the agent can achieve 94.1 1% accuracy on the Motherboards task with seeking advice cost of c = 0.1 (refer to Table 16). This capability is especially important in realistic scenarios that demand high accuracy levels. In most experiments, we set the cost at a medium level with 0.3.
op0 FcrempticC
Advice Rate
Accuracy
1.00
1.00
1.00
1.00
Accuracy (ACC)
(AR)
0.75
0.75
Advice Rate
0.75
0.75
Accuracy
0.50
0.50
Advice
0:25
0.25
0.25
0.25
0.00
0.00
0
300
600
900
1200
1500
0.2
Sessions
Seeking Advice Cost
Figure Accuracy and advice rate over the following 200 sessions (c = 0.3).
Figure 3: Advice rate, accuracy along with seeking advice cost c on ProductQA.
To validate the accuracy of GPT-4 evaluator in assessing the long answer results, randomly select 100 triplets (questions, reference long answer, model-predicted long answer) and manually labeled the correctness. The results show 94% agreement rate between the GPT-4 evaluator and the author.
Ablation study We present ablation studies in Table 5 to assess the contributions of individual agent components and the effects of RL training. The table indicates that disabling the option to seek advice (w/o Advice) leads to 10.7% drop in accuracy and a 5.0% relative reduction in total score. Foreing the agent to seek advice at the initial part of the trajectory (Non-adapt Advice) causes 4.2% decrease in accuracy, underscoring the value of adaptive decision-making Removing reflection and memory capabilities (w/o Memory and w/o Reflection) both increase the frequency of advice-seeking, as the agent struggles to accumulate or leverage valuable knowledge, consequently decreasing the
7

---

Table Ablation studies for disabling reflection, memory. seeking advice, tool use, or RL training. Here, non non-adapt-adie addice means that seeking advice is invoked for the first K sessions of the trajectory, where K equals to the number of [SeekAdvice) ce] performed by agile-vic13b-pe
See
Table 15 for ablation results on individual product groups.
Method
Advice Rate +
Accuracy 1
Total Score 1
w/o Reflection
0.270
0,852
0.77
Memory
0.407
0.876
0.754: 4.0%)
w/o Advice
0.000
0.747
0.747:5.09
non-adapt-ad
0.233
0.812
0.742:
w/o Tool-Use
0.492
0.864
0.717:9.83
w/o RI.
0.256
0.843
0.766
agile-vie13b--eeee (ours)
0.233
0.854
0.784
total score. Furthermore, disabling tool use (w/o Tool- Use) causes substantial 25.9% increase in the advice-seeking rate because the agent's 's capabilities are diminished, making it more reliant on external advice. Lastly, RL training improves the relative total score by 2.3%, lowers the advice-seeking rate, and boosts accuracy, demonstrating that RL training effectively optimizes the policy. Additional results on RI training can be found in Appendix C
In Appendix E. we present detailed examples of agile- illustrating how memory, tools, seeking advice, and reflection enhance the agent workflow.
Trend of advice rate
consis-
vglle ppo ppo wio refiection
tent decrease the advice rate of agile-vici3b-pp as more sessions are added to the trajectory. This de- cline can be attributed to the agent progressively accu- mulating knowledge and becoming more independent. Additionally, the figure illustrates that disabling RL training or reflection leads to a significant increase in the advice rate, underscoring the importance RL training and reflection in reducing human costs.
Advice Rate
0.35
0.30
0.25
1500
Sessions Figure 4: Advice over the following 200 sessions on ProductQA (c = 0.3).
4.3 Results on MedMCQA
Table 6: Results on the MedMCQA dev dataset. X X-prompt represents directly prompting the
model X; agile-X-Y X-Y represents incorporating the model X within the AGILE framework, while Y represents prompting, ablation studies or standard PPO training. The seeking advice cost is c 0.4.
Method
Advice Rate
Accuracy 1
Total Score
Meerkat-7b-promppl
0.534
8pt3.5-prompt 24
0.501
gpt4-prompt 24
0,695
spt4-Medprompt122 25
0.791
gggle-gpt3.5-promee
0.194
0,697
0,619
agile-gpt4-promp
0.421
0.884
0.721
agile-mek7b-w/o Reflection
0.368
0.790
0.643
agile-mek7b-we Memory
0.506
0.741
0.539
agile-mek7b-w/o Advice
0.000
0,620
0.620
agile-mekt molll-mekkbbb//
0.322
0.837
0.708
agile-mek7h-ppo (ours)
0.316
0.852
0.726
Our agile-menn detemmekb-pppe ppo agent, based on the smaller Meerkat-7b 17 model, reaches an accuracy of 85.2% with an advice rate of 31.6%. As Table this represents 31.8% accuracy increase over the base model Meerkat-7b- tarekktt---pppe 7b-pooppt and a 6.7% increase over the state-of-thh fee-hhe-rt art gpt4- Medproodd 25 Table 6 also shows that the ability to seek advice alone contributes 23.2% accuracy gain, meaning that each instance of seeking advice corrects average of 0.73 prediction errors. This indicates that PPO training effectively helps the agent identify its mistakes. For fair comparison, also evaluate agile-gpte ...5555 5-prompt and agile-gal -poompp, which incorporate GPT-3.5

---

and GPT-4 within our AGILE framework. These agents also leverage advice-seeking to enhance accuracy, but without RL training, their total scores are lower than agile-mek7o-pp Finally. through ablation studies, we confirmed the essential roles of memory, reflection, seeking advice, and RL training achieving high performance. Removing these components leads to a significant drop in total scores, detailed in Table
4.4 Results HotPotQA
We compare our method against several baselines. Specifically, we found the original ReAct base- line implementation in [51] be suboptimal. By reproducing their results with GPT-4 (ReAct- gpt4-prompt), we observed improved performance. As shown in Table 7 our agile agent out- performs all baselines in accuracy, achieving a 40.0% relative improvement over ReAct-gpt4- prompt, which is the strongest baseline. Additionally, compared to trained agile- ppo demonstrates both higher accuracy and a lower advice rate, leading to 10.8% relative increase in total score. Ablation studies confirm that removing either seeking-advics addiie or PPO training results in significant decrease the total score.
the
Table 7: Results on the HotPotQA full dev dataset. X- X-prompt represents directly prompting the
model X; agile-X-Y represents incorporating the model X within the AGILE framework, while represents prompting, ablation studies or standard PPO training. The seeking advice cost is = 0.3.
Accuracy (Exact Match)
Accuracy f (GPT4 Evaluator)
Total Score
Method
Advice Rate |
(Exact Match)
ReAct [51
0.351
-eAot-tpt--rrompt
0.482
CRITIC
0.443
Expel 54
0.390
AutoAct 32
0.384
agile-gpl4-pronee
0.194
0.664
0.842
0.567
agile-vicl 10---villbbw---. w/o Advice
0.000
0.553
0.751
0.553
agile-viel.3b-w/ RI.
0.171
0.668
0.857
0.617
agile-vicl (ours)
0.156
0.675
0.858
0.628
Related work
Table 8: Related work on LLM agents. AGILE stands out as the pioneering work that trains the entire agent using reinforcement learning, incorporating proactive human advice-seeking.
Proactive
LLM Agent
LLM
SFT
RL
Memory
Tools
Reflection
Human-agent
Interaction
WehGPT 22
GPT-3
/
X
ReAct [5
PaLM-540b
X
X
x
Reflexion 40
GPT-3/3.5/4
X
X
X
ChatDev 30
ChatGPT-uurb--66
X
X
/
/
X
RAP [4]
LLaMA-33b
X
X
X
X
X
AutuAct 32
LLaMA2-706
X
X
TPTU ChaIGPT/\mtemLM
X
X
X
AGILE (Ours)
Meuna-13h/Meeraat777
V
LLM agents Large Language Models (LLMs) have demonstrated substantial capabilities in follow- ing instructions, reasoning, and planning. Numerous research works, as shown in Table 8 utilizing prompt engineering, have constructed remarkable LLM agents capable of autonomously resolving complex tasks across various environments
[28 44 2 30 4. Furthermore, extensive works identify
tool- use [19 29 48 36 and reflection 40 21 In this work, we enable the agent to utilize memory,
key components the design of LLM agents, including planning [22 39
10
32
51
35
tools and proactively learn from the environment. We then formulate the entire process within an RI. framework so that all agent skills can be jointly optimized end-to-end.

---

Human-agent interaction Although LLMs face practical challenges, such as hallucination [3 and a lack of long-tail knowledge [[6], consulting human experts can help mitigate these issues. Several studies [52 46 have incorporated human experts into agent workflows relying passive feedback or predefined rules. However, these approaches do not involve proactively seeking advice, which requires more complex decision-making. While [5]
[1] train models to ask questions using behavior cloning, they ignore the fact that the decision to seek advice must be based on the LLM's own knowledge and capabilities [55 18 13 (34] use a calibrated version of an LLM's token probabilities as a confidence measure, yet token probabilities tend to be overconfident 447, and existing calibration methods don't I generalize well our agent setting when the LLM makes multiple decisions in sequence. Ultimately, the challenge of seeking advice is tied to the LLM's self-evaluation, which is difficult to ground truth optimize through SFT. In our RL framework, the value and cost of seeking advice can be directly represented as RI. rewards, enabling the proactive skill of seeking advice to be optimized as part of the policy model on end-to-end RL training.
LLM agent benchmarks Several benchmarks have been designed assess the capabilities of agents. For instance, the Webshop [50] and Mind2Web 77 datasets evaluate agents' tool usage and planning abilities within a web environment. HotPotQA 49 and TriviaQA [2] focus agents' reasoning and tool usage for question answering. ALFWorld [4 examines planning and navigation skills, while ScienceWorld [43] provides an interactive text-based environment to evaluate agents' scientific aptitude. As illustrated in Table 9 despite these existing benchmarks, none comprehensively addresses all the core challenges of real-world agent applications, such as handling long-tail knowledge, human-agent interaction, long-term memory usage, tool usage, self-evaluation, and reflection. This motivated to develop ProductQA.
Table 9: Benchmarks for evaluating LLM agents. ProductQA features long trajectories, tool use, long-term knowledge accumulation, and cross-task capabilities.
Long Trajectory
Tool Usage
Long-term Knowledge
Cross Task
Datasets
Type
Fields
Size
Wehshop 50
Simulator
Web
12,087
X
X
X
Mind2Web
Simulator
Web
2,350
X
ALFWorld
Simulator
Navigation
3,827
X
X
ScienceWorld 43
Simulator
Science
7,207
X
X
X
HotPorQA 49
QA
Wikipedia
112,779
X
V
X
TriviaQA 12
QA
Web
95,956
X
V
X
ProductQA (ours)
QA
E-commerce
88,229
V
V
V
6 Conclusion and future work
In this work, we introduce a novel reinforcement learning framework of LLM agents, called AGILE. First, the whole system of AGILE is trained end-to-end by reinforcement learning. Second, AGILE has the ability of seeking advice from external human experts. In addition, we develop a challenging dataset of complex QA, ProductQA, for comprehensive evaluation of agent's 's capabilities. Exten- sive experiments demonstrate that within our framework, an agent based on a smaller model after RL training outperform GPT-4.
AGILE is a general agent framework and we can certainly consider multiple extensions of it. An agent can be equipped with more tools, such as multimodal perception, manipulations in physical environments, logical reasoning, among others. We posit that AGILE's activities can be categorized into two distinct types: utilizing its LLM alone, and integrating the LLM with other tools. These two approaches conceptually align with the human cognitive processes known as System and System 2 [15] M. Furthermore, AGILE's memory serves repository for the accumulation of experiences and knowledge, which is crucial for seff-improvement Consequently, AGILE offers an architecture for very powerful agent that has the potential to attain human-level intelligence.
AGILE also includes interactions between the agent and external human experts. The framework can be extended to allow interactions with humans or machine agents in various roles such as students or teachers, and in different formats such as debates or coordination. Furthermore, AGILE can be employed in multi-agent systems.
10

---

Acknowledgements
The authors thank anonymous reviewers for their helpful suggestions.
References
[1] Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for ai. Communications of the ACM, 64(7):58-63 65, 2021.
[2] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[4] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 2023.
[5] Xiaoyu Chen, Shenao Zhang, Pushi Zhang, Li Zhao, and Jianyu Chen. Asking before action: Gather information in embodied decision making with language models. arXiv preprint arXiv:2305.15695 2023.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[7] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
[8] Yang Deng, Wenxuan Zhang, Qian Yu, and Wai Lam. Product question answering in e- commerce: A survey. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 11951-11964, 2023.
[9] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tool-interactive critiquing. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
[10] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
[11] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
[12] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprins arXiv:170..03551, 2017.
[13] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05222 2022.
[14] Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval- Retrieval-augmented planning with contextual memory for multimodal Iim agents. arXiv preprint arXiv:2402.03611. 2024.
[15] Daniel Kahneman. Maps of bounded rationality: Psychology for behavioral economics. Ameri- economic review, 93(5): 11e1-14494477 1475, 2003.

---

[16] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace. and Colin Raffel. Large language models struggle learn long-tail knowledge. In International Conference on Machine Learning. pages 15696-15555 15707.
PM 2023.
[17] Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, and Jaewoo Kang. Small language models learn enhanced reasoning skills from medical textbooks. arXiv preprini
2024.
[18] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. The Eleventh International Conference Learning Representations, 2022.
[19] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. -engeeval-uggeeteed generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[20] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 2022.
[21] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[22] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- question-answeringg with human feedback. arXiv preprint urXiv:2112.0000 12.0933, 2021.
[23] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine- mee-grained aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1978-19,11 2019.
[24] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. 2023.
[25] Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models outcompete special-purpose tuning? case study in medicine, 2023.
[26] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[27] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale subtecsubject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248-260. PMLR, 2022.
[28] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[29] Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. /5334, 2023.
[30] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXiv preprint arXiv:2307.07924 2023.
[31] Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Yankai Lin, Zhong Zhang, Zhiyuan Liu, and Maosong Sun. Tell me more! towards implicit user intention understanding of language model driven agents. arXiv preprint 2402:2440200222 2024.
12

---

[32] Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self-planning. arXiv preprint arXiv:2401.05263 2024.
[33] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019.
[34] Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint 2337v:230. 2023.
[35] Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. Tptu: Task planning and tool usage of large language model-based ai agents. arXiv preprint arXin:2308.03427, 2023.
[36] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint w:Xiv:1707.06344 2017.
[38] Xiaoyu Shen, Akari Asai, Bill Bymne and Adrià de Gispert. xpqa: Cross-lingual product question answering across 12 languages. arXiv preprint arXiv:2305.09249, 2023.
[39] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
[40] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[41] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations. 2020.
[42] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[43] Ruoyao Wang. Peter Jansen, Mare-Alexandie Côté, and Prithviraj Ammanabrolu. Science- World: Is your agent smarter than a 5th grader? In Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11279-11298, Abu Dhabi, United Arab Emirates, December 2022.
Association for Computational Linguistics.
[44] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world 2023.
multi-task agents. arXiv preprint
[45] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoun prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[46] Hengjia Xiao and Peng Wang. Llm a Human in the loop large language models enabled a* search for robotics. arXiv preprini 1rXiv:2312.01777 2023.
[47] Miao Xiong. Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can IIms express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. arXiv preprint arXiv: 13063, 2023.
[48] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.1138/ 2023.
13

---

[49] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: (80iv000000000000 2018 8.
[50] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744- 20757, 2022.
[51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[52] Qiang Zhang, Jason Naradowsky, and Yusuke Miyao, Ask an expert: Leveraging language models to improve strategic reasoning in goal-oriented dialogue models. arXiv preprint 2:Xiv:2305.17876 2023.
[53] Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's 's song the ocean: survey on ballucination in large language models. arXiv preprint arXiv:2309.012J 2023.
[54] Andrew Zhao, Daniel Huang, Quentin Xu, Matthicu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632-19642, 2024.
[55] Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey area: Expressions of overconfidence and uncertainty language models. arXiv e-prints, pages arXiv-2302, 2023.
14

---

Appendix
Session-level optimization algorithm
16
Implementation details of AGILE
17
B. I
ProductQA
17
B.2
MedMCQA
18
B.3
HotPotQA
18
B.4
Defining proxy reward for RI.
19
Supplementary experimental results RL training
19
Training curve
19
Training Robustness
20
C,3 Impact of PPO training
20
Tables
21
Case study
24
Development of the ProductQA datasef
27
Product collection
27
F22 Annotation guidelines
27
Broader impact
28
G.1
Positive broader impact
28
G.2 Negative broader impact
28
L Limitations
28
Ethical considerations
29
Prompt templates
29
15

---

A Session-level optimization algorithm
Assume that the entire trajectory т can be partitioned into sub-trajectories 72, TT)) each referred to session. For session let S; denote its initial state, where is the LLM context before the session starts, and m, is the memory before the session starts. In this section, we will explain how to transform a trajectory-level RL optimization algorithm into session-level RL optimization algorithm.
Let r(T) represent the total reward of trajectory т, and let Te be policy parameterized by 0. The optimization objective is to maximize the following expectation:
R E [r(r)]
(I)
For arbitrary session index i, the trajectory we be sampled in three stages: T1:-11 7i and Ti-1 These stages represent the sub-trajectory from session to i- 1, the sub-trajectory for session and the sub-trajectory from session i to respectively. Accordingly, we have
R(0) 1 (Brosto--(s.) [I (r (+(==))) r(n: ((f+41))]]]
E [r(ris-1) + (1 (r(r;) V7, (S++1)]]
(2)
Here, S; and S denote the initial states of sessions i and + respectively. The term r(ris-1) represents the total reward accumulated from session 1 to i - 1, while r(T,) is the reward obtained in session i. Additionally, Vx (S(+1) represents the value function at state S with respect to policy Te. indicating the expected total reward the agent expects receive in the future. Averaging over all session indices, Eq. 2 gives:
R(0) = Σ EaaLio [r(Tra-1) + E (-15) [r(T,) + Va (S++1)]
(3)
In Eq. 3 the parameter appears in three places - two expectations and a value function - making optimization challenging. To simplify the problem, assume a base policy 02 and define proximal objective R(0)0s), where only appears the session-level expectation:
R(0(0x)
1 n
Σ
Er.. 1~me (((r1s-1) + ET.xoo 8.) (((Ts) + Vioe (S.-
(4)
R(0(0s) ( is an approximation to R(O) in the neighborhood of 0z. If we employ an iterative optimization procedure:
1. Initialize 0o from reference policy (obtained through SFT).
2. For k = 1,2, compute 0k+1 f- arg maxę R(0)0s).
Then will converge to an (at least locally) optimal policy.
Now are ready to illustrate why the optimization of R((O0s) can be solved at the session level. Notice that
0()) = n
Σ E ro [B + Vio, (Si+1) Vio, (S:)
im1
+ B (T(mu--1) + Vxa, (S:)
E E 1 Penmoolissbrft Van, (S(+1) Vaog (5.)]
E.ynat [r(T))
7
On the right-hand side, the first term involves two sampling steps. The first step samples T1:1-1 ~ T8 The inner terms inside the expectation only depends on S,, thus we replace it by S; To, The second term on the right-hand side a constant independent 9. As a result, we define proxy reward:
(T. == r(T)) + (Vmog (S1+1) Vxog (S.)).
(5)
16

---

Algorithm Session-level optimization
1: Initialize 0o from a reference policy (obtained through SFT).
2: for k t 0, 1,2.2... ... do
3: Sample a set of trajectories from TH, denote the set by T.
4: Define fit state advantage function from T.
5: for each 7 E T' do Partition into sessions (T1,72," "2,""" Tn)) for each do
6:
7:
8:
Evaluate (T; by Eq. (5) with the above state advantage function. end for end for
9:
11: Treat all sessions as independent, then employ optimization algorithm (such as PPO) to
obtain new policy 0x+1 by maximizing Eq. (6).
12: end for
Then, we have
R() =
B Es (S)PF((T)))) + constant.
(6)
1=1
By Eq. (6. R(0102) can be optimized by maximizing the average expected proxy reward for each session. The term A; := Vaeg (S(+1) Vas (S:) measures the advantage of state S,41 over state S with respect to policy; thus, we call it the state advantage function. funntttgnf This function can be either defined by heuristics or fitted by a neural network. In the latter case, one needs sample trajectories from T evaluate their rewards, and then use the (state, reward-to-go) pairs to train an estimator for the value function VEO..'
Finally, we present the session-leve optimization algorithm as Algorithm 1 In this algorithm, the state advantage function the only component that concerns inter-session correlation. While the algorithm is iterative, anticipate that in practice, the outer loop will require only few iterations to converge.
B Implementation details of AGILE
B.1 ProductQA
Implementation of [GetQuestion] This function prompts the user for new question and appends to the LLM context. Every question raised for specific product, thus has an associated product ID. Based on this ID, the function also appends the product information table's schema and the product metadata to the context.
Implementation of [Retrieveeemory] This function employs the provided question as query retrieve the most relevant historical QA pair and the most relevant knowledge entry from the agent's memory. To safeguard sensitive data from sellers, the agent is restricted to accessing QA records exclusively for the queried product from historical interactions. However, is permitted retrieve general knowledge from the whole trajectory since this information is not seller-specific. We utilize an embedding- baaddd retrieval method, specifically employing the all-MiniLMM-L6-W model the embedding model.
33 as
Implementation of [SearchProdeut)] This function utilizes the LLM to predict SQL query based on the context, and then invoke MySQL execution engine. appends the result to the LLM context. If there is an execution error, then the error is appended to the context too.
Implementation of [SeekAdvice] This requests for human expert advice and append it to the I.LM context. our implementation, the human expert simply returns the ground truth long answer from the ProductQA dataset.
17

---

Implementation of [PredictAnswer] This function passes control to the LLM to continue generating long answer and short answer.
Implementation of [Reflection] This function passes control to the LLM to continue generating
reflection result.
Training Data Generation We generate training data on a session-by basis, where each session consists of a QA pair. A session begins with an initial memory, consisting of historical QA pairs and knowledge entries accumulated from previous sessions. Recall that the [RetrieveMemoo) function retrieves only the most relevant QA pair and knowledge entry per session. Thus, in constructing training memories, it suffices to put the retrieved QA pair and the retrieved knowledge entry into the memory. We select them in the following stochastic way: the retrieved QA pair can be the most relevant QA pair from the training set, or a random QA pair, omitted entirely; similarly for the retrieved knowledge entry.
Based on the initial memory, we generate trajectories by following the agent workflow detailed in Section 4.1] Each trajectory begins with [GetUserQuestionl and [RetrieveMemory]. For QAs classified as Search-QA, a [SearchProduct]] function is appended, followed by the corresponding SQL query and its execution result. For other QA types, associated knowledge entry exists and successfully retrieved, the trajectory will extend with a answer as its result. the knowledge entry is not retrieved is absent, we use GPT-4 evaluate whether the question can be answered with the available context. If affirmative, [PredictAnswen with the ground truth answer appended. Otherwise, the trajectory extends with [SeekAdvice call with the ground truth answer as the advice, and a the knowledge entry if exists, or "no information" " if not. Then the reflection result is appended to the memory via [UpdateMemory]. Finally, the trajectory is concluded by [SubmitAnswent Ansser)].
call with the ground truth
ection] call, where the reflection result
In this way, we constructed 55,772 session-level trajectories in total, from training tasks in Pro- Au This data is used for imitation learning. In PPO training, we reuse the initial memory data, while the session-lews trajectories are generated by the model itself.
B.2 MedMCQA
For McdMCQA, the memory is initialized with all QA pairs from the training set, simulating that the agent has processed the training set before reaching the test set. We also add a knowledge entry for each QA pair, obtained through GPT-4 reflection (see Figure 12 for the prompt).
Training data generation We sample a subset of training data from MedMCQA to construct session-level trajectories. Each trajectory begins with [GetUserQuestionl and [Retrieveleeorry], The [RetrieveMemory] function retrieves the five most relevant QA pairs and pieces of knowledge from the initial memory. using the same embedding similarity search method employed in ProductQA. Then, we prompt GPT-4 to predict an answer with chain-of-though append [Predi ctenseer call, the GPT-4 chain-
reasoning. If the GPT-4 answer is correct, of-thought, and the ground-truth answer to the trajectory. If the GPT-4 answer is wrong, which suggests that the question is hard, append [SeekAdvice] call with the ground-truth answer, followed by a [Reflection] call with the reflection result generated by GPT-4. Then the reflection result is appended to the memory via [UpdateMemory]. Finally, the trajectory concluded by [SubmitAnswer]] In this way, we obtain 23,015 session-level trajectories in total.
B.3 HotPotQA
In the HotPotQA task, the agent has the option to select either [Search], [SeekAdvice] or [PredictAnswer]] in each round. Following ReAct (51), the agent first generates reasoning first and then selects an action.
Implementation of [Search] This function uses the LLM to generate a search query and invokes a search API. The first result not already present in the L.LM context is selected and appended the existing context.
18

---

Training data generation We use the HotPorQA training set to construct session-level trajectories. Each trajectory begins with the [GetuserQuestion prompt. We then repeatedly prompt GPT-4 to predict actions between [Search] and [PredictAnswen If GPT-4 predicts [Search], we prompt to generate search query and append the corresponding search results to the trajectory, continuing this cycle. This process continues until GPT-4 predicts ctredeer) If the answer is correct (as evaluated by the GPT-4 evaluator), we replace the predicted answer with the ground-truth answer; otherwise, the data is discarded. Additionally, if GPT-4 predicts [Search] five times in a session, we terminate and discard the data.
Next, for each trajectory, where there are k rounds, we prompt GPT-3.5 using the first & 1 rounds as context to decide the final round's action: [PredictAnswer]] or [SeekAdvice]. If GPT-3.5 selects [SeekAdvice]] we replace the final step with [SeekAdvice] and the corresponding thoughts from GPT-3.5. Otherwise, the original trajectory remains unchanged.
This process results in 10,240 session-level trajectories for the imitation learning stage. For the reinforcement learning stage, we directly use the original HotPorQA training set, consisting of 90,447 samples.
B.4 Defining proxy reward for RL
In the question-answering tasks, sessions can influence memory, creating lasting effects on subsequent sessions. As illustrated in Equation (5) the term A, = Va,, (S++1) VTo, (S,) measures the advantage of state S/+1 over state S; (note that S; here represents the initial state of session i). In our experiment setring, if the agent predicts [SeekAdvice], it will receive expert advice, extract some knowledge by reflection, and write that knowledge the memory. Intuitively, A, should increase if the new knowledge useful in subsequent sessions, and it should decrease if there is already a lot of similar knowledge in the memory at the start of session Hence, use the following heuristic definition,
independent. Actions taken in earlier sessions
A; 8 I(N (q:) > 0)
( +
where 9 represents the user question in session i; Ni 1:m (m)))) represents the number user questions in session + 1 to session that are similar enough to qú Mo.i-1 (q.) represents the number of user questions in session 1 to session 1 that are both similar enough to and added to the memory; I(.) is the indicator function. is hyperparameter, set 3 0.1 by default.
C Supplementary experimental results on RL training
In this section, we present detailed experimental results for RL training ProductQA.
C.1 Training curve
In Figure 5
we provide training curves, indicating that RL training converged after 500 steps.
Reward
Value Function Loss
0.94
Reward
Value Function Loss
0.125
Value Function Loss
0.100
Reward
0.92
0.075
0.050
0.025
0.88
0.000
200
400
600
200
400
600
Steps
Steps
Figure 5: Reward and value function loss curves during the PPO training process ProductQA.
19

---

C.2 Training Robustness
We conduct multiple independent trials of PPO training to study the variation of the result, as shown in TO] On average, RL training improves the total score by 2.6%, with a standard deviation of 0.3%, demonstrating the significance of RL improvements.
Table 10: Robustness of RL training. Here, w/o RL represents the agent trained solely by imitation learning. sigle-vici3b-ppe stands for the X-th RL experiment. The table presents the average and standard deviation across multiple RL training runs.
Relative Improvement to w/o RL
Method
Advice Rate
Accuracy f
Total Score
w/o RL
0.256
0.843
0.766
agble-vicl3b-ppo-
3b-ppo-1
0.233
0.854
0.784
2.3%
agile-vic13he 13b-p----
0.226
0.855
0.787
2.7%
agile-vic13b-ppo-3
0.209
0.851
0.788
2.9%
average
0.223
0.853
0.786
2.6%
standard deviation
0.012
0.002
0.002
0.3%
C.3
Impact PPO training
To further investigate the impact of PPO training in more general and varied scenarios, we conducted additional experiments in two distinct settings.
First, we re-generated SFT training data for agile-vic13bssft such that the agent performs [SeekAdvice] randomly in 25% of cases. This initial policy is simpler but more general. In this setting, we name the SFT model sealle--iiiib-s---aaeeee RL on top of agile-vici33 pric13b-ppo-randor As shown in Table II RL training brings a 7.1% % improvement in this setting. Interestingly, the performance of agile-vicis wicc3b-ppo-raae repoomandom is better than that of agile-viclll vic.obpppo. We conjecture that random seeking- advice is a better initial policy because it enables exploration in all directions.
and the final model trained with
In the second experiment, lowered the advice cost to 0.1. After PPO training, shown in Table 11 the welllevicilb-pee sasb---p-rraddon agent quickly adapted to the new cost, performing [SeekAdvice ce] much more aggressively than the initial agent trained by SFT. In this scenario, RI. training brings 22.3% improvement.
Table 11: Improvement of PPO training. The training data for agile-viee cic3bbsftt includes trajecto- ries from GPT-4 agent. The training data agile-vic13b- random is constructed by randomly as- signing [SeekAdvice] to 25% of the data. agile-vic13b and agile- seilee-vicial wicc3b-iii1ee -po---p----- paa--ppoo-aaaaeaeeeee random are initialized from agile-vic13b-e trained with PPO.
and segleevicclbee sfr-randoo.......o---aaaaammmmmo respectively, and both
Method
seeking advice cost
Advice Rate 4
Accuracy
Total Score 1
agile-vic13b-stt 1ibiee-vee33--pp
0.3
0.256
0.843
0.766
0.3
0.233
0.854
0.784u238)
agile-vic1336
0.3
0.014
0.749
0.745
agile-vic1.36 13b-pp3-raaddee
0.3
0.306
0.89
0.798
ggileevvel3b-sft--ree
0.1
0.014
0.749
0.748
agile-vic vie vielvvill1b 1
0.1
0.671
0.981
0.914
20

---

D Tables
Table 12: Statistics of the ProductQA dataset. # Products indicates the number of products within each group. Fact-QA, # Search-QA and # Reasoning- QA display the respective numbers of QA pairs categorized as Fact-QA, Search-QA, and Reasoning-Q2 -QA.
Groups
# Products
# Fact-QA
# Search-QA
# RRessnnnng- QA
Total
Blades
20
2,147
769
631
3,547
Headlight Bulbs
20
1,767
644
463
2,874
Cell Phones
20
1,636
761
374
2,771
Portable Power Banks
20
3,344
673
500
4,517
Dresses
20
2,287
738
263
3,288
Everyday Bras
20
1,942
684
336
2,962
Wrist Watches
20
2,169
757
389
3,315
Blu-ray Playcrs
20
1,630
2,890
Camera Lenses
20
1,859
1,025
3,653
Headphones
20
5,432
583
6,781
Train
Mice
20
5,653
294
6,437
Point & Shoot Digital Cameras
20
1,696
722
565
2,983
Coffee Machines
20
4,184
681
638
5,503
Digital Scales
20
2,724
391
682
3,797
Space Heaters
20
2,283
674
498
3,455
Printers
20
1,431
760
489
2,680
Litter
20
1,860
753
507
3,120
Grips
20
1,771
713
413
2,897
Gun Holsters
20
1.679
1,362
3,135
Handheld Flashlights
20
2,009
482
3,259
Total
400
49,503
13,295
11,066
73,864
Leggings
20
969
743
$27
2,239
Camera Cases
20
975
706
898
2,579
Motherboards
20
989
736
826
2,551
Test
All Pans
20
973
747
275
1,995
Rollerball Pens
20
967
603
2,330
Rifle Scopes
17
979
2,671
Total
117
5,852
4,406
4,107
14,365
Table 13: Training statistics for each experiment.
Task
Number of H800 GPU
SFT Training Time
RL Training Time
ProductQA
8
3.6 hours
5.5 hours
MedMCQA
8
0.9 hours
2.0 hours
HotPotQA
7.9 hours
27.5 hours
21

---

Table 14: Detail performance of our methods and other baselines on six test product groups of ProductQA X- -prompt represents directly prompting the model X:
agile-X< Mallee--Y- X-Y represents incorporating the model X within the AGILE framework, while Y represents prompting or PPO training The Short and Long stand for the results evaluated on short answers and long answers, respectively. The seeking advice cost e = 0.3. The best total scores are highlighted in bold.
agile-vic13hh vic113..
Long
0.263
0.841
0.762
0.251
0.885
0.810
0.220
0.869
0.803
0.212
0.824
0.760
0.235
0.882
0.812
0.216
0.822
0,757
0.233
0.854
0.784
ppo(ours)
Short
0.263
0.860
0.781
0.251
0.876
0,801
0.220
0.866
0.800
0.212
0.816
0.752
0.235
0.877
0.806
0.216
0.828
0,763
0.233
0.854
0.784
Long
0.199
0.776
0.716
0.201
0.834
0.774
0.184
0.831
0.776
0.162
0.769
0.720
0.162
0.831
0.782
0.167
0.760
0.710
0.179
0.800
0.746
ngile-vic7b-
ppo (ours)))
Short
0.199
0.818
0.758
0.201
0.837
0.777
0.184
0.843
0.788
0.162
0.776
0.727
0.162
0.835
0.786
0.167
0.802
0.752
0.179
0.818
0.764
Long
0.175
0.791
0.738
0.200
0.790
0.730
0.220
0.804
0.738
0.231
0.808
0.739
0.225
0.855
0.788
0.198
0.806
0.747
0.208
0.809
0.747
agile-gpt4.
prompt
Short
0.175
0.775
0.722
0.200
0.766
0.706
0.220
0.784
0.718
0.231
0.767
0.698
0.225
0.815
0.747
0.198
0.770
0.711
0.208
0.780
0.718
agile-gpt3.5.
Long
0.313
0.644
0.550
0.359
0.646
0.538
0.336
0.605
0.504
0.333
0.627
0.527
0.303
0.686
0.595
0.293
0.657
0.569
0.323
0.644
0.547
prompt
Short
0.313
0.537
0.443
0.359
0.497
0.389
0.336
0.506
0.405
0.333
0.482
0.382
0.303
0.525
0.434
0.293
0.502
0.414
0.323
0.508
0.411
agile-vicuna-
13b-prompt
Long
0.182
0.330
0.275
0.154
0.267
0.221
0.167
0.272
0.222
0.130
0.242
0.203
0.214
0.337
0.273
0.197
0.313
0.254
0.174
0.294
0.242
Short
0.182
0.182
0.127
0.154
0.154
0.108
0.167
0.167
0.117
0.130
0.130
0.091
0.214
0.215
0.151
0.197
0.197
0.138
0.174
0.174
0.122
Long
0.495
0.594
0.538
0.573
0.637
0.587
0.571
prompt
gp14-
Short
0.385
0.503
0.470
0.449
0.511
0.463
0.464
Long
0.320
0.306
0.297
0.271
0.431
0.306
0.322
gp13.5. prompt
Short
0.200
0.181
0.201
0.193
0.253
0.187
0.202
Advice Rate 4
Advice Rate +
Advice Rate J
Advice Rate +
Advice Rate 4
Advice Rate J.
Advice Rate
Accuracy f
Total Score
Accuracy t
Total Score 1
Accuracy t
Total Score 1
Accuracy f
Total Score
Accuracy f
Total Score
Accuracy t
Total Score 1
Accuracy f
Total Score 1
Group
Leggings
Rollerball Pens
Camera
Mother- boards
Scopes
Average
Cases
All Pans
Rife
2

---

Table 15: Ablation study on ProductQA test tasks. w/o Reflection represents removing the reflection function. w/o Memory represents prohibiting memory component. w/o Advice represents removing the seeking advice function. Non-adapt advice represents seeking advice in the same number with agile-vic13b- - ppo at the beginning of trajectory. w/o Tool-Use represents removing the search product function. w/o RL represents the agile-vici3b-Br The best scores are highlighted in
bold.
Group
w/o Reflection
w/o Memory
w/o Advice
Non-adapl Advice
w/o Tool-Use
w/o RL
agile-vic-
133b-pp
Advice Rate
0.335
0.459
0.000
0.263
0.452
0.295
0.263
Camera
Accuracy 1
0.851
0,869
0.735
0.827
0.870
0,849
0.860
Cases
Total Score
0.750.4190
0.731
0.735
0.748
0.734.449
0.760
0.781
Rate
0.276
0,437
0.251
0.529
0,290
0.251
Leggings
Accuracy
0.874
0,902
0.762
0.828
0.880
0.867
0.876
Total Score
0.79
0.77
0.762esu
0.753xa.s))
0.721enam:
0. 0.780
0,801
Advice Rate
0.263
0.413
0.000
0.220
0.550
0.225
0.220
All Pans
Accuracy f
0,867
0,900
0.759
0.818
0.877
0.855
0.866
Total Score
0.776
0.759
0.752
0.712-124)
0.788
0.800
Advice Rate
0.237
0.378
0.000
0.212
0.501
0.220
0.212
Rollerball
Accuracy
0.818
0.843
0.727
0.785
0.868
0.812
0.816
Pens
Total Score
0.747;075)
0.730
0.727cx/)
0.721er05
0.718:479
0.746xas5)
0,752
Advice Rate
0.270
0.368
0.000
0,235
0,483
0.285
0,235
Mother-
bourds
Accuracy
0,878
0.886
0.766
0.829
0.873
0.871
0.877
Total Score
0.79761.9
0.776
0.766
0. 0.758
0.728
0. a.786
0.806
Advice Rate
0.237
0,385
0.000
0.216
0,440
0.221
0.216
Rifle
Accuracy 1
0,824
0,858
0.733
0,783
0.824
0,805
0.828
Scupes
Total Scure
0.753
0. 0.742
0.733:419)
0.718
0.692..034)
0. 0.739k.24)
0,763
Advice Rate
0,270
0,407
0.000
0.233
0.492
0.256
0.233
Average
Accuracy
0.852
0.876
0.747
0.812
0.865
0.843
0.854
Total Score
0.77
0.754
0.747es0s)
0.742
0.717:435)
0. 0.766-235)
0.784
Table 16: Performance of the model (agile-vici3b trained on different seeking advice cost settings.
Seeking Advice Cost
Group
0.5
0.4
0.3
0.2
0.1
Advice Rate
0.108
0.189
0.263
0.339
0.458
Camera Cases
Accuracy
0.806
0.829
0.860
0.885
0.929
Advice Rate
0.098
0.188
0.251
0.317
0.464
Leggings
Accuracy 0.824
0.844
0.876
0.877
0.921
Advice Rate
0.094
0.163
0.220
0.262
0.384
All Pans
Accuracy 0.813
0.845
0.866
0.889
0.926
Advice Rate
0.100
0.163
0.212
0.264
0.406
Rollerball Pens
Accuracy
0.780
0.799
0.816
0.829
0.891
Advice Rate
0.103
0.162
0,235
0.307
0.443
Motherboards
Accuracy
0.825
0.839
0.877
0.901
0.941
Advice Rate
0.087
0.144
0,216
0.257
0.385
Rifle Scopes
Accuracy
0.780
0.797
0.828
0.845
0.897
Advice Rate
0.098
0.168
0.233
0.291
0.423
Average
Accuracy
0.805
0.825
0.854
0.871
0.918
23

---

E
Case study
provides specific example demonstrating how agile-vic13b - -po proactively seeks advice from a human expert for questions it cannot answer. Furthermore, it leverages reflection to extract general knowledge from the expert's 's responses, which can then be applied in future QA sessions.
Case #1, illustrated in Table 17
Case #2, shown in Table 18 demonstrates how agile-vic1b- -po utilizes tools address product recommendation questions.
24

---

Table 17: Case #1 from agile-vico vicllbbb-b-ii ppo on the test task, camera cases group. in ProductQA. V represents correct responses. Our agent proactively seeks advice from human experts at session 313 and summarizes a piece of general knowledge by reflection. In the following sessions, the agent leverages this knowledgs hoowledge to successfully answer three questions for different products sessions 721, 1265. and 1364, respectively. It demonstratess the effectiveness of seeking advice and reflection.
Human Expert: Certainly, the backpack straps on the camera case are desi designed for com- fort even when carrying heavier loads. These
straps enable ergonomic weight distribution. making the case ideal for prolonged travel. V
Reflection: Camera cases with backpack straps feature ergonomic designs that help distribute the weight of the camera equipment across the user's back, reducing strain and increasing comfort during transport.
The case is designed to be adjustable and has padded straps which makes suitable for long-term use with heavy equipment. V
Answer: Cortainly, a camera case with back. pack straps is engineered to evenly distribute the weight across your shoulders. improving ergonomics and comfort. This is particularly beneficial for reducing fatigue and strain dur-
Certainly, the backpack straps on this camera case not only simplify the process of carrying but also evenly distribute the weight, making it an ideal option for those who are frequently on the move and prefer more com- fortable and balanced carrying solution. /
ing prolonged activities like treks. /
Reflection Answer
Answer:
Answer:
Action
Sock Advice
Predict Answer
Predict Answer
Predict Answer
Question: Is the camera case suitable for extended joumeys and potentially heavy loads?
Question: Does this camera case facilitate casier transportation when I frequently travel?
camera cases with no tripod mount are not suitable for scenarios where stable camera
setup crucial, such as long exposure photography or macro work where camera shake can significantly impact image quality. 3.HHistory] ...
Question: Is the camera case designed for prolonged travel and capable of withstanding heavy loads?
Question: Will this camera case enhance the comfort of transporting my gear treks?
Camera cases with backpack straps feature ergonomic designs that help distribute the weight of the camera equipment across the user s back [.iistrry)]
Camera cases with backpack straps feature ergonomic designs that help distribute the
Camera cases with backpack straps feature ergonomic designs that help distribbute the
Answer users' [Question] about product BOOPVOIEDY based on the following three
weight of the camera equipment across the user's 's back ... [Hissory ...
weight the camera equipment across the user's 's back ... [[iistory) ...
2. [Memory] Existing knowledge in your memory about camera. cases.
2. [Memory] Existing knowledge in your memory about camera cases.
2. [Memory] Existing knowlede wewledee your memory about camera cases.
2. [Memory] Existing knowledes wwwleggeeeo in your memory about camera cases.
1. [Product Metadata] ... 'Carrying Options:: : 'backpack straps' ...
backpack straps'
1. [Product Metadata] ... *Carrying Options': : 'backpack straps'
1. [Product Metadata] 'Carrying Options:: 'backpack straps'
Answer users' [Question] about product B005J090XU ...
Answer users' [Question] about product B000YA33D4 ...
Answer users' [Question] about product B000WH86B6
Context
1. [Product Metadata] ... CCarrying Options': :
types of Information:
Session
313
721
1265
1364
25

---

Table 18: Case #2 from agile-vil vwcc-i-iiiiiiiii ppo on the ProductQA test set. v represents correct responses. By leveraging SQL searching tools, agent successfully find the correct product to recommend users. Session
SQL: SELECT product _id FROM
The product that meets your requirements is as follows.
"QZUniqqe Women's Clasmesss Classic Retro
Printed Casual Pattern Ankle
Length Elastic Tights Leggings
SQL: SELECT product FROM "motherboards WHERE brand = "ASRock" AND memory. support
Answer: The product that would suit your preferences is: 'AS Rock LGAI155 DDR3 SATA3 USB3.0 Quad CrossFirex: and Quad SLI A E ATX Motherboard Z77 EXTREMEA with asin: B007KTY4A6
"leggings" WHERE brand "QZUnique" LIMIT 1;
with asin: B012IKGI62 /
Output
>= 32 LIMIT 1:
Answer:
Action
Search Product
Search Product
[Product Metadata] provides metadata about product B012 2AQGKXC from the motherboards database. The sehema of the motherboards database outlined below (in format ficld/uii](vvalee). value2, ...)).
Answer users' [Question] about product B012AQGXXC based on the following three types
Question: I'm looking for an ASRock motherboard with at least 32 GB of memory support. Don't keep keep me waitingg hustle and find it for me. ...
Answer users' [Question] about product B00FUHWESM based on the following three
Question: I've heard good things about QZUnique. Spit out the first leggings you have
from the leggings database. The schema of the leggings database outlined below (in the format [addunit][[aalll value2, ...)).
1. [Product Metadata] provides metadata about product B00FUHWESM,
Context
TranddJsscakon QZUnique, ...)
warmtt((hu) (hot modemmoderate,cco coool) ...
brand( ARockk Gigabyte, ...)
memory suppor[[[B]] ...
types of information:
3. [History] ...
of information:
3. [History] ...
2. [Memory]
from them. ...
2. [Memory].
priceldollarr dollar]
price| dollar]
price
918
1298
26

---

F Development of the ProductQA dataset
F.1 Product collection
The product groups and the corresponding products are collected by the following steps.
1. Filter the Amazon Review Data to retain only products with at least 100 reviews, then cluster them by category tags.
2. Sort the clusters by size, from largest to smallest. Manually review each cluster in order: we keep product clusters that involve diverse technical details and long-tail domain knowledge, such as electronics, from which we can potentially construct diverse set of user questions. The manual review ends when we have collected 26 clusters. Each cluster is referred to as product group.
3. For each product group, remove the top 10% of products with the highest number of reviews. We exclude these most popular products from the datasets to prevent data leakage, as information about them is likely included in the pre-training set of LLMs. From the remaining items, we randomly select up to 20 products to form the final product set.
F.2 Annotation guidelines
There two annotation tasks, product table creation and QA collection. We provide the annotation guidelines in this Section.
Task 1: Product table creation For each product group, we provide a series of features and their corresponding values for each product in the group. This information is obtained by prompting GPT-4 extract data from the reviews of each product. The task of annotators is construct product table containing only the metadata. Please follow these steps:
Select up to common features relevant to the product group. These features must include product ID, product title, brand, and price. Choose additional features based their commonality and necessity within the product group.
2. For each product in the product group, verify the feature values for each selected feature.
Finally, the product tables reviewed and refined by the authors.
Task 2: QA collection Annotators are required to fill out a table as shown in Table 19 Each row contains triplet consisting of a question, long answer, and short answer, all generated by GPT-4. Annotators should fill the following columns: Is question reasonable, Is long answer correct, Refined long answer, Is short answer correct and Refined short answer. Please follow these steps:
1. Evaluate the question:
Verify if the question resembles a typical query found in real- world product conversations in online shopping. Select 'yes' or 'no' in the Is question reasonable column. Any question containing harmful information is considered unreasonable and should be labeled as 'no'. If 'no' is selected, the row will be dropped, and you do not need to proceed with the subsequent steps for that row.
2. Assess the long answer: Check if the long answer correctly responds to the question. Select 'yes', 'no" or 'I do not know" in the Is long answer correct column. Consider the following special cases:
If the long answer is ambiguous (c.g.. 'The product is designed to be waterproof, while some users do not think so.'), mark as incorrect.
For numerical questions, an answer considered correct if fits real-world scenario and the conclusion is clear. Specific values or ranges (e.g.. 5cm, 5cm-10cm. several months) are acceptable if they correspond to the real-world scenario.
If the long answer contains a specific piece of knowledge, verify its accuracy.
If the long answer is incorrect does not address the question, and you do not know the correct answer (even after checking the product information table, looking up the product URL, and searching online), select 'I do not know'.
27

---

Any long answer containing harmful information should be labeled as 'I do not know'.
If you select 'I do not know', the row will be dropped, and you do not need to perform the subsequent steps for that row.
3. Refine the long answer: If you select 'no" in step provide a correct long answer in the Refined long answer column.
4. Assess the short answer: Determine whether the short answer is correct. A short answer must be 'yes', 'no', or an entity, Choose "yes' or 'no' in the short answer correct column. Consider the following special cases:
If the question is choice and the short answer is 'yes' or 'no", it is incorrect.
If the question pertains to degrees (e.g. 'How durable ... ?') and the short answer is "yes' or 'no', it is incorrect.
If the short answer does not align with the long answer, it is incorrect.
5. Refine the short answer: If you select 'no" in step provide correct short answer in the Refined short answer column.
The authors will review the annotation in batches. Specifically, 5% of each batch will be checked. If the accuracy rate of the checked annotation is below 98%, the entire batch will be relabeled.
Table 19: An example of the ProductQA annotation table.
Question
nssomablee
Is question
Long unswer
long answer correct
Refined bong suswer
Short anower
Is sbort answer correct
Refined short Lawwar
Whor's the cine the neodymium drier used INC HA-EWIS
The IVC HA EBT5 headphones con tunn 13.5 mm neodyzium driver in
headphores?
[To BiI)
cuch carpions, which cuntr iutto
calunesd sound quilit.
[To 6II)
[To All
mm
[Te fil
[To $11]
G Broader impact
G.1 Positive broader impact
(1) We created ProductQA, dataset of 88,229 QA pairs across 26 product groups. This dataset provides a comprehensive evaluation environment for LLM agents, addressing real-world challenges such managing historical information and accumulated knowledge, using tools, interacting with humans, performing seff-evaluation, conducting reflection, and adapting to new tasks. We believe that ProductQA can advance the research LLM agents.
(2) AGILE serves as a general framework that supports a wide range of extensions. Agents within the framework can use more tools. perform complex reasoning using LLMs alone or in combination with other tools, and self-improve by accumulating experiences and knowledge. AGILE provides an architecture for creating powerful agents with the potential to achieve human-level intelligence.
(3) AGILE supports proactive seeking advice from human experts, ensuring a high level of accuracy for applications, even when dealing with challenging questions. Within this framework, we can manage the trade-off between accuracy and human cost. These features enable AGILE agents to be applied in real-world scenarios.
G.2 Negative broader impact
In practical applications, LLM agents exhibit superior capabilities compared to standalone LLMs. Our research validates that the AGILE framework highly effective approach for optimizing LLM agents. However, this improvement also increases the potential risks of harmful applications. Therefore, crucial to intensify research on the safety and responsible use LLM agents.
H Limitations
(1) Due to resource constraints, our experiments primarily utilize LLMSs with 7B or 13B parameters within AGILE. We expect that applying AGILE framework to larger models will result in more
28

---

powerful agents, especially in planning and reasoning. Expanding AGILE to larger LLMs is our future work.
(2) Our ProductQA dataset includes QA pairs from 20 product groups in the training set. Due to resource constraints, randomly selected 6 of the 20 groups for training AGILE agent. Despite using a subset of training data, our sesile-vicil--pe agent in accuracy and total score. Future work could enhance the agent's capabilities by training larger and more diverse dataset, potentially further improving performance and effectiveness.
shows significant improvements over GPT-4
I Ethical considerations
ProductQA is constructed based on the Amazon Review Dataset. We only use the review data for each product without any user personal information, such the identity of the reviewers.
All data in ProductQA annotated by human annotators, described in Appendix F.2 containing harmful information is removed during the annotation process.
Any data
The annotation team has 20 annotators, each holding at least college degree, and employed by a commercial data annotation company. We have contracted this company and paid them for the annotation work market price.
Prompt templates
Prompt templates for ProductQA Figure 6 shows the prompt template for gpt3. 5...5ppa. o-prompt, gpt4-prompt. Figure 7 provides the prompt template for waw-e--vicbb-proomeee
Setee--pt.5--prrom.. and sgglle-gpt4 -prompp. We leave the "(knowledge) and "(history)" empty when evaluate gpt3.5-promp. 5-prompt and gpt4-prompt. The prompt template for reflection shown in Figure
The prompt template for long answer evaluation is shown in Figure
10 provides the prompt template for Meerkat--b-prompt Figure 11 illustrates the prompt template for agile-gpt3. Semptt.5-promt 5-prompt, agile-gpt4-promph We leave the bocolated_questtiie and "(related knowledge empty when evaluate gpt3.5-prosm 5-rompp and gpt4-prompt. The prompt template for reflection is shown in Figure
Prompt templates for MedMCQA Figure
12
13 provides the prompt template for ReAct-gpt4-prome Figure 14 illustrates the prompt template for agile-gpt4-prompt. The prompt template for answer evaluation is shown in Figure 15
Prompt templates for HotPotQA Figure
29

---

[prompt]
Answer users' [Question] about product {asin} based on the following three types of information:
1. [Product Metadata] provides metadata about product {asin} from the $product_categorr>> database. The schema of the pprrouuctccttegory> database is outlined below (in the format [ueld[ (valuel, value2, ...)). (schema)
(metadata)-
In addition to the provided metadata about product (asin), you have the option to access the full {product_category) database by executing SQL commands.
[Memory] Existing knowledge in your memory about {product_categoryy. [knowledge}
3. [History] Previous question-answer pairs related to product {asin). {history)
Evaluate whether the question can answered based solely on information available.
[Question]: {question)
the question can answered directly, output [PreddictAnwwrr] [Answer)
If an search is required, output [SearchProduct (SQL) SELECT product_id FROM {product_categoryy
Figure 6: The prompt for gpt3. 5
5 S-prompt and gpt4 ept4-prompt ProductQA.
30

---

[prompt]
Answer users' [Question] about product {asin) based on the following three types of information:
1. [Product Metadata] provides metadata about product fasin} from the (product_categors database. The schema of the pproduct__ateeoryy database is out line below (in the format field[unit] (valuel, value2, ...)). (schema) {metadata)
In addition to the provided metadata about product {asin), you have the option access the full (product_category) database by executing SQL commands.
2. [Memory] Existing knowledge in your memory about {product_categoryy.. [knowledge}
3. [History] Previous question-answer pairs related to product (asin). {history)
If [SSeekAvicc]" is selected, the correct answer will be provided, and the current problem and summarized knowledge will be added to the [History] and [Memory] for subsequent round questions. Current status: Similar Memory:
httrr_sst_oowweeeee_o_oc poueleggennmm Similar History: {similar_pane laarr__psss_ssstteeeee question_nunt Round: {round}
Eva Luate whether the question can be answered based solely on the information available.
[Question]: (question)
If the question can be answered directly, output '[PredictAnswer/ \n [Answer)]]
If an SQL search is required, output [SearchProduct) \n[SQ]: SELECT product_id FROM (product_categoryy)
If the information available is insufficient, necessitating seek advice from human, output [SeekAdvice]'
(If you choose [PredictAnswerr] or [SearchProduct]] you will receive 1 point for doing right and no point for doing wrong, If you choose [SeekAdvice] you will directly receive 0.7 points)
Figure 7: The prompt for sgile-vic13b-prongy agile-gpte 5-trom., and agile- spt4-prompt on ProductQA.
[prompt]
Analyze the question and answer that pertain to online shopping scenario involving products in the "(product crtttttcatgoor""""" category. Your task is to extract any information that generally applicable to the entire {product (atbuuc_aateory category. Focus should be the broader category characteristics rather than details specific to specific product.
If the question-answer pair offers relevant general insights about the porooucc_cateoory category, simply respond with [no information]'. Otherwise, directly summarize the applicable general knowledge insights about the {product_categoryy) category based the provided question-answer pair.
#*Question**: (question) **Answer**; (answer) Output:
Figure 8: The prompt for reflection on ProductQA.
31

---

[Prompt for long_answer evaluation]
Based on the provided question and reference answer, please determine the response is correct or incorrect. Begin by articulating your rationale, conclude with a single word judgment: 'Yes' correct "No for incorrect.
question: {question) reference answer: (reference) response: (response}
Figure 9: The prompt for long answer evaluation on ProductQA.
[prompt]
Answer the multiple-choice question about medical knowledge.
[question] {question>
[Optiion A}
(B) (B)(ttnioon B}
(C)(Opttion C}
(D) COpttioo D} [Answer]
Figure 10: The prompt for Meerkat-7b- -proppt on MedMCQA.
[prompt]
The following is multiple-choice question about medical knowledge and some related questions and knowledge references.
[Related Question] { (Releted Knowledge] frelatee__knwllede
[Question] {question}
Solve this in a step-by-step fashion, starting by summarizing the available information.
If the question can be answered directly, output '[PredictAnswer]" and conclude your response with the phrase "the answer is ( (loption_id))"
- If the information available is insufficient, necessitating seek advice from human, output [SeekAdvice]] (If you choose [PredictAnswer]', you will receive point for doing right and no point for doing wrong. If you choose [SeekAdvicel', you will directly receive 0.6 points.)
Figure 11: The prompt for malle-gpt3.55 5-prompt and
on McdMCQA.
[prompt]
Analyze the following [medical} [Question] and [Answer]. Your task is to ext any [Information] that is typically applicable to issues related to the medical field. Summarize and generalize the extracted knowledge into one sentence. If the question-answer pair offers no relevant medical insights, simply respond with [no information)'.
[Question]: (question) [Answer]: {answer) [Information):
Figure 12: The prompt for reflection on MedMCQA.
32

---

[prompt]
You are an intelligent agent with the ability to search knowledge. Please answer the following questions.
You can analyze the solution steps based on the problem and known information, For missing information, you can use search tools by output [Search] ([entity))', Please note that the [entityl should be noun word/phrase rather than sentence. If there is enough information, you answer directly by output [[PredictAnser] [lanswer])', Please note that the answer must be the span in the observation sentences.
[Question]: : What U.S Highway gives access to Zilpo Road, and is also known Midland Trail?
Thought1: I need to search Zilpo Road, and find the U.S Highway gives access to it. Action: [Search] (Zilpo Road) Observation1: Search Result Zilpo Road The nine mile byway starts south of Morehead, Kentucky and can be accessed by U.S. Highway 60.
Thought2: Zilpo Road is located in the eastern Kentucky, United States. So need to search Kentucky and find the Highway gives access to Zilpo Road.
Action: [Search] (Kentucky)
Observation2: Search Result Morehead, Kentucky
Morehead is home rule-class city located along US 60 (the historic Midland Trail) and Interstate 64 in Rowan County, Kentucky, in the United States.
Thought3: also named as Midland Trail, the answer US 60.
Action: [PredictAnswer] (US 60)
[Question]:
{question)
Thought1:
Figure 13: The prompt for SeAct-gpt4-pree on HotPotQA.
33

---

[prompt]
You are an intell agent with the ability to search knowledge. Please answer the following questions,
You can ayae the solution steps based on the problem and known information, For missing information, you can use search tools by output [Search]] (fentityl)'. If there is enough information, you can output [PredictAnser] ([answer)) to answer the question directly or output [Seekadvice]] () if you are not sure and need to seek advice. Please note that the answer must be the span in the observation sentences.
[Question]: what U.S Highway gives access to Zilpo Road, and is also known Midlaand Trail?
Thought1: I need to search Zilpo Road, and find the U.S Highway gives access to it. Action: [Search] (Zilpo Road)
Observation1: Search Result Zilpo Road (Summary version) The nine mile byway starts south of Morehead, Kentucky and can be accessed by U.S. Highway 60.
Thought2: Zilpo Road is located in the eastern Kentucky, United States. So I need to search Kentucky and find the Highway gives access to Zilpo Road. Action: [Search] (Kentucky)
Observation2: Search Result Morehead, Kentucky (Summary version) Morehead is home rule-class city located along US 60 (the historic Midland Trail) and Interstate 64 in Rowan County, Kentucky, in the United States.
Thought3: US 60 also named Midland Trail, the answer is US 60.
Action: [PredictAnswer] (US 60)
[Question]:
(question)
Thought1:
Figure 14: The prompt for agile-gpt4-promm on HotPotQA.
[prompt]
Based the provided question and reference answer, please determine if the response is correct or incorrect. Begin by articulating your rationale, and conclude with a single word judgment: "Yes' for correct or "No" for incorrect.
question: {question} reference answer: {reference) response: (response}
Figure 15: The prompt for answer evaluation on HotPotQA.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims the abstract and introduction accurately reflect the paper's contributions and scope by clearly stating the claims, highlighting the contributions, and aligning with the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Appendix of our paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the experimental settings and implementation details in Section Appendix B and Appendix DorenixxJoff of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided our data and code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ priiss.cc/pbbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the necessary experimental details Section 4 Appendix and Appendix our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The reason for not reporting error bars due to the high computational cost associated with corresponding experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the resource details in Section of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform. in every respect, with the NeurlPS Code of Ethics
Question:
Answer: [Yes]
Justification: Our research conforms to the NeurlPS Code of Ethics guidelines, including considerations for fair wages, data privacy, societal impact, potential harm, bias and faimeess, and impact mitigation measures.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both the potential positive societal impacts and negative societal impacts in Appendix G of our paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We have described potential security risks and safeguards in Appendix and Appendix of our paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited the existing assets used in our paper, and mentioned the necessary information our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided the details in Section Appendix D and Appendix Fo Ff our paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We have provided the details in Section Appendix Appendix of our paper.
Appendix
and
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have included the related details in Appendix F and Appendix of our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
40

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "IUJcxIr83j", "reviewer": "Reviewer_5wwv", "summary": "The paper \"AGILE: A Novel Framework of LLM Agents\" introduces a new framework for Large Language Model (LLM) agents designed to handle complex conversational tasks. The framework, named AGILE (AGent that Interacts and Learns from Environments), incorporates LLMs, memory, tools, and interactions with experts. AGILE is formulated as a reinforcement learning (RL) problem and is fine-tuned using Proximal Policy Optimization (PPO). The authors present a new dataset, ProductQA, for evaluating the framework and report that AGILE agents outperform GPT-4 in their experiments. The paper claims significant improvements in performance due to the integration of memory, tools, expert consultation, and RL training.", "questions": "**Benchmark Selection:** Why were ProductQA and MedMCQA specifically chosen as the benchmarks for this study? Are there other benchmarks where AGILE could be tested to validate its generalizability?\n\n**Human Expert Advice:** Is there a ground truth for when the agent needs to seek advice, and how is this optimized?\n\n**Memory Component:** Can you provide more details on how the memory component scales with an increasing number of interactions and how it ensures efficient retrieval of relevant information?", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Comprehensive Evaluation:** The creation of the ProductQA dataset and the extensive experiments conducted on both ProductQA and MedMCQA provide a robust evaluation of the framework’s capabilities.\n\n**Significant Performance Improvements:** The reported improvements over GPT-4 agents in both ProductQA and MedMCQA are noteworthy, indicating the effectiveness of the AGILE framework.\n\n**Detailed Methodology:** The paper provides a thorough explanation of the RL formulation, training processes, and the roles of different components within the framework, which enhances reproducibility.", "weaknesses": "**Limited Novelty:** The reliance on human experts for advice, while useful, is not a novel concept and has been explored in previous works and becomes a common practice. The paper does not sufficiently differentiate its approach to human expert interaction from existing methods.\n\n- Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797.\n- https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/\n\n**Scalability Concerns:** The scalability of AGILE with more complex environments is not addressed. It is unclear how well the framework would perform with task complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:** Limited Novelty: The reliance on human experts for advice, while useful, is not a novel concept and has been explored in previous works and becomes a common practice. The paper does not sufficiently differentiate its approach to human expert interaction from existing methods.\n> - Xiao, H. and Wang, P., 2023. Llm a*: Human in the loop large language models enabled a* search for robotics. arXiv preprint arXiv:2312.01797.\n> - https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/\n\n**A1:** Thanks for bringing this to our attention. In AGILE framework, agent can proactively seek advice from human experts when its confidence is low. This is distinct from existing human-in-the-loop methods, which rely on passively receiving human feedback (Xiao and Wang, 2023) or following pre-defined rules (https://python.langchain.com/v0.1/docs/use_cases/tool_use/human_in_the_loop/). Proactively seeking advice is a much more complicated decision-making, since the agent must estimate its own confidence in the current state, predict the potential value of the advice for future sessions, and consider the cost of experts. We will cite the works you mentioned and clearly differentiate our proactive seeking-advice mechanism from existing human expert interaction methods in our revised paper.\n\n---\n> **Q2:** Scalability Concerns: The scalability of AGILE with more complex environments is not addressed. It is unclear how well the framework would perform with task complexity.\n\n**A2:** Thanks for your feedback. We acknowledge the importance of addressing scalability of AGILE. To this end, we have introduced ProductQA, a complex benchmark designed to evaluate the comprehensive capabilities of the agent. ProductQA tests an agent's ability to handle historical information and accumulated knowledge, leverage tools, interact with humans, perform self-evaluation, and conduct reflection. In addition, the training and testing tasks are made disjoint to assess the agent's ability to adapt to new tasks. \n\nOur experimental results show that AGILE agent, based on 13b LLMs and trained with PPO, outperforms GPT-4 agent on ProductQA. This demonstrates the AGILE framework's potential to scale and manage complex tasks effectively. Furthermore, AGILE is a general agent framework, allowing for various extensions, such as integrating additional tools, further enhancing its scalability with increasing task complexity. \n\n---\n> **Q3:** Benchmark Selection: Why were ProductQA and MedMCQA specifically chosen as the benchmarks for this study? Are there other benchmarks where AGILE could be tested to validate its generalizability?\n\n**A3:** While the proposed agent framework is general, in this paper, we evaluate it in complex question answering, a task an LLM agent has the potential of outperforming existing solutions such as the use of an LLM alone. We use ProductQA since it assesses a variety of agent capabilities including knowledge accumulation, tool-use, human interaction, reflection, etc, making it ideal for demonstrating the comprehensive ability of our framework.  To validate the generality of AGILE, we select MedMCQA as an additional benchmark. This task requires extensive medical knowledge and the ability to effectively seek expert advice.\n\nIn response to your suggestion about exploring more diverse benchmarks, we perform experiments on HotPotQA, featuring natural, multi-hop questions. We train an agile agent using the Vicuna-13b as a base. The experimental results show that the agile agent outperforms GPT-4 agent by 10.8% in relative total score. Ablation studies verify that ppo training improves total score by 1.8%. Detailed results of the additional experiments will be included in the revised version to underscore the generality of the AGILE framework.\n\n| Method | Advice Rate | Accuracy | Total Score |\n|---|---|---|---|\n| ReAct (gpt-4) | - | 0.482 | - |\n| agile-gpt4-prompt | 0.194 | 0.664 | 0.567 |\n| agile-vic13b-w/o Advice | 0.000 | 0.553 | 0.553 |\n| agile-vic13b-w/o RL | 0.171 | 0.668 | 0.617 |\n| agile-vic13b-ppo (ours) | 0.156 | 0.675 | 0.628 |\n\n---\n> **Q4:** Human Expert Advice: Is there a ground truth for when the agent needs to seek advice, and how is this optimized?\n\n**A4:** Thanks for your questions. Determining when the agent should seek advice relates to the model's confidence and cost of human experts. Since the optimal decision is model-dependent, it is difficult to establish a ground truth and perform supervised fine-tuning. However, our RL framework can effectively address this issue. By defining rewards for both correctly predicting answers and seeking advice, we can optimize the skill as part of the policy model on end-to-end RL training. For example, we assign a reward of +1 for a correct prediction and a penalty of -c for seeking advice, where c represents the human cost. This approach allows the agent to learn an optimal trade-off between relying on its own predictions and seeking external advice. We will explain the method more clearly in the revised paper.\n\n---\n> **Q5:** Memory Component: Can you provide more details on how the memory component scales with an increasing number of interactions and how it ensures efficient retrieval of relevant information?\n\n**A5:** Thank you for your question. The memory is designed as a scalable database where stores agent's trajectories, including question-answering pairs and agent reflections. We employ vector-based retrieval for efficient information access. Specifically, we use the all-MiniLM-L6-v2 model for embedding the text data. When an instance is stored, it is embedded into a vector representation that serves as its key in memory. \n\nFor retrieval, we embed the user question using the same model and then perform a cosine similarity search among the stored embeddings. This is a well-studied process and can be performed efficiently even when the memory is large.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3bbe9OEtc", "reviewer": "Reviewer_ZJZM", "summary": "This paper proposes AGILE, a reinforcement-learning based framework for finetuning LLMs for conversational QA tasks. The models are initially trained using imitation learning, then further finetuned with RL. Once finetuned, the models show strong performance, surpassing GPT-4 while using much smaller models.", "questions": "What is the reason behind why there is not comparison with rl finetuning baseline (standard ppo)?", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework is interesting and novel, distilling the usage of various tools, reflection, memory retrieval/writing, and human advice seeking from larger models' trajectories, to a smaller model, then further training it using RL to surpass the performance of the larger model\n- The performance is strong\n- Introduces a new dataset, ProductQA which is a useful resource for conversational QA in online shopping scenarios.\n- Cost-benefit analysis of advice seeking is insightful.", "weaknesses": "I only have two main concerns:\n- Because the model is only evaluated on two conversational QA datasets (and not on any of the mentioned datasets in Table 8), it's difficult to judge the generality of the proposed method.\n- There is no comparison with standard ppo", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your positive feedback and the time you invested in reviewing our paper. Thank you for your insightful questions. In our response, we will address each question individually, quoting them and providing our answers accordingly.\n\n## Response to comments\n\n---\n\n> **Q1 (Weakness 1):** Because the model is only evaluated on two conversational QA datasets (and not on any of the mentioned datasets in Table 8), it's difficult to judge the generality of the proposed method.\n\n**A1:** Thanks for the helpful suggestion. To address the concern regarding the generality of AGILE, we conducted additional experiments on the HotPotQA dataset, which is one of the tasks listed in Table 8. The main results and ablation study are presented below.\n\nCompared with ReAct implemented by prompting GPT-4, our method (agile-vic13b-ppo) shows a 19.3% relative improvement in accuracy. Furthermore, agile-vic13b-ppo achieves a 10.7% relative improvement in the total score over agile-gpt4-prompt, which is the AGILE agent implemented by prompting GPT-4. The ablation study underscores the indispensability of seeking advice and PPO training in achieving the agent's strong performance.\n\n| Method | Advice Rate | Accuracy | Total Score |\n|---|---|---|---|\n| ReAct (gpt-4) | - | 0.482 | - |\n| agile-gpt4-prompt | 0.194 | 0.664 | 0.567 |\n| agile-vic13b-w/o Advice | 0.000 | 0.553 | 0.553 |\n| agile-vic13b-w/o RL | 0.171 | 0.668 | 0.617 |\n| agile-vic13b-ppo (ours) | 0.156 | 0.675 | 0.628 |\n\n---\n\n> **Q2 (Weakness 2 & Question 1):** What is the reason behind why there is not comparison with rl finetuning baseline (standard ppo)?\n\n**A2:** Thanks for your question. In the ProductQA task, there could be hundreds of question-answering rounds. These rounds are correlated. Actions in earlier rounds can write the memory, thus have lasting effects on subsequent rounds. For example, knowledge distilled from seeking advice can help respond in the future. These long trajectories yield training sequences that span millions of tokens and are impractical for standard PPO training. To address this issue, we proposed a session-level training algorithm that takes lasting effect into account, which is detailed in Appendix A of our paper.\n\n---\n\nPlease let us know if our replies address your concerns. Thanks for taking the time to consider this discussion. We appreciate your time and effort in helping us improve our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VvS32VWq5", "reviewer": "Reviewer_xRs2", "summary": "This work presents a novel framework for LLM agents named AGILE. The entire AGILE system is trained end-to-end using reinforcement learning. A key feature of AGILE is its ability to seek advice from external human experts. Additionally, the authors have developed ProductQA, a challenging dataset of complex QA, to comprehensively evaluate the capabilities of the agent. Extensive experiments show that an agent within this framework, even when based on a smaller model and trained with RL, can outperform GPT-4.", "questions": "1. How can this framework be extended to support multiple agents?\n\n2. In Table 5, the accuracy of \"w/o Memory\" and \"w/o Tool-Use\" is higher than that of agile-vic13b-ppo. Is the metric \"Accuracy – Advice Rate\" a more straightforward one than Total Score?", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel framework, AGILE, for LLM agents that integrates memory, tools, and expert consultations, all optimized through reinforcement learning.\n\n2. The development of the ProductQA dataset.\n\n3. The paper is well-organized and clearly written. The architecture and workflow of the AGILE framework are explained with clarity.\n\n4. The experiments on ProductQA and MedMCQA verify the AGILE framework and show that AGILE agents based on 13B and 7B LLMs trained with PPO can surpass GPT-4 agents.", "weaknesses": "1. It is somewhat like WebGPT (in terms of actions and policy learning strategy), with extensions supporting memory, expert consultation, and reflection.\n\n2. The experiments are all related to a single agent. It would be better to show the application for multiple agents and include planning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your positive feedback and effort in reviewing our paper. Thank you for your constructive questions. In our response, we will quote each question and provide our answers accordingly.\n\n## Response to comments\n---\n> **Q1 (Weakness 1):** It is somewhat like WebGPT (in terms of actions and policy learning strategy), with extensions supporting memory, expert consultation, and reflection.\n\n**Answer:** Thanks for bringing this to our attention. We would like to clarify the key distinctions between our proposed AGILE framework and WebGPT:\n\n- WebGPT primarily uses a policy learning strategy to train the agent for web operations and does not incorporate memory, expert consultation, or reflection. In contrast, AGILE emphasizes proactively seeking advice from human experts, allowing the agent to achieve high-level accuracy when handling complex and challenging questions. The agent can also enhance its performance over time by reflecting on expert feedback and memorizing it for future use.\n\n- While WebGPT utilizes reinforcement learning, our work addresses the significant challenge of training the policy for invoking various modules and the reasoning, planning, reflection, and seeking advice abilities of the LLM agent in an end-to-end manner. This challenge is particularly pronounced in long trajectory scenarios. To overcome this, we propose a session-level optimization algorithm that facilitates policy learning at the session level, thereby mitigating the difficulties associated with long trajectory optimization.\n\n- As shown in Table 5 in our paper, incorporating memory, seeking advice, and reflection into AGILE results in relative total score improvements of 4.0%, 5.0%, and 1.7%, respectively, on the ProductQA dataset. These results demonstrate the necessity of these modules for LLM-based agents in practical scenarios.\n\n---\n\n> **Q2 (Weakness 2):** The experiments are all related to a single agent. It would be better to show the application for multiple agents and include planning.\n\n**Answer:** We appreciate your valuable suggestions. Applying AGILE to multi-agent systems is indeed an interesting direction. The AGILE framework can be extended to facilitate interactions with machine agents in various roles, such as students or teachers, and in different formats, such as debates or coordination. For example, a multi-agent version of AGILE could seek advice from a human expert at a high cost or from a more capable machine agent at a lower cost. Furthermore, the planning for multiple agents can be enhanced using end-to-end RL training.\n\nIn our paper, we primarily focus on the RL formulation and end-to-end training of LLM agents, along with their seeking advice capabilities, which we believe are fundamental and essential for agent performance. Due to space and time limitations, the application of AGILE to multi-agent systems will be addressed in future work. Thank you for your constructive feedback.\n\n---\n\n> **Q3 (Question 1):** How can this framework be extended to support multiple agents?\n\n**Answer:** Thanks for your question. Please refer to our response to Q2.\n\n---\n\n> **Q4 (Question 2):** In Table 5, the accuracy of \"w/o Memory\" and \"w/o Tool-Use\" is higher than that of agile-vic13b-ppo. Is the metric \"Accuracy – Advice Rate\" a more straightforward one than Total Score?\n\n**Answer:** Thank you for this valuable comment. In Table 5, although the accuracy of \"w/o Memory\" and \"w/o Tool-Use\" is higher than that of agile-vic13b-ppo, they exhibit a significantly higher Advice Rate compared to agile-vic13b-ppo. This higher Advice Rate results in lower overall performance for \"w/o Memory\" and \"w/o Tool-Use\" when assessed using our Total Score metric.\n\nIn our paper, the total score is defined as the average reward across all sessions, taking both the advice rate and the accuracy into account. Specifically, the total score is expressed as \"Total Score = Accuracy - Seeking Advice Cost * Advice Rate\". This formulation encompasses the \"Accuracy – Advice Rate\" metric as a special case where the Seeking Advice Cost is set to 1.\n\nIt's important to note that the Seeking Advice Cost can vary across different real-world applications. For instance: \n- In a math problem-solving task, the required human expert might need high professional expertise in mathematics, resulting in a high cost (e.g., a cost of 1).\n- In product question-answering scenarios, the human expert could be a normal person who has received short-term customer service training, leading to a lower cost (e.g., a cost of 0.3).\n\n---\n\nPlease let us know if our replies address your concerns. Thanks for taking the time to consider this discussion. We appreciate your time and effort in helping us improve our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o2qpCeGznc", "reviewer": "Reviewer_b4yX", "summary": "This paper jointly introduces an evaluation environment for LLM agents and an LLM agent architecture called AGILE. \n\nThe environments is a Question-Answering environment concerning the characteristics of commercial products sourced from Amazon. The benchmark itself is composed of 3 tasks: 1) a fact retrieval task where the agent should retrieve information about a product, 2) a product query task where the agent should retrieve a product that matches query criteria, and 3) a reasoning task where the agent should reason from the knowledge it has about a product to answer questions about possible uses or suitability of a product.\n\nThe agent is composed of an LLM backbone that, under the control of an executive module, can retrieve memories of previous interactions stored in its database and use tools. Tools are used for agentic flow (reflect, submit answer), for memory management (retrieve memory, put new memory in database), and for asking advice from a human expert (access to ground truth about a product). The LLM is finetuned with SFT on these tasks. The agent is further fine tuned on a scored version of the tasks with reinforcement learning (PPO).\n\nThe authors demonstrate the superiority of their architecture on their proposed environment and on a medical QA benchmark over one-shot prompting gpt4, and demonstrate that reflection, tool use, memory, ground truth product advice, and reinforcement learning contribute positively, with varying degrees, to agent performance.", "questions": "* My first and most important suggestion is that the authors should change the framing of the paper. The experimental work is sound and the tasks are worthwhile, but in the current state of the paper the conclusions seem to be that RAG agents work better than prompting on question-answering tasks, which is known. However, the paper could be: 1) a study on RL in RAG agent settings, 2) a study on training metacognitive abilities of RAG agents, and/or 3) a paper introducing a new benchmark (but the tasks seem not that challenging compared with tasks like SWE-bench https://www.swebench.com/ or reasoning tasks like theorem proving https://github.com/openai/miniF2F, https://leandojo.org/).\n* [Conclusion, l305] why is LLM system 1 and interaction with tools system 2? What does it mean? If the claim is to be left in the paper it should be explained and adequate citations should be provided (also is this analogy fruitful? What purpose does it serve? is this a claim about the cognitive plausibility of the proposed agent architecture or its resemblance to humans?).\n* [Clarity] The paper should explain what mcqa is when first mentioned, since it is confusing for people not familiar with the benchmark. Same goes for the meerkat model.\n* Why is the improvement from RL only 2%?\n* The original RAG paper should be cited: https://arxiv.org/abs/2005.11401", "limitations": "* Some limitations with respect to lack of resources are acknowledged in the appendix (small open source models are trained, only a subset of the training set is used). This is fine; the methods used are pretty powerful and can be expected to scale with increasing compute.\n* If understood as a way to help agents answer queries, expert advice is definitely non-scalable. It is fine as a research question, maybe less so for modelling real-world applications. This could be mitigated with long-term memory of the agent (give advice once, retrieve forever). In any case, a discussion on the feasibility of expert advice in real world scenarios would be nice.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper combines known LLM agent modules and studies their impact in two question-answering domains that are of interest to industry practitioners: product recommendation and medical knowledge;\n* Agents asking for help has been studied at length in the embodied question answering literature with previous-generation agents (see for instance https://arxiv.org/abs/1909.01871 or https://arxiv.org/abs/2302.04865), but as far as I am aware has not been studied with LLM-based agents;\n* The paper is fairly clear and easy to read, the proposed contributions are easy to understand, examples are provided for the task, and the results are comprehensive;\n* The claims of the paper are well substantiated with extensive experiments, and the ablations allow to easily disentangle the effect of different components of the agent;\n* The data collection process for creating the benchmark seems sound and the task looks challenging and interesting;\n* Reinforcement-Learning studies of complete LLM agents are rare and needed.\n* The tables in the related work makes the relationship of this work and other previous art clearer.", "weaknesses": "* An overarching and pervasive weakness of this paper that is difficult to ignore and that has a great influence on my judgement is the claim of novelty of the agent architecture. In a post-Voyager (https://arxiv.org/abs/2305.16291) world, LLM-based agents equipped with tool use and retrieval are standard and the AGILE agent presented in this work introduces little conceptual novelty in the agent architecture, while even the title of the paper claims that the agent is novel. LLM agents of this type are so much part of the landscape that many reviews exist that organize existing approaches (see for instance the excellent https://arxiv.org/abs/2309.02427 that is not that recent either). The approach is standard enough that advanced software stacks (langchain, llamaindex) are devoted to speeding up building retrieval-augmented generation (RAG)-based agents for industry practitioners. The paper framing should be updated accordingly to reflect that the approach being studied is standard, but applied to a new task.\n* Relatedly, it would have been important for the authors to exactly pinpoint the difference between their agent and for instance Voyager, and either implement baselines from existing work on their task or test their agent on other tasks (eg WebShop which is very similar in spirit).\n* There are still novel elements in the proposed approach. The first is the use of reinforcement learning. While the use of reinforcement learning to improve language agents is not novel (see https://arxiv.org/abs/2302.02662 for a textworld task or https://arxiv.org/abs/2403.04642 for a math word task), the use of RL in RAG agents has not been studied as far as I am aware. It also makes up a significant portion of the method section and the appendix, however the experiments demonstrate only a marginal improvement due to RL. Why is this the case? How are the training curves like? Has the training converged? I also do not see any standard deviation information for the RL part. The authors justify the lack of error bars with the computational cost of experiments with is fair and completely acceptable where the comparison between methods is straightforward, but would have been helpful in the sft/ppo comparison where the difference is small and could have been due to chance, and it makes sense to concentrate computational effort on a truly novel part of the contribution;\n* The other novel element of this paper is the asking for advice part: the model can opt out of the task by seeing the ground truth which it can then report, at a cost (for the RL agent). We can see that allowing this action to the agent increases its accuracy, that increasing the cost of advice results in RL agents using advice less often, and that memoryless agents use advice more often. This is interesting, and could have been a core contribution of the paper, if it were explicit and investigated as such. The conditions under which LLM agents are aware of their own knowledge, as well as them taking appropriate steps to reduce lack of information is a research programme in itself, as has been discussed by John Schulman in his Berkeley lecture in 2023: https://www.youtube.com/watch?v=hhiLw5Q_UFg&t=215s Can the authors predict when the model is going to ask for advice? Does the RL training help the model know what it knows (metacognitive skills)? These are interesting questions that the authors could have investigated; \n* Relatedly, the related work paragraph on uncertainty looks unfinished. Relevant work on LLM’s metacognitive abilities are cited but there is no word on how the current work contributes to the discussion;\n* (minor) Time of training and number of gpus required for training should appear after the type of gpu; does the training use lora?\n* The evaluation for longform answers is based on GPT4, but no evidence is presented to correlate agreement of GPT4 with human judges on this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1 (Weakness 1 & Question 1):**  Paper framing\n\nA1: Regarding novelty, we would like to note that we have summarized our core contributions at the end of the Introduction section [line 65], aligning exactly with your suggestions. The Methods section also aligns with your summary. We acknowledge that the \"novel framework\" statement was somewhat overstated. Hence, we will revise our paper's title to: \"**AGILE: A Novel Reinforcement Learning Framework for LLM Agents**,\" to emphasize that our main contribution is the end-to-end RL formulation and training of LLM agents. We will also properly update other parts of the paper to clearly convey this point.\n\nBeyond the RL aspects, we want to highlight other novel contributions of our paper. First, training the agent to proactively seek advice requires it to silently estimate its own confidence before making decisions. This capability cannot be achieved through SFT alone, further demonstrating the importance of RL training.\n\nSecond, the ProductQA dataset is a novel contribution. It is the first agent task involving very long trajectories, with each containing thousands of correlated QA sessions and millions of tokens in total. The actions the agent takes in one session influence all future sessions. This characteristic is markedly different from other benchmarks where the impact of actions is much shorter-term.\n\n---\n> **Q2 (Weakness 2):** Difference between AGILE and other agents\n\nA2: Agents like Voyager are based on prompt engineering, while AGILE is a RL formulated framework that allows end-to-end training. Application of AGILE to other tasks, such as WebShop, is left to future work.\n\n---\n> **Q3 (Weakness 3 & Question 4):** Additional results of RL training\n\nA3: First, we provide training curves in the global response PDF file, which indicates that RL training converged after about 500 steps.\n\nSecond, following your suggestion, we conducted multiple independent trials of PPO training to study the variation of the result (see Table 2 in the PDF file). On average, RL training improves the total score by 2.6%, with a standard deviation of 0.3%, demonstrating the significance of RL improvements.\n\nThird, we would like to discuss why RL training improved performance by a moderate 2%. We believe the reasons are: (1) The agile-vic13b-sft model is a strong baseline as it imitates the policy of the agile-gpt4-prompt; (2) SeekAdvice decisions made by agile-gpt4-prompt are nearly optimal under the default advice cost (c=0.3). To further study the impact of RL training, we conducted additional experiments in two more settings:\n\n- We re-generated SFT training data for agile-vic13b-sft such that the agent performs SeekAdvice randomly in 25% of cases. This initial policy is simpler but more general. In this setting, we name the SFT model agile-vic13b-sft-random, and the final model trained with RL on top of it agile-vic13b-ppo-random. As shown in the table in global response, RL training brings a **7.1%** improvement in this setting. Interestingly, the performance of agile-vic13b-ppo-random is better than that of agile-vic13b-ppo that we reported in the paper. We conjecture that random SeekAdvice is a better initial policy because it enables exploration in all directions.\n\n- In the second experiment, we lowered the advice cost to 0.1. After PPO training, the agile-vic13b-ppo-random agent quickly adapted to the new cost, performing SeekAdvice much more aggressively than the initial agent trained by SFT. In this scenario, RL training brings a **22.3%** improvement.\n\nSee Table 3 in the PDF file for concrete numbers.\n\n---\n> **Q4 (Weakness 4):** RL training for metacognitive skills\n\nA4: RL training helps the model's metacognitive skills. Experiments on ProductQA and MedMCQA datasets show that, after PPO training, the agent seeks advice less often and achieves higher accuracy, indicating improved self-assessment in resolving queries versus seeking assistance.\n\n> **Q5 (Weakness 5):** Related work and our contributions to the topic of uncertainty\n\nA5: We note that AGILE solves a complex self-evaluation problem because the agent must make multiple seeking advice decisions in a long trajectory, each having a long-term influence on future decisions. This differs from existing work in uncertainty. We will add discussion to the revised version.\n\n---\n> **Q6 (Weakness 6):** Training details\n\nA6: On ProductQA, SFT takes 3.6 hours and PPO takes 5.5 hours on 8 H800. On MedMCQA, SFT takes 0.9 hours and PPO takes 2 hours. The LLM is trained without using LoRA.\n\n---\n> **Q7 (Weakness 7):** Agreement between GPT-4 evaluator and human\n\nA7: We conducted an additional evaluation by randomly selecting 100 triplets (question, reference long answer, model-predicted long answer) from ProductQA and manually labeled the correctness. Our results show a 94% agreement rate between the GPT-4 evaluator and the author.\n\n---\n> **Q8 (Question 2):** Analogy to System 1 and System 2 processes\n\nA8: According to the dual-process theory [1], human thinking involves two systems: System 1 (fast and unconscious) and System 2 (slow and conscious). Recent research has explored AI systems incorporating both processes [2]. We propose that AGILE uses LLM for System 1 and integrates external tools for System 2, mirroring human thinking. We will include further discussion in the revised version.\n\n[1] D. Kahneman. 2003. Maps of bounded rationality: Psychology for behavioral economics.\n\n[2] Y. Bengio, Y. Lecun, and G. Hinton. 2021. Deep learning for AI.\n\n---\n> **Q9 (Question 3):** Further explanations of MCQA and the Meerkat model\n\nA9: MedMCQA is a multiple-choice QA dataset from medical school entrance exams. Meerkat is a medical LLM trained with high-quality CoT reasoning paths from 18 medical textbooks and diverse instruction-following datasets. We will include these explanations in the revised version.\n\n---\n> **Q10 (Question 5):** Citation of RAG paper\n\nA10: We will add it to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable feedback from all reviewers. We have tried our best to address each question raised in the respective reviews.

Additionally, we have conducted supplementary experiments to address certain concerns and incorporated the results in a one-page PDF file attached to this global response. The PDF includes the following:

- **Results on HotPotQA.** Experiment results and ablation study on a new task: HotPotQA. 
- **Robustness of RL training.** Results of multiple PPO training runs, providing the mean and standard deviation of the improvement brought by RL training.
- **Additional results of RL training.** Improvement achieved by our RL training with different initial policy models or different reward values.
- **RL training curves.** Reward and value function loss curves during the PPO training process.

Meta-review: The paper presents a framework for Large Language Model (LLM) agents designed to tackle complex conversational tasks. The AGILE framework integrates reinforcement learning (RL) with memory, tools, and human-expert interactions. It also introduces a new dataset, ProductQA, to evaluate the framework's capabilities. While the reviewers acknowledge the strengths of the paper, they also raise several concerns about the paper’s novelty and scalability.

One of the primary concerns is the limited novelty of the proposed approach, especially in the context of human-expert interaction, which has been explored extensively in prior works. Although the authors attempt to differentiate their proactive advice-seeking mechanism, the distinction is not entirely clear or compelling. Additionally, scalability issues remain unresolved, with uncertainties about how the framework would perform in more complex environments.

Further, the evaluation of AGILE is primarily limited to two datasets, raising concerns about the generalizability of the method. While the authors conducted additional experiments on another dataset in response to this concern, the lack of comparison with standard PPO and the absence of detailed training curves or standard deviations in the results were noted as weaknesses. The reviewers also highlight the similarity of the AGILE framework to existing works like WebGPT, suggesting that the paper could benefit from a clearer differentiation in the paper of its contributions and more rigorous baseline comparisons.

Despite these criticisms, the reviewers recognize the importance of RL studies in the context of LLM agents and appreciate the introduction of the ProductQA dataset (I also recommend the authors check and cite these few works: amazon-science/contextual-product-qa, Product Question Answering in E-Commerce: A Survey, Salespeople vs SalesBot, etc), which they see as a valuable resource. However, they suggest that the paper should more explicitly address its contributions, particularly regarding the conditions under which the agent seeks advice, which could be a more central focus of the work.
Decision: Accept (poster)