ReAct Meets ActRe: Autonomous Annotation of Agent Trajectories for Contrastive Self-Training

Authors: zonghan  yang
peng  li
ming  yan
ji  zhang
fei  huang
yang  liu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 07:40:25

Abstract: Language agents have demonstrated autonomous decision-making abilities by reasoning with foundation models. Recently, efforts have been made to train language agents for performance improvement, with multi-step reasoning and action trajectories as the training data. However, collecting such trajectories still requires considerable human effort, by either artificial annotation or implementations of diverse prompting frameworks. In this work, we propose A$^\mathbf{3}$T, a framework that enables the Autonomous Annotation of Agent Trajectories in the style of ReAct. The central role is an ActRe prompting agent, which explains the reason for an arbitrary action. When randomly sampling an external action, the ReAct-style agent could query the ActRe agent with the action to obtain its textual rationales. Novel trajectories are then synthesized by prepending the posterior reasoning from ActRe to the sampled action. In this way, the ReAct-style agent executes multiple trajectories for the failed tasks, and selects the successful ones to supplement its failed trajectory for contrastive self-training. Realized by policy gradient methods with binarized rewards, the contrastive self-training with accumulated trajectories facilitates a closed loop for multiple rounds of language agent self-improvement. We conduct experiments using QLoRA fine-tuning with the open-sourced Mistral-7B-Instruct-v0.2. In AlfWorld, the agent trained with A$^3$T obtains a 1-shot success rate of 96\%, and 100\% success with 4 iterative rounds. In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts. A$^3$T agents significantly outperform existing techniques, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.

Main paper: 
 introduction: 
 The rapid development of Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023; Team et al., 2023; Jiang et al., 2024) has led to the prosperity of language agents. Leveraging the ability of LLMs, language agents have demonstrated impressive performances in diverse decision-making scenarios by interacting with the environments autonomously (Wang et al., 2023; Mirchandani et al., 2023; Zheng et al., 2024; Wu et al., 2024) .Recently, increasing efforts have been made to train language agents with open-sourced LLMs. The multi-step trajectories that describe the entire task-solving process of a language agent are used as training data, which consist of environmental observations, internal reasoning texts, and external actions. The collection of such trajectories is therefore essential, which are currently categorized into two paradigms in Fig. 1 (a) and (b) . The first paradigm is to leverage expert demonstrations (Yao et al., 2022) . However, the expense of human labor hampers the scalability of the approach. Another paradigm is implementing different agent frameworks to gather diverse trajectories with proprietary LLMs (Qin et al., 2023; Zeng et al., 2023; Chen et al., 2023; Aksitov et al., 2023) . However, the exploration coverage in the training data is still upper-bounded by the full set of prompting techniques. Besides, implementing diverse agent frameworks requires considerable human efforts and proprietary LLM calls (Yang et al., 2024) . To ease the data collection process in diverse scenarios, Yin et al. (2023) and Zhang et al. (2024) propose unified data formats by elucidating the comprising submodules in agent trajectories. However, as obtained by converting human-annotated data or one single defaulted prompting scheme, the agent trajectories are still limited in diversity and scalability. Considering that an environment automatically returns observations and rewards with action inputs, it should serve as an infinite data generator. While Song et al. (2024) propose an exploration-based agent framework for self-improvement, the gathered trajectories consist of only interleaved external actions and environmental observations, without textual rationales that could steer better behavior of language agents. We ask the following question: Can a language agent autonomously gather high-quality trajectories from the external environment, with textual annotations suitable for further self-training?In this work, we propose A 3 T, a framework that enables Autonomous Annotation of Agent Trajectories in the style of ReAct (Yao et al., 2023) for self-improvement with minimal human supervision. The central idea is to exploit both the in-context language ability and the decision-making ability of a language agent: To collect diverse trajectories, an agent could randomly sample external actions from the action space at arbitrary steps. However, the corresponding reason for the sampled action should be annotated for a ReAct-style agent. To facilitate this, we propose ActRe, an act-then-reason prompting agent that explains the reason for the sampled action. With ActRe, the ReAct-style agent composes extra reasonthen-act trajectories for each failed task by inversely prepending the ActRe-prompted reason to the randomly sampled action. After the execution of each composed trajectory, the agent receives a terminal reward from the environment, which automatically annotates the quality of the trajectory. The gathered successful trajectories are then supplemented with the failed trajectory by the ReAct-style agent alone for contrastive self-training, where we use policy gradient methods (Williams, 1992) with binarized rewards for LLM fine-tuning. As new agents are trained, more trajectories can be gathered and accumulated, which forms a closed loop for the self-improvement of language agents as shown in Fig. 1-(c ). We validate our A 3 T framework in the textual embodied environment AlfWorld (Shridhar et al., 2021) and the online shopping environment WebShop (Yao et al., 2022) . We use QLoRA (Dettmers et al., 2023) to fine-tune Mistral-7B- Instruct-v0.2 (Jiang et al., 2023) in the training experiments. Experimental performances demonstrate significant improvement over state-of-the-art agent techniques: On AlfWorld, our trained agent achieves a 96% success rate in unseen scenarios with a single trial. On WebShop, the success rate of our agent reaches 49%, which matches the average human performance (50%). In the setting of iterative refinement, after four rounds of data collection and contrastive self-training, the accumulated success rate becomes 100% on AlfWorld and 54.8% on WebShop, narrowing the gap with human experts (59.6% on WebShop). A 3 T paves the way for agents with improved autonomy through the closed loop of self-annotation and contrastive self-training. 
a 3 t for closed-loop self-improvement: 
 In this section, we introduce the closed-loop self-improvement for agents facilitated by the A 3 T framework. The loop contains two parts: autonomous trajectory annotation with the ActRe agent (Sec. 2.1), and the contrastive self-training process (Sec. 2.2).Agents are able to gather diverse trajectories by exploration. However, for language agents like ReAct, the actions are inferred by first reasoning with LLMs. When the agent randomly samples an action that differs from the self-inferred one, a modified reason is needed to compose a full reason-then-act trajectory. Yao et al. (2023) show that humans can modify the reasoning in the trajectory and prompt a ReAct-style agent for desirable actions. As humans can provide in-progress supervision, such a human-in-the-loop process still lacks scalability.To automate the process, we propose a complementary ActRe prompting agent to synthesize the modified reasons by leveraging the in-context language ability of an LLM. ActRe inverts the causality of ReAct: while ReAct conditions the external action with a reason a priori, ActRe explains the reason a posteriori for an external action. The synergy of ActRe with ReAct facilitates the autonomous annotation of textual reasons: when the language agent randomly samples an external action, the reason for the sampled action is obtained by querying the ActRe prompting agent. The synthetic reason is then used as the condition of the sampled action for the ReAct-style agent. The progress of a trajectory is synchronized between the ReAct-style agent and the ActRe prompting agent, with the only difference in the order of intermediate reasoning and actions. The detailed workings are depicted below: Denote o t , RS t , EA t as the environmental observation, internal reasoning, and external action at the t-th step, respectively. The trajectory of a ReAct-style agent reads: .. It is important to note that the external action serves as the prior in our framework, rather than the posterior in ReAct. This implies that the sampling space for our framework is equivalent to the external environment and not skewed by the prior of LLM reasoning. At the end of each trajectory, the environment provides a terminal reward R to indicate whether the execution is successful. The reward R automatically annotates the quality of the entire trajectory. In this way, the language agent autonomously composes diverse ReAct-style trajectories without human annotation effort, paving the way for self-training.Language agents are trained by fine-tuning an LLM with the accumulated trajectories. While supervised fine-tuning (SFT) with high-quality data is widely adopted (Zhou et al., 2023b; Singh et al., 2023) , in this work, we improve the awareness of the agent about the contrast between failed and successful trajectories in the same task with policy gradient methods (Williams, 1992) . In the context of ReAct-style language agents, a gathered trajectory τ with T steps reads τ = {o 1 , a 1 , o 2 , a 2 , • • • , o t , a T }, with o t in token strings representing the t-step environmental observation, and a t representing the textual action of the agent being either the internal reasoning RS t or the external action EA t . Given a total of M trajectories {τ m }, we maximize the following objective as the estimation of policy gradient:EQUATIONwith R(τ m ) ≤ 1 as the score of the trajectory τ m , and p θ as the LLM with parameters θ to be fine-tuned. While traditional policy gradient methods omit the p θ (o m t |a m <t , o m <t ) world modeling part, in our work, we keep the term and tune p θ to learn a joint model of action and world modeling. This instructs the LLM to better align with the tasks and the environment.For the gathered trajectories in each task, we filter the composed ones that result in unsuccessful task completion. This ensures that all the failed trajectories generated solely by the agent are paired with successful trajectories in the same task, and all the successful trajectories are retained in the set. Assume that in the same task, we have K ≥ 1 successful trajectories τ 1 s , τ 2 s , • • • τ K s and a failed trajectory τ f . Then Eq.( 1) for the K+1 trajectories can be structured asEQUATIONwhere we use the fact that R(τ k s ) = 1 for all k as they are successful trajectories. According to Eq. (2), we have the following remarks about shaping the reward of the failed trajectory: Remark 1. When R(τ f ) = 0, Eq. ( 2) is reduced to the objective of supervised fine-tuning with only the successful trajectories, which is equivalent to Zhou et al. (2023b) and Singh et al. (2023) . Remark 2. When R(τ f ) = -1, the coefficient of the second part (supervised fine-tuning on the failed trajectory) ∇ log p θ (τ f ) is zeroed. The objective becomes a weighted average of supervised fine-tuning on successful trajectories (the first part), and likelihood contrast between each pair of successful/failed trajectories (the third part). Remark 3. When R(τ f ) = -1 and K = 1, the coefficient of the first part (supervised fine-tuning on the successful trajectories) is zeroed out as well, leaving the objective into a single likelihood contrast (the third part) between trajectory pairs. According to Rafailov et al. (2023) Table 2 : Success rate (%) on each task type of AlfWorld, with iterative refinement on each of the 134 unseen evaluation scenarios. "Iters" denotes the minimum iterations of test-time refinement required to achieve the reported results 1 , and "accum." means the best reward of the trajectories accumulated since the 0-th Round for each task. After the self-training process in Round 3, the success rate of the 4-shot A 3 T agent already reaches 100%.In implementation, we binarize the reward of the failed trajectories with R(τ f ) = -1.To address Remark 3, we let the agent collect multiple successful trajectories via diverse exploration to satisfy K > 1. After training, the new agent would follow Sec. 2.1 to gather more annotated trajectories. The trajectory set then continually grows as looping more rounds of data collection and agent training. For the training in each round, we use the accumulated trajectory set to fine-tune an LLM with Eq. ( 1). Another implementation detail is that in the initial round, we use 1-shot ReAct prompting to gather the training trajectories instead of exploration and annotation for bootstrapping. The failed trajectory for each task is directly excluded as it is not paired with sampled successful trajectories. Eq. ( 1) is therefore reduced to ReAct supervised fine-tuning in Yao et al. (2023) for the training in Round 0. The latter rounds leverage explored trajectories via autonomous annotation, and self-training by Eq. ( 1) with binarized rewards. Other details are covered in Appendix A. 
experiments: 
 We conduct experiments on two benchmarks to valid the effectiveness of A 3 T: the textual embodied environment AlfWorld (Shridhar et al., 2021) , and the online shopping environment WebShop (Yao et al., 2022) . The two benchmarks require a language agent to perform multi-step decision-making to accomplish a certain goal introduced in each task.In A 3 T, we loop for 4 rounds of trajectory collection and agent training, with the initial round using ReAct prompting as the bootstrap of training data. No trajectories are gathered from testing tasks for training. We use gpt-3.5-turbo-instruct-0914 to implement the initial ReAct prompting, as well as the ActRe prompting agent that helps the trajectory composition in the latter 3 rounds. We compare our A 3 T framework with multiple strong baselines, including methods like advanced prompting frameworks using GPT-4, specialized LLMs by full fine-tuning, and gpt-3.5-turbo-1106 fine-tuning. The results are reported in the following sections.Alfworld is a textual embodied environment where an agent needs to accomplish a highlevel goal by reasoning about its situation and performing sequential low-level actuation.Covering 6 task types, the benchmark provides 3,553 tasks for training and 134 held-out tasks for unseen scenarios evaluation. We use 660 out of the 3,553 tasks for our experiments: 600 for training and 60 for validation. In each round, 40 trajectories are composed for each training task failed by the policy agent. See Appendix A for other implementation details.Baseline methods are divided into two categories: the methods that make only a single trial in each test task, and the methods that perform iterative refinement in a test task. In the former category, we select BUTLER (Shridhar et al., 2021) with supervised training over 10 5 expert trajectories on each task type. We also select LM-BUTLER (Micheli & Fleuret, 2021) that fine-tunes a full GPT2-medium by collecting expert demonstrations with the interactions from all the 3,553 tasks (with interleaved observations and external actions in each trajectory). We also compare with the best version of the fully fine-tuned AgentLM (Zeng et al., 2023) in the AlfWorld task (AgentLM-70B), which leverages trajectories from all 3,553 training tasks in AlfWorld and other tasks in different benchmarks. The ReAct prompting (Yao et al., 2023) is also categorized into this category, and we also rerun the method with gpt-3.5turbo-instruct-0914, following their setting to use 6 distinct prompts and report the best performance. In the latter category, we select Reflexion (Shinn et al., 2023 ) that prompts GPT-3.5 to self-reflect with failed trials. We also compare with RAFA (Liu et al., 2023) , a principled iterative planning framework using GPT-4 as the critic.Tables 1 and 2 show the performance comparison of our framework. For the single trial setting, the overall success rate of our agent reaches 96% at 2-nd round and matches the prior SoTA (LM-BUTLER). However, our agent is trained with a QLoRA of 26M parameters and 600 training tasks, while LM-BUTLER is fine-tuned from a full GPT2-medium of 355M parameters and all 3,553 training tasks. Besides, our agent demonstrates constant performance improvements with the 4 rounds in the held-out seen evaluation scenarios and outperforms LM-BUTLER (Table 15 in Appendix C.1). For the iterative refinement setting, our agent obtains 100% success by accumulating the decision-making trials of all the 4 trained agents from each round. The accumulated trajectory set accounts for the significance of the performance. Table 3 shows that the success rate of the trajectories composed by the agent on the training tasks improves continually. More details are covered in Appendix C.1.WebShop is an online shopping environment where an agent needs to purchase the most suitable item according to a provided instruction. The agent should navigate through a sequence of query searches and button clicks on the website to accomplish the task.  
comparisons with react-only sampling: 
 The A 3 T framework relies on ActRe with action sampling for trajectory annotation. In this section, we compare our design on WebShop with the sampling process conducted by the ReAct policy agent only. As Round 0 in our framework does not involve sampling, we start the comparison with our framework since Round 1, and use gpt-3.5-turbo-instruct-0914 with temperature as 0.7 under the fair API call budget. For Rounds 2 and 3, we use the self-trained ReAct policy agents to conduct trajectory sampling and compare them with our framework. To compare the collected trajectories by the two methods, we further define two metrics for quantitative analysis:• Outperformance Rate (OP) computes the percentage of training scenarios where the best trajectory gathered by ActRe has a higher reward than that by ReActonly sampling. Let the reward of the best trajectory in the i-th scenario gathered by ActRe be R(i)ActRe , and that by ReAct-only be R(i)ReAct-only , With the number of training scenarios in total as N, we defineOP = ∑ N i=1 1 R (i) ActRe ≥ R (i)ReAct-only /N. 1). The parenthesized suffixes represent the reward configuration for the trajectories in Eq. ( 1). For example, "(= 1; -1)" means keeping the R = 1 trajectories, and assign the failed non-composed trajectories with R(τ f ) = -1.The training trajectory set is shared among different runs in the same round, except for reward threshold constraints other than R = 1. 
ablation studies: 
 In this section, we conduct ablation studies for the self-training techniques proposed in Section 2.2, and fine-tune the proprietary gpt-3.5-turbo-1106 for further comparisons.We conduct ablated experiments on WebShop, as it provides a real-valued reward ranging from 0 to 1. We first study the effect of different reward thresholds for training trajectory filtering. In A 3 T, only the trajectories with reward R = 1 are used in training. We change the constraint into R ≥ 0.75 and R ≥ 0.5 and compare the performance of the initial round with supervised ReAct fine-tuning. According to Table 8 , the best performance is still obtained with R = 1, which agrees with the findings of Zhou et al. (2023b) .We proceed to ablate the policy gradient technique with binarized rewards and conduct experiments for training in Round 1. The first type to be compared with is supervised fine-tuning. We implement supervised training with the successful trajectories only (namely R(τ f ) = 0 in Eq. ( 2)). This echoes the practice adopted by Singh et al. (2023) . We also prepend the trajectories with the label conditions "Success" / "Fail" in the training data and conduct Round-1 supervised training. For the comparisons with policy gradient Eq. (2), we alternatively set R(τ f ) to be the original reward provided by WebShop, or to be 0.1 following the practice of Wang et al. (2024) . When using the original reward, we also include another setting by relaxing the reward threshold constraint to be R ≥ 0.75. The comparisons are shown in Table 8 . Conclusions can be drawn that: (1) Policy gradient methods lead to higher promotion in task performance than supervised training methods. This resembles the effectiveness of RLHF (Ouyang et al., 2022) on top of supervised fine-tuning.(2) The binarized rewards (±1) used in A 3 T lead to a significant improvement of success rate. We leave the incorporation of advanced RL (Zhou et al., 2024) and RLAIF (Lee et al., 2023; Chen et al., 2024; Hosseini et al., 2024; Yuan et al., 2024) algorithms into A 3 T as future work.While all of the experiments we previously reported are conducted with Mistral-7B-Instruct-v0.2 and QLoRA finetuning, in this section, we also validate A 3 T with gpt-3.5-turbo-1106 finetuning, the proprietary service provided by OpenAI. As the initial trajectory set for Round-0 training in A 3 T is obtained by ReAct prompting with gpt-3.5-turbo-instruct-0914, the starting point for the two base LLMs is the same. Tables 9 and 10 11 , the quality of the accumulated trajectories composed by the proprietary LLM is on par with those composed by the opensourced LLM. After Round-1 self-training, the open-sourced model achieves an even higher test success rate. This is attributed to the proprietary service providing only the supervised fine-tuning option, while also indicating the importance of contrastive fine-tuning in A 3 T. 
conclusion: 
 In this work, we propose A 3 T, a framework that enables the autonomous annotation of agent trajectories in the style of ReAct for contrastive self-training. The key factor in the trajectory annotation process is the ActRe prompting agent, which produces the textual rationales given arbitrary external actions. Together with ActRe and environmental feedback, the ReAct-style agent autonomously synthesizes trajectories for self-training. In the contrastive self-training process, we leverage the policy gradient methods with binarized rewards to boost the task success rate. Extensive experiments on AlfWorld and WebShop have demonstrated the superiority of A 3 T over multiple strong baselines. 
 
 Appendix: We implement autonomous trajectory composition in Rounds 1-3, with Round 0 as the ReAct fine-tuning bootstrapping process. The trained agent for Rounds 1-3 also follows the style of ReAct. Shown in Yao et al. (2023) , humans can modify the mistaken internal reasoning to correct the action behavior of a ReAct agent. In our work, the ActRe prompting agent replaces the human role as a rationale annotator for the ReAct policy agent. The ReAct agent decides whether to sample a novel action with probability p. If the action is not to be sampled, then the ReAct policy agent takes the action conditioned on internal reasoning. Otherwise, the sampled action is sent to the ActRe agent as a query for posterior explanations. In the implementation, we select p = 0.5. If consecutive 3 actions taken by the policy agent itself are invalid, the external action is sampled one more time with the query of ActRe for reason synthesis. In AlfWorld, we let the agent gather 40 trajectories to improve the possibility of attaining at least one successful trajectory for each failed task. In WebShop experiments, due to budgetary limits, we adopt another strategy by first forcing the policy agent to collect 3 trajectories, and then terminate when a successful trajectory is gathered, or the total number of annotated trajectories reaches 20 for a single task. The prompts for ActRe in AlfWorld and WebShop are shown in Tables 12 and 13 , respectively.As for self-training, we use QLoRA finetuning with the open-sourced Mistral-7B-Instruct-v0.2 throughout the main experiments. We set the rank to be 16 and α to be 32 in the QLoRA configurations. The tuned modules are q_proj and v_proj only. We use the nf4 quantization without double quantization in our experiments with QLoRA. The optimizer is set to be the 32-bit paged AdamW. This allows the experiments to be conducted on 2-4 NVIDIA 3090 GPUs with 24GB memory or 2 NVIDIA A100 GPUs with 40GB memory. We set the learning rate as 1e-4, and train 10 epochs for the initial Round 0 (ReAct supervised fine-tuning with successful trajectories). In Rounds 1-3, we fine-tune the LLM for 6 epochs with Eq. 1 (R(τ f ) = -1) and conduct checkpoint averaging over the entire training process.Observation: Your task is to: put some spraybottle on toilet. You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.Action: think[To solve the task, I need to find and take a sparybottle, then put it on toilet.] Observation: OK. Action: act[go to cabinet 1] Reason for the action: [First I need to find a spraybottle. A spraybottle is more likely to appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1) . I can check one by one, starting with cabinet Reason for the action: [For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy.]Table 13 : The ActRe prompt for the WebShop tasks. ActRe inverts the causality of ReAct, and provides the posterior reason for the external action.Figs. 2, 3, 4, and 5 show several cases of trajectories composed with ActRe on WebShop.WebShop Instruction: i am looking for a table and chair set that is white and easy to assemble, and price lower than 260.00 dollars [Search] Obs 1:search[white table and chair set easy to assemble] Act 1:[Back to Search] Page 1 (Total results: 50) [Next >] [B09MDPY329]… [B0727PL2CK]… [B01KZTJN58]… [B099WGDBBC]… [B07YT6WR15]… [B08F7TFRVQ]… [B091927CNJ]… [B08JCM6TLJ]… [B09P35QDY7]… [B00CE69IGA]…Obs 2: Act 1:[Back to Search] Page 1 (Total results: 50) [Next >] [B09QCYNF9R]… [B09QD1GRX5]… [B09Q8LDJJ9]… [B09QPLB4L2]… [B09PDHC7BG]… [B09NPWD8GM]… [B09NPWZLQQ]… [B09NT1MF62]… [B09NT2486L]… [B09NT25D7R]…Obs 2: Act 4:You have clicked 7.5.Obs 4:click[Buy Now]Act 5:You have clicked black.Obs 5:Your score (min 0.0, max 1.0): 0.5Obs 6:(b) Act 1:[Back to Search] Page 1 (Total results: 50) [Next >] [B09QCYNF9R]… [B09QD1GRX5]… [B09Q8LDJJ9]… [B09QPLB4L2]… [B09PDHC7BG]… [B09NPWD8GM]… [B09NPWZLQQ]… [B09NT1MF62]… [B09NT2486L]… [B09NT25D7R]…Obs 2:think[B09NPWZLQQ is a 7.5 no. high heel for women and the price is lower than $130.00.]Act 2: Obs 3:think[For 7.5 no. high heel for women, the item has option '7.5' and seems good to buy.] Act 1:[Back to Search] Page 1 (Total results: 50) [Next >] [B00QB6TW9C]… [B01HP6EQMO]… [B00I9N4IQW]… [B01HP6DYRM]… [B01JEFY2LE]… [B09HHMQFD7]… [B09PWGKJQ5]… [B07H6RVMNL]… [B07CYD69J9]… [B09DRGV127]…Obs 2: Act 3:click[< Prev] click[B07CYD69J9]Act 4: EQUATION[Next >] [B00QB6TW9C]… [B01HP6EQMO]… [B00I9N4IQW]… [B01HP6DYRM]… [B01JEFY2LE]… [B09HHMQFD7]… [B09PWGKJQ5]… [B07H6RVMNL]… [B07CYD69J9]… [B09DRGV127]…Obs 2: According to the results, the performance of the ReAct-only method becomes better on the validation scenarios across different rounds, demonstrating the effectiveness of the contrastive self-training method used to train the models in Rounds 1 3. However, it is shown that our framework achieves the better success rate on test scenarios. Furthermore, the consistency of outperformance is further demonstrated in the iterative refinement evaluation setting. This suggests better generalization, which might be attributed to a more sufficient coverage of high-quality trajectories gathered by our framework.Case Study. The ReAct agent trained by our framework in Round 1 demonstrates the autonomous behaviour listed in Table 20 .This case demonstrates the complex reasoning ability of returning to the previous page and selecting another item. Such behavior is fostered thanks to the ActRe-gathered training data, which results in a final reward of 0.25. By comparison, all the SFT baselines (Round=0) and the agents with ReAct-only self-training achieve 0 reward at this test case. By comparing the gathered trajectories of both our framework and ReAct-only sampling, we found that the Prev]" action in WebShop does not exist in the successful trajectories gathered by ReAct-only sampling in our runs, while such action has been covered in the successful trajectories annotated by our framework. The case study further supports the superiority of our framework in terms of environmental awareness, as the agent is more sufficiently grounded by directly sampling external actions for trajectory annotation, without the limitation of LLM internal reasoning priors of the external world.The refinement iterations of Reflexion and RAFA are measured in their publicized logs: https://github.com/noahshinn/reflexion/blob/main/alfworld_runs/root/reflexion_run_ logs/env_results_trial_10.json and https://github.com/agentification/RAFA_code/blob/main/ ALFWorld/run_logs/env_results_trial_7.json.The pricing is listed in https://openai.com/pricing 


Decision: Summary:
The paper introduces A3T, a framework for autonomously annotating agent trajectories in the ReAct style using an ActRe prompting agent. It enhances self-training for language agents by explaining actions and synthesizing new trajectories. A3T surpasses strong baselines in experiments on AlfWorld and WebShop, demonstrating its effectiveness in improving autonomous decision-making for agents.

Overall, the concerns raised by the authors on additional experiments comparing with ReAct, alternatives for initial trajectories, diversity and scalability of the trajectories are well-addressed. ChatGPT. Their experiments also demonstrate that ActRe achieves a broader coverage of high-reward trajectories compared to sampling with ReAct alone.

While the technique is not very novel, the idea of A3T has a lot of significance in handling scale and providing a new experimental base on closed loop solution for the future work.

Pros:
1. The experiments highlight the importance of this approach compared to supervised fine-tuning and policy gradient with non-binarized rewards.
2. The A3T framework handles scalability issues in gathering expert demonstrations and implementing diverse agent frameworks for trajectory annotation.

Cons:
1. The technique is not very novel as such, as it is very similar to REINFORCE.

Novelty:
This work presents a closed-loop system for continuous improvement, where A3T iteratively enhances agent performance through autonomous annotation and contrastive self-training. Extensive experiments validate the framework's robustness and versatility.

Significance:
1. The paper presents a thorough exploration of existing literature and the obstacles in developing completely autonomous agents with LLMs. A3T uses policy gradient techniques with binary rewards in its contrastive self-training process to significantly improve task success rates. With this motivation, the authors present a closed loop solution with ActRe which opens up a lot of promising avenues to explore RL techniques in this paradigm extensively in the future.
2.  It uses the ActRe prompting agent for autonomous trajectory annotation, reducing human input and dependence on proprietary LLM calls.

[comments from the PCs] Please add the ablations and comparison you conducted during the discussion period to the paper. Also, please conduct more detailed analyses on the types of trajectories that are sampled initially leading to SR and diversity of actions. This can help understand where the improvements are coming from.

Reviews: review: paper_summary: paper_summary: This paper proposes A3T, a framework for autonomous annotation of agent trajectories in the style of ReAct, which enables contrastive self-training of language agents. The key component of A3T is the ActRe prompting agent, which provides textual rationales for arbitrary external actions. The ReAct-style agent leverages ActRe and environmental feedback to synthesize trajectories for self-training. The contrastive self-training process utilizes policy gradient methods with binarized rewards to improve the task success rate. The authors conduct experiments on AlfWorld and WebShop, demonstrating the superiority of A3T over multiple strong baselines, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.
strength_weakness: strength_weakness: - While the authors mention the potential of incorporating advanced RL and RLAIF algorithms into A3T as future work, the current study does not explore these directions. Investigating the integration of more sophisticated RL techniques could further enhance the performance and sample efficiency of the A3T framework.
-  The initial trajectory set for Round-0 training in A3T is obtained by ReAct prompting with gpt-3.5-turbo-instruct-0914. While this provides a starting point for the self-training process, it may limit the diversity of the initial trajectories. The paper does not explore alternative methods for generating the initial trajectory set, such as using different prompting techniques or combining trajectories from multiple sources. Investigating the impact of the initial trajectory set diversity on the final performance could provide insights into the importance of this factor and potential ways to improve it.
paper_summary: paper_summary: This work introduces the A3T framework, which autonomously annotates agent trajectories in the ReAct style using an ActRe prompting agent. The primary idea behind this work is to have an agent that explains the reasons behind arbitrary actions, allowing the ReAct-style agent to synthesize new trajectories by combining these explanations with sampled actions.

The paper has been clearly written, with the motivation and solution fleshed out in reference to the existing works. The paper clearly identifies the challenge of collecting multi-step reasoning and action trajectories, and emphasizes the issues of autonomous decision-making.

Under extensive experiments on AlfWorld and WebShop, the authors clearly show improvements over ReAct, and conduct numerous ablation studies to study the different components of their proposed algorithm. I believe, the findings in the paper will be valuable to the community.
strength_weakness: strength_weakness: As-such, I don't have many concerns with the papers.
paper_summary: paper_summary: This paper presents $A^3$T, a framework that enables the autonomous Annotation of Agent Trajectories in the style of ReACT. The idea is to modify the trajectories with an ActRe prompting agent, which would give the reasoning for any actions. In this way, the ReAct-style agent executes multiple trajectories for specific tasks and can select successful ones together with failed ones for contrastive self-training. 
The contrastive self-training is realized by policy gradient with binarized rewards. Empirical results show that the method can achieve significant improvements compared with existing techniques and methods.
strength_weakness: strength_weakness: - My major concern is that the authors only conduct experiments on two tasks. It would be great to show the advantage over other method on more tasks or benchmarks such as toolbench or toolqueries.
questions: questions: - How do you choose $t$ when you do synthetic data/trajectories generation?
paper_summary: paper_summary: The paper proposes a self-training method for language agents. They propose to gather new reasoning-action trajectories from an agent by first sampling an action and then generating a rationale given the sampled action at each step. Then, they train models on both successful and unsuccessful trajectories with reinforcement learning and investigate designing rewards for these trajectories. Results on AlfWorld and WebShop demonstrate the effectiveness of the method.
strength_weakness: strength_weakness: 1. Since ActRe is one of their claimed contributions, I would expect 1) better motivations and 2) more ablations and analyses on this. The current motivation of ActRe is not well-supported by the experiments, it is possible that ReAct can sample different reasoning trials that lead to diverse actions as well. In addition, ablations and analyses on this should be added. For example, self-training with ReAct-style sampling only is an important baseline that should be evaluated in all of their evaluation settings. Also, because prior work [1] demonstrates that combining different sampling strategies can lead to better performance, more discussion on previous work and more comparisons are required.
2. I do not see much difference between their contrastive self-training method with REINFORCE. I appreciate the efforts in designing the rewards and sampling strategies, but there seems to be a lack of methodological contributions.
3. While the paper mentions that for the previous work, their agent trajectories are limited in diversity and scalability, I do not see they directly solve such limitations in the experiments.


[1] Chen et al., FireAct: Toward Language Agent Fine-tuning, 2023.

Comments: title: Thank you for your response! (2/2)
Thank you for your response! (1/2)
Follow-up clarification about the limitations of ReAct-only sampling
comment: ### More experiments to compare with ReAct-only self-training
___

The following results depict the Table 4, with the task performances of the ReAct-only self-training method added. We have omitted the prior work baselines for simplicity. The best results are listed in **bold**, and the second best results are listed with [brackets].

| Test Trial         | Round | Method     | Valid.    | Test.     |
|--------------------|-------|------------|-----------|-----------|
| Single             | 0     | ReAct-only | 69.1 / 43.2 | 72.4 / 45.8 |
| Single             | 1     | ReAct-only | 69.2 / 43.2 | 73.1 / 47.0 |
| Single             | 2     | ReAct-only | 70.0 / 43.5 | **74.3** / 47.8 |
| Single             | 3     | ReAct-only | **70.5** / **44.0** | [73.9] / 47.2 |
| Single             | 0     | Ours       | [70.1] / 41.0 | 72.4 / 45.4 |
| Single             | 1     | Ours       | 69.7 / 43.0 | 73.1 / **49.0** |
| Single             | 2     | Ours       | 69.0 / [43.8] | 73.0 / [48.0] |
| Single             | 3     | Ours       | 69.1 / 42.8 | [73.9] / **49.0** |
|
| Iterative (accum.) | 1     | ReAct-only | 73.9 / 49.7 | 76.1 / 50.4 |
| Iterative (accum.) | 2     | ReAct-only | 75.8 / [52.0] | 77.4 / 52.6 |
| Iterative (accum.) | 3     | ReAct-only | **76.6** / **53.5** | [78.0] / [53.4] |
| Iterative (accum.) | 1     | Ours       | 74.0 / 47.3 | 76.6 / 51.6 |
| Iterative (accum.) | 2     | Ours       | 75.1 / 49.5 | 77.8 / [53.4] |
| Iterative (accum.) | 3     | Ours       | [75.9] / 51.3 | **78.2** / **54.8** |

According to the results, the performance of the ReAct-only method becomes better on the validation scenarios across different rounds, demonstrating the effectiveness of the contrastive self-training method used to train the models in Rounds 1~3. However, it is shown that our framework achieves the better success rate on test scenarios, and the consistency of outperformance is further demonstrated in the iterative refinement evaluation setting. The reason might be that the trajectories gathered with ActRe enjoy higher scenario-wise **Success Coverage** and **Outperformance Rate** compared with those by ReAct-only sampling, and overfitting is thus reduced when training with the data gathered with ActRe. 

We hope the added experiments can further address your concerns, and we will add these experiments as well as relevant discussions to the paper.



### Other weaknesses
___

***Weakness 2. Lack of methodological contributions.***

As mentioned in our paper, the primary contribution of our work is the closed-loop framework for agents to self-improve. In this framework, agents autonomously annotate the rationales for the sampled actions from the environment in each trajectory, and self-improve through contrastive self-training. While we indicated in Section 2.2 that we follow the idea of policy gradient for self-training, we conduct the rearrangement in Eq.(2), which has inspired us with its relationship to SFT and contrastive training (Remarks 1~3). We have also conducted corresponding ablation studies in Sec. 4.1 to validate the relationship and the superiority of contrastive self-training. As indicated in Sec. 4.1, we leave the integration of more advanced RL techniques for future work.

***Weakness 3. Diversity and scalability issues of agent trajectories.***

Our ActRe allows a ReAct agent to directly sample actions from the environment by annotating the rationales in each trajectory. By grounding in the environment, our framework can in theory collect trajectories with arbitrary action allowed in the environment at each time step. Furthermore, the sampling process of ActRe can be executed in parallel for different trajectories, since the probability at each time step to sample an action and query ActRe is fixed as $p=0.5$ (Appendix A). 

According to the comparison of the collected trajectories (see the augmented Table 6), the trajectories collected by ActRe are of consistently higher reward and success rate than ReAct-only sampling that relies on the internal reasoning of LLMs only. Besides, Reflexion suggested in FireAct does not yield improved performance on WebShop because of the environmental complexity (indicated by the Reflexion paper in Appendix B.1). These experiments and facts validate that our framework collects agent trajectories with improved diversity and scalability. We will add relevant discussions to our paper.
Thank you for providing us with the discussion opportunity! 

We are glad to know that you appreciate the experiments we added. We have been trying our best to run additional experiments to cover all the evaluation settings in the main experiments of WebShop. In the following, we provide the full comparison between ReAct-only self-training and our framework (A$^3$T with ActRe) following Tables 4 (task performance) and 6 (data quality) in our paper. For a fair comparison, we use the "SFT with initial ReAct-only sampling" model in our previous rebuttal as the Round=0 for ReAct-only, and "contrastive self-training with initial ReAct-only sampling" model in our earlier rebuttal as the Round=1 for ReAct-only. We have proceeded to use the Round=1 trained model to do ReAct-only sampling on the training scenarios, and use the accumulated trajectories to train the Round=2 ReAct-only model with contrastive self-training; The same goes for the Round-3 ReAct-only model.

### Supporting the claimed motivation
___

We start with the augmented Table 6, which compares the quality of the accumulated data between our framework and ReAct-only sampling. As the initial ReAct-only sampling was conducted with fair comparison to the sampling with ActRe in Round 1, we omit Round 0 of ours in the following Table.

| Round | Method     | Reward R (\*100) | Success Rate (%) | %{R$\ge$0.75} | %{R$\ge$0.5} |
|-------|------------|-----------------|------------------|------------|------------------|
| 1     | ReAct-only | 81.6            | 55.3             | 69.2       | 91.3             |
| 1     | Ours       | 85.2            | 61.1             | 75.8       | 94.3             |
|
| 2     | ReAct-only | 83.0            | 58.5             | 71.3       | 92.2             |
| 2     | Ours       | 88.9            | 69.4             | 82.6       | 96.3             |
|
| 3     | ReAct-only | 83.3            | 59.1             | 71.8       | 92.4             |
| 3     | Ours       | 90.6            | 73.9             | 85.0       | 96.8             |

According to the results, our framework collects the trajectories with consistently higher rewards and success rate. These quantitative results support our motivation, as the successful / high-reward trajectories could be collected only when the agent system is grounded in the environment with better awareness, and our ActRe has gathered better trajectories in this sense than the ReAct-only method that relies only on the internal reasoning of LLMs. 

We also define two scenario-wise comparison metrics to show the advantage of ActRe in data collection:

- **Outperformance Rate** computes for the percentage of training scenarios where the best trajectory gathered by ActRe has a higher reward than that by ReAct-only sampling. Let the reward of the the best trajectory in the $i$-th scenario gathered by ActRe be $R_{\mathrm{ActRe}}^{(i)}$ , and that by ReAct-only be $R_{\mathrm{ReAct-only}}^{(i)}$. With the number of training scenarios in total as $N$, we define
 
$$\mathrm{Outperformance~Rate} = \frac{{\sum_{i=1}^{N} \mathbf{1}} \\{R_{\mathrm{ActRe}}^{(i)} \ge R_{\mathrm{ReAct-only}}^{(i)} \\}}{N}$$

- **Success Coverage** calculates the percentage of training scenarios where ReAct-only sampling collects successful trajectories, and ActRe achieves success as well. Let $S_{\rm ReAct-only}$ be the set of scenarios where ReAct-only sampling succeeds, and $S_{\rm ActRe}$ goes similar for ActRe, we define

$$\mathrm{Success~Coverage} = \frac{|S_{\rm ActRe} \cap S_{\rm ReAct-only}|}{|S_{\rm ReAct-only}|}$$

Computing the two metrics for the accumulated trajectories of all rounds, we have the following table:

| Round | Outperformance Rate | Success Coverage |
|-------|---------------------|------------------|
| 1     | 92.1                | 89.9             |
| 2     | 96.4                | 95.0             |
| 3     | 98.0                | 97.2             |

This additionally demonstrates that ActRe obtains a better coverage of high-reward trajectories over ReAct-only sampling, showing its superiority of environmental awareness and supporting our motivation.
Thank you for your response!

I appreciate the additional experiments, but I would expect more results from the ReAct-only baseline as mentioned in my review. I understand that this is difficult to complete during the short rebuttal period, but I do believe this is an important baseline and comparisons should be made in most of their settings.

The added case study is a good starting point to support the claimed motivation, yet the results are relatively preliminary and more quantitative and qualitative results can be added.

In summary, I like the provided experiments, but I would expect more of them for the paper to be published. I'm also curious about how the authors address my second and third points in the weakness section.
We'd like to add further clarification about the limitations of ReAct-only sampling, especially to rectify the potentially overclaiming statement of "`click[< Prev]` *action in WebShop is never performed by ReAct-only sampling*" in our rebuttal.

The statement was drawn by observing the set of *successful* trajectories obtained by ReAct-only sampling: The `click[< Prev]` action does not exist in this set. When observing all the trajectories gathered by ReAct-only sampling, we indeed find that the `click[< Prev]` action has been performed. However, in our runs, all the trajectories containing the `click[< Prev]` action do not reach the full reward. Compared with the trajectory set gathered by ActRe, the ReAct-only sampling falls short of environmental understanding in terms of the possibility of `click[< Prev]` leading to success. Therefore, we'd like to update the statement in our rebuttal to "`click[< Prev]` *action in WebShop does not exist in the successful trajectories gathered by ReAct-only sampling in our runs*". 

Following the ablation studies in Section 4.1, the $R=1$ trajectories in the training set do not cover the `click[< Prev]` action for ReAct-only sampling. On the contrary, the `click[< Prev]` action often appears in the successful trajectories set with ActRe sampling. This explains the case study shown in our rebuttal.

The motivation for ActRe proposed in our rebuttal still holds: Our framework better grounds the agent by rectifying the policy of the LLM itself with uniform action sampling from the environment when exploring the training scenarios. Such practice shares similar intuition with optimistic exploration for the less-chosen actions, as we complement additional exploration probabilities to them.