ReAct Meets ActRe: Autonomous Annotation of Agent Trajectories for Contrastive Self-Training
Authors: Zonghan Yang
Peng Li
Ming Yan
Ji Zhang
Fei Huang
Yang Liu
Venue: COLM (2024)
Keywords: autonomous annotation, language agent, self-training, contrastive
TL;DR: We propose a framework that enables the autonomous annotation of agent trajectories to facilitate contrastive self-training for language agents, outperforming existing agent frameworks and prompting methods with GPT-4.

Abstract: Language agents have demonstrated autonomous decision-making abilities by reasoning with foundation models. Recently, efforts have been made to train language agents for performance improvement, with multi-step reasoning and action trajectories as the training data. However, collecting such trajectories still requires considerable human effort, by either artificial annotation or implementations of diverse prompting frameworks. In this work, we propose A$^\mathbf{3}$T, a framework that enables the Autonomous Annotation of Agent Trajectories in the style of ReAct. The central role is an ActRe prompting agent, which explains the reason for an arbitrary action. When randomly sampling an external action, the ReAct-style agent could query the ActRe agent with the action to obtain its textual rationales. Novel trajectories are then synthesized by prepending the posterior reasoning from ActRe to the sampled action. In this way, the ReAct-style agent executes multiple trajectories for the failed tasks, and selects the successful ones to supplement its failed trajectory for contrastive self-training. Realized by policy gradient methods with binarized rewards, the contrastive self-training with accumulated trajectories facilitates a closed loop for multiple rounds of language agent self-improvement. We conduct experiments using QLoRA fine-tuning with the open-sourced Mistral-7B-Instruct-v0.2. In AlfWorld, the agent trained with A$^3$T obtains a 1-shot success rate of 96\%, and 100\% success with 4 iterative rounds. In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts. A$^3$T agents significantly outperform existing techniques, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.

Published as a conference paper at COLM 2024
ReAct Meets ActRe: Autonomous Annotation of Agent Trajectories for Contrastive Self-Training
Zonghan Yang', Peng Lifx, Ming Yan Ji Zhang?, Fei Huang?, Yang Liu
Department of Computer Science and Technology, Tsinghua University, Beijing, China
2 Institute for Al Industry Research (AIR), Tsinghua University, Beijing, China
3 Institute of Intelligent Computing, Alibaba Group
4 Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China yangzh20@mails tsinghua. edu.cn
Abstract
Language agents have demonstrated autonomous decision-making abilities by reasoning with foundation models. Recently, efforts have been made to train language agents for performance improvement, with multi-step reasoning and action trajectories the training data. However, collecting such trajectories still requires considerable human effort, by either artificial annotation implementaiions of diverse prompting frameworks. In
this work, we propose A framework that enables the Autonomous Annotation of Agent Trajectories in the style of ReAct. The central role is an ActRe prompting agent, which explains the reason for an arbitrary action. When randomly sampling an external action, the ReAct-style agent could query the ActRe agent with the action obtain its textual rationales. Novel trajectories are then synthesized by prepending the posterior reasoning from ActRe to the sampled action. In this way, the ReAct-style agent executes multiple trajectories for the failed tasks, and selects the successful ones to supplement its failed trajectory for contrastive self-training. Re- alized by policy gradient methods with binarized rewards, the contrastive self-training with accumulated trajectories facilitates a closed loop for mul- tiple rounds of language agent self-improvemmnt. We conduct experiments using QLoRA fine-tuning with the open-sourced Sistral-7B-Instrucee
In AlfWorld, the agent trained with A obtains a 1-shot success rate of 96%, and 100% success with 4 iterative rounds. In WebShop, the 1-shot performance of the A agent matches human average, and 4 rounds of iter- ative refinement lead to the performance approaching human experts. A agents significantly outperform existing techniques, including prompting with GPT-44, advanced agent frameworks, and fully fine-tuned LLMs.
Introduction
The rapid development of Large Language Models (LLMs) (OpenAI 2023: Touvron et al.,
2023: Team et al. 2023 Jiang et al. 2024) has led to the prosperity of language agents. Leveraging the ability of LLMs, language agents have demonstrated impressive perfor- mances in diverse mecision-making scenarios by interacting with the environments au- tonomously (Wang al., 2023 Mirchandaní et al. 2023 Zheng al. 2024 Wu al. 2024).
Recently, increasing efforts have been made to train language agents with open-sourced L.MSs. The multi-step trajectories that describe the entire task-solving process of a language agent are used as training data, which consist of environmental observations, internal rea- soning texts, and external actions. The collection of such trajectories is therefore essential, which are currently categorized into two paradigms in Fig. D (a) and (b). The first paradigm to leverage expert demonstrations (Yoo et al. 2022) However, the expense human labor
* *ooreespodiing authors: P.I.. (Ipeeng@airtssngguu.eeu... and Y.I... (liuyang201 (@tsinghua.edu.cm)

---

Published as a conference paper at COLM 2024
(a)
(b)
Act
Act
Diverse Prompting
Human
Environment
Human
Agent
Environment
Obs.
Obs.
(c)
Contrastive Self- Self-Training
Act
Act
Traj (ReAc) only)
Dbs; Reason, Acts
0bs Reason, Act.
Roward-0 X
Traj Actia)
Obsg Reason, Act,
ActRe
ReAct
Environment
Obs, Reason" Act'
Reward-1
Reason
Obs.
Autonomous Trajectory Annotation
Figure 1: Upper: Two common paradigms to collect trajectories for language agents. (a) Tra- jectories are artificially annotated as human demonstrations, (b) Trajectories are gathered by deploying policy agents that reason and act in the language form. However, both paradigms require considerable human effort in either data annotation or different implementations of agent frameworks, thus lacking scalability in the data collection process. Lower: (c) Our T framework. A enables the Autonomous Annotation of Agent Trajectories in ReAct style with an ActRe agent, facilitating the closed loop of contrastive self-training.
hampers the scalability of the approach. Another paradigm is implementing different agent frameworks to gather diverse trajectories with proprietary LLMs (Qin et al. 2023: Zeng
al. 2023 Chen et al. 2023 Aksitov et al. 2023). However, the exploration coverage in the training data still upper-bounded by the tull set of prompting techniques. Besides, im- plementing diverse agent frameworks requires considerable human efforts and proprietary
LLM calls Yang et al. 2024). To ease the data collection process in diverse scenarios, Yin et al.) (023) and Zhang et al. (2024) propose unified data formats by elucidating the comprising submodules agent trajectories. However, as obtained by converting aanan-annotated data or one single defaulted prompting scheme, the agent trajectories are still limited in diversity and scalability. Considering that an environment automatically returns observa- tions and rewards with action inputs, it should serve as an infinite data generator. While Song et al. (222) propose an exploration-based agent framework for eelf-improvement, the gathered trajectories consist of only interleaved external actions and environmental obser- vations, without textual rationales that could steer better behavior of language agents. We ask the following question: Can language agent autonomously gather high-quality trajectories from the external environment, with textual annotations suitable for further self- self-training?
In this work, we propose A a framework that enables Autonomous Annotation of Agent Trajectories in the style of ReAct (Yao et al., 2023) for self-improvement with minimal human supervision. The central idea is to exploit both the in-context language ability and the decision-making ability of a language agent: To collect diverse trajectories, an agent could randomly sample external actions from the action space at arbitrary steps. However, the corresponding reason for the sampled action should be annotated for ReAct-style agent. To facilitate this, we propose ActRe, an act-then-reason prompting agent that explains the reason for the sampled action. With ActRe, the ReAct-style agent composes extra reason- then-act trajectories for each failed task by inversely prepending the ActRe-prompted reason to the randomly sampled action. After the execution of each composed trajectory, the agent receives terminal reward from the environment, which automatically annotates the quality of the trajectory. The gathered successful trajectories are then supplemented with the failed trajectory by the ReAct-style agent alone for contrastive self-training, where policy
2

---

Published as a conference paper at COLM 2024
gradient methods (Williams, 1992) with binarized rewards for LLM fine-tuning. As new agents are trained, more trajectories can be gathered and accumulated, which forms a closed loop for the self-improvement of language agents as shown in Fig. (c).
We validate our A framework in the textual embodied environment AlfWorld (Shridhar iet al. 2021) and the online shopping environment WebShop (Yao et al.; 2022). We use
QLoRA (Dettmers et al. 2023) to fine-tune Misttal--B-Insttruct-v (Jiang et al. 2023) in the training experiments. Experimental performances demonstrate significant improvement over state-of-the-arr agent techniques: On AlfWorld, our trained agent achieves 96% success rate in unseen scenarios with a single trial. On WebShop, the success rate of our agent reaches 49,, which matches the average human performance (50%). In the setting of iterative refinement, after four rounds of data collection and contrastive self-training, the accumulated success rate becomes 100% on AlfWorld and 54.8% on WebShop, narrowing the gap with human experts (59.6% on WebShop). A paves the way for agents with improved autonomy through the closed loop of self-annotation and contrastive self-training.
2 A for Closed-Loop Self-Improvement
In this section, we introduce the closed-loop self-improvement for agents facilitated by the A ' framework. The loop contains two parts: autonomous trajectory annotation with the ActRe agent (Sec. 2.1, and the contrastive self-training process (Sec. 2.2
2.1 Autonomous Trajectory Annotation with ActRe
Agents are able to gather diverse trajectories by exploration. However, for language agents like ReAct, the actions are inferred by first reasoning with LLMs. When the agent randomly samples an action that differs from the self-inferred one, a modified reason is needed to compose full reason-then reasoning in the trajectory and prompt a ReAct-style agent for desirable actions. As humans can provide in-progress supervision, such human-in-the-loop process still lacks scalability.
trajectory. Yao et al. (2023) show that humans can modify the
To automate the process, we propose a complementary ActRe prompting agent to synthesize the modified reasons by leveraging the in-context language ability of an LLM. ActRe inverts the causality of ReAct: while ReAct conditions the external action with a reason a priori, ActRe explains the reason a posteriori for an external action. The synergy of ActRe with ReAct facilitates the autonomous annotation of textual reasons: when the language agent randomly samples an external action, the reason for the sampled action is obtained by querying the ActRe prompting agent. The synthetic reason is then used as the condition of the sampled action for the ReAct-style agent. The progress of a trajectory is synchronized between the ReAct-style agent and the ActRe prompting agent, with the only difference in the order intermediate reasoning and actions. The detailed workings are depicted below:
Denote 0,, RS;, EA, as the environmental observation, internal reasoning, and external action at the f-th step, respectively. The trajectory of ReAct-style agent reads:
01, RS,, E.A, 0 RS EAt11
The synchronized ActRe prompting agent has the following trajectory:
0₁, E,A,, RSt, 0++1, EAH+1, RS
Now when the ReAct-style agent explores for a different external action at f + step by changing EA4+1 into EA411, the corresponding internal reasoning RS,+1 should be altered as well. This is achieved by querying the ActRe prompting agent:
0₁, EA,, RS,, 0t+1, EA > RS
then the synthesized RS,41 and the sampled EA441 compose new ReAct trajectory:
Ot, RS,, E.A 0t1 RS EA
3

---

Published as a conference paper at COLM 2024
It is important to note that the external action serves as the prior in our framework, rather than the posterior in ReAct. This implies that the sampling space for our framework is equivalent to the external environment and not skewed by the prior of LLM reasoning. At the end of each trajectory, the environment provides a terminal reward R to indicate whether the execution is successful. The reward R automatically annotates the quality of the entire trajectory. In this way, the language agent autonomously composes diverse ReAct-style trajectories without human annotation effort, paving the way for self-training.
2.2 Contrastive Self-Training
Language agents are trained by fine-tuning an LLM with the accumulated trajectories. While supervised fine-tuning (SFT) with high-quality data is widely adopted (Zhou et al. 2023b: Singh et al., 2023), in this work, we improve the awareness of the agent about the contrast between tailed and successful trajectories in the same task with policy gradient methods (Williams 1992 In the context of ReAct-style language agents, a gathered trajec- tory T with steps reads T = {o1, 001,01,0,,440 02, A2, ... , ar), with o; in token strings representing the 1-step environmental observation, and a representing the textual action of the agent being either the internal reasoning RS, or the external action E.A,. Given a total of M trajectories {T"),, we maximize the following objective as the estimation of policy gradient:
1 M
(0) M m=1
Σ
R(T")
V log Po(T")
(1)
1 M
T Σ V log Po a" o + Vo Po am)))
Σ
R(T
with R(T") < 1 as the score of the trajectory T" and pe as the LLM with parameters 0 to be fine-tuned. While traditional policy gradient methods omit the Po " om,,) world modeling part, in our work, we keep the term and tune pe to learn a joint model of action and world modeling. This instructs the LLM to better align with the tasks and the environment.
For the gathered trajectories in each task, we filter the composed ones that result in un- successful task completion. This ensures that all the failed trajectories generated solely by the agent are paired with successful trajectories in the same task, and all the successful trajectories are retained in the set. Assume that in the same task, we have K 2 successful trajectories T T², ... r and a failed trajectory Ty. Then (1 for the K+1 trajectories can be structured as
log (o + R(Ty) V loggo(Tz)
1 - R(TT)
K
1+ R(T))
Σ
V VlogPa(TT)
v VlogPe(TT)
(2)
2K
1 - R(TT))
V log (T()) V log (ogy))
2K
where we use the fact that R(x²) = for all as they are successful trajectories. According to Eq. (2), we have the following remarks about shaping the reward of the failed trajectory:
Remark 1. When R(T₁) = 0, Eq. 2 is reduced to the objective of supervised fine-tuning with only the successful trajectories, which equivalent to Zhou et al. (2023)) and Singh et al. (2023).
Remark 2. When R(T,) 11, the coefficient of the second part (superoised fine-tuning on the failed trajectory) V log PT(TT) ) is zeroed. The objective becomes weighted average of supervised fine-tuning on successful trajectories (the first part), and likelihood contrast between each pair of successfulffailed trajectories (the third part).
Remark 3. When R(T,) -1 and K 1, the coefficient of the first part (superoised fine-tuning on the successful trajectories) is zeroed out as well, leaving the objective into single likelihood contrast (the third part) between trajectory pairs. According to Rafaiioo et al. (2023), this leads to poor performance because of training instability.
4

---

Published as a conference paper at COLM 2024
Method
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
B Shridhar et al. 2021
46
39
74
100
22
24
37
AgentLM Zeng
2023
86
LM-BUTLER [ichelli & Fleuret 2021
96
97
96
90
100
94
96
ReAct, Yao et at,
58
86
78
41
71
ReAct, (Our rerun)
87
100
88
A (Round=0)
77
100
95
94
47
A (Round=1)
96
94
96
95
89
94
94
A (Round=2)
100
100
100
95
89
82
96
A (Round=3)
100
100
100
Table 1: Success rate (%) on each task type of AlfWorld, with a single trial on each of the 134 unseen evaluation scenarios. "BUTLERg" and "ReAct," denotes the best performance of 8/6 trials following prior work. The A agents are trained with 600 out of the total 3,553 training tasks in AlfWorld. ": The best result reported in Zeng et al. (2023).
Method
Iters.
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
Reflexion (Shinn al.
2023
96
94
100
95
100
100
97
RATA (Liu et al, 2023
100
97
100
100
100
100
A (Round=0)
77
100
47
A (Round=1, accum.)
96
94
100
95
100
100
97
A (Round=2, accum.)
100
100
100
A (Round=3, accum.)
100
100
100
100
100
100
100
Table 2: Success rate (%) on each task type of AlfWorld, with iterative refinement on each of the 134 unseen evaluation scenarios. "Iters" denotes the minimum iterations test-time refinement required to achieve the reported results") and "accum." means the best reward of the trajectories accumulated since the 0-th Round for each task. After the self-training process in Round 3, the success rate of the 4-shot A agent already reaches 100%.
In implementation, we binarize the reward of the failed trajectories with R(T)) =
To address Remark 3, we let the agent collect multiple successful trajectories via diverse exploration to satisfy K > After training, the new agent would follow Sec. t. gather more annotated trajectories. The trajectory set then continually grows as looping more rounds of data collection and agent training. For the training in each round, we use the accumulated trajectory set to fine-tune an LLM with Eq. (1). Another implementation detail that in the initial round, we use 1-shot ReAct prompting to gather the training trajectories instead of exploration and annotation for bootstrapping. The failed trajectory for each task is directly excluded as is not paired with sampled successful trajectories. Eq. (I) is therefore reduced to ReAct supervised fine-tuning in Yao et al. (2023) for the training in Round O. The latter rounds leverage explored trajectories via autonomous annotation, and oofftrrainigg by Eq. (1 with binarized rewards. Other details are covered in Appendix A
Experiments
We conduct experiments on two benchmarks to valid the effectiveness of A the textual embodied environment AlfWorld (Shridhar et al., (202)), and the online shopping environ- ment WebShop (Yao et al. 2022 The two benchmarks require language agent to perform a multi-step decision-makinn to accomplish a certain goal introduced in each task.
In TT, we loop for rounds of trajectory collection and agent training, with the initial round using ReAct prompting as the bootstrap of training data. No trajectories are gathered from testing tasks for training. We use apt-3.5-turbo-instree prompting, as well as the ActRe prompting agent that helps the trajectory composition in the latter rounds. We use the open-sourced Eustral-7B-Instructte with QLoRA (Dettmers et al. 2023) finetuning for the training experiments.
to implement the initial ReAct
(Jiang et al. 2023

---

Published as a conference paper at COLM 2024
Round
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
0
76
85
92
85
86
77
82
100
100
100
100
100
99
99
100
100
100
100
100
100
100
3 100
100
100
100
100
100
100
Table 3: The success rate (%) of the accumulated trajectory set in terms of each task type in the training scenarios of AlfWorld. With the agent autonomously annotating diverse trajectories, the data quality for contrastive self-training progressively improving.
We compare our A framework with multiple strong baselines, including methods like advanced prompting frameworks using GPT-4, specialized LLMs by full fine-tuning, and spt-3.5-turbo-1066 fine-tuning. The results are reported in the following sections.
3.1 AlfWorld
Alfworld is textual embodied environment where an agent needs accomplish high- level goal by reasoning about its situation and performing sequential low-level actuation. Covering 6 task types, the benchmark provides 3,553 tasks for training and 134 held-out tasks for unseen scenarios evaluation. We use 660 out of the 3,553 tasks for our experiments: 600 for training and 60 for validation. In each round, 40 trajectories are composed for each training task failed by the policy agent. See Appendix A for other implementation details.
Baseline methods are divided into two categories: the methods that make only a single trial in each test task, and the methods that perform iterative refinement in test task. In the former category, we select BUTLER (Shridhar et al. 2021) with supervised training over 10 expert trajectories on each task type. We also select LM-BUTLER (Micheli & Fleuret 2021 that fine-tunes full GPT2-medium by collecting expert demonstrations with the interactions from all the 3,553 tasks (with interleaved observations and external actions in each trajectory).
We also compare with the best version of the fully fine-tuned AgentLM (Zeng et al.) 2023) in the AlfWorld task (AgentLLM-70B) which leverages trajectories from all 3,553 training tasks in AlfWorld and other tasks in different benchmarks. The ReAct prompting YYao al. 2023) is also categorized into this category, and we also rerun the method with gpt-3.5- turbo-instruct-O9 following their setting to use 6 distinct prompts and report the best performance. In the latter category, we select Reflexion (Shinn et al., 2023 that prompts GPT-3.5 to self-reflect with failed trials. We also compare with RAFA (Liu et al., 2023 principled iterative planning framework using GPT-4 as the critic.
Tables and show the performance comparison of our framework. For the single trial setting, the overall success rate of our agent reaches 96% at 2-nd round and matches the prior SoTA (LM-BUTLER). However, our agent is trained with a QLoRA of 26M parameters and 600 training tasks, while LM-BUTLER is fine-tuned from full GPT2-medium of 355M parameters and all 3,553 training tasks. Besides, our agent demonstrates constant performance improvements with the 4 rounds in the held-out seen evaluation scenarios and outperforms LM-BUTLER (Table 15 in Appendix C.1]. For the iterative refinement setting, our agent obtains 100% success by accumulating the decision-making trials of all the trained agents from each round. The accumulated trajectory set accounts for the significance of the performance. Tsble that the success rate of the trajectories composed by the agent on the training tasks improves continually. More details are covered in Appendix C.1
3.2 WebShop
WebShop is an online shopping environment where an agent needs to purchase the most suitable item according a provided instruction. The agent should navigate through sequence of query searches and button clicks on the website to accomplish the task.
'The refinement iterations of Reflexion and RAFA are measured in their publicized and httprpaaaa..a.o.ooooooaaaotooata AL L..Noo///ruu.loogs/eveeeessssss.

---

Published as a conference paper at COLM 2024
Test Trial
Method
Valid.
Test.
ReAAt (Yao et al.
2023
66.6/40.0
ReAct (Our rerun)
63.7/34.7
68.5/40.0
WebGUM (Furuta et al 2024
67.5/45.0
Single
AgentLM ZZng et al.
2023
70.8/
A (Round=0)
70.1/41.0
72.4/45.4
AT (Round=1)
69.7/43.0
73.1/49.0
A (Round=2)
69.0/43.8
73.0/48.0
A (Round=3)
69.1/42.8
73.9/49.0
LATS30 (Zhou et al. 2023a
75.9/38.0
A²T₂ (Round=1, accum.)
74.0/47.3
76.6/51.6
Iterative
A T (Round=2, accum.)
75.1/49.5
77.8/53.4
A (Round=3, accum.)
75.9/51.3
78.2/54.8
Human (average) (Yao et al 2022
75.5/50.0
Human (expert)
Yao et al. 2022
82.1/59.6
Table 4: Reward (x 100) and success rate (x 100%) on the validation and the test sets in WebShop. The best result reported in Zeng et al. (2023). "LATS30" or "A²T4" denotes using 30 test trials. The A agents are trained with 2,300 out of the total 11,587 training and validation tasks in WebShop. The single-shot A matches averaged human performance, while the multi-shot A closes the performance gap with human experts.
LLM
easy
hard
all
GPT-3.5-Tuubo-16k--h66 GPT-4-Turbo-0613' GPT-4-32N-0613-
52.8
50.6
52.2
Prompting
67.6
67.4
67.6
67.5
69.6
68.1
XLAM-v0.1"
[Zhang et 2024
53.2
51.2
52.4
A (Round=0)
74.1
66.5
72.0
Training
A (Round=1)
73.6
73.2
73.5
A (Round=2)
71.6
74.2
72.3
A (Round=3)
72.5
73.9
72.9
Table 5: The average reward (x100) in the easy easy/hard split for WebShop by AgentBoard (Ma et al. 2024). *; Results reported in Liu et al. (2024). The single-shot A model significantly surpasses prompting methods with the most capable LLMs like GPTT4.
Round
Reward R (x100)
Success Rate (%)
(R 2 0.75)
% 2 0.5}
68.5
40.0
51.6
77.5
1
85.2
61.1
75.8
94.3
2
88.9
69.4
82.6
96.3
90.6
73.9
85.0
96.8
Table 6: The reward, success rate, and percentages of reward above thresholds 0.75 and 0.5 of the accumulated trajectory set in the training tasks of WebShop. With the agent autonomously composing diverse trajectories, the data quality is continually increasing.
success. The benchmark provides 11,587 tasks for training and validation, and 500 held-out tasks as testing scenarios. We use 2,700 out of the 11,587 tasks for our experiments, with 2,300 for training and 400 for validation. 20 trajectories are composed for each training task failed by our trained agent in each round. Other training details are listed in Appendix
WebShop provides a real-valued reward R E [0, 1), with R
Baseline methods are still divided by whether or not to perform test-time iterative re- finement. For the setting of single test trial, we compare with ReAct prompting and WebGUM (Furuta et al.) 2024) by jointly fine-tuning ViT visual encoder and F--LL Recently, AgentBoard (Ma et al.) 2024) offers an easy hasy/ split of the first 251 test tasks in WebShop for better evaluation, and Liu et al. (0024) report the benchmarked results of
7

---

Published as a conference paper at COLM 2024
Round Reward (x100)
Success Rate (%)
% % 0.75)
% 2 0.5)
%OP
%SC
81.6 (-4.2%)
55.3 (-9.5%)
69.2
91.3
92.1
89,9
83.0 (-6.6%)
58.5 (-15.7%)
71.3
92.2
96.4
95.0
83.3 (-8.1%)
59.1 (-20.0%)
71.8
92.4
98.0
97.2
Table 7: The reward, success rate, and percentages of reward above thresholds 0.75 and 0.5 of the accumulated trajectory set by the ReAct-only sampling process, as well as the calculated OP and SC metrics. The relative gaps of the rewards and success rates are computed with those achieved by ActRe in Table 6
XLAM-v0.1 (Zhang et al. 2024) with multi-task full finetuning of Mixtral-6x77-Insttuee v0.1 iiang et al. 2024). We also include XLAM-v0.1 (Zhang et al., 2024) as a single-shot baseline and report the performance comparison on AgentBoard. While LUMOS (Yin et al.,
2023) shares a similar spirit with xLAM-v0.1, the WebShop task is treated as an unseen scenario in their setting. To conduct a fair comparison, we do not compare ours with LU- MOS. For the setting that allows test-time iterative refinement, Reflexion has been claimed to be ineffective in [Shinn et al. 2023). We compare ours with LATS (Zhou et al. 2023a), prompting-basee language agent with multiple rounds of self-reflection and tree search.
Tables and 5 demonstrate the significance of A agents. With a single test trial, the A agent matches averaged human performance (reward: 73,9 v.s. 75.5; success rate: 49.0% v.s. 50.0%). With shots of test trials, A achieves a 54.8% success rate, closing the gap with human expert performance (59.6%). The 1-shot T agent also outperforms prompting with GPT-4-22-00613 in both the easy and the hard split of WebShop from AgentBoard. Table 6 further shows the quality improvement of the accumulated trajectories across multiple rounds of A AT. Case studies for annotated trajectories, as well as the dataset statistics for each round of training are reported in Appendices B and C.2 respectively.
Comparisons with ReAct-only Sampling
The A framework relies on ActRe with action sampling for trajectory annotation. In this section, we compare our design on WebShop with the sampling process conducted by the ReAct policy agent only. As Round 0 in our framework does not involve sampling, we start the comparison with our framework since Round 1, and use spt-3.5-turbo-instrus with temperature as 0.7 under the fair API call budget. For Rounds 2 and 3, we use the self-trained ReAct policy agents to conduct trajectory sampling and compare them with our framework. To compare the collected trajectories by the two methods, further define two metrics for quantitative analysis:
Outperformance Rate (OP) computes the percentage of training scenarios where the best trajectory gathered by ActRe has higher reward than that by ReAct- only sampling. Let the reward of the best trajectory in the i-th scenario gathered
by ActRe be R (i) ActRe and that by ReAct-only be ReAct-ont only" With the number of
R())
training scenarios in total N, we define OP ΣN 1{ R ActRe
RRAAt only /N.
Success Coverage (SC) calculates the percentage of training scenarios where ReAct- only sampling collects successful trajectories, and ActRe achieves success as well. Let SReAct-only and Actte be the set of scenarios where ReAct-only sampling and ActRe succeed, respectively, we define SC
|SeeAct-nly n S //eActt-oolyl
According to the results in Table 7, our framework collects the trajectories with consistently higher rewards and success rates. The results of outperformance rate and success coverage also demonstrate that ActRe obtains a better coverage high-reward trajectories over ReAct- only sampling. This shows the superiority of our framework in terms of environmental awareness: since external actions are directly sampled from the environment to achieve sufficient grounding, framework bypasses the potentially limited environmental prior by LLM reasoning. Case studies of the gathered trajectories, as well as the performance comparison on WebShop between ReAct-only and ours, are attached in Appendix C.3
8

---

Published as a conference paper at COLM 2024
Training Scheme
Valid.
Test.
A (Round=0)
Supervised with R 1.0
70.1/41.0
72.4/ /4.4/
Ablated (Round=0)
Supervised with R > 0.75
68.2/37.5
71.5/43.2
Ablated (Round=0)
Supervised with R 2 0.5
68.1/37.8
72.5/44.
A (Round=1)
PG with binarized rewards (= 11)
69.7/43.0
73.1/49.0
Ablated (Round=1)
Supervised with R = 1.0
69.4/42.5
73.2/45.6
Ablated (Round=1)
Supervised with label conditions
69.5/41.3
72.4/45.4
Ablated (Round=1)
PG with original rewards (= 1;ori)
69.5/41.3
73.4/4 73.4/47.0
Ablated (Round=1)
PG with original rewards (2 0.75; ori)
69.3/42.3
72.2/46 46.6
Ablated (Round=1)
PG with binarized rewards 1;0.1)
70.2/42.8
73.0/46.6
Table 8: Ablated experiments of the self-training techniques in A on WebShop. "PG"
stands for the policy gradient technique in Eq. (1. The parenthesized suffixes represent the reward configuration for the trajectories in Eq. ( For example, "(= 1; -1)" means keeping the R = trajectories, and assign the failed non-composed trajectories with R(Ty) The training trajectory set is shared among different runs in the same round, except for reward threshold constraints other than R
Ablation Studies
In this section, we conduct ablation studies for the self-training techniques proposed in Section 2.2. and fine-tune the proprietary gpt-3.5-turbo-1100 for further comparisons.
5.1 Variants of the Self-Training Techniques
We conduct ablated experiments WebShop, it provides real-valued reward ranging from 0 to 1. We first study the effect different reward thresholds for training trajectory filtering. In ATT, only the trajectories with reward = are used in training. We change the constraint into R > 0.75 and R > 0.5 and compare the performance of the initial round with supervised Aet fine-tuning. According to Table 8 the best performance is still obtained with R 1, which agrees with the findings of Zhou et al. (2023b).
We proceed to ablate the policy gradient technique with binarized rewards and conduct experiments for training in Round The first type be compared with is supervised fine-tuning. We implement supervised training with the successful trajectories only (namely R(T,) 0 in Eq. 2 This echoes the practice adopted by Singh et al. (2023). We also prepend the trajectories with the label conditions "Success" / "Fail" in the training data and conduct Round-1 supervised training. For the comparisons with policy gradient Eq. (2), we alternatively set R(Ty) to be the original reward provided by WebShop, or be 0.1 following the practice of Wang et al. 2024). When using the original reward, we also include another setting by relaxing the reward threshold constraint to be R > 0.75. The comparisons are shown Table S Conclusions can be drawn that: (1) Policy gradient methods lead to higher promotion in task performance than supervised training methods. This resembles the effectiveness of RLHF (Ouyang et al. 2022) on top of supervised fine-tuning. (2) The binarized rewards (=11 used in T lead to a significant improvement of success rate. We leave the incorporation of advanced RL (Zhou et al. 2024) and RLAIF (Lee et al. 2023 Chen
et al. 2024 Hosseini al. 2024 Yuan et
2024 algorithms into A as future work.
5.2
Experiments with gpt-3.5-turbo-1106 fintuning
While all of the experiments we previously reported are conducted with Mistral-7B-IInsrruct v0.2 and QLoRA finetuning, in this section, we also validate A with gpt-3.5-turbo-110 finetuning, the proprietary service provided by OpenAl. As the initial trajectory set for Round-0 training in A is obtained by ReAct prompting with the starting point for the two base L.MSs is the same.

---

Published as a conference paper at COLM 2024
Base LLM
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
Miistral-B-IInstrucee -
96
77
100
95
94
47
86
100
81
96
86
78
53
84
Table 9: Success rate (%) each task type of AlfWorld after the Round-0 (supervised) training with both of the base LLMs.
Base LLM
Setting
Valid.
Test.
Round=0
70.1/41.0
724/4454
Mistral-7B-Instrutt--00
Round=1
69.7/43.0
73.1/49.0
Round=0
69.4/42.0
73.6/48.0
gpt-3.5-turbo-1106
Round=1 (supervised)
69.7/44.
72.7/46.8
Table 10: Reward (x 100) / success rate (x (00%) on validation and test sets of WebShop using different base LLMs. For Round-1 training with gpt-3.5-turbo-1100 we conduct supervised training with R = 1 filtering, as only SFT is offered in the service.
Base LLM
Reward R (x (x100) Success Rate (%)
%(R 2 0.75)
% 2 0.5)
1istral-7B--nsstrut--0222 9p0-3.5-turbo-106
85.2
61.1
75.8
94.3
85.4
62.5
76.1
94.3
Table 11:
Quality comparison of the accumulated trajectory set in Round for WebShop, which is composed by the policy agent with the open-sourced or the proprietary LLM.
Tables and 10 report the performance comparison of Round-0 supervised training between the open-sourced and the proprietary LLMs. In AlfWorld, the performance of the QLoRA fine-tuned Mistral-7B-Instruct-00.. even surpasses that of the proprietary spt-3.5tturbo-1066 fine-tuning service. In WebShop, the proprietary gpt-3.5-turbo-111 1olurbo-1166 finetuning performs better in Round-0 supervised training. We then let the two models separately compose diverse trajectories for their self-training. Because of the expense of inferring with the finetuned 1-tt----tuurb--1066 mode compose 10 trajectories for each failed training task (20 with the open-sourced LLM). Shown in Table TT the quality of the accumulated trajectories composed by the proprietary LLM is on par with those composed by the open- sourced LLM. After Round-1 self-training, the open-sourced model achieves an even higher test success rate. This attributed to the proprietary service providing only the supervised fine-tuning option, while also indicating the importance of contrastive fine-tuning in A
Conclusion
In this work, we propose A A a framework that enables the autonomous annotation of agent trajectories in the style of ReAct for contrastive self-training. The key factor in the trajectory annotation process is the ActRe prompting agent, which produces the textual rationales given arbitrary external actions. Together with ActRe and environmental feedback, the ReAct-style agent autonomously synthesizes trajectories for self-training. In the contrastive self-training process, we leverage the policy gradient methods with binarized rewards to boost the task success rate. Extensive experiments on AlfWorld and WebShop have demonstrated the superiority of A²T over multiple strong baselines.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (No. 61925601, 62276152). We thank Kaiming Liu, An Liu, Zijun Liu, Xuanyu Lei, and many others for discussions and feedback, and Ziwei Chi for the assistance with the figures. We thank all the anonymous reviewers for their valuable suggestions.
2The pricing is listed in httpp:://ppnaat httpp/pppan.....ppppp..paeeeeeee
10

---

Published as a conference paper at COLM 2024
References
Renat Aksitov, Sobhan Akbhao Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Koppa- rapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, al. Rest meets re- act: Self-imppovement for multi-step reasoning Ilm agent. arXiv preprint ar Xft::2312..00003 2023.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint ar arXii:2310.05915 2023,
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-
tuning converts weak language models to strong language models, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Xiv preprint arXiv:2305.1453 2023.
Hiroki Furuta, Kuang- Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixi- ang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models. In The Twelfth International Conference on Learning Representations, 2024.
URL/httts://openreiiee wtw.:/eerreeieeeeeooruuuuummeeee www.:nottoo.umm.mmmm Fiworrr
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXit:2402.06457, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arxiv preprint ar xrXi::2310.06822 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. ar preprint (et:2401.04084 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint ar fit::230900067,, 2023.
Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang, Reason for future, act for now: A principled framework for autonomous 1lm agents with provable sample efficiency. arXiv preprint ar Xiix:209.17388 2023.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al. Agentlite: A lightweight library for building and advancing task-oriented Ilm agent system. arXio preprint ar arXio:2402.15533, 2024.
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn I1m agents. Xiv preprint ar arXiv:2401.13178 2024.
Vincent Micheli and Francois Fleuret. Language models are few-shot butlers. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp- 9312-9318, Online and Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 10.18653/v http://112221..mmmmaati httpp::/becl.nthooogg...rr 2021 www.e-nnnnlmmeeeeee mai-maainn.. 734
URL
Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. In Proceedings of the 7th Conference on Robot Learning (CoRL., 2023.
OpenAL. GPT-4 technical report. CoRR, habb:/2303.08744, 2023 doi: 10.48550/ ARXIV.2303. 08774. URL hhttps:///dol www.or/....////aaeeee//////////e 23550/arXiis 48550/2rKx..288..67774
11

---

Published as a conference paper at COLM 2024
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arxio preprint ar (rXi:22307.16889 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu
Yao. Reflexion: language agents with verbal reinforcement learning, In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL httpps://openreviee.... net/
Forum?
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Inter- active Learning. In Proceedings of the International Conference on Learning Representations 2010.. 03768
(ICLR), 2021.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXio preprint ar arXio:2312.06585, 2023.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-bass baased trajectory optimization for Ilm agents. ar Xio preprint arXtv:2403.02502, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arxiv preprint arXtt:2312.11805, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama Open foundation and fine-tuned chat models. Xrv preprint 7xXix2337.009288 2023.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Open- chat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2024.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Xiu preprint ar Arx10-2305.16291, 2023.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-256, 1992.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement, 2024.
Zonghan Yang, An Liu, Zijun Liu, Kaiming Liu, Fangzhou Xiong, Yile Wang, Zeyuan Yang, Qingyuan Hu, Xinrui Chen, Zhenhe Zhang, Fuwen Luo, Zhicheng Guo, Peng Li, and Yang Liu. Towards unified alignment between agents, humans, and environment. Xiv preprint ar Xiv:2400.007744 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Adounces in Neural Information Processing Systems,, volume 35, pp. 20744-2001 207577. Curran Associates, Inc., 2022.
12

---

Published as a conference paper at COLM 2024
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference Learning Representations, 2023. URL https:h/opeerrevieeeee////// net/
Brrum?id=WE_vluuUL>>
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. ar preprint arXiv:2311.05657, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. rela-rewarding language models. arXiv preprint #rXiv:2401.10026, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. ar preprint arXiv:2310.12823, 2023.
Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training pipeline for effective agent learning, arXiv preprint arXie:2402.15506 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-Av(ision) is generalist web agent, if grounded. arviv preprint ar Xii:24401.01611, 2024.
Andy Zhou, Kai Yan, Michal Salapentokh-Rothman Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2023a.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2023b.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. Xiv preprint trXiv:2402.19446, 2024.
13

---

Published as a conference paper at COLM 2024
A Implementation Details
We implement autonomous trajectory composition in Rounds 1-3, with Round 0 as the ReAct fine-tuning bootstrapping process. The trained agent for Rounds 1-3 also follows the style of ReAct. Shown Yao Yao et al. (2023), humans can modify the mistaken internal
reasoning to correct the action behavior of ReAct agent. In our work, the ActRe prompting agent replaces the human role as a rationale annotator for the ReAct policy agent. The ReAct agent decides whether to sample novel action with probability p. If the action is not to be sampled, then the ReAct policy agent takes the action conditioned on internal reasoning. Otherwise, the sampled action is sent to the ActRe agent as a query for posterior explanations. In the implementation, we select p = 0.5. If consecutive 3 actions taken by the policy agent itself are invalid, the external action is sampled one more time with the query of ActRe for reason synthesis. In AlfWorld, we let the agent gather 40 trajectories to improve the possibility of attaining at least one successful trajectory for each failed task. In WebShop experiments, due to budgetary limits, we adopt another strategy by first forcing the policy agent to collect 3 trajectories, and then terminate when a successful trajectory is gathered, or the total number of annotated trajectories reaches 20 for a single task. The prompts for ActRe in AlfWorld and WebShop are shown in Tables T2 and 13 respectively.
As for self-training, we use QLoRA finetuning with the open-sourced Misttal--B-Instruut- v0.2 throughout the main experiments. We set the rank to be 16 and a to be 32 in the QLoRA configurations. The tuned modules are q_proj and v_proj only. We use the nf4 quantization without double quantization in our experiments with QLoRA. The optimizer is set to be the 32-bit paged AdamW. This allows the experiments to be conducted on 2-4 NVIDIA 3090 GPUs with 24GB memory or 2 NVIDIA A100 GPUs with 40GB memory. We set the learning rate as le-4, and train 10 epochs for the initial Round (ReAct supervised fine-tuning with successful trajectories). In Rounds 1-3, we fine-tune the LLM for 6 epochs with Eq. ( = -1) and conduct checkpoint averaging over the entire training process.
The ActRe in-context prompt for AlfWorld
Observation: Your task is to: put some spraybottle on toilet. You are in the middle
a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and towelholder 1. think[[oo solve the task, I need to find and take a sparybottle, then put it on toilet.]
Action:
Observation: OK.
Action: act[go to cabinet 1]
Reason for the action: [First I need to find spraybottle. A spraybottle is more likely to
appear in cabinet (1-4), countertop (1), toilet (1), sinkbasin (1-2), garbagecan (1). I can check one by one, starting with cabinet 1.] On the cabinet 1, you see a cloth 1, a soapbar a soapbottle Action: act[go to cabinet 2]
Observation:
Observation:
The cabinet 2 is closed.
Action: act[open cabinet 2]
Observation: You open the cabinet 2. The cabinet is open. In it, you see candle 1, and spraybottle 2.
Action: act/take spraybottle 2 from cabinet 2]
Reason for the action: [Now find a spraybottle (2). Next, I need to take it.]
Observation: You pick up the spraybottle 2 from the cabinet 2.
Action: actfgo to toilet 1]
Reason for the action: [Now take a spraybottle (2). Next, need to put it in toilet 1.] Observation: On the toilet 1, you see soapbottle 2.
Action: act(put spraybottle 2 /n/ on toilet 1]
Observation: You put the spraybottle 2 on the toilet 1.
Table 12: The ActRe prompt for the AlfWorld tasks. ActRe inverts the causality of ReAct, and provides the posterior reason for the external action.
14

---

Published as a conference paper at COLM 2024
The ActRe in-context prompt for WebShop
Webshop
Instruction:
would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars [Search]
Action: search[3 ounce bright citrus deodorant sensitive skin]
Observation:
[Back to Search]
Page (Total results: 50)
[Next >]
[B07SGWRCIJ]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin,
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[BU7SGTKVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin,
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[B0SKBV]4XN]
Barrel and Oak Aluminum- Deodorant, Deodorant for Men, Essential Oil-Based Scent, Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
$15.95
Action: CBCc(B077GWRCC]]
Reason for the action: [B078GWRCI] and BU78GIKVXY are bright citrus deodorant less then 50 dollars. can check B078GWRCIJ first.]
Observation:
|Back to Search]
[< Prev]
scent [assorted scents [bright citrus][calming lavender[[ginger fresh][simply non-scents]
size [travel set (4- [3ccac]][33 ounce (pack of 1)][3-ounce [2-pack)]
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin,
Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
Price: $10.99
Rating: N.A.
[Description]
[Feturrs]]
[Reviews]
[Buy Now]
Action: click[bright citrus)
Reason for the action: [For ounce bottle of bright citrus deodorant for sensitive skin, the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy.] Observation: You have clicked bright citrus.
Action: click[3 ounce (pack of 1)]
Reason for the action: [For ounce bottle of bright citrus deodorant for sensitive skin, the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy.] You have clicked ounce (pack of 1).
Observation:
Action: click|Buy |Buy Now]
Reason for the action: [For ounce bottle of bright citrus deodorant for sensitive skin,
the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy.]
Table 13:
The ActRe prompt for the WebShop tasks. ActRe inverts the causality of ReAct, and provides the posterior reason for the external action.
15

---

Published as a conference paper at COLM 2024
B
Case Studies of the Composed Trajectories
Figs.
3
and
show several cases of trajectories composed with ActRe on WebShop.
WebShop
Instruction:
Obs
i an looking for table and chaír that easy to assemble, and price lower than 260, 60 dollars [Search]
Act tehich table and chair set to assemble]
Search]
Page (Total results: 50)
think(BB9wDPY322 and B0727PL2CK white table and chair sets easy to assenble and less Act dollars. I con check B03NDPY329 first.]
[Next >]
Obs
[B81K2TON58]_
[B0SNDPY3297.
[899WWGDBBC].
[80727PL 2CK]]
[B07YT6WR15
[08F7TFRVO]
click(B09WDP322g]
[881997CMJ]-
[B8GJOMSTL]]
[09P3500Y7].
[B80CI69IGA]..
[Back to Search]
think[Far
easy assemble chair set, the iten has options "white blue pink' and seems good to buy
[s
[prinay] (wiite
(a)-Act 3:
[white primory] Humble Crow, te/Pinory
click[white blue pink]
Kids Activity Building Block Play Table & Chairs
Obs 3:
Price: $49.99
Roting: N.A. [Description]
shinkLFor
set easy
assemble and less than 260 dallars, the iten has aptions "ahico moneeee and seens good to buy.
[Features]
(b)-Act
[Reviews]
[Buy Now]
click[white prinory]
(a)-Obs
You have clicked white blue pink.
(a)-Act click[Buy Now]
(b)-Obs
You have clicked white primary.
(b)-Act
click[Buy Nor]
(a)-Obs
Your scone (min 0,0, max 1.0): 686666666666666
Your (min 1.0): 1.0
Ob)sObs 5:
Figure 2: Trajectory comparison on the 518-th task of WebShop. (a) the failed trajectory by the trained agent at the 0-th Round; (b) the composed trajectory assisted with ActRe. The trained policy agent fails to choose the correct option in the item content page. Success is obtained by clicking "[white primary]", and ActRe annotates for the sampled action.
16

---

Published as a conference paper at COLM 2024
(a)
Instruction:
Obs
highh
worten, no. price Tower than 130.00
Looking
Act search[7.3 high momen]
dollars
[3earch]]
Search]
Page (Total results:
Bhiok[npSQCYWF9 Bo9QDiGRX high
dollars. can BP9OCYAF3R first.]
Oba 2:
[B000
80908LD1
[809 86
[809N0071 00
[809N72486]
[889NT2507R]
[Back Search]
size (8))5)888
color [bloacd] [whiite]
Momen' Seny Fashion Slide
Stipper Cross-strap Plotform
think[Far 7.5 no. high heel monen, the iten has options '7.5' and "bllack"" and searss good Act to buy
Medges Beach
Fp
Flop Lightweiight
Support Slide Sandals Indoor
Outddor VeyWIV
click(7.5) 53
Price: $100.0
Ratina: N.A.
Description]
[Foauuros]
[evt.ows] [auy Now
Obs
have icked 2,5.
Act click(lock
Obs
You have clicked black.
Act 5: click[Duy Nor]
Oba 6:
Your score Orin 8.0, nox 1.8): 8.5
(b)
MebShop
Instruction:
Obs
Lookiing no. high
Act search(7.5 high woreo]]
price 130.00
dollars
[Search]
Search]
Page (Totol. Dicxt: [B0
thinkl is no. high heel For
naren and the price is Lowen than $130.00.
Obs 2:
[89908LD119) [RP9OP
click(ReINNWLL00
88990HC786
|009NM0Z2 00
[SPNNTINES
[089NT2486L].
[B89NT2507R]
Search]
[x Prev]
size [6.5][2][25]]0
7SMikknn Monen's Over The
Boots
Flostic Chunkyy Heels Thigh
think[for 7.5 high heel, for the iter has option 'y.5' and good buy
High Boots SLip-on Round Toe
Obs
Fashion Sexy Winter Hlgh
3:
Heel Knee High Long Booties
click[7.5]
Dress Shoes for Club Party Price: $32.19
Roting: N.A. [Description]
[Fcctures]
[evvi Duyy Nowd
think[Far 7.5 no, high heel. for wonen, the
iter has option '7.5" and seens good to buy
Obs
You have ilked 7.5,
Act 4:
Click[Buy koi]
Obs 5:
score (rin 8.8, 1.8): 1.8
Figure 3: Trajectory comparison on the 512-th task of WebShop. (a) the failed trajectory by the trained agent at the 0-th Round; (b) the annotated trajectory assisted with ActRe. The action of clicking the 7-th item "[B09QCYNF9RJ" is explored in the new trajectory, and the reason highlighted in green generated by ActRe.
17

---

Published as a conference paper at COLM 2024
(a)
WebShop
Instruction:
Obs 1:
1 an Looking for a qued core tablet that is certified refurbished. price lower [Search]
Act 1:
searchIquad core tablet certified refurbished]
[Back Search]
Page (Total results: 50)
BOO and B01HPGE((M) are qued care tablets certified refurbished and less than 190.00 dollars. can check B00086TW9C first..]
[Next >]
Obs 2:
[
[B01HPGEOMO]
Act 2:
[008I9N4ΣQW]
[B@1HPEDYRM
[B01)ErYYLE]
[689/))0007]
click[D00086T9CC]
[809PM5К305]
[887H6RVeNI_
[B07CYD69.8]]
[B890RGV127].
[Back to Search] [< Prev]
RCA Vayager Quad Care
think[ The iten to fit criteria and budget.. purchase it.]
Toblet BLACK
Obs
Rating: N.A.
Price: $100.0
3:
[Description]
click[Buy Now]
[Features]
ews]
[Buy Now.
Obs
1.8): 0, 6666666666666666
(b)
WebShop
Instruction:
Obs 1:
i an Looking for a qued core tablet that is certified refurbished, price lower 190.00 dollars [Search]
Act 1: search[quad core tablet certifled refurbished]
[Rack Search]
Page (Total results: 50) [Next >]
BOONKTB0 and B01HPGE(MM are quad core tablets certifled refurbished and less than 190.00 dollars. can check B80086TW9C first.]
Obs 2:
0080086T9913
[881HPGEQMD]
Act 2:
[808I9N41QW]
[01HHPEDYRM]
[B2
[089010FFD7]
click[D0006619CC
[887H6RUMUL
[907CVD69.9]
[B890R6V127]
[Back to Search] [ Prev]]
RCA Voyager Quad Core
Android Tablet BLACK
Think[To back the previqus page and sheck options...
Obs
Price: $100.0
3:
Rating: N.A.
[Description]
click[< Prev]
[Features]
[Buy Now]
[Back Search]
Page (Total results: 50)
think(po7CvD69Jp is quad tablet certified refurbished
[Next >]
Obs
[B08Q86TW9C]_
[B01HPGEQMM]...
Act 4: dallars. I can check B87CYD6939 first.]
[B00Σ9NΑΙQЙ]
BB11PPDDYW
[801]]EFY2LE]
[D09
click[667CVD690g]
[809PHGK]05]
687H6RVMNL
[807CV06939]
[B890RGV127]
[Hack to Search]
Prev]]
think[To go back to the previous poge and check other options.]
Microsoft Surface 706-
Obs 5:
00001 10.8 Inch 128 GB sSp Toblet (Stlver) (Renewed)
click[= Prev]
Price: $305.0
Rating: N.A. [Description]
[Features]
[Revi ews]
[Buy Now]
Figure 4: Trajectory comparison the 754-th task of WebShop. (a) the failed trajectory by the trained agent at the (-th Round; (b) the annotated trajectory assisted with ActRe. The trajectory (b) continued in Fig.
18

---

Published as a conference paper at COLM 2024
[Back to Search]
Page (Total results: 50)
BhinkkBoOQB6WWCC and B01HPGEQMD are quod core tablets certifled refurbished and less than 190.80 dollars. can check B00086TWBC first.]
[Next
Obs
[B00086TM9C]
[B0HHPGENM]]
Act 6:
[80019N410W]
[BIHPEDYYMM
[B01JEFY2LE]
[089m00FD
click[Bo006619CC
[809PH65105]]
[887H6RUMN_]]
[807CYD69J9]-
[8890R6V127].
[Rack Snarch] [s Prev]
RCA Voyager Quad Core
think[To back to the previous page and check other options.]
Android Tablet BLACK
Obs
$100.0
Act 7:
Rating: [Description]
click[< Prev]
[Features]
[Neviews]
[Ruy Now]
Search]
Page (Total results: 50)
think[To go to the next poge and check other options.J clickDlext: =]
[Next >]
Obs
[800086T19(]]
[801HP6:QM0].
Act
[B00I9NAIQW]
[B01HPEDYRM]]
[B01]EFY2LE]
[B89HIMMFD7
[в09РиBК105
[B87HGRUMeIL
[B87CYD69J8]
[B890RSV127].
[Back to Search]
Page (Total results: 50)
thi [h47 and B07JD4PGXQ are Dell Vostra 460 Desktop Tawer Corputers with quod processors less than 150. dollars. can 607]CHРНTH first.]
[< Prev]
[Next >]
Obs
[B09FG1QXLG) -]
[807STFLVY8]
9:
[CHPHII
[807]D4PGX0]
B08KSGMK4T
click[p0/)CHPHIHQ
[B08K5K3MKK
887L3229944
[801N64DHE0]
3880UBPEXE)
[lack to Search]
[< Prev]
Desktop
Tower Computer Super Fast
think(This
Desktop Tower
Quad Core Intel Core 17-2600
Computer seens like a good deal. for less than 100 dol dalla can click on "Buy Now" to purchase it.]
3,4GHz CPU, 8GB DDR3 SORAM,
Obs 10:
500 G8 HDO, ninss Pro
Act 10:
Price: $
Roting: N.A.
click[Buy Now]
[Description]
[Features]
[Revi ews]
[Buy Now]
Your score (min 0.0, 1.0):
Obs 11:
Figure 5: The second part of the composed trajectory on the 754-th task of WebShop. This case demonstrates some sophisticated behavior achieved by the synergy of the ReAct- style policy agent and the ActRe prompting agent. The policy agent alone terminates the trajectory by directing purchasing the first item ([B0QQBTT99C) on the first page, obtaining an imperfect reward of 0.6667. By the randomly sampling actions and querying the ActRe agent to synthesize the reasons, the annotated trajectory shows that the policy agent initially clicks the first item on Page 1 as well, but then goes back to seek for other options (the reason in Act 3 highlighted in green the first synthetic reason by ActRe). Then the agent clicks the 9-th item (B00CCY69)JJ) on Page 1, but then goes back and clicks the 1-st item, and returns to the search results of Page once again. After that, the agent chooses to go to the next page, and then selects the 4-th item ((B0)CCРHHHH) and finally makes purchase. This item results in a perfect match with the provided instruction.
19

---

Published as a conference paper at COLM 2024
C
Additional Experimental Results
C.1
AlfWorld
Table
covers the detailed task statistics in each split for our experiments.
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
Train
135
98
74
92
59
142
600
Valid.
12
10
7
14
60
Test' (held out, seen)
35
27
16
25
13
24
140
Test (held out, unseen)
24
23
21
134
Table 14:
The statistics of tasks used in training, validation, and testing. *: For all experi- mental results except for Table we report the success on the 134 held-out unseen test scenarios following the settings in previous work.
Tables 15 and 16 exhibit more experimental results on the 140 held-out seen testing tasks
(official split) and the 60 validation tasks (our use).
Method
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
LM-BUTLER Micheli & Fleuret 2021
97
89
100
80
77
92
90
A (Round=0)
86
67
94
80
85
75
80
A (Round=1)
92
67
A (Round=2)
97
89
100
88
92
79
91
A3T (Round=3)
100
96
100
96
77
79
93
Table 15: Success rate (%) on each task type of AlfWorld, with single trial on each of the 140 in-distribution test scenarios. Our agents outperform LM-BUTLER on this test split.
Method
Pick
Clean
Heat
Cool
Examine
PickTwo
Total
A (Round=0)
11/12
9/10
5/7
11/14
7/8
9/9
52/60
A (Round=1)
11/12
10/10
6/7
13/14
7/8
8/9
55/60
A (Round=2)
12/12
10/10
7/7
13/14
8/8
8/9
58/60
A (Round=3)
12/12
10/10
7/7
12/14
8/8
7/9
56/60
Table 16: The number of our 1-shot successful / all tasks of each type in our validation split AlfWorld.
Table 17 shows the sentence statistics of the training datasets for each round of LLM finetuning on AlfWorld. The low percentage of the "failed" sentences (which are assigned with R(T,) 11) in the total training set empirically satisfies the constraint of K > in
Remark 3 of Section 2.2
Round
#Total
#Failed
#Failed /HTotal (%)
2,130
0
0.0
2,669
99
3.7
2
4,066
177
4.4
4,995
219
4.3
Table 17: The sentence statistics of the training datasets for LLM finetuning in each round of our trajectory collection and self-training on AlfWorld. The training set contains sentences with -1 weights for Round 0, as Round 0 performs supervised fine-tuning with ReAct prompting trajectories as bootstrapping. The details of trajectory collection are covered in Appendix
20

---

Published as a conference paper at COLM 2024
C.2
WebShop
Table 18 shows the sentence statistics of the training datasets for each round of LLM finetuning on WebShop. The percentage of the "failed" sentences is about 10%, which still satisfies the constraint of K > 1 in Remark 3 of Section 2.2 empirically with stable training.
Round
#Total
#Failed
@Railed/#Tole (%)
981
0
0.0
3,431
336
9.8
5,719
694
12.1
3
8,550
1,122
13.1
Table 18: The sentence statistics of the training datasets for Mistral-7B-Innttrutto22 fine- tuning in each round of our trajectory collection and self-training on WebShop. Round corresponds with supervised fine-tuning with ReAct trajectories, and thus uses no failed sentences. The details of trajectory collection are covered in Appendix A
C.3 Comparisons with ReAct-only Self-Training
Table 19 shows the comparison performances on WebShop by ReAct-only self-training and our framework.
Test Trial
Method
Round
Valid.
Test.
0
69.1/43.2
72.4/45.8
69.2/43.2
73.1/47.0
ReAct-only
70.0/43.5 2
74.3/47.8
3
70.5/44.0
73,9/47.2
Single
0
70.1/41.0
72.4/45.4
1
69.7/43.0
73.1 73.1/49. /49.0
Ours
2
69.0/43.8
73.0/48.0
69.1/42.8 3
73.9/49.0
1
73.9/49.7
76.1/50.4
ReAct-only
2
75.8/52.0
77AA/2.6
76.6/53.5 3
78.0/53.4
Iterative
1
74.0/47.3
76.6/51.6
Ours
75.1/49.5
77.8/53.. 53.4
3
75.9/51.3
78,2/ 14.2/54.8
Table 19: Performance comparisons between ReAct-only self-training and our framework WebShop. Similar to Table 4 reward (x 100) and success rate (x 100%) on the validation and the test sets are reported.
According to the results, the performance of the ReAct-only method becomes better on the validation scenarios across different rounds, demonstrating the effectiveness of the contrastive self-training method used to train the models in Rounds 13. However, it is shown that our framework achieves the better success rate on test scenarios. Furthermore, the consistency of outperformance is further demonstrated in the iterative refinement evaluation setting. This suggests better generalization, which might be attributed to more sufficient coverage of high-quality trajectories gathered by our framework.
Case Study. The ReAct agent trained by our framework in Round 1 demonstrates the autonomous behaviour listed Table 20
This case demonstrates the complex reasoning ability of returning to the previous page and selecting another item. Such behavior is fostered thanks to the ActRe-gathered training data, which results in final reward of 0.25. By comparison, all the SFT baselines (Round=0) and the agents with ReAct-only self-tre training achieve reward at this test case. By comparing the gathered trajectories of both framework and ReAct-only sampling, found that
21

---

Published as a conference paper at COLM 2024
Table 20: The list of actions taken by our ReAct agent trained in Round 1 at test case #328.
search[loveseat flat packed wood finish]
think[ GCJ and BO7J4S5JMS are flat packed wood finish loveseats less than 240 dollars.
can check B0855KIGCJ first.]
+ click[Bo855K1GG (the first item)
- think[For a flat packed wood finish loveseat, the item has options 'flat packed' and 'wood finish' but does not seem to have those specifications. I can go back to the previous page and check other options.]
click[< Prev]
+ LS is flat packed wood finish loveseat less than 240 dollars. can check BU7J4S5JMS first.]
+ CJick[B074S5JJMS (the second item)
think[For flat packed wood finish loveseat, the item has options '4" and 'dark walnut finish' and seems good to buy.]
+ click[4"]
+ click[dark walnut finish]
click[Buy Now]
the "click[< Prev]" action in WebShop does not exist in the successful trajectories gathered by ReAct-only sampling in our runs, while such action has been covered in the successful trajectories annotated by our framework. The case study further supports the superiority of our framework in terms of environmental awareness, as the agent is more sufficiently grounded by directly sampling external actions for trajectory annotation, without the limitation of LLM internal reasoning priors of the external world.
22


Reviews:
[{"review_id": "o2PWopWLK6", "reviewer": "Reviewer_PnAD", "summary": "This paper proposes A3T, a framework for autonomous annotation of agent trajectories in the style of ReAct, which enables contrastive self-training of language agents. The key component of A3T is the ActRe prompting agent, which provides textual rationales for arbitrary external actions. The ReAct-style agent leverages ActRe and environmental feedback to synthesize trajectories for self-training. The contrastive self-training process utilizes policy gradient methods with binarized rewards to improve the task success rate. The authors conduct experiments on AlfWorld and WebShop, demonstrating the superiority of A3T over multiple strong baselines, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions!\n\n***Weakness 1. Lack of investigation of more sophisticated RL techniques.***\n\nAs stated in our paper, the major contribution of our work is the closed-loop framework for agents to self-improve, where agents directly sample actions from the environment, autonomously annotate the rationales, and self-improve by contrastive self-training. We leave the integration of more advanced RL techniques for future work.\n\n***Weakness 2. Explorations of alternative methods for generating the initial trajectory set to improve its diversity.***\n\nWe first consider alternative prompting techniques like Reflexion, which is also suggested by the FireAct [1] work. However, as indicated by the Reflexion paper [2] in Appendix B.1, Reflexion does not yield improved performance on WebShop because of the environment complexity.\n\nTo improve the diversity of the initial trajectory set, we also experiment with ReAct-only sampling for multiple times, and assign the same API calls budget as that for ActRe in Round 1 of our framework. After collection, ReAct-only sampling obtains Reward=$81.6$ and Success Rate $=55.3$%. The quality of these gathered trajectories is better than that of the initial trajectory set in the paper (Table 6, Round 0: Reward=$68.5$ and Success Rate $=40.0$%). After the initial SFT process in Round 0, the trained ReAct agent obtains **R**/**SR**=$72.4$/$45.8$% on test tasks, while ours in Round=0 achieves **R**/**SR**=$72.4$/$45.4$% on test tasks. \n\nThis suggests a better agent initialization could be achieved by providing more diverse initial trajectories. However, this comes at the cost of more API calls, and the cost would be higher when integrating trajectories from other sources, *e.g.*, human annotators. In contrast, the main purpose of our work is to propose the closed-loop framework for agents to annotate trajectories *autonomously*, and we have demonstrated the effectiveness of the framework when starting from a limited initial trajectory set. Due to time and budgetary limits, we will add more diverse trajectories into the initial set, and conduct more comprehensive studies on it in the future. We will also add the related discussions to our paper.\n\n[1] Chen et al., 2024. FireAct: Toward Language Agent Fine-tuning.\n\n[2] Shinn et al., 2023. Reflexion: Language Agents with Verbal Reinforcement Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed A3T framework addresses the scalability issue in collecting expert demonstrations and the limitations of implementing diverse agent frameworks for trajectory annotation. By leveraging the ActRe prompting agent, A3T enables autonomous trajectory annotation, reducing human effort and proprietary LLM calls.\n- The contrastive self-training process in A3T utilizes policy gradient methods with binarized rewards, which effectively boosts the task success rate. The ablation studies demonstrate the significance of this technique compared to supervised fine-tuning and policy gradient with original rewards.", "reasons_to_reject": "- While the authors mention the potential of incorporating advanced RL and RLAIF algorithms into A3T as future work, the current study does not explore these directions. Investigating the integration of more sophisticated RL techniques could further enhance the performance and sample efficiency of the A3T framework.\n-  The initial trajectory set for Round-0 training in A3T is obtained by ReAct prompting with gpt-3.5-turbo-instruct-0914. While this provides a starting point for the self-training process, it may limit the diversity of the initial trajectories. The paper does not explore alternative methods for generating the initial trajectory set, such as using different prompting techniques or combining trajectories from multiple sources. Investigating the impact of the initial trajectory set diversity on the final performance could provide insights into the importance of this factor and potential ways to improve it.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ygBfjkfo2", "reviewer": "Reviewer_Db4p", "summary": "This work introduces the A3T framework, which autonomously annotates agent trajectories in the ReAct style using an ActRe prompting agent. The primary idea behind this work is to have an agent that explains the reasons behind arbitrary actions, allowing the ReAct-style agent to synthesize new trajectories by combining these explanations with sampled actions.\n\nThe paper has been clearly written, with the motivation and solution fleshed out in reference to the existing works. The paper clearly identifies the challenge of collecting multi-step reasoning and action trajectories, and emphasizes the issues of autonomous decision-making.\n\nUnder extensive experiments on AlfWorld and WebShop, the authors clearly show improvements over ReAct, and conduct numerous ablation studies to study the different components of their proposed algorithm. I believe, the findings in the paper will be valuable to the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review! We will continue to contribute to the community by improving our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's strength lies in its clean exposition to the existing literature and the issues that need to be addressed for completely autonomous agents with LLMs. The closed-loop system for continuous self-improvement is a standout feature. A3T iteratively refines the agent’s performance through autonomous annotation and contrastive self-training. The validation of the framework through comprehensive experiments in different environments (AlfWorld and WebShop) shows its robustness and generalizability.", "reasons_to_reject": "As-such, I don't have many concerns with the papers.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LL7SScYXHL", "reviewer": "Reviewer_aDjj", "summary": "This paper presents $A^3$T, a framework that enables the autonomous Annotation of Agent Trajectories in the style of ReACT. The idea is to modify the trajectories with an ActRe prompting agent, which would give the reasoning for any actions. In this way, the ReAct-style agent executes multiple trajectories for specific tasks and can select successful ones together with failed ones for contrastive self-training. \nThe contrastive self-training is realized by policy gradient with binarized rewards. Empirical results show that the method can achieve significant improvements compared with existing techniques and methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive comments!\n\n***Weakness 1. Experiments with more tasks and benchmarks.***\n\nThank you for the suggestions! Due to time and budgetary limits, we plan to experiment with more tasks and benchmarks in the future.\n\n***Question 1. Choice of t when synthesizing trajectories.***\n\nIn our framework, at each time step, the finetuned ReAct agent determines whether to (i) rely on its own reasoning-then-action or to (ii) sample a novel action and query ActRe for reasons with probability $p=0.5$ (Appendix A). Therefore, the $t$ that ActRe is queried with a sampled action is randomly chosen in a trajectory with $p=0.5$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a simple yet effective method for generating synthetic trajectories without human annotations/supervision.\n- The policy improvement method includes both successful and unsuccessful trajectories, which I think would be the key to improving performance (similar to DPO). \n- The authors conducted experiments on AlfWorld and Webshop and achieved better empirical performance.", "reasons_to_reject": "- My major concern is that the authors only conduct experiments on two tasks. It would be great to show the advantage over other method on more tasks or benchmarks such as toolbench or toolqueries.", "questions_to_authors": "- How do you choose $t$ when you do synthetic data/trajectories generation?", "extra_scores": {}}, {"review_id": "b207nUjyd8", "reviewer": "Reviewer_QZXM", "summary": "The paper proposes a self-training method for language agents. They propose to gather new reasoning-action trajectories from an agent by first sampling an action and then generating a rationale given the sampled action at each step. Then, they train models on both successful and unsuccessful trajectories with reinforcement learning and investigate designing rewards for these trajectories. Results on AlfWorld and WebShop demonstrate the effectiveness of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions!\n\nWe conduct self-training experiments with ReAct-only sampling on WebShop. \n\nThe major finding from the ablation study is that ReAct-only sampling fails to cover all the actions in the environment, *e.g.*, the `click[< Prev]` action in WebShop is never performed by ReAct-only sampling. This leads to better motivation for ActRe: ***our framework directly samples the actions from the environment to achieve sufficient grounding, making up for the limitation of LLMs in environmental awareness when reasoning***. The following experiments support this motivation:\n\nQuantitative comparisons: \n- For the comparison of collected trajectories, under the fair settings of API call budgets, ReAct-only sampling obtains Reward=$81.6$ and Success Rate $=55.3$%, which is lower than those by ActRe (Reward **R**=$85.2$ and Success Rate **SR**$=61.1$% in Table 6).\n- For the comparison of performance after training:\n  - Using SFT with trajectories via ReAct-only sampling to train ReAct agents leads to **R**/**SR**=$72.4$/$45.8$% on test tasks, while ours in Round=0 achieves **R**/**SR**=$72.4$/$45.4$% on test tasks. This suggests that ReAct-only sampling could provide a better initial trajectory set at the cost of more API calls.\n  - Using contrastive self-training with trajectories via ReAct-only sampling to train ReAct agents leads to **R**/**SR**=$73.1$/$47.0$% on test tasks, while ours in Round=1 achieves **R**/**SR**=$73.1$/$49.0$%. This suggests the effectiveness of both ActRe sampling and contrastive self-training.\n\nCase study: \n\n- The ReAct agent trained by our framework in Round=1 autonomously conducts the following actions in the #328 testing task: click the first item --> `click[< Prev]` according to its own reasoning --> click the second item --> purchase, which shows its more complex reasoning ability thanks to the ActRe-gathered training data, and results in a final reward of 0.25. By comparison, all the SFT baselines (Round=0) and the agents trained with ReAct-only sampling achieve 0 reward.\n\n*Other Weaknesses*: While FireAct suggests also incorporating Reflexion, Reflexion does not yield improved performance on WebShop because of the environmental complexity (indicated by the Reflexion paper in Appendix B.1). We also leave more advanced RL techniques for future work.\n\nWhile the rebuttal length is limited, We will add more discussions to the paper, and are willing to address your concerns more thoroughly during the next period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They propose a plausible way to generate self-training data for language agents.\n2. They base most of their models on strong open-sourced models, which adds credibility to the results and encourages reproducibility of the work.\n3. The paper is well-written.", "reasons_to_reject": "1. Since ActRe is one of their claimed contributions, I would expect 1) better motivations and 2) more ablations and analyses on this. The current motivation of ActRe is not well-supported by the experiments, it is possible that ReAct can sample different reasoning trials that lead to diverse actions as well. In addition, ablations and analyses on this should be added. For example, self-training with ReAct-style sampling only is an important baseline that should be evaluated in all of their evaluation settings. Also, because prior work [1] demonstrates that combining different sampling strategies can lead to better performance, more discussion on previous work and more comparisons are required.\n2. I do not see much difference between their contrastive self-training method with REINFORCE. I appreciate the efforts in designing the rewards and sampling strategies, but there seems to be a lack of methodological contributions.\n3. While the paper mentions that for the previous work, their agent trajectories are limited in diversity and scalability, I do not see they directly solve such limitations in the experiments.\n\n\n[1] Chen et al., FireAct: Toward Language Agent Fine-tuning, 2023.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary:
The paper introduces A3T, a framework for autonomously annotating agent trajectories in the ReAct style using an ActRe prompting agent. It enhances self-training for language agents by explaining actions and synthesizing new trajectories. A3T surpasses strong baselines in experiments on AlfWorld and WebShop, demonstrating its effectiveness in improving autonomous decision-making for agents.

Overall, the concerns raised by the authors on additional experiments comparing with ReAct, alternatives for initial trajectories, diversity and scalability of the trajectories are well-addressed. ChatGPT. Their experiments also demonstrate that ActRe achieves a broader coverage of high-reward trajectories compared to sampling with ReAct alone.

While the technique is not very novel, the idea of A3T has a lot of significance in handling scale and providing a new experimental base on closed loop solution for the future work.

Pros:
1. The experiments highlight the importance of this approach compared to supervised fine-tuning and policy gradient with non-binarized rewards.
2. The A3T framework handles scalability issues in gathering expert demonstrations and implementing diverse agent frameworks for trajectory annotation.

Cons:
1. The technique is not very novel as such, as it is very similar to REINFORCE.

Novelty:
This work presents a closed-loop system for continuous improvement, where A3T iteratively enhances agent performance through autonomous annotation and contrastive self-training. Extensive experiments validate the framework's robustness and versatility.

Significance:
1. The paper presents a thorough exploration of existing literature and the obstacles in developing completely autonomous agents with LLMs. A3T uses policy gradient techniques with binary rewards in its contrastive self-training process to significantly improve task success rates. With this motivation, the authors present a closed loop solution with ActRe which opens up a lot of promising avenues to explore RL techniques in this paradigm extensively in the future.
2.  It uses the ActRe prompting agent for autonomous trajectory annotation, reducing human input and dependence on proprietary LLM calls.

[comments from the PCs] Please add the ablations and comparison you conducted during the discussion period to the paper. Also, please conduct more detailed analyses on the types of trajectories that are sampled initially leading to SR and diversity of actions. This can help understand where the improvements are coming from.
Decision: Accept