Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents (2024)

Abstract: Large Language Models (LLMs) have become integral components in various autonomous agent systems. In this study, we present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Contrary to previous studies that exclusively train on successful expert trajectories, our method allows agents to learn from their exploration failures. This leads to improved performance through an iterative optimization framework. During the exploration phase, the agent interacts with the environment while completing given tasks, gathering failure trajectories to create contrastive trajectory pairs. In the subsequent training phase, the agent utilizes these trajectory preference pairs to update its policy using contrastive learning methods like DPO (Rafailov et al., 2023). This iterative cycle of exploration and training fosters continued improvement in the agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin. Furthermore, an examination of task-solving efficiency and potential in scenarios lacking expert trajectory underscores the effectiveness of our approach.1

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated impressive capabilities in addressing complex interactive tasks through action planning for interactions with environments and tools (Wang et al., 2023a; Xi et al., 2023). Systems using ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) as principal controllers have been developed for a range of applications. These include web browsing (Deng et al., 2023; Zhou et al., 2023), embodied tasks (Yao et al., 2022b; Lin et al., 2023), multi-modal reasoning (Lu et al., 2023), and complex question
*Corresponding Authors. 1Code & Data: https://github.com/Yifan-Song793/ETO.
answering. However, recent research suggests that open-source LLMs are considerably less effective than GPT-4 in constructing agents (Liu et al., 2023; Wang et al., 2023d; Mialon et al., 2023).
The standard approach for constructing open LLM agents involves imitation learning, which finetunes LLMs based on expert trajectories. Behavioral cloning (BC) (Pomerleau, 1991) is a simple yet effective imitation learning technique that derives a policy through supervised learning from observation-action pairs. Recent efforts (Chen et al., 2023; Zeng et al., 2023), including Agent LUMOS (Yin et al., 2023), have explored the use of BC to develop open LLM agents by implementing supervised fine-tuning (SFT) on expert trajectories. These methods employ the teacher-forcing algorithm to train LLMs, enabling them to learn a policy for generating subsequent actions based on observations and past actions. However, these SFT methods, which rely entirely on expert demonstrations, may yield sub-optimal policies due to inadequate exploration of target environments, thereby limiting their generalizability.
The process of human learning not only in-
7584
volves observing successful demonstrations but also includes experiencing and exploring failures through trial-and-error interactions with the environment. Drawing inspiration from this, we propose a novel learning approach for LLM agents, which we call Exploration-based Trajectory Optimization (ETO). Unlike previous methods that solely rely on successful trajectories, our approach capitalizes on the exploration failures of the current policy to enhance the learning process of an agent.
In particular, we first use SFT-based behavioral cloning to construct a base agent, as depicted in Figure 1. During the exploration phase, this base agent interacts with the target environment to undertake a set of given tasks and receive feedback from the environment. We gather failed trajectories from the base agents and pair them with expert trajectories previously collected for those tasks. In the subsequent training phase, we apply the DPO loss (Rafailov et al., 2023) to fine-tune the LLM policy with these contrastive trajectory pairs, thereby further improving the agent. The ETO can be expanded to multiple rounds by collecting failure cases from previously ETO-tuned agents.
We assessed our approach using three representative datasets: WebShop (Yao et al., 2022a) for web navigation, ScienceWorld (Wang et al., 2022) for simulated science experiments, and ALFWorld (Shridhar et al., 2021) for embodied household tasks. Across these datasets, our ETO consistently exceeded the performance of SFT behavioral cloning and other robust baselines by a significant margin, thereby demonstrating the effectiveness of learning from exploration failures. Furthermore, our approach demonstrated an impressive performance improvement of 22% over SFT on the challenging out-of-distribution test set in ScienceWorld, indicating its strong generalization capability. Our analysis also highlighted the task-solving efficiency of our method, as it achieves higher rewards with fewer action steps. In extreme scenarios where expert trajectories are not available, our ETO still delivers impressive performance in a self-play mode, further emphasizing the potential of our approach.
In this paper, our contributions are as follows: • Method. We introduce exploration-based tra-
jectory optimization, ETO, a learning algorithm which iteratively collects failure trajectories and refines the agent policy via contrastive learning.
• Evaluation. Extensive experiments on three complex interactive tasks show that our method
outperforms SFT behavioral cloning and other strong baselines by a large margin.
• Analysis. We conduct in-depth analysis to carefully validate the effectiveness of ETO in multiple aspects, including out-of-distribution generalization, action efficiency, and its feasibility without the need for expert trajectories.

heading: 2 Task Formulation
text: The agent task with environment feedback can be formalized as a partially observable Markov decision process (POMDP) (U ,S,A,O, T ,R) with instruction space U , state space S , action space A, observation space O, transition function T : S×A → S, and reward function R : S ×A → [0, 1]. Note that in our LLM-based agent scenario, U ,A,O are subsets of natural language space.
Given a task instruction u ∈ U , the LLM agent with parameter θ generates the action a1 ∼ πθ(·|u) ∈ A according to its policy πθ. The action incurs a change in the latent state space st ∈ S , and an execution feedback as observation ot ∈ O. Then the agent generates the corresponding action in the t+ 1 step at+1 ∼ πθ(·|u, a1, o1, ..., ot−1, at) ∈ A. The interaction loop repeats until the task completes or exceeds the maximum steps, and the trajectory is denoted as:
e = (u, a1, o1, ..., on−1, an) ∼ πθ(e|u), (1) πθ(e|u) = n∏
j=1
πθ(aj |u, a1, o1, ..., oj−1), (2)
where n is the trajectory length. Finally, the final reward r(u, e) ∈ [0, 1] is computed, with 1 representing successful task completion.

heading: 3 Method
text: Our method, ETO, starts by training a base agent through behavioral cloning. Based on the base agent, our framework continually enhances the policy from trial and error in an iterative manner.

heading: 3.1 Behavioral Cloning
text: Behavioral cloning (BC) has demonstrated promising results through supervised fine-tuning on the expert interaction trajectory data, serving as a solid starting point for building a powerful agent. In this work, we employ ReAct-style (Yao et al., 2022b) trajectory to conduct BC, which additionally generates Chain-of-Thought (CoT) rationales (Wei et al., 2022) before each action. Considering that the
CoT and action are generated together in the ReAct framework, we use a to represent the action with CoT for simplicity. Given an expert trajectory dataset D ={ (u, e)(i)
}|D| i=1
, where |D| is the number of trajectories, we fine-tune an LLM on auto-regressive loss to get the base agent πbase:
LSFT(πθ) = −Ee∼D [πθ(e|u)] , (3) where e = (u, a1, o1, ...on−1, an) ∼ D is an expert interaction trajectory.
Since πθ(e|u) = ∏n
j=1 πθ(aj |u, ..., oj−1), in practice, we first concatenate the instruction, actions and observations in trajectory e as a text sequence t:
t = concat(u, a1, o1, ..., on−1, an)
= (t1, t2, ..., tl) , (4)
where tk is the k-th token in the result sequence. Then the probability of the trajectory in Eq. (3) can be obtained by directly computing the probability of actions with tokens in task description and observations masked:
πθ(e|u) = − ∑
k
log πθ(tk|t<k)× 1(tk ∈ A),
(5) where 1(tk ∈ A) is an indicator function about whether tk is a token belonging to actions produced by the agent.

heading: 3.2 Learning From Exploration Failures
text: Behavioral cloning exclusively depends on expert trajectories and lacks the ability to explore the environment, leading to sub-optimal policies. To train a more powerful agent, it is important for the model to also explore failure trajectories. To achieve this, a viable approach is reinforcement learning, which empowers agents to actively explore the environment to get rewards and refine the policy through trial and error (Ouyang et al., 2022):
max πθ
Eu∼D,e∼πθ(e|u) [r(u, e)]−
βDKL [πθ(e|u) || πref(e|u)] , (6)
where the KL term with weighting parameter β controls the deviation from the base reference policy πref , i.e., the base agent πbase. In practice, the agent to be trained πθ is also initialized to πbase. Then the optimization problem in Eq. (6) can be solved via RL methods such as PPO (Schulman et al., 2017; Ouyang et al., 2022).
However, directly applying online RL on LLM agents will present practical challenges such as instability and low efficiency (Shen et al., 2023; Rafailov et al., 2023). Therefore, we instead design an iterative offline learning framework and train the agent with contrastive trajectory data. As shown in Figure 2, the training process can be formulated in an iterative exploration-training loop. In the
Algorithm 1: ETO: Exploration-based Trajectory Optimization for LLM Agents
Input: D = { (u, a1, o1, ...on−1, an) (i) }
: expert trajectory dataset for behavioral cloning, T1: number of behavioral cloning steps, I: number of iterations for ETO, T2: number of steps in training phase, πθ: initial LLM policy.
Output: Final policy πθ // Behavioral cloning for i = 1 to T1 do Optimize θ on BC objective: LSFT(πθ) = −Ee∼D [πθ(e|u)]
// Iteratively learning from exploration failures for i = 1 to I do
πbase = πθ; πref = πθ Get base agent trajectories on D: ê = (u, â1, ô1, ..., ôm−1, âm) ∼ πbase(e|u) Compare rewards of ê with expert trajectory e to get the failure-success pair: ew ≻ el | u Construct contrastive trajectory dataset: Dp = { (u, ew, el) (i) }
for j = 1 to T2 do Optimize θ on trajectory contrastive objective:
LDPO(πθ;πref) = −E(u,ew,el)∼Dp
[ log σ
( β log πθ(ew|u)πθ(el|u) − β log πref(ew|u) πref(el|u)
)]
return πθ
exploration phase of ETO, the agent explores the environment to collect failure trajectories. During the training phase, the agent learns the contrastive information from the “failure-success” trajectory pairs to update the policy.
Exploration Phase In this phase, the base agent πbase explores the environment to get the trajectories on the instructions of training data for BC:
ê = (u, â1, ô1, ..., ôm−1, âm) ∼ πbase(e|u). (7)
The environments then return a reward r̂ corresponding to the trajectory ê.
Then we construct failure-success trajectory pairs, denote as ew ≻ el | u, based on the final rewards. Here, ew and el represent the trajectories with higher and lower rewards, chosen from the expert trajectory e and agent-generated trajectory ê respectively. Note that we only collect pairs where two trajectories have different rewards. If both ê and e successfully complete the task, we simply discard the pair. Finally, we get the contrastive trajectory dataset Dp = { (u, ew, el)
(i) }|Dp| i=1 .
Training Phase In this phase, the agent policy is updated by modeling the contrastive failure-success information in the trajectory pair data.
Given trajectory pair ew ≻ el | u, the failuresuccess relation can be modeled via Bradley-Terry
(BT) (Bradley and Terry, 1952) model:
p(ew ≻ el|u) = exp (r(u, ew))
exp (r(u, ew)) + exp (r(u, el)) .
(8) Under the optimal policy πr(e|u) of Eq. (6), the reward function in the environment can be written as (Peng et al., 2019; Rafailov et al., 2023):
r(u, e) = β log πr(e|u) πref(e|u) + β logZ(x), (9)
where Z(u) = ∑ e πref(e|u) exp ( 1 β r(u, e) ) is the partition function. Substitute Eq. (9) into Eq. (8) to get the BT model over policy:
p(ew ≻ el|u) =
σ ( β log
πθ(ew|u) πθ(el|u) − β log πref(ew|u) πref(el|u)
) ,
(10)
where σ is the sigmoid function. Then the optimal policy πθ can be achieved by applying maximum likelihood: LDPO(πθ;πref) =
− E(u,ew,el)∼Dp
[ log σ ( β log
πθ(ew|u) πθ(el|u) − β log πref(ew|u) πref(el|u)
)] .
(11) This optimization objective aims to increase the likelihood of the success trajectories ew and decrease the likelihood of failure trajectories el, with
a constraint term to maintain the basic agent capabilities. Moreover, as a reformulation of RL objective Eq. (6), Eq. (11) is directly maximizing the final reward while avoiding the need to perform RL optimization.
Iteration To further improve the agent’s performance, ETO adopts an iterative explorationtraining manner. After the training phase, the agent policy can be used to gather new failure cases and create contrastive trajectory pairs. These new data are then used to further enhance the agent through trajectory contrastive learning. The complete learning process of ETO is shown in Algorithm 1.

heading: 4 Experiments
text: In this section, we conduct extensive experiments to validate the effectiveness of ETO. Our method demonstrates superior performance compared to baselines across three datasets, and it exhibits enhanced advantages when dealing with out-ofdomain unseen tasks. The analysis further showcases the efficiency of our method. Furthermore, our method also holds promise in scenarios where expert trajectories are unavailable.

heading: 4.1 Experimental Settings
text: Datasets We conduct experiments on three representative agent datasets, WebShop (Yao et al., 2022a) for web navigation, ScienceWorld (Wang et al., 2022) for embodied science experiments, and ALFWorld (Shridhar et al., 2021) for embodied house holding tasks. Both WebShop and ScienceWorld environments provide dense final rewards ranging from 0 to 1, while ALFWorld only provides binary rewards indicating whether the task is completed. All three environments can be formally described as partially observable Markov decision processes. For details of the datasets and the expert trajectory collection process, please refer to Appendix A.
The statistical information of our datasets is presented in Table 1. It is important to mention that, in addition to the in-distribution test sets, both ScienceWorld and ALFWorld contain test sets that include out-of-distribution unseen variations. These additional test sets allow us to assess the generalization capabilities of different agents.
Training Setup We mainly use Llama-2-7BChat (Touvron et al., 2023) as the base model for building LLM agents. To provide more comprehen-
sive results, we also conduct experiments on Llama2-13B-Chat and Mistral-7B (Jiang et al., 2023). We utilize the AdamW optimizer (Loshchilov and Hutter, 2017). For the SFT phase, the batch size is 64 and the learning rate is set to 1e-5 with 3% warm up and a cosine scheduler. Then the base agent will explore once for each instance in the training set to collect failure trajectories. For the training phase of ETO, the batch size is 32 and the learning rate is set to 1e-6. The β in DPO loss is set to 0.1 for WebShop and ScienceWorld, 0.5 for ALFWorld. The learning epochs of SFT phase and training phase in ETO are set to 3. The number of iterations of ETO is set to 2 for WebShop and ScienceWorld, 1 for ALFWorld. All experiments are conducted on 8 NVIDIA A100 80G GPUs.
Baselines We compare ETO with SFT behavioral cloning and other post-imitation baseline methods. 1) SFT (Chen et al., 2023; Zeng et al., 2023) conducts behavioral cloning on expert trajectories, which is the base agent for ETO and other baselines. 2) Best-of-N sampling employs SFT base agent and selects the trajectory with the best reward within N samplings. Here we set N to 10. 3) RFT (Rejection sampling Fine-Tuning) (Yuan et al., 2023) is a strong baseline which adds the success trajectories to the expert trajectory dataset and trains the agent on new augmented trajectories. 4) PPO (Proximal Policy Optimization) (Schulman et al., 2017) is an RL method directly optimizing the SFT agents to maximize the final task reward. We also include GPT-3.5-Turbo (OpenAI, 2022), GPT-4 (OpenAI, 2023), and untuned Llama-2-7BChat for comparison.
Evaluation All methods are evaluated using the ReAct-style interaction format (Yao et al., 2022b), with CoT rationale generated before the action. See Appendix E for the detailed prompts. We add 1- shot in-context example in the instruction prompt for each task. The decoding temperature of the LLMs is set to be 0.0 for deterministic generation,
except for Best-of-N method. We mainly employ Average Reward as the metric, which represents the average reward of all task instances in the test set. We also report Success Rate in Appendix B for reference.

heading: 4.2 Results
text: Table 2 presents the performance comparison of ETO and baselines on three agent datasets. As shown, ETO demonstrates a significant improvement over SFT imitation learning, leading to an average reward increase of 8% and 9.5% for WebShop and ScienceWorld. Our method also outperforms all other baselines on all datasets. On WebShop dataset, ETO even outperforms GPT-4 on the average reward, showing the extraordinary performance of our method. Although the RFT method also exhibits improvement compared to SFT, its performance remains constrained as it is an augmented version of behavioral cloning and solely learns from success trajectories. This comparison indicates the comparison between failure and expert trajectories is essential to improve the performance of the agent. Meanwhile, though PPO gains improved performance on WebShop, it struggles to achieve satisfactory results on the other two datasets due to the inherent instability in RL optimization procedures, particularly on ALFWorld dataset which only provides binary final rewards. In Appendix D, we present case studies to show the task-solving trajectories of our method.
Notably, our approach showcases enhanced advantages in out-of-domain unseen scenarios, achieving an impressive performance boost of 20% on ScienceWorld-Unseen. Moreover, ETO exhibits strong effectiveness on the unseen scenarios
in ALFWorld and outperforms the RFT and PPO baselines, both of which suffer from performance degradation. These results underscore that learning by trial and error can further enhance the agent’s generalization capabilities, particularly in out-ofdistribution unseen scenarios.
Results on Different LLMs To further demonstrate the effectiveness of our method, we present the results based on other base LLMs, including Llama-2-13B-Chat and Mistral-7B. Table 3 showcases the consistent improvement in agent performance achieved by ETO across different LLMs. Notably, when compared to Llama-2-7B, the 13B model displays a relatively smaller performance gain on both datasets, suggesting that our method can provide greater benefits to weaker agents. Despite Mistral-7B is a more powerful LLM than Llama-2-13B, it falls short of Llama-2-7B after either SFT or ETO. This finding indicates that there is not a strong correlation between the basic LLM capabilities and the agent capabilities.
Analysis on Efficiency We evaluate the tasksolving efficiency of agents in ScienceWorld environment, which provides fine-grained subgoals for each task. The reward of a task is updated
upon the accomplishment of a subgoal. By assessing the agent’s ability to achieve higher rewards within fewer action steps, we can determine its efficiency. Figure 3 showcases the score trajectories of ScienceWorld-Seen test set, comparing ETO with the SFT base agent, and the oracle agent. As depicted, ETO can reach higher rewards in fewer action steps than the SFT base agent. Interestingly, in certain cases like 15-90 and 19-23, our method outperforms even the oracle agent, reaching a score of 100 earlier. These results demonstrate that by learning from failure trajectories, our method also acquires a more powerful task-solving efficiency.

heading: 4.3 Ablation of Iterations
text: In this section, we present a study on the impact of iteration numbers in ETO. The results are shown in Figure 4. As depicted, ETO demonstrates the ability to enhance the performance of agents in the first two iterations on both the WebShop and ScienceWorld datasets. However, further increasing the iterations does not lead to continuous improvement. Instead, the performance starts to decline after the third iteration. Regarding the ALFWorld dataset, only the first iteration of ETO shows an improvement. Surprisingly, the performance on the second and third iterations even falls behind that of the SFT base agent.
To explain this, it is important to note that the learning process of ETO relies on a fixed expert trajectory set, and the exploration phase of the agent is performed on the same training set. Consequently, the diversity and quantity of failure-success contrastive trajectory data are constrained. Initially, the policy can be improved by learning from past mistakes. However, the model gets overfitting on
the contrastive information in subsequent iterations, resulting in a decline in performance. In the case of ALFWorld, the coarse-grained binary rewards further hinder the agent from getting improvement from iterative training. As a potential solution, future work could explore the incorporation of GPT-4 to dynamically construct more diverse contrastive trajectory data.

heading: 4.4 Strategy of Contrastive Data Construction
text: In this section, we delve deeper into the contrastive trajectory pair construction strategy used in our method. In Section 3.2, we directly learn from failure-success trajectory pairs (Eq. (11)), referred to trajectory-wise contrastive. Alternatively, inspired by previous work (Lightman et al., 2023), we introduce a fine-grained variation of ETO that captures step-wise contrastive information by comparing “good-bad” action pairs. To achieve this, we use the expert trajectory to conduct teacher forcing for first t − 1 steps, and then have the agent predict the action of t-th step. Then the quality of t-th action is determined by the final rewards. We also implement a mixture variation by combining the above two strategies. For further details regarding the step-wise variation of ETO, please refer to Appendix C.
The comparison of different methods is presented in Table 4. As the results demonstrate,
trajectory-wise contrastive yields the best performance. On the other hand, we observed that stepwise contrastive modeling tends to be less stable, necessitating a lower learning rate and a higher constraint parameter β to maintain the basic capabilities of the agent. This instability may be attributed to the inaccurate estimation of the action quality, as we simply utilize the final rewards to construct step-wise contrastive pairs. Moreover, the performance of mixture strategy also falls short compared to trajectory-level contrastive.

heading: 4.5 Self-Play w/o Expert Trajectory
text: In this section, we explore a challenging scenario where no expert trajectory is available. In such cases, the agent is compelled to explore the environment and depend on self-play to enhance its capabilities. To achieve this, we eliminate the behavioral cloning phase of ETO and allow the LLM agent to explore the environments using a decoding temperature of 1.0. Subsequently, we compare different trajectories associated with the same instruction based on their final rewards, creating trajectory preference data. Finally, the agent is trained exclusively on the preference data generated by itself.
On WebShop, the untuned Llama-2-7B-Chat achieves relatively high rewards. Thus, we use this dataset to conduct the experiment. We also employ rejection sampling fine-tuning (RFT) as a baseline. The results in Table 5 show that ETO alone does not improve performance without behavioral cloning. In contrast, RFT shows promising ability to enhance the agent’s capabilities without relying on expert trajectories. However, when combining RFT with ETO, we observe a further enhancement in the agent’s performance. These findings suggest that in scenarios without expert trajectories, it may be beneficial to first employ RFT and then allow the agent to learn from exploration failures. These results further highlight the potential of our method when expert trajectories are unavailable.

heading: 5 Related Work
text: Imitation Learning Imitation learning is a learning paradigm where an agent learns a policy by mimicking expert demonstrations (Hussein et al., 2017; Fang et al., 2019). A prevalent approach in imitation learning is behavioral cloning (BC) (Pomerleau, 1991), which utilizes expert trajectories to learn a direct mapping from states to actions. There are various methods to mitigate
the limitations of BC (Ross et al., 2011; Ross and Bagnell, 2014). Our method, ETO, shares a similar spirit with DAgger (Ross et al., 2011), an approach used to enhance the agent’s performance by learning from failure cases. However, unlike DAgger which gathers additional expert trajectories on agent-failed cases, ETO improves the policy through learning from contrastive trajectory pairs.
LLM Agents With the various emergent abilities of LLMs, researchers have explored building agent systems based on LLMs (Xi et al., 2023). Recent projects such as AutoGPT (Richards, 2023), BabyAGI (Nakajima, 2023), and RestGPT (Song et al., 2023) have employed LLMs as core controllers, building powerful agent frameworks capable of solving realistic tasks. While GPTs have shown strong agent intelligence, open-source LLMs still lag far behind (Liu et al., 2023; Wang et al., 2023d). To bridge this gap, recent studies, including FireAct (Chen et al., 2023), AgentTuning (Zeng et al., 2023), and Lumos (Yin et al., 2023), construct expert trajectory data from teacher agents (e.g., GPT-4) and perform BC on opensource LLMs. Taking a step further, Aksitov et al. (2023) refine the agent through iterative BC on success trajectories generated by the previous policy. Concurrently with our work, Yang et al. (2023) use the DAgger framework (Ross et al., 2011) and also employ DPO loss to develop embodied multimodal agents.
LLM Policy Learning Learning from preference has shown promise for learning an enhanced LLM policy, particularly in LLM alignment research. Reinforcement Learning from Human Feedback (RLHF) is a method that learns a reward model
and then utilizes proximal policy optimization to update the policy model (Christiano et al., 2017; Ouyang et al., 2022). Despite its attractive advantages, RLHF presents limitations regarding training efficiency and instability. To address these issues, Rafailov et al. (2023) reformulate the optimization objective of RLHF, introducing the DPO loss to directly model preferences. Similar to our work, ReST (Gulcehre et al., 2023) iteratively generates new samples from the current policy and refines the policy using offline RL methods. Recent studies have explored the application of LLM policy learning in other domains (Lightman et al., 2023; Wang et al., 2023b). For example, Wang et al. (2023c) train a step-wise reward model to improve the performance of LLMs in mathematical reasoning. The comparison of ETO with several alternatives in Table 6 highlights the strength of our method in LLM agent policy learning.

heading: 6 Conclusion
text: In this work, we present ETO, a method aimed at enhancing the capabilities of LLM agents. Our approach allows the agent to learn by trial and error, thereby improving the performance of the base agent acquired through behavioral cloning. ETO uses an exploration-training iteration framework. During the exploration phase, the agent explores the environment, gathering failure trajectories and constructing trajectory preference pairs. Subsequently, in the training phase, the agent learns from the preference information using DPO loss. This iterative process of exploration and training enables further improvement in the agent’s performance. Extensive experiments on three agent datasets demonstrate our method outperforms behavioral cloning and strong baselines by a large margin. Moreover, our method exhibits remarkable efficiency and shows great potential in scenarios where expert trajectories are unavailable.
Limitations
Our method, ETO, demonstrates effective learning of powerful LLM agents through trial and error. However, it is important to acknowledge several limitations of this work. 1) ETO simplifies the comparison of failure-success trajectories by assuming that the agent generates wrong actions right from the beginning. However, in realistic cases, the agent may start executing incorrect actions from some intermediate step. If we can identify when the agent makes a bad action (e.g., â3 at 3-th step), we should then collect the expert trajectory for the remaining actions at>3. Unfortunately, most current environments do not contain such information, making it challenging to conduct action-wise or process-level reward modeling. A potential solution is to employ GPT-4 to identify the bad action and construct fine-grained contrastive trajectory data. 2) This work primarily focuses on developing specialized LLM agents for a specific agent task, with limited exploration into the construction of strong generalized agents. For future work, we will investigate the transferability of the policies trained by ETO and try to apply our method in a multi-task training scenario.
Ethics Statement
This work fully complies with the ACL Ethics Policy. We declare that there are no ethical issues in this paper, to the best of our knowledge.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful comments on this paper. This work was partially supported by National Key R&D Program of China (No. 2022YFC3600402).

heading: A Datasets
text: WebShop WebShop (Yao et al., 2022a) is an online shopping website environment where agents navigate the platform to make purchases based on user instructions. Once the agent selects the "buy" action, the environment provides a final reward, which is calculated based on the matching heuristics of the product’s attributes and price.
ScienceWorld ScienceWorld (Wang et al., 2022) is a text-based virtual environment centered around accomplishing elementary science experiments, including 10 different task types such as thermodynamics and electrical circuits. The agents need to be grounded in embodied interactive environments to engage with and comprehend scientific concepts through practical experience. Each task in ScienceWorld includes several optional subgoals, and the overall final reward is computed based on the achievement of these subgoals.
The original test set in ScienceWorld consists of critical unseen task variations. For instance, in the training set, the task may involve boiling water, whereas in the test set, the task is to boil lead. Consequently, we employ the original test set to evaluate our model’s generalization performance on unseen scenarios. We utilize the original development set as our test set with seen scenarios. We exclude Task-9 and Task-10 due to their excessively long task-solving trajectories. Following Lin et al. (2023), we use the first 10 instances for task types with more than 10 test variations for fair and cost-effective comparisons.
ALFWorld ALFWorld (Shridhar et al., 2021) consists of interactive TextWorld environments that parallel embodied worlds in the ALFRED (Shridhar et al., 2020) dataset. In this environment, agents are required to explore and complete high-level house-holding instructions. The original ALFWorld dataset comprises both seen and unseen evaluation sets. The seen set is designed to assess in-distribution generalization, whereas the unseen set with new task instances measures out-of-distribution generalization of the agents.
CoT Annotation Webshop and ALFWorld provide a few human-annotated trajectories for imitation learning. We also employ GPT-4 as the teacher agent to explore in the WebShop environment and select trajectories which have a reward greater than 0.7. ScienceWorld environment provides heuristic searching algorithms to generate golden trajectories for each sub-task. Since the original trajectories do not contain CoT information for each action step, we use GPT-4 to generate the corresponding rationales.

heading: B Success Rate
text: We report the success rate of our experiments in Table 7. Note the definition of success rates of three tasks are different. For WebShop, success rate is defined as the portion of instances where final reward is 1.0. For ScienceWorld, the original paper does not provide the definition of success rate. However, according to its official environment, a trajectory is considered success if the environment reaches a pre-defined latent state where the reward may not be exactly 1.0. For ALFWorld, since it only provides binary final rewards, success rate is equal to average final reward.

heading: C Details for Step-Wise Contrastive
text: We implement a variation of ETO which learns from contrastive good-bad action pairs. Specifically, for a task instruction u with expert trajectory e = (u, a1, ..., on−1, an), we utilize teacher forcing for the first t− 1 steps (a1, o1, ...at−1, ot−1), and let the agent predict the actions from t-th step to get the trajectory:
ê = (u, a1, o1, ..., ot−1, ât, ôt, ..., ôm−1, âm) (12)
The environments return a reward r̂ for the trajectory ê. If we denote the golden trajectory for the first t− 1 steps as e(t−1), then the good-bad action pairs aw ≻ al | u, e(t−1) is constructed based on the final rewards. Here, aw and al represent the actions with higher and lower final rewards, chosen from (at, ât)
respectively. Then the contrastive relation of the action pair can also be utilized in DPO loss to improve the policy:
LDPO(πθ;πref) = −E [ log σ ( β log
πθ(aw|u, e(t−1)) πθ(al|u, e(t−1)) − β log πref(aw|u, e(t−1)) πref(al|u, e(t−1))
)] . (13)

heading: D Case Study
text: Here, we present case studies to compare ETO agents with SFT behavioral cloning agents. We illustrate examples of WebShop and ScienceWorld in Figure 5 and 6, respectively.
In the WebShop scenario, the SFT agent failed to select the “3pc” color option, resulting in a suboptimal trajectory. Conversely, ETO demonstrated the ability to learn from past failures and adeptly chose the appropriate attribute option.
In the ScienceWorld example, the task is to find a(n) animal. The SFT agent, however, persisted in performing the incorrect action, focusing on a non-living object instead. Additionally, despite generating a correct CoT, the SFT agent’s action selection was inconsistent with the CoT rationale. In contrast, ETO successfully accomplished the task, showcasing the effectiveness of trial and error.

heading: E Prompt for Evaluation
text: We show the instruction prompts for WebShop, ScienceWorld, ALFWorld in Figure 7, 8, 9, respectively.
Instruction Prompt for WebShop
You are doing a web shopping task. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation and a list of available actions, you have to respond to an action based on the state and instruction. You can use search action if search is available. You can click one of the buttons in clickables. An action should be one of the following structure: search[keywords] or click[value]
If the action is not valid, perform nothing. Keywords in search are up to you, but the value in click must be a value in the list of available actions. Remember that your keywords in search should be carefully designed.
Your response should use the following format: Thought: I think ... Action: click[something]

heading: Future Work
text: For future work, we will investigate the transferability of the policies trained by ETO and try to apply our method in a multi-task training scenario