Large Language Models Can Self-Improve At Web Agent Tasks

## Abstract

Abstract Training models to act as agents that can effectively navigate and perform actions in a complex environment, such as a web browser, has typically been challenging due to lack of training data. Large language models (LLMs) have recently demonstrated some capability to navigate novel environments as agents in a zero-shot or few-shot fashion, purely guided by natural language instructions as prompts. Recent research has also demonstrated LLMs have the capability to exceed their base performance through self-improvement, i.e. fine-tuning on data generated by the model itself. In this work, we explore the extent to which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. In WebArena, an agent must autonomously navigate and perform actions on web pages to achieve a specified objective. We explore fine-tuning on three distinct synthetic training data mixtures and achieve a 31% improvement in task completion rate over the base model on the WebArena benchmark through a self-improvement procedure. We additionally contribute novel evaluation metrics for assessing the performance, robustness, capabilities, and quality of trajectories of our fine-tuned agent models to a greater degree than simple, aggregate-level benchmark scores currently used to measure self-improvement.

## 1 Introduction

Large language models (LLMs) have demonstrated impressive capabilities in a variety of natural language processing (NLP) tasks such as summarization and question answering (Radford et al., ; Raffel et al., ; Brown et al., ) through zero-shot and few-shot prompting techniques (Ouyang et al., ; Wei et al., ). However, prompting techniques alone are insufficient to enable LLMs to act as agents and navigate environments in order to solve complex, multi-step, long-horizon tasks (Yao et al., ). Fine-tuning LLMs to perform such tasks is also infeasible due to the scarcity of training data suitable for these tasks. Acquiring data for sequential decision-making and complex interactions is not only time-consuming, but also costly. Additionally, automatic evaluation of trajectories (or sequences of actions) taken by an agent is also difficult (Dinu et al., ).
The absence of metrics that accurately capture the efficacy of each step in a sequence complicates the assessment of incremental improvements or degradations in an agent’s performance.

A number of proposed self-improvement techniques have demonstrated that LLMs can use zero-shot and few-shot prompting to achieve performance above the baseline without any additional supervised training data (Huang et al., ; Chen et al., ). In place of supervised data as a learning signal, many of these techniques use a self-critique technique (Weng et al., ; Yuan et al., ), or obtain a critique through interactions with tools or environments (Gou et al., ). While self-improvement techniques have shown promise on standard NLP benchmark tasks like machine translation or question answering (Han et al., ; Huang et al., ; Chen et al., ), their efficacy has not yet been thoroughly investigated for long-horizon tasks that require multi-step interactions with a complex and realistic environment.

WebArena (Zhou et al., ) is a recently proposed benchmark wherein an LLM agent is required to solve tasks using a web browser. One example WebArena task is to use the OpenStreetMap website to answer the question “What is the minimum travel time by car from CMU to University of Pittsburgh?”. Such a task requires an agent to complete a sequence of steps on the website, including entering a start location, entering a destination location, submitting a form, and then, reasoning over the result. The sequence of steps selected by an agent is called a trajectory. Unlike existing benchmarks, WebArena tasks are realistic and diverse, require dynamic interaction, and require navigating a complex environment. The baselines presented by Zhou et al. () demonstrate that while LLMs are capable of interacting with this environment, even the strongest baseline, GPT-4 (OpenAI et al., ), is only able to solve ~14% of the tasks. This demonstrates that WebArena is a challenging benchmark even for the strongest frontier models (Chiang et al., ).

In this paper, we introduce new techniques that allow LLM agents to better perform complex, multi-step tasks via self-improvement. We detail different strategies for self-improvement that all involve fine-tuning the LLM agent on its own generations (synthetic data) and inducing a signal for learning by employing unsupervised techniques like self-critique to selectively filter training examples. To better understand the effect of our self-improvement, we introduce two auxiliary metrics: 1) a measure to analyze capabilities acquired and lost by the agent and 2) an extension of the VERTEX score (Dinu et al., ) to measure the quality of variable-length agent trajectories. These metrics allow finer-level assessment of improvements and degradations than aggregate-level benchmark scores.

Figure: Figure 1: We generate synthetic data to fine-tune LLM agents to accomplish WebArena tasks such as “Add this product to my wishlist”. Step 1: We first collect an initial set of trajectories, filter out low-quality trajectories in an unsupervised fashion, and keep the remainder as synthetic in-domain examples. We prompt our base LLM to generate novel out-of-domain tasks along with hypothetical solution trajectories by providing a few in-domain examples. Step 2: We then fine-tune our base LLM agent on each of the three distinct synthetic training data mixtures and evaluate performance.
Refer to caption: /html/2405.20309/assets/x1.png

In summary, our contributions are:

- •
We propose and detail procedures for collecting and generating synthetic training examples for complex, multi-step tasks involving interaction with an environment. We explore collecting in-domain synthetic examples of trajectories as well as generating synthetic examples of solution trajectories for novel, out-of-domain tasks.
- •
We show that the performance of LLM agents improves after fine-tuning on this synthetic data, demonstrating that self-improving techniques work for a new class of tasks. We analyze three synthetic training data mixtures and find all three mixtures improve performance, with the best performing mixture yielding a 31% improvement over the base LLM agent on the WebArena benchmark.
- •
We propose auxiliary metrics to understand the effect self-improvement has with respect to acquiring new capabilities and to evaluate variable-length trajectories produced by agents through an extension of the VERTEX score. These metrics provide nuanced insights not captured by aggregate-level benchmark scores currently used to evaluate self-improvement, allowing us to better assess the effect self-improvement has on multiple dimensions: performance, robustness, capability acquisition, and the quality of generated trajectories.

## 2 Synthetic Data Collection and Generation

Self-improvement techniques for large language models typically involve using the model’s own generations to create synthetic few-shot examples (Han et al., ) or synthetic fine-tuning data (Huang et al., ). These techniques amplify knowledge, correct behaviors, and introduce regularization (Pham et al., ), often leading to an overall boost in performance. The self-generated examples are often filtered, post-edited, or ranked with a set of unsupervised techniques such as self-critique to introduce a signal for learning and improvement (Weng et al., ; Patel et al., ; Chen et al., ; Yuan et al., ). For multi-step agent tasks, the environment itself can additionally provide the LLM agent a way to detect failure in a fully unsupervised manner, which provides another useful signal for learning (Gou et al., ; Yuan et al., ; Song et al., ).

Using the WebArena benchmark (Zhou et al., ), we define and experiment with both in-domain synthetic training examples and out-of-domain synthetic training examples for web agent tasks, and fine-tune on three different synthetic data mixtures: Mixture A (in-domain synthetic examples only), Mixture B (both in-domain and out-of-domain synthetic examples), and Mixture C (out-of-domain synthetic examples only). Figure [1](#S1.F1) illustrates our process.

#### In-Domain Synthetic Data :

For all tasks in WebArena, we collect an initial set of trajectories using the base model. We filter out any trajectories where the model self-detected failure (self-critique) or failure was detectable in the environment and keep the remainder. We denote the remaining set of trajectories as plausible trajectories, where the model may or may not have completed the task successfully. Since lower-quality trajectories where the model outright failed to complete the task have been filtered out through self-detection, we hypothesize this remaining higher-quality set of plausible trajectories can serve as reasonably high-quality in-domain synthetic examples for fine-tuning. Similar to the self-improvement prior work we discuss earlier, the collection of this data is completely unsupervised and no ground-truth labels are utilized for filtering and selection.

#### Out-of-Domain Synthetic Data :

We also evaluate whether the base model can generate completely novel tasks, objectives, web pages, and solution trajectories that can serve as useful training examples. We use the plausible trajectories as few-shot examples in a prompt for the base model to generate completely new tasks along with potential solution trajectories. To ensure the model generates examples with sufficient diversity and to improve generalization, we prompt the model to generate out-of-domain synthetic examples that are dissimilar from existing tasks and objectives as well as generate tasks for different websites than the set of 6 websites covered by the WebArena benchmark.

### 2.1 In-Domain Synthetic Data Collection

The WebArena environment can be formulated as a partially observable Markov decision process: $\mathcal{E}=\langle\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T}\rangle$, where $\mathcal{S}$ represents the state space, $\mathcal{A}$ represents the action space, $\mathcal{O}$ represents the observation space, and $\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ is the deterministic transition function (Zhou et al., ). An agent model $\mathcal{M}$ produces a next action $a_{t}\in\mathcal{A}$ provided an objective represented by some natural language intent $\mathbf{i}$, the current observation $o_{t}\in\mathcal{O}$, and the previous action taken $a_{t-1}\in\mathcal{A}$: $(\mathbf{i},o_{t},a_{t-1})$. This continues for $T$ time steps until the agent produces a stop action or the environment produces an error or stop condition. The model $\mathcal{M}$ we select for our experiments is the Qwen-1.5-72B-Chat model (Bai et al., ), which at the time of this work is a highly ranked(^1^11[https://chat.lmsys.org/?leaderboard](https://chat.lmsys.org/?leaderboard)) and competitive open source LLM (Chiang et al., ) that is accessible for fine-tuning. Further choice of inference parameters and other configuration details can be found in Appendix [A](#A1).

Given this definition, we propose a procedure for sampling a set of in-domain synthetic training examples $\mathcal{D}_{\textsc{In-Domain}}$ where each training example is structured as $(\mathbf{i},o_{t},a_{t-1})\rightarrow a_{t}$. These examples are sampled from a filtered set of trajectories collected by an initial run of the base agent model $\mathcal{M}$ over all tasks in WebArena:

Figure: Algorithm 1 Collect In-Domain Synthetic Training Examples $\mathcal{D}_{\textsc{In-Domain}}$

We filter out low-quality trajectories where the model produced a generation stating the task to be “impossible” or that it “cannot” make progress (a form of self-critique). Additionally, we filter out any trajectories where the model produced stop[N/A], stop[No ...], or stop[], indicating when the model may have refused to provide an answer. Finally, we also filter out any trajectories where the WebArena environment encountered an error or the model failed to produce a valid, parsable generation. The final dataset of synthetic examples is balanced by randomly sampling an equal number of initial steps ($t=0$), final steps ($t=T$), and intermediate steps ($t=1\ldots(T-1)$) from the plausible trajectories in $\mathcal{P}$. In Table [1](#S2.T1), we display how effective this unsupervised filtering process is by measuring the accuracy, precision, and recall of the 58 remaining trajectories kept in $\mathcal{P}$ from the 812 total trajectories to assess the proportion of correct/incorrect examples in $\mathcal{D}_{\textsc{In-Domain}}$.

**Table 1: Metrics on the proportion of trajectories that successfully completed the task in the set of plausible trajectories kept in $\mathcal{P}$ after filtering out low-quality trajectories. Approximately 43% of trajectories in $\mathcal{P}$ successfully completed the task, up from ~7% with no filtering, indicating useful learning signal is introduced by filtering using self-critiques and information from the environment.**
| Set of Trajectories | # | Accuracy | F1 | Precision | Recall |
| --- | --- | --- | --- | --- | --- |
| All Trajectories | 812 | 0.071 | 0.133 | 0.071 | 1.000 |
| Plausible Trajectories $\mathcal{P}$ | 58 | 0.919 | 0.431 | 0.431 | 0.431 |

### 2.2 Out-of-Domain Synthetic Data Generation

Using examples from $\mathcal{D}_{\textsc{In-Domain}}$ as seed examples, we prompt our base LLM $\mathcal{M}$ to synthetically generate completely novel tasks, objectives, web pages, and solution trajectories to produce $\mathcal{D}_{\textsc{Out-of-Domain}}$.

Figure: Algorithm 2 Generate Out-of-Domain Synthetic Training Examples $\mathcal{D}_{\textsc{Out-of-Domain}}$

When generating new objectives, we use 4 few-shot examples (two objectives sampled from tasks in WebArena and two sampled from previously generated objectives). We use 2 few-shot examples when generating previous actions, next actions, and observations (web pages in the form of accessibility trees). We use a temperature of 1.0 and set top-p to 1.0 during generation. Detailed information on the prompts used for generating $\mathcal{D}_{\textsc{Out-of-Domain}}$ can be found in Appendix LABEL:sec:appendix_prompts. When generating novel objectives, we specifically prompt the model to generate objectives that are dissimilar to the example objectives to encourage out-of-domain generations. We also ensure each novel objective has $<0.70$ cosine similarity with any objective previously generated using the all-distilroberta-v1 sentence similarity model (Reimers and Gurevych, ; Liu et al., ; Sanh et al., ) to promote diversity. Table [2](#S2.T2) gives examples of out-of-domain objectives that our method generated.

**Table 2: A sample of the novel objectives generated compared with the objectives found in WebArena. A full sample of a generated out-of-domain synthetic example can be found in Appendix [B](#A2).**
| Objectives in WebArena Benchmark | Generated Out-of-Domain Objectives |
| --- | --- |
| •<br>Tell me the total cost of my latest pending order? (Shopping)<br>•<br>Compare the time for walking and driving route from AMC Waterfront to Univ of Pittsburgh (Maps)<br>•<br>Check out the most recent open issues (GitLab)<br>•<br>Which customer has placed 2 orders in the entire history? (Shopping Admin)<br>•<br>$\ldots$ | •<br>Locate and purchase a subscription to The Economist digital edition (https://store.economist.com/...)<br>•<br>Find the nutrition facts for a Grilled Chicken Caesar Salad from Chili’s (http://www.chilis.com/...)<br>•<br>Find the active coupons for a one-year subscription to Adobe Creative Cloud (https://www.couponcabin.com/...)<br>•<br>Subscribe to the premium plan for Grammarly to unlock advanced writing features. (https://www.grammarly.com/...)<br>•<br>$\ldots$ |

## 3 Evaluation

We perform evaluation using the standard metrics proposed by the WebArena benchmark like functional correctness (Zhou et al., ) as well as evaluate with new auxiliary metrics we propose that give more nuanced insight into an agent’s performance.

### 3.1 Functional Correctness Score

Functional correctness is the standard metric proposed by the WebArena benchmark that is a simple binary task completion score (0 or 1) averaged over all 812 tasks in the benchmark.

### 3.2 Capability Score (New)

While WebArena contains 812 unique task instances, these 812 tasks are instantiated using natural language intent templates like “What is the minimum travel time by car from {{location1}} to {{location2}}?”. Therefore, many tasks actually test the same capability. Aggregate-level metrics like the functional correctness score may be misleading since improvements may only be due to the model becoming more robust at solving capabilities it already could solve versus demonstrating the ability to solve new capabilities that were previously unsolvable. There are 241 unique templates in WebArena that are used to instantiate 812 tasks. Moreover, some of these templates are simple paraphrases of each other. For example, “What is the estimated driving time between {{city1}} and {{city2}}?” is a paraphrase of the prior template. Using a sentence similarity model (^2^22We use the all-distilroberta-v1 sentence similarity model (Sanh et al., ).), we iteratively group these templates into a set of unique capabilities. Each template is grouped with any existing capability if it has a similarity of > 0.60 with any template in the group, otherwise the template is added to a new capability group. This results in 136 unique capabilities (see Appendix [F](#A6)). A model receives a score of 1 for each capability group with at least one successful task completed, otherwise it receives a score of 0 (^3^33Since we do not count trivial tasks as a successful completion, a single successful completion of a capability provides sufficient evidence of acquisition. We discuss robustness and consistency separately in Section [5](#S5).). The capability score is then the averaged over all 136 capabilities.

We note, however, that a number of tasks in the WebArena benchmark are trivial tasks and can be solved by a trivial baseline agent or weak model that performs no actions and only immediately exits by always generating stop [N/A]. In the capability score computation, we do not count such trivial tasks as evidence a model can perform the capability as these are degenerate cases of the capability.

### 3.3 VERTEX DTW fragments VERTEX DTW \text{VERTEX}_{\text{DTW}} Score (New)

Both functional correctness and the capability score only evaluate task completion, however, they do not assess the quality of entire trajectories, therefore, a measure that is sensitive to incremental improvements and degradations in trajectories, independent of task completion, is desirable. We extend the recently proposed VERTEX score (Dinu et al., ), which measures the similarity of two relational trajectories by using embeddings to compare node distributions within a computational graph. The VERTEX score integrates the semantic meaning across the distributional path by computing at each node the cross-similarity between the generated embeddings and embeddings sampled from a reference distribution. An ideal reference distribution would be ground-truth reference trajectories produced by humans for all of the WebArena tasks. In absence of this, we use a larger, stronger model, GPT-4 (OpenAI et al., ), to collect three reference trajectories for each task.

One obstacle to the straightforward application of the VERTEX score is the assumption that both trajectories are of the same length. Agents operating in complex environments, however, are not constrained to a fixed-length for the trajectories they produce. Therefore, we propose modification in the computation of the VERTEX score that enables comparison of sequences with different lengths. Our extension consists of an additional alignment step prior to calculating the VERTEX score for the aligned trajectories.
First, we embed all steps of a trajectory $\mathcal{X}$ as $e_{t}=f(o_{t},a_{t})\in R^{d}$, where $f$ is an embedding model(^4^44We use the all-mpnet-base-v2 embedding model (Song et al., ).) with embedding dimension $d$.
The embedding model $f$ is independent of both the model that generated the reference trajectories as well as the model that generated the test trajectories.
Then, we use Dynamic Time Warping (DTW) (Berndt and Clifford, ) to align two embedded trajectories $\tilde{\mathcal{X}}_{m}=(e_{0},\dots,e_{i},\dots,e_{m})\in R^{m\times d}$ and $\tilde{\mathcal{X}}_{n}=(e_{0},\dots,e_{j},\dots,e_{n})\in R^{n\times d}$ with length $m$ and $n$, respectively.
Consequently, we refer to our proposed measure as $\text{VERTEX}_{\text{DTW}}$.
DTW returns an alignment path $\nu$ of length $T$, where each $e_{i}\in\tilde{\mathcal{X}}_{m}$ is aligned with a corresponding $e_{j}\in\tilde{\mathcal{X}}_{n}$, preserving the order in their respective trajectory.
This order preservation occurs because once a node is matched, it is excluded from potential new matches, maintaining the integrity of the temporal alignment.
As a scoring function for DTW, we choose cosine distance.
In addition to the alignment step, we introduce a linear distance decay factor that decreases the contribution of aligned embeddings if they are far apart in the original trajectories.
Once two trajectories are aligned, we compute the VERTEX score by Eq. (4) in Dinu et al. () with the addition of the distance decay.
Therefore, the $\text{VERTEX}_{\text{DTW}}$ score is computed as:

$$ $s(\tilde{\mathcal{X}}_{\text{ref}},\tilde{\mathcal{X}}_{\text{test}},\nu):=\frac{1}{T}\int_{t_{0}}^{t_{T}}\big{[}\min(\max(0,\frac{1}{1+|i_{\nu_{t}}-j_{\nu_{t}}|}\ \widetilde{\text{MMD}^{2}}(e^{\nu_{t}}_{\text{ref}},e^{\nu_{t}}_{\text{test}})-z_{\mathrm{rand}}),1)\big{]}dt,$ (1) $$

where $i_{\nu_{t}}$ and $j_{\nu_{t}}$ are the position indices in the alignment path $\nu$ at time $t$, $\mathcal{\tilde{X}}_{\text{ref}}$ and $\mathcal{\tilde{X}}_{\text{test}}$ are aligned trajectories of embeddings from the reference set and the model under test, respectively, and $z_{\mathrm{rand}}$ is a baseline correction from a random baseline(^5^55We use the trivial agent implementation described in Section [4.1](#S4.SS1) for baseline correction in our computation.).
Furthermore, if we have multiple reference sequences for a given task, we compute the $\text{VERTEX}_{\text{DTW}}$ score for every reference sequence and choose the maximum score, under the assumption that they describe different paths for solving the task.

## 4 Experiments

We perform a number of experiments fine-tuning agent models on the synthetic training data mixtures we discuss in Section [2](#S2) and assess the extent to which the agent model has self-improved over base agent model $\mathcal{M}$ with our evaluation metrics. Table [3](#S4.T3) displays the results of these experiments.

### 4.1 Baseline Agent Performance

As baselines, we evaluate our base agent model $\mathcal{M}$ as well as implement a trivial agent that always outputs stop [N/A]. A number of tasks in WebArena can be solved by this trivially implementable agent or a weak model that always refuses to continue and exits immediately, therefore, our trivial agent baseline helps discriminate which tasks being completed successfully should contribute to an agent being meaningfully capable when computing the capability score.

### 4.2 Self-Improvement Fine-Tuned Agent Performance

We fine-tune our base agent model $\mathcal{M}$ on the 3 synthetic dataset mixtures previously discussed: 1) $\mathcal{D}_{A}=\mathcal{D}_{\textsc{In-Domain}}$ 2) $\mathcal{D}_{B}=\mathcal{D}_{\textsc{In-Domain}}\cup\mathcal{D}_{\textsc{Out-of-Domain}}$ and 3) $\mathcal{D}_{C}=\mathcal{D}_{\textsc{Out-of-Domain}}$ with a straightforward auto-regressive loss using QLoRA (Dettmers et al., ; Hu et al., ):

$$ $L_{\text{FT}}(\theta)=-\mathbb{E}_{[(\mathbf{i},o_{t},a_{t-1}),a_{t}]\sim\mathcal{D}}\left[\log P_{\theta}(a_{t}\mid(\mathbf{i},o_{t},a_{t-1}))\right]$ $$

to produce $\mathcal{M}_{A}$, $\mathcal{M}_{B}$, and $\mathcal{M}_{C}$. We perform a 90/10% train-validation split of our datasets and train with an early stopping patience of 5 epochs, using a batch size of 16 examples and a learning rate of 1e-5. Further details about training configuration and hyperparameters can be found in Appendix [A](#A1).

### 4.3 Iterative Self-Improvement Fine-Tuned Agent Performance

We also experiment with iterative self-improvement (Chen et al., ) to assess whether further improvement can be gained from a subsequent round of our self-improvement procedure. We perform this experiment on Mixture A. It is conceivable that after fine-tuning on $\mathcal{D}_{A}^{1}$, filtering from a set of trajectories with higher performance might yield a stronger set of plausible trajectories (^6^66To maximize data for iterative self-improvement, during filtering, we also fallback to checking the base model trajectory for a task if the self-improved model’s trajectory for a task is filtered out.) to produce $\mathcal{D}_{A}^{2}$. Mixtures B and C are less likely to demonstrate improvement over a subsequent round since the fine-tuned models are not specifically trained to generate better synthetic out-of-domain examples.

**Table 3: Evaluation metrics on WebArena for baseline agents and self-improved agent models.**
| Agent Model | Functional Correctness | Capability | $\textbf{VERTEX}_{\textbf{DTW}}$ |
| --- | --- | --- | --- |
| Baseline Agents |  |  |  |
| Trivial Agent | 4.68 | 0.00 | - |
| Base Agent Model ($\mathcal{M}$) | 7.14 | 15.44 | 0.35 |
| Self-Improved Agents |  |  |  |
| Agent Model Fine-Tuned on Mixture A ($\mathcal{M}_{A}$) | 8.87 | 19.12 | 0.38 |
| Agent Model Fine-Tuned on Mixture B ($\mathcal{M}_{B}$) | 9.36 | 19.12 | 0.35 |
| Agent Model Fine-Tuned on Mixture C ($\mathcal{M}_{C}$) | 6.16 | 16.91 | 0.28 |
| Iterative Self-Improved Agents |  |  |  |
| Agent Model 2x Fine-Tuned on<br>Mixture A ($\mathcal{M}_{A}^{2}$) | 8.37 | 16.91 | 0.37 |

## 5 Discussion

We summarize key results from our experiments as well as discuss insights towards the efficacy of our self-improvement procedures for complex, multi-step tasks like web agent tasks.

#### Can models self-improve at web agent tasks?

We find fine-tuning on both Mixtures A and B improve overall benchmark performance with the best performing mixture, Mixture B, completing 18 more tasks correctly, a 31% relative improvement ($7.14\rightarrow 9.36$). Training on all Mixtures A, B, and C demonstrate self-improvement on at least one metric, with $\mathcal{M}_{C}$ showing a gain on capability score.

#### Do self-improved agents acquire new capabilities?

We find agent models can acquire new capabilities through self-improvement, however, they also may lose the ability to perform some capabilities. In net, all of our self-improved agents acquire more capabilities than they lose. We find fine-tuning on both Mixtures A and B improve the capability score equally and lead to the largest net acquisition of capabilities demonstrating 5 more capabilities than the base agent model, a 24% relative improvement ($15.44\rightarrow 19.12$). We find all agent models demonstrate at least one new capability that no other agent model demonstrates, for example, only $\mathcal{M}_{C}$ successfully completes the “Fork {{repo}}” capability on the GitLab website. Interestingly, we find that the majority of capabilities acquired by $\mathcal{M}_{A}$ and $\mathcal{M}_{C}$ are mutually exclusive, suggesting in-domain synthetic examples and out-of-domain synthetic examples improve acquisition of different capabilities. We list all capabilities $\mathcal{M}_{A}$, $\mathcal{M}_{B}$, and $\mathcal{M}_{C}$ acquire and lose compared to $\mathcal{M}$ in Appendix [C](#A3).

#### Are self-improved agents more robust?

For $\mathcal{M}_{B}$, we find a larger improvement in functional correctness (31%) than in capability score (24%), which supports that the agent model is improving at more consistently succeeding at tasks belonging to the same capability, an indicator of one type of robustness. $\mathcal{M}_{C}$ is less robust by the same measure. Moreover, the capability analysis in Appendix [C](#A3) also shows both $\mathcal{M}_{A}$ and $\mathcal{M}_{B}$ after self-improvement still demonstrate the majority of capabilities demonstrated by the base agent model $\mathcal{M}$, whereas $\mathcal{M}_{C}$ only demonstrates a minority. This would indicate $\mathcal{M}_{A}$ and $\mathcal{M}_{B}$ more reliably maintain the capabilities of the base agent model after self-improvement, a measure of robustness that would be useful in deployed settings where users of agent models may desire stability in performance.

#### Is there an effect on the quality of generated trajectories?

Fine-tuning on Mixtures A and B show no degradation in the quality of generated trajectories and show small improvement towards the reference on $\text{VERTEX}_{\text{DTW}}$. Fine-tuning on Mixture C degrades the the quality of generated trajectories from the reference. Training on the out-of-domain synthetic examples allows $\mathcal{M}_{C}$ to demonstrate some unique capabilities no other agent model demonstrates, however, inspecting trajectories from $\mathcal{M}_{C}$, we find this comes with trade-offs. For example, compared with $\mathcal{M}_{A}$, we find $\mathcal{M}_{C}$ produces longer trajectories (~1.6x) and produces more invalid actions (~3.9x). In comparison with $\mathcal{M}$, $\mathcal{M}_{A}$ and $\mathcal{M}_{B}$ do not greatly increase trajectory length (~1.1x and ~1.3x) or the rate of invalid actions (~1x and ~1.3x), further explaining the quality difference $\text{VERTEX}_{\text{DTW}}$ highlights. Due to lack of human reference, the reliability of this evaluation is limited which we discuss in Section [7](#S7). In Appendix [D](#A4), we compute variants of $\text{VERTEX}_{\text{DTW}}$, weighting by capability and filtering out trivial tasks. We find these variants make little difference in the relative ranking of agent models.

#### Can models iteratively self-improve at web agent tasks?

Our results are consistent with prior works such as Chen et al. () and Feng et al. () and we find diminishing returns to successive rounds of self-improvement and training on synthetic data. While the agent model after a second round of self-improvement outperforms the base agent model, it does not perform any better than agent models with a single round of self-improvement. We analyze the set of plausible trajectories in the second round in Appendix [E](#A5) and find that while more synthetic training examples can be collected, they are of lower quality and contain a higher proportion of failed trajectories.

## 6 Related Work

#### Self-Improvement

A number of techniques have been proposed for self-improving LLMs (Huang et al., ; Weng et al., ; Madaan et al., , inter alia). Some self-improvement techniques (Han et al., ; Gulcehre et al., ; Singh et al., ; Chen et al., ; Yuan et al., ) involve self-distillation (Zhang et al., ), a special form of knowledge distillation (Hinton et al., ) where the teacher and student are the same model. A growing trend of works (Wang et al., ; Gunasekar et al., ) similarly prompt LLMs to generate synthetic fine-tuning data.

#### LLM Agents

A number of prompting techniques proposed (Kojima et al., ; Wei et al., ; Yao et al., ; Shinn et al., ) can improve an LLM agent’s performance, however, these techniques are orthogonal to self-improvement fine-tuning. Chen et al. () introduces a technique for supervised fine-tuning of LLM agents. Sodhi et al. () and Lai et al. () introduce handcrafted subprompts or supervised techniques that improve performance on WebArena.

#### Self-Improving Agents

Bousmalis et al. () demonstrates self-improving embodied agents for complex robotics tasks. Aksitov et al. () introduces a method for self-improving agents on a simpler multi-step question answering task. Concurrently, Song et al. () proposes a similar procedure of filtering trajectories and fine-tuning, but primarily focuses on supervised filtering, does not explore generating novel tasks and synthetic data, and evaluates on less realistic and complex benchmarks. Pan et al. () explores using vision models for critique to improve on WebArena.

## 7 Limitations and Broader Impacts

While we find self-improvement fine-tuning techniques can improve performance by reinforcing correct actions and decisions of an underlying model, these techniques can also further reinforce incorrect actions and biases of the underlying model. Some human or supervised filtering may mitigate this drawback, however, in this paper we focus our investigation on the efficacy and quality of unsupervised self-improvement as producing datasets for such complex tasks is difficult and expensive. Our analysis of capabilities is limited by our method to group tasks by the intent template used and cosine similarity. It is possible other strategies may produce more optimal groups to measure capabilities. Our $\text{VERTEX}_{\text{DTW}}$ score utilizes a stronger model’s generations (GPT-4) as a reference, however, human references would significantly improve the reliability of this evaluation. While WebArena spans many different types of realistic tasks and websites (shopping, online forums, maps, etc.), a future direction for this work might involve evaluation on larger, and more diverse benchmark.

## 8 Conclusion

In this work, we explore whether large language models can self-improve beyond their base performance at complex, long-horizon web agent tasks. We conclude self-improvement can increase the performance and robustness of agent models and allow agent models to acquire new capabilities. We also find it is possible for self-improvement to yield these benefits with minimal degradation to the quality of trajectories. The self-improvement procedures we propose are a promising step towards boosting the performance of LLMs in complex, multi-step agent environments such as web environments, without relying on supervised training data. We release our code, evaluation metrics with references, synthetic datasets, and model trajectories.

## Acknowledgements

This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids (FFG- 899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo), Software Competence Center Hagenberg GmbH, Borealis AG, TÜV Austria, Frauscher Sensonic, TRUMPF, the NVIDIA Corporation and ExtensityAI.

The authors gratefully acknowledge the HPC RIVR consortium (www.hpc-rivr.si) and EuroHPC JU (eurohpc-ju.europa.eu) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (www.izum.si).

The authors gratefully acknowledge OpenAI for providing research credits and computing resources for this research.

## Appendix

## Appendix A Training and Inference Details

**Table 4: Hyperparameters selected for fine-tuning experiments.**
| Hyperparameter | Value |
| --- | --- |
| Model | Qwen/Qwen1.5-72B-Chat |
| Hardware | 2x NVIDIA RTX A6000 |
| Distributed Protocol | PyTorch FSDP |
| Data Type | torch.bfloat16 |
| Quantization | 4-bit (nf4), double quantized |
| LoRA | all-linear, r=8<br>lora_alpha=8<br>lora_dropout=0.0 |
| Optimizer | adamw_torch |
| Learning Rate | 1e-5 |
| Weight Decay | 0.01 |
| Learning Rate Scheduler | linear |
| Warmup Steps | 0 |
| Batch Size | 16 |
| Train-Validation Split | 90/10% |
| Early Stopping Threshold | 0.0 |
| Early Stopping Patience | 5 epochs |

**Table 5: Parameters used during inference, we follow the default parameters for inferencing set by the WebArena benchmark [Zhou et al., ].**
| Inference Parameter | Value |
| --- | --- |
| Model | Qwen/Qwen1.5-72B-Chat |
| Hardware | 4x NVIDIA RTX A6000 |
| Data Type | torch.bfloat16 |
| Quantization | 4-bit (nf4), double quantized |
| Prompt Template | p_cot_id_actree_2s |
| Temperature | 1.0 |
| Top-P | 0.9 |
| Max New Tokens | 384 |

## Appendix B Sample of Generated Out-of-Domain Synthetic Training Example

**Table 6: A selected full sample of a generated novel, out-of-domain synthetic training example in $\mathcal{D}_{\textsc{Out-of-Domain}}$.**
| Objective | Subscribe to the premium plan for Grammarly to unlock advanced writing features. |
| --- | --- |
| URL | https://www.grammarly.com/plans |
| Plan | 1. Click on the “Go Premium” button located at the top right corner of the page.<br>2. Scroll down to review the available plans (if needed).<br>3. Click on the “Choose Plan” button for the preferred premium plan.<br>4. Provide payment information, such as credit card details, in the respective input fields.<br>5. Enter your billing information, including name, address, and zip code (if required).<br>6. Type your email address again to confirm it.<br>7. Create a password for your Grammarly account and type it into the password field.<br>8. Check the box to agree to the terms and conditions.<br>9. Click on the “Start Your Free Trial” (or similar) button if you qualify for a trial.<br>10. Review the confirmation message and wait for the subscription activation.<br>11. Stop when the subscription is successfully activated and the premium features are unlocked. |
| Selected Step in Plan | 3 |
| Previous Action | click [1098] where [1098] is Generic 'Premium' |
| Next Action | Let’s think step-by-step. To proceed with subscribing to the premium plan, the user must now choose their plan. They have already clicked on the 'Premium' plan option. The next logical action is to click on the 'Choose Plan' button to finalize the selection and proceed to the payment page. In summary, the next action I will perform is ```click [1105]``` to select the premium plan and move to the next step in the subscription process. |
| Observation | [1] RootWebArea 'Grammarly plans - Premium, Business & Student Writing Tools' focused: True<br>[125] Generic 'Grammarly plans - Premium, Business & Student Writing Tools'<br>\tabto0.5cm[127] link 'Privacy Policy'<br>\tabto0.5cm[130] link 'Terms of Use'<br>\tabto0.5cm[133] link 'Cookies'<br>\tabto0.5cm[136] link 'TrustArc Privacy Notice'<br>\tabto0.5cm[587] link 'Language'<br>\tabto0.5cm[1054] link 'United States English'<br>\tabto0.5cm[592] link 'Logout'<br>\tabto0.5cm[595] link 'Invite'<br>\tabto0.5cm[598] link 'Get the Extension'<br>\tabto0.5cm[128] link 'Login'<br>\tabto0.5cm[601] link 'Sign Up'<br>\tabto0.5cm[580] Generic 'Plans'<br>\tabto1cm[583] div 'Free'<br>\tabto1.5cm[586] link 'Get Started'<br>\tabto1cm[590] Generic 'Premium'<br>\tabto1.5cm[1098] Generic 'Premium' haspopup: submenu expanded: False<br>\tabto1.5cm                 [640] link 'View Features'<br>\tabto1.5cm                 [643] StaticText 'Start Now!'<br>\tabto1.5cm                 [1105] button 'Choose Plan'<br>\tabto1cm[1089] Generic 'Business'<br>\tabto1.5cm[649] link 'View Pricing'<br>\tabto1.5cm[652] StaticText 'Get Quote'<br>\tabto1cm[1092] Generic 'Student'<br>\tabto1.5cm[646] link 'View Pricing'<br>\tabto1.5cm[649] StaticText 'Start Now!'<br>\tabto0.5cm[566] Generic 'Write with confidence, wherever you work'<br>\tabto1cm[570] StaticText 'Powerful writing tools for work, school, and everything in between.'<br>\tabto1cm[574] button 'Learn More &amp; Try it Free'<br>\tabto0.5cm[1052] table '<br>\tabto1cm[569] row '<br>\tabto1.5cm[574] rowheader 'Free'<br>\tabto1.5cm[575] gridcell '<br>\tabto1cm[568] row '<br>\tabto1.5cm[573] rowheader 'Premium'<br>\tabto1.5cm[574] gridcell '<br>\tabto1cm[567] row '<br>\tabto1.5cm[572] rowheader 'Business'<br>\tabto1.5cm[573] gridcell '<br>\tabto1cm[566] row '<br>\tabto1.5cm[571] rowheader 'Student'<br>\tabto1.5cm[572] gridcell ' |

## Appendix C Capability Analysis

**Table 7: Capabilities acquired and lost compared to the base agent model $\mathcal{M}$, along with the net change in the total number of capabilities demonstrated, for each self-improved fine-tuned agent model.**
| Agent Model | Net Change | Capabilities Acquired | Capabilities Lost |
| --- | --- | --- | --- |
| $\mathcal{M}_{A}$ | +5 |  |  |
| $\mathcal{M}_{B}$ | +5 |  |  |
| Continued on next page$\ldots$ |  |  |  |
| $\mathcal{M}_{C}$ | +2 |  |  |

## Appendix D Full VERTEX DTW fragments VERTEX DTW \text{VERTEX}_{\text{DTW}} Score Results

**Table 8: Variants of the $\text{VERTEX}_{\text{DTW}}$ score metric: 1) computed over all trajectories 2) weighting the trajectories by capability 3) weighting the trajectories by capability and filtering out trajectories for trivial tasks.**
| Agent Model | $\textbf{VERTEX}_{\textbf{DTW}}$ | $\textbf{VERTEX}_{\textbf{DTW-bycap}}$ | $\textbf{VERTEX}_{\textbf{DTW-notrivial}}$ |
| --- | --- | --- | --- |
| Baseline Agents |  |  |  |
| Base Agent Model ($\mathcal{M}$) | 0.35 | 0.40 | 0.38 |
| Self-Improved Agents |  |  |  |
| Agent Model Fine-Tuned on Mixture A ($\mathcal{M}_{A}$) | 0.38 | 0.42 | 0.42 |
| Agent Model Fine-Tuned on Mixture B ($\mathcal{M}_{B}$) | 0.35 | 0.40 | 0.40 |
| Agent Model Fine-Tuned on Mixture C ($\mathcal{M}_{C}$) | 0.28 | 0.33 | 0.34 |
| Iterative Self-Improved Agents |  |  |  |
| Agent Model 2x Fine-Tuned on<br>Mixture A ($\mathcal{M}_{A}^{2}$) | 0.37 | 0.41 | 0.43 |

## Appendix E Iterative Self-Improvement Plausible Trajectories

**Table 9: Metrics on the proportion of trajectories that successfully completed the task in the set of plausible trajectories kept in $\mathcal{P}$ after filtering out low-quality trajectories for each iterative round of self-improvement. On the second round of self-improvement, we keep 131 plausible trajectories making our potential synthetic training dataset larger, however, the accuracy and P/R/F1 metrics indicate it would be a lower quality dataset to fine-tune on.**
| Set of Trajectories | # | Accuracy | F1 | Precision | Recall |
| --- | --- | --- | --- | --- | --- |
| All Trajectories | 812 | 0.071 | 0.133 | 0.071 | 1.000 |
| Plausible Trajectories $\mathcal{P}^{1}$ | 58 | 0.919 | 0.431 | 0.431 | 0.431 |
| Plausible Trajectories $\mathcal{P}^{2}$ | 131 | 0.825 | 0.317 | 0.252 | 0.429 |

## Appendix F Capabilities in WebArena

In this appendix, we list the grouping of tasks into “capabilities” we find in WebArena using the automated method we describe in Section [3.2](#S3.SS2). These tasks are grouped by the intent template used by WebArena to create the task as well as cosine similarity to group paraphrases detected by a sentence similarity model. We do not perform manual modifications to the groups and instead solely rely on automated techniques. We acknowledge grouping of natural language task objectives into capability areas is subjective and discuss this a limitation in Section [7](#S7):

| Capability #1:<br>•<br>What are the top-{{n}} best-selling product in {{year}}<br>Capability #2:<br>•<br>Tell me the the number of reviews that our store received by far that mention term “{{term}}”<br>Capability #3:<br>•<br>What brands appear most frequently among the top search terms?<br>•<br>List the top {{n}} search terms in my store<br>Capability #4:<br>•<br>Telll me the grand total of invoice {{id}}.<br>Capability #5:<br>•<br>Presents the monthly count of successful orders {{period}} in MM:COUNT format<br>Capability #6:<br>•<br>What’s the total number of items sold in the most recent {{k}} orders?<br>Capability #7:<br>•<br>Show all customers<br>Capability #8:<br>•<br>Give me the {{Attribute}} of the products that have {{N}} units left<br>Capability #9:<br>•<br>Get the total payment amount of the last {{N}} {{status}} orders<br>Capability #10:<br>•<br>Find the customer name and email with phone number {{PhoneNum}}<br>Capability #11:<br>•<br>Tell me the {{attribute}} of the customer who has the most cancellations in the history<br>•<br>Which customer has completed the {{quantifier}} number of orders in the entire history?<br>•<br>Show me the {{information}} of the customer who is the most unhappy with {{product}}<br>Capability #12:<br>•<br>How many reviews our shop received {{time}}?<br>•<br>What is the total count of {{status}} reviews amongst all the reviews?<br>Capability #13:<br>•<br>Preview the {{name}} theme for my shop<br>Capability #14:<br>•<br>Mark all {{brand}} shirts on sale | Capability #15:<br>•<br>Disable {{product}} from the site, they are facing some quality issues.<br>Capability #16:<br>•<br>{{action}} the price of {{config}} by {{amount}}<br>•<br>{{action}} the price of this product by {{amount}}<br>Capability #17:<br>•<br>Update the description of {{product}} to highlight the real user positive reviews by quoting the comments<br>Capability #18:<br>•<br>Cancel order {{id}}<br>Capability #19:<br>•<br>Change the page title of “{{old-heading}}” page on my site to “{{heading}}”.<br>Capability #20:<br>•<br>Notify {{name}} in their most recent pending order with message “{{message}}”<br>Capability #21:<br>•<br>Update order #{{order}} with the {{service}} tracking number {{tracking}}<br>Capability #22:<br>•<br>Make all {{product}} as out of stock<br>Capability #23:<br>•<br>Modify the address of order #{{order_id}} to {{address}}<br>Capability #24:<br>•<br>Add new {{option}} {{value}} to {{base_setting}} of {{product}}<br>Capability #25:<br>•<br>Lookup orders that are {{status}}<br>•<br>Get the {{attribute}} of the {{status}} order<br>Capability #26:<br>•<br>Add a simple product named {{product}} with {{stock}} in stock, available in size {{size}} and color {{color}}, priced at ${{price}}<br>Capability #27:<br>•<br>Draft a new marketing price rule for {{topic}} that offers {{rule}} for all customers<br>Capability #28:<br>•<br>Today is 3/15/2023, generate a {{report}} {{time_span}}<br>•<br>Create a {{type}} report from {{start_date}} to {{end_date}} | Capability #29:<br>•<br>We’ve received {{quantity}}, update the inventory.<br>Capability #30:<br>•<br>Approve the positive reviews to display in our store.<br>Capability #31:<br>•<br>Delete all {{review_type}}<br>Capability #32:<br>•<br>Tell me the full address of all {{airport_type}} that are within a driving distance of {{radius}} to {{start}}<br>Capability #33:<br>•<br>What is the {{information}} of {{location}}<br>•<br>I will arrive {{place}} soon. Provide the name of a {{target1}} in the vicinity, if available. Then, tell me the {{information}} to {{target2}} from the hotel.<br>Capability #34:<br>•<br>What is the zip code of {{place}}?<br>Capability #35:<br>•<br>Given the following locations, {{place_list}}, what would be the optimal route to travel through them all in order to minimize total travel time? Please note the journey begins at the first place listed.<br>Capability #36:<br>•<br>Which US states border {{state}}?<br>Capability #37:<br>•<br>Where is the nearest {{places}} to {{start}}, and what is the walking distance to it?<br>•<br>Find the walkway to the closest {{store}} from {{location}}.<br>•<br>How long does it take to walk from {{start}} to {{end}}?<br>•<br>Tell me the closest {{place1}}(s) to {{place2}}<br>Capability #38:<br>•<br>From my stay at {{hotel}}, what’s the estimated driving time to reach {{place}}?<br>•<br>What is the minimum travel time by car from {{location1}} to {{location2}}?<br>•<br>What is the duration required to first walk from {{place_A}} to {{place_B}}, and then drive to {{place_C}}?<br>•<br>Show me the walking distance from nearby hotels to {{location}} that take at most {{n}} minutes?<br>•<br>What is the estimated driving time between {{city1}} and {{city2}}? |
| --- | --- | --- |
| Capability #39:<br>•<br>From my stay at {{hotel}}, what’s the estimated driving time to reach {{place}}?<br>•<br>What is the estimated driving time between {{city1}} and {{city2}}?<br>•<br>I am at CMU Pittsburgh, how long it takes to drive to the nearest {{location}}<br>•<br>Check if the {{place}} in pittsburgh can be reached in one hour by car from {{location}}<br>Capability #40:<br>•<br>Find the {{space}} around {{location}}<br>•<br>Find the walkway to the closest {{store}} from {{location}}.<br>•<br>Tell me the closest {{place1}}(s) to {{place2}}<br>•<br>Where is the nearest {{location}} from {{location2}} {{condition}}<br>Capability #41:<br>•<br>What is the {{information}} of {{location}}<br>•<br>Tell me the coordinates of {{location}} in DD format<br>Capability #42:<br>•<br>How much time does it take from Pittsburgh to Philadelphia by car?<br>Capability #43:<br>•<br>Show the route from SCS CMU in Pittsburgh to the location where the Declaration of Independence and Constitution were signed<br>Capability #44:<br>•<br>Pull up the description page of {{location}} on Map<br>•<br>What is the {{information}} of {{location}}<br>Capability #45:<br>•<br>I am arriving at Carnegie Mellon University. Find the nearby US Citizenship and Immigration Services and the walking distance to the nearest Social Security Administration from US Citizenship and Immigration Services<br>Capability #46:<br>•<br>I am arriving at Pittsburgh Airport. Show me the name of a Hyatt hotel if there is any nearby. Tell me the names of supermarkets that are within 15mins driving from the hotel<br>Capability #47:<br>•<br>Measure distance between {{location/address_1}} and {{location/address_2}} by walking<br>•<br>Get directions from {{location/address_1}} to {{location/address_2}} using {{transportation}} options.<br>Capability #48:<br>•<br>List out reviewers, if exist, who mention about {{description}}<br>Capability #49:<br>•<br>Today is 6/12/2023. Tell me how many fulfilled orders I have {{period}}, and the total amount of money I spent. | Capability #50:<br>•<br>Tell me the status of my latest order and when will it arrive<br>Capability #51:<br>•<br>What is the date when I made my first purchase on this site?<br>Capability #52:<br>•<br>I have jaw bruxism problem, show me something that could alleviate the problem.<br>Capability #53:<br>•<br>What is the price range for products from {{brand}}?<br>•<br>What is the price range of {{product}} in the One Stop Market?<br>Capability #54:<br>•<br>How much I spent on {{category}} shopping during {{time}}<br>Capability #55:<br>•<br>What is the {{option}} configuration of the {{product}} I bought {{time}}<br>•<br>I previously ordered some {{product}} {{time}} and later cancelled. Can you reorder it for me?<br>Capability #56:<br>•<br>I have a lot of Nintendo Switch game cards now, help me find the best storage option to fit all {{num}} cards<br>Capability #57:<br>•<br>What are the main criticisms of this product? Please extract the relevant sentences.<br>Capability #58:<br>•<br>What do customers say about {{product_type}} from {{manufature}}<br>Capability #59:<br>•<br>Buy the best rating product from “{{category}}” category with at least 5 reviews and the product is least expensive<br>•<br>I am doing a market survey for one stop market, show me the most expensive product from {{product_category}} category<br>•<br>Buy the highest rated product from the {{product_category}} category within a budget {{dollar_value}}.<br>Capability #60:<br>•<br>Search for “{{keyword}}”<br>Capability #61:<br>•<br>List the full product names of slide slippers from Nike and tell me the price range of the available products<br>Capability #62:<br>•<br>Look up the most recent models of XBox controllers released between 2020-2021?<br>Capability #63:<br>•<br>Show the least expensive {{product}} with a minimum storage capacity of {{min_storage}}.<br>Capability #64:<br>•<br>Show the most recent {{status}} order<br>•<br>Get the order number of my most recent {{status}} order | Capability #65:<br>•<br>Which number to call for the customer service?<br>Capability #66:<br>•<br>How much refund I should expect from my order canlled in {{time}}? I only kept the AC-DC Adapter and the shop told me that I cannot get the shipping fee back<br>Capability #67:<br>•<br>Show me the “{{product}}” listings by {{sorting_order}}.<br>Capability #68:<br>•<br>How much did I spend on shopping at One Stop Market {{time}}? They gave me a 20% discount on the total amount for orders exceeding $200 in cash<br>Capability #69:<br>•<br>Tell me when I last ordered my {{description}}?<br>Capability #70:<br>•<br>List products from {{product_category}} category by {{order}} price<br>•<br>Show me products under ${{price}} in “{{product_category}}” category<br>Capability #71:<br>•<br>Show me the {{info}} for order number {{order_number}}.<br>Capability #72:<br>•<br>find discounted items.<br>Capability #73:<br>•<br>Summarize customer reviews for {{product}}.<br>Capability #74:<br>•<br>List the customer names who thinks EYZUTAK phone cases are of good looking<br>•<br>Who gave {{stars}} for phone cases from EYZUTAK<br>Capability #75:<br>•<br>What is the rating of {{product}}<br>Capability #76:<br>•<br>Add the product with the lowest per unit price from my open tabs to the shopping cart<br>Capability #77:<br>•<br>Add {{product}} to my wish list<br>•<br>Add this product to my wishlist<br>•<br>Add a {{product}} to my wish list.<br>Capability #78:<br>•<br>Subscribe to the newsletter of OneStopMarket<br>Capability #79:<br>•<br>I recently moved, my address is {{address}}, update my information on OneStopShopping accordingly<br>•<br>Change the delivery address for my most recent order to {{address}}.<br>Capability #80:<br>•<br>Rate my recent purchase of {{product}} with {{num_star}} stars, using my nickname {{nickname}}? |
| Capability #81:<br>•<br>Fill the “contact us” form in the site for a refund on the {{product}} I bought, stating that it broke after just three days of use. Also, ensure to include the order number #{{order_id}} and the product SKU. Don’t submit yet, I will check.<br>•<br>Draft a refund message via their “contact us” form for the {{product}} I bought {{time}}. It broke after three days of use. The shop requires the order id, the reason and the amount to refund in the message. Don’t submit yet<br>Capability #82:<br>•<br>Draft an email to the shop owner via their contact us function for a coupon as {{reason}}<br>Capability #83:<br>•<br>Tell me the count of comments that have received more downvotes than upvotes for the user who made the latest post on the {{forum}} forum.<br>Capability #84:<br>•<br>Among the top {{number}} post in “{{subreddit}}” forum, {{description}}<br>Capability #85:<br>•<br>Change my reddit bio to “{{content}}”<br>Capability #86:<br>•<br>Reply to {{position_description}} with my comment “{{content_description}}”<br>Capability #87:<br>•<br>Create a new forum named {{name}}, with a description of {{description}}, and include {{sidebar_list}} in the sidebar?<br>Capability #88:<br>•<br>Open the thread of a trending post on the forum “{{subreddit}}” and subscribe.<br>•<br>Upvote the newest post in {{subreddit}} subreddit<br>Capability #89:<br>•<br>Create a discussion post about “{{topic}}” in a relevant subreddit and ask users for their opinions with the simple prompt, “your opinion”<br>•<br>Find a subreddit focused on topics related to {{topic}}, and post my question, “{{question}}” there<br>•<br>Post my question, “{{question}} in a subreddit where I’m likely to get an answer<br>Capability #90:<br>•<br>Post a review of my recent reading “{{book}}” in the r/books with my comment “{{content}}”.<br>Capability #91:<br>•<br>Re-post the image of {{content}} in this page to {{subreddit}} subreddit and note “from /f/pics”<br>Capability #92: |  |  |

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - What is the definition of plausible trajectories and high-quality?
- What are the possible oblations against the proposed type of data?
soundness: 2
strengths: - well written.
summary: The papers is about having an agent for environments with lack of training data like web browsing. In that way, LLM can be a good candidate. In this work, they introduced a new technique in which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. To better understand the effect of the self-improvement method, they introduced two auxiliary metrics: 1) a measure to analyze capabilities acquired and lost by the agent and 2) an extension of the VERTEX score to measure the quality of variable-length agent trajectories.
weaknesses: - The innovation of the work is very limited! 
- Lack of enough experiments! The experiments are not comprehensive! It should consider a good analysis across different datasets and careful reasoning of what is going there! For example, having complete oblations studies versus other possibilities.
- Lack of comparison with baselines! There are a lot of works of self-improvement and self-correctness! There should be considered as baselines. 
- This work needs a proper and further study and analysis! This is an incomplete work to be published!

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: While the study is promising, its impact would be further enhanced by addressing the concerns regarding the selection of certain variables used in the experiments and evals and model diversity. 

### Suggestions for Improvement:
- **Expand Hyperparameter Exploration:** Providing a sensitivity analysis or a justification for the selection of hyperparameters used in generating synthetic data would enhance the transparency and reproducibility of the results.
- **Broader Agent Evaluation:** Including diverse model architectures in the experiments could help validate the proposed self-improvement techniques across a range of LLMs, potentially increasing the impact and applicability of the findings.

I would be happy to increase the score post author responses.
soundness: 3
strengths: ### strenghts:
1. **Capabilities through Self-Improvement:** The paper demos how LLMs can extend their capabilities through self-improvement techniques, particularly in the context of complex, long-horizon web agent tasks. This ability to acquire new capabilities while largely retaining existing ones is notable.
2. **Appropriate Eval Metrics:** The introduction of novel metrics, and scores to evaluate the quality of trajectories, adds depth to the evaluation process. These metrics provide a nuanced view of the models' performance, moving beyond simple task completion metrics and enabling a more detailed assessment of the agent's robustness and capabilities. the additions seem appropriate  enough to capture both success, relevance, and efficiency of trajectoriesl
3. **Synthetic Data Utilization:** The use of both in-domain and out-of-domain synthetic data for fine-tuning is commendable. This approach not only addresses the challenge of data scarcity but also demonstrates a systematic mechanism for enhancing agent model generalization.
summary: solving complex, multi-step, long-horizon tasks using prompting or fine-tuning poses various challenges. The paper presents a sound approach to enhance the capabilities of LLMs through self-improvement on web automation task. They propose evaluation techniques and effective use of in-distribution and out-of-distribution synthetic data, showcasing some performance gains.
weaknesses: Good to address:
1. **Hyperparameter Selection:** The paper lacks a clear justification for the choice of hyperparameters in the synthetic data generation process and generating new objectives. e.g. using 4 or 2 few-shot samples, temperature value, and perhaps just one sentence on why 0.7 cosine similarity used is better. Any missing details may lead readers to question the replicability and robustness of the results. A more thorough analysis or rationale for these choices would strengthen the paper.
2. **Limited Model / Results:** The study's reliance on a single or fewer models limits its generalizability. Expanding the experiments to include a subset of models on appropriate agent frameworks that came out recently, such as smaller or different architectures, could provide insights into how these techniques scale or vary across different model sizes and types. I am thinking aloud if the results section can be tightened and elaborated more. But I will wait to see if my other reviewers has some feedback or ideas on this front.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: How many examples are in Mixture C? How does the result scale with the number of examples used in Mixture C? Does it improve as more examples are provided?
soundness: 1
strengths: - The paper shows that you can use in-domain trajectories as few-shot examples to generate novel out-of-domain trajectories, which will be useful in producing synthetic data for finetuning agents.
- The paper proposes a new metric $\text{VERTEX}_{\text{DTW}}$ which measures the similarity of an agent’s trajectory w.r.t. a reference (in this paper, the reference is a gpt-4 trajectory). This can be useful for providing a soft metric that has less false negatives than the WebArena functional correctness evaluation.
summary: This paper proposes a method of improving LLMs for web agent tasks. The method involves prompting an existing LLM as an agent to tackle tasks in WebArena. The trajectories are then filtered down to retain those that are likely to be successful. The model is then finetuned (with QLoRA) on these trajectories. In the paper, alternative trajectory generation methods are also explored, such as using the successful WebArena trajectories to generate out-of-domain trajectories on other websites. In addition, the authors propose a new metric for measuring the similarity of an agent trajectory, which has benefits over using the functional correctness evaluation in WebArena.
weaknesses: My primary concern with this paper is that its evaluation setup is not fair. The models are essentially *training on the test set*, and doing a pass@2 on the evaluation set (these results align with the ablation experiments in the paper, where training on Mixture C actually makes the agent worse). Consider the filtered subset of in-domain examples (mixture A). It has a very high accuracy rate (0.919), which is expected as tasks that errored out or the model failed to solve are filtered out. In the WebArena benchmark, models often only stop when they are successful, so the false negative rate is low. If you view mixture A as groundtruth samples, finetuning on this subset is essentially allowing the model to memorize the examples that have passed, while ignoring those that it failed on the first attempt. When you test such a model, this can be viewed as essentially taking a second sample for the failed tasks. This is not a fair comparison to baseline models: a fairer comparison would be the same baseline model but an accuracy score of its pass@2 (i.e., take the best run out of two samples using the same filtering criteria). [1] actually does this with a prompted value function and gets a similar relative improvement (29%) compared to this work, and they do not do any training on the samples. If the authors can (1) compare their results against pass@2 of the agent (while resetting the WebArena environment after the first run, to avoid bias), and (2) perhaps run their model on a different benchmark, e.g., Mind2Web, VisualWebArena, WorkArena, or WebLINX, to show generalization, this would alleviate some of my concerns.

In addition, it would also be good to have error bars on the evaluation results as WebArena can be noisy, as the absolute difference between the baseline and the self-improved agent is low (7.14 to 9.36) and could be within the error range.

The proposed $\text{VERTEX}_{\text{DTW}}$ metric is evaluated against gpt-4 generated outputs, which possibly introduces model bias in the evaluation. WebArena has [human demonstrations](https://github.com/web-arena-x/webarena/blob/main/resources/README.md#12212023-human-trajectories), and it would be good to compare against these instead.


Overall I think the evaluation of the paper can be greatly improved (by measuring on more benchmarks, to avoid train-test contamination), as well as more ablation experiments. For these reasons I don’t think that the paper in its current shape is sufficiently complete.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: There are a few other points that I would appreciate if the authors could clarify.

**(1)** The authors claim that the procedure for filtering in-domain self-refinement data is entirely unsupervised, but then seem to contradict this point later on.

In Lines 135-136: “Similar to the self-improvement prior work we discuss earlier, the collection of this data is completely unsupervised and no ground-truth labels are utilized for filtering and selection.”

Then, in Lines 194-196: “Finally, we also filter out any trajectories where the WebArena environment encountered an error or the model failed to produce a valid, parsable generation”
    
If I understand the procedure correctly, the highlighted parts of the filtering process are not truly unsupervised — they leverage signals that are external to the model, e.g. feedback from the environment. If so, the first claim should be adjusted accordingly.
    
**(2)** On lines 290-292, the authors write: “We note, however, that a number of tasks in the WebArena benchmark are trivial tasks and can be solved by a trivial baseline agent or weak model that performs no actions and only immediately exits by always generating stop [N/A].” 

This seems like a major weakness of the benchmark. What proportion of the WebArena tasks have this degeneracy property? How many of the <10% of tasks solved by tested LLM agents are degenerate? From my understanding of the text, degenerate tasks **are** filtered from capability score computation, but **are not** filtered from/identified during evaluation. Is this correct?
soundness: 2
strengths: - The study of generalizable techniques for improving LLM agents is of interest to the community. While many previous works have studied the advantages and limitations of self-refinement for reasoning and simple decision-making tasks, there has been limited work on harder benchmarks with longer decision-making horizons, like Web Arena. These tasks represent the frontier of state-of-the-art LLM capabilities. As a result, developing more effective methods for self-refinement on such problems is especially important.
- The paper is polished, clear, and fairly well written. The authors expound on all experimental details.
- The default metric for LLM performance on WebArena is the average completion rate of environment tasks. The authors make an effort to address the lack of granularity in this metric by introducing two auxiliary metrics for assessing LLM agentic performance. These auxiliary metrics add some nuance to the analysis, though this could be strengthened by increasing the number of experiment replicates and/or increasing the inference-time sample size, as models are evaluated at high sampling temperature.
summary: The authors study whether self-refinement (i.e. finetuning on self-generated data) can improve the performance of large language models on web agent tasks via the WebArena benchmark. This benchmark tests agents on a set of web page decision-making tasks. In such setting, self-refinement data consists of trajectories of interleaved environment observations and LM agent actions. The authors test three different ways of generating  trajectories for self-refinement: (A) fully “in-domain,” using train set tasks from the Web Arena environment only; (B) on a “in-domain” and “out-of-domain”  mixture, using train set tasks from WebArena as well as synthetic tasks generated by the LM agent; (C) fully “out-of-domain,” using only synthetic tasks. All “in-domain” trajectories are filtered using self-critique, for LM agent generation errors, and environment errors. The authors finetune a single LLM on each of these three data mixtures and compare the resultant LLM agents’ performance on WebArena under three different metrics. They show that self-refining on data mixtures A and B result in some improvements to performance on 3/3 metrics, while finetuning on mixture C produces a slight drop in performance on 2/3 metrics. Finally, the authors show that running an additional round of self-refinement with the best performing data generation strategy (mixture A) causes a drop in performance on 3/3 metrics, compared to results after a single round.
weaknesses: While the questions posed by this paper are timely, the experimental results have limited scope and are much more mixed than the text would suggest. While the title reads “LLMs can self-improve at Web Agent tasks,” the results seem to suggest that the performance margins are extremely limited. 

- Throughout the paper, the authors cite a 31% improvement over the base LLM agent as evidence for the power of self-improvement. However, this 31% is a relative score, corresponding to a mere ~2.2%  improvement in raw score from a completion rate of 7.14/100 to 9.36/100. The statistical significance of this improvement is unclear for two reasons: (1) the scale of standard error over 0-1 task completion score is omitted from Table 3; (2) according to Table 5, the authors sample from models during inference with a high temperature (T = 1). Sampling rather than using greedy decoding introduces an additional source of statistical error to results, which is similarly not quantified.
- In the introduction, the authors write: “We analyze three synthetic training data mixtures and find all three mixtures improve performance” (Lines 101-102).  This claim is misleading, since as shown in Table 3, self-refining on the third data mixture (C, “out-of-domain” tasks only) produces decreases in performance on 2/3 of the studied metrics.
- The authors show that iterative self-refinement produces a drop on all 3/3 Web Arena LLM agent performance metrics, even after the careful filtering procedures (Table 3, last row). As a reader, this result powerfully contradicts the tone of the title.
- On lines 404-406, the authors write: “Interestingly, we find that the majority of capabilities acquired by M_A and M_C are mutually exclusive, suggesting in-domain synthetic examples and out-of-domain synthetic examples improve acquisition of different capabilities.” This point would be extremely interesting if strengthened, but does not seem to be rigorously supported by the results. Currently, it seems to be being made on the basis of a single random seed of self-refinement in each setting, i.e. using a single sampled set of high temperature synthetic task generation + agent trajectories + LLM fine-tuning. In my view, to confirm these results, evaluation should be conducted with at minimum ~3-5 seeds in each data mixture setting.
- Overall the experiments have limited scope. Evaluations are conducted with a single seed for each data generation variant and a single LLM.  

Though the results of this paper are mostly negative, I still believe they are of interest to the community. I would be willing to increase my score if (1) the authors can provide an estimate of statistical error for all reported scores; and (2) the tone of the paper is adjusted to reflect the strength and scope of the experimental results.

Meta-review: This paper performs self-improvement in LLM on the Web Arena tasks which is a challenging web navigation planning task for LLMs agents. Two main concerns were raised during the reviewing process. The first was the question of fairness in evaluation (by reviewer neMT) and the second was experimental results being weak or even misleading (reviewer 1yvG). 

The fairness concern is that the agents are being trained on the same prompt/task distribution as the ones on which they are being tested. Essentially, the agent generates a set of trajectories on a given distribution of which some of the trajectories are eliminated using a self-critique signal and this data is used for training (mixture a). There is also additionally an option to generate out-of-distribution data (mixture b) and mix it with the in-distribution filtered data (mixture c). If the self-critiquing works, then it is clear that we should expect gains when training on it and then evaluating a similar distribution. A much more powerful result would have been to show that the agent can solve novel tasks after being trained. Reviewer neMT suggested doing pass@2 evaluation with baseline agents which the authors did not report. I think the comparison with RL tasks like Atari or Mujoco is tenuous. I think there is some value in overfitting to a given webpage/environment but I think the results then aren't very surprising as we know that self-critiquing helps and so training on filtered data is bound to give improvement.

The second concern is that gains are small ~2.2% absolute, using mixture C hurts, and evaluations are conducted with a single seed. Of this, authors have clarified that capability score is the most important where even with mixture C there is some gain over the base agent. However, the use of a single seed makes it harder to read a strong signal from the experiments especially when the gains are small. I do agree that training 70b models is expensive. 

For these reasons, I am currently not recommending acceptance. I suggest authors to either focus on the generalization aspect and show that the agents can solve newer tasks than where they were trained. Or, add more experiments to show stronger and more reliable gains or include more domains. Ideally, both should be done. I do like the choice of WebArena and I think this paper is in an important direction.

Decision: Reject