Large Language Models Can Self-Improve At Web Agent Tasks

Authors: Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison-Burch, Sepp Hochreiter
Primary area: foundation or frontier models, including LLMs
Keywords: llm, llms, synthetic data, web agents, agents, self-improvement, unsupervised learning

Abstract:
Training models to act as agents that can effectively navigate and perform actions in a complex environment, such as a web browser, has typically been challenging due to lack of training data. Large language models (LLMs) have recently demonstrated some capability to navigate novel environments as agents in a zero-shot or few-shot fashion, purely guided by natural language instructions as prompts. Recent research has also demonstrated LLMs have the capability to exceed their base performance through self-improvement, i.e. fine-tuning on data generated by the model itself. In this work, we explore the extent to which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. In WebArena, an agent must autonomously navigate and perform actions on web pages to achieve a specified objective. We explore fine-tuning on three distinct synthetic training data mixtures and achieve a 31% improvement in task completion rate over the base model on the WebArena benchmark through a self-improvement procedure. We additionally contribute novel evaluation metrics for assessing the performance, robustness, capabilities, and quality of trajectories of our fine-tuned agent models to a greater degree than simple, aggregate-level benchmark scores currently used to measure self-improvement.

Decision: reject
Reviewer scores: 3, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729966451417

thread_type:
official_review

<thread_object>
creator: reviewer_1yvg

<content>
summary:
The authors study whether self-refinement (i.e. finetuning on self-generated data) can improve the performance of large language models on web agent tasks via the WebArena benchmark. This benchmark tests agents on a set of web page decision-making tasks. In such setting, self-refinement data consists of trajectories of interleaved environment observations and LM agent actions. The authors test three different ways of generating  trajectories for self-refinement: (A) fully “in-domain,” using train set tasks from the Web Arena environment only; (B) on a “in-domain” and “out-of-domain”  mixture, using train set tasks from WebArena as well as synthetic tasks generated by the LM agent; (C) fully “out-of-domain,” using only synthetic tasks. All “in-domain” trajectories are filtered using self-critique, for LM agent generation errors, and environment errors. The authors finetune a single LLM on each of these three data mixtures and compare the resultant LLM agents’ performance on WebArena under three different metrics. They show that self-refining on data mixtures A and B result in some improvements to performance on 3/3 metrics, while finetuning on mixture C produces a slight drop in performance on 2/3 metrics. Finally, the authors show that running an additional round of self-refinement with the best performing data generation strategy (mixture A) causes a drop in performance on 3/3 metrics, compared to results after a single round.

strengths:
- The study of generalizable techniques for improving LLM agents is of interest to the community. While many previous works have studied the advantages and limitations of self-refinement for reasoning and simple decision-making tasks, there has been limited work on harder benchmarks with longer decision-making horizons, like Web Arena. These tasks represent the frontier of state-of-the-art LLM capabilities. As a result, developing more effective methods for self-refinement on such problems is especially important.
- The paper is polished, clear, and fairly well written. The authors expound on all experimental details.
- The default metric for LLM performance on WebArena is the average completion rate of environment tasks. The authors make an effort to address the lack of granularity in this metric by introducing two auxiliary metrics for assessing LLM agentic performance. These auxiliary metrics add some nuance to the analysis, though this could be strengthened by increasing the number of experiment replicates and/or increasing the inference-time sample size, as models are evaluated at high sampling temperature.

weaknesses:
While the questions posed by this paper are timely, the experimental results have limited scope and are much more mixed than the text would suggest. While the title reads “LLMs can self-improve at Web Agent tasks,” the results seem to suggest that the performance margins are extremely limited. 

- Throughout the paper, the authors cite a 31% improvement over the base LLM agent as evidence for the power of self-improvement. However, this 31% is a relative score, corresponding to a mere ~2.2%  improvement in raw score from a completion rate of 7.14/100 to 9.36/100. The statistical significance of this improvement is unclear for two reasons: (1) the scale of standard error over 0-1 task completion score is omitted from Table 3; (2) according to Table 5, the authors sample from models during inference with a high temperature (T = 1). Sampling rather than using greedy decoding introduces an additional source of statistical error to results, which is similarly not quantified.
- In the introduction, the authors write: “We analyze three synthetic training data mixtures and find all three mixtures improve performance” (Lines 101-102).  This claim is misleading, since as shown in Table 3, self-refining on the third data mixture (C, “out-of-domain” tasks only) produces decreases in performance on 2/3 of the studied metrics.
- The authors show that iterative self-refinement produces a drop on all 3/3 Web Arena LLM agent performance metrics, even after the careful filtering procedures (Table 3, last row). As a reader, this result powerfully contradicts the tone of the title.
- On lines 404-406, the authors write: “Interestingly, we find that the majority of capabilities acquired by M_A and M_C are mutually exclusive, suggesting in-domain synthetic examples and out-of-domain synthetic examples improve acquisition of different capabilities.” This point would be extremely interesting if strengthened, but does not seem to be rigorously supported by the results. Currently, it seems to be being made on the basis of a single random seed of self-refinement in each setting, i.e. using a single sampled set of high temperature synthetic task generation + agent trajectories + LLM fine-tuning. In my view, to confirm these results, evaluation should be conducted with at minimum ~3-5 seeds in each data mixture setting.
- Overall the experiments have limited scope. Evaluations are conducted with a single seed for each data generation variant and a single LLM.  

Though the results of this paper are mostly negative, I still believe they are of interest to the community. I would be willing to increase my score if (1) the authors can provide an estimate of statistical error for all reported scores; and (2) the tone of the paper is adjusted to reflect the strength and scope of the experimental results.

questions:
There are a few other points that I would appreciate if the authors could clarify.

**(1)** The authors claim that the procedure for filtering in-domain self-refinement data is entirely unsupervised, but then seem to contradict this point later on.

In Lines 135-136: “Similar to the self-improvement prior work we discuss earlier, the collection of this data is completely unsupervised and no ground-truth labels are utilized for filtering and selection.”

Then, in Lines 194-196: “Finally, we also filter out any trajectories where the WebArena environment encountered an error or the model failed to produce a valid, parsable generation”
    
If I understand the procedure correctly, the highlighted parts of the filtering process are not truly unsupervised — they leverage signals that are external to the model, e.g. feedback from the environment. If so, the first claim should be adjusted accordingly.
    
**(2)** On lines 290-292, the authors write: “We note, however, that a number of tasks in the WebArena benchmark are trivial tasks and can be solved by a trivial baseline agent or weak model that performs no actions and only immediately exits by always generating stop [N/A].” 

This seems like a major weakness of the benchmark. What proportion of the WebArena tasks have this degeneracy property? How many of the <10% of tasks solved by tested LLM agents are degenerate? From my understanding of the text, degenerate tasks **are** filtered from capability score computation, but **are not** filtered from/identified during evaluation. Is this correct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730236075609

thread_type:
official_review

<thread_object>
creator: reviewer_nemt

<content>
summary:
This paper proposes a method of improving LLMs for web agent tasks. The method involves prompting an existing LLM as an agent to tackle tasks in WebArena. The trajectories are then filtered down to retain those that are likely to be successful. The model is then finetuned (with QLoRA) on these trajectories. In the paper, alternative trajectory generation methods are also explored, such as using the successful WebArena trajectories to generate out-of-domain trajectories on other websites. In addition, the authors propose a new metric for measuring the similarity of an agent trajectory, which has benefits over using the functional correctness evaluation in WebArena.

strengths:
- The paper shows that you can use in-domain trajectories as few-shot examples to generate novel out-of-domain trajectories, which will be useful in producing synthetic data for finetuning agents.
- The paper proposes a new metric $\text{VERTEX}_{\text{DTW}}$ which measures the similarity of an agent’s trajectory w.r.t. a reference (in this paper, the reference is a gpt-4 trajectory). This can be useful for providing a soft metric that has less false negatives than the WebArena functional correctness evaluation.

weaknesses:
My primary concern with this paper is that its evaluation setup is not fair. The models are essentially *training on the test set*, and doing a pass@2 on the evaluation set (these results align with the ablation experiments in the paper, where training on Mixture C actually makes the agent worse). Consider the filtered subset of in-domain examples (mixture A). It has a very high accuracy rate (0.919), which is expected as tasks that errored out or the model failed to solve are filtered out. In the WebArena benchmark, models often only stop when they are successful, so the false negative rate is low. If you view mixture A as groundtruth samples, finetuning on this subset is essentially allowing the model to memorize the examples that have passed, while ignoring those that it failed on the first attempt. When you test such a model, this can be viewed as essentially taking a second sample for the failed tasks. This is not a fair comparison to baseline models: a fairer comparison would be the same baseline model but an accuracy score of its pass@2 (i.e., take the best run out of two samples using the same filtering criteria). [1] actually does this with a prompted value function and gets a similar relative improvement (29%) compared to this work, and they do not do any training on the samples. If the authors can (1) compare their results against pass@2 of the agent (while resetting the WebArena environment after the first run, to avoid bias), and (2) perhaps run their model on a different benchmark, e.g., Mind2Web, VisualWebArena, WorkArena, or WebLINX, to show generalization, this would alleviate some of my concerns.

In addition, it would also be good to have error bars on the evaluation results as WebArena can be noisy, as the absolute difference between the baseline and the self-improved agent is low (7.14 to 9.36) and could be within the error range.

The proposed $\text{VERTEX}_{\text{DTW}}$ metric is evaluated against gpt-4 generated outputs, which possibly introduces model bias in the evaluation. WebArena has [human demonstrations](https://github.com/web-arena-x/webarena/blob/main/resources/README.md#12212023-human-trajectories), and it would be good to compare against these instead.


Overall I think the evaluation of the paper can be greatly improved (by measuring on more benchmarks, to avoid train-test contamination), as well as more ablation experiments. For these reasons I don’t think that the paper in its current shape is sufficiently complete.

questions:
How many examples are in Mixture C? How does the result scale with the number of examples used in Mixture C? Does it improve as more examples are provided?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580038268

thread_type:
official_review

<thread_object>
creator: reviewer_xuok

<content>
summary:
solving complex, multi-step, long-horizon tasks using prompting or fine-tuning poses various challenges. The paper presents a sound approach to enhance the capabilities of LLMs through self-improvement on web automation task. They propose evaluation techniques and effective use of in-distribution and out-of-distribution synthetic data, showcasing some performance gains.

strengths:
### strenghts:
1. **Capabilities through Self-Improvement:** The paper demos how LLMs can extend their capabilities through self-improvement techniques, particularly in the context of complex, long-horizon web agent tasks. This ability to acquire new capabilities while largely retaining existing ones is notable.
2. **Appropriate Eval Metrics:** The introduction of novel metrics, and scores to evaluate the quality of trajectories, adds depth to the evaluation process. These metrics provide a nuanced view of the models' performance, moving beyond simple task completion metrics and enabling a more detailed assessment of the agent's robustness and capabilities. the additions seem appropriate  enough to capture both success, relevance, and efficiency of trajectoriesl
3. **Synthetic Data Utilization:** The use of both in-domain and out-of-domain synthetic data for fine-tuning is commendable. This approach not only addresses the challenge of data scarcity but also demonstrates a systematic mechanism for enhancing agent model generalization.

weaknesses:
Good to address:
1. **Hyperparameter Selection:** The paper lacks a clear justification for the choice of hyperparameters in the synthetic data generation process and generating new objectives. e.g. using 4 or 2 few-shot samples, temperature value, and perhaps just one sentence on why 0.7 cosine similarity used is better. Any missing details may lead readers to question the replicability and robustness of the results. A more thorough analysis or rationale for these choices would strengthen the paper.
2. **Limited Model / Results:** The study's reliance on a single or fewer models limits its generalizability. Expanding the experiments to include a subset of models on appropriate agent frameworks that came out recently, such as smaller or different architectures, could provide insights into how these techniques scale or vary across different model sizes and types. I am thinking aloud if the results section can be tightened and elaborated more. But I will wait to see if my other reviewers has some feedback or ideas on this front.

questions:
While the study is promising, its impact would be further enhanced by addressing the concerns regarding the selection of certain variables used in the experiments and evals and model diversity. 

### Suggestions for Improvement:
- **Expand Hyperparameter Exploration:** Providing a sensitivity analysis or a justification for the selection of hyperparameters used in generating synthetic data would enhance the transparency and reproducibility of the results.
- **Broader Agent Evaluation:** Including diverse model architectures in the experiments could help validate the proposed self-improvement techniques across a range of LLMs, potentially increasing the impact and applicability of the findings.

I would be happy to increase the score post author responses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675578574

thread_type:
official_review

<thread_object>
creator: reviewer_ap9e

<content>
summary:
The papers is about having an agent for environments with lack of training data like web browsing. In that way, LLM can be a good candidate. In this work, they introduced a new technique in which LLMs can self-improve their performance as agents in long-horizon tasks in a complex environment using the WebArena benchmark. To better understand the effect of the self-improvement method, they introduced two auxiliary metrics: 1) a measure to analyze capabilities acquired and lost by the agent and 2) an extension of the VERTEX score to measure the quality of variable-length agent trajectories.

strengths:
- well written.

weaknesses:
- The innovation of the work is very limited! 
- Lack of enough experiments! The experiments are not comprehensive! It should consider a good analysis across different datasets and careful reasoning of what is going there! For example, having complete oblations studies versus other possibilities.
- Lack of comparison with baselines! There are a lot of works of self-improvement and self-correctness! There should be considered as baselines. 
- This work needs a proper and further study and analysis! This is an incomplete work to be published!

questions:
- What is the definition of plausible trajectories and high-quality?
- What are the possible oblations against the proposed type of data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519936202

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank the reviewers for their positive reception and valuable feedback which helped us to substantially improve and strengthen our work. We further thank all reviewers for their consensus on novelty and impactfulness of our work on an important task that will be of interest to the ICLR community.

We are especially invigorated by the positive evaluation: 

- From **reviewers xuoK, neMT, and 1yvG** for addressing the **relevance of nuanced evaluation metrics**, including auxiliary metrics and **VERTEX_DTW**, which offer more granular and robust assessments of LLM agent performance compared to standard metrics.  
- From **reviewers xuoK and neMT** for emphasizing the effective use of **synthetic data**, demonstrating its value in addressing data scarcity and enhancing model generalization through both in-domain and out-of-domain trajectories.  
- From **reviewers xuoK and 1yvG** for highlighting the paper’s focus on **self-improvement techniques for LLMs**, particularly in tackling **long-horizon tasks** like WebArena, a frontier of LLM capabilities.  
- From **reviewers aP9e and 1yvG and xuoK** for recognizing that the paper is **well-written and polished**, with clear exposition of experimental details.  

In response to the comments and suggestions, we have provided a detailed and thorough rebuttal for each reviewer, have uploaded a revised PDF with proposed revisions/additions requested by reviewers, and provide a summary of the major points for convenience:

- We noted that we did not experiment / explore different hyperparameter settings for some parameters in “Limitations”.
- We have adjusted the tone and narrowed a few sentences with claims that could be interpreted to be overstated.
- We have provided results on an **additional model**, Llama-3-70B, showing the procedure generalizes to different models. 
-  We also provide **confidence intervals and highlight significance**.

We believe these significant revisions now adequately address the stated concerns of reviewers, who were willing to increase their scores, and we are happy to make further adjustments at each reviewer’s discretion in our final revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper performs self-improvement in LLM on the Web Arena tasks which is a challenging web navigation planning task for LLMs agents. Two main concerns were raised during the reviewing process. The first was the question of fairness in evaluation (by reviewer neMT) and the second was experimental results being weak or even misleading (reviewer 1yvG). 

The fairness concern is that the agents are being trained on the same prompt/task distribution as the ones on which they are being tested. Essentially, the agent generates a set of trajectories on a given distribution of which some of the trajectories are eliminated using a self-critique signal and this data is used for training (mixture a). There is also additionally an option to generate out-of-distribution data (mixture b) and mix it with the in-distribution filtered data (mixture c). If the self-critiquing works, then it is clear that we should expect gains when training on it and then evaluating a similar distribution. A much more powerful result would have been to show that the agent can solve novel tasks after being trained. Reviewer neMT suggested doing pass@2 evaluation with baseline agents which the authors did not report. I think the comparison with RL tasks like Atari or Mujoco is tenuous. I think there is some value in overfitting to a given webpage/environment but I think the results then aren't very surprising as we know that self-critiquing helps and so training on filtered data is bound to give improvement.

The second concern is that gains are small ~2.2% absolute, using mixture C hurts, and evaluations are conducted with a single seed. Of this, authors have clarified that capability score is the most important where even with mixture C there is some gain over the base agent. However, the use of a single seed makes it harder to read a strong signal from the experiments especially when the gains are small. I do agree that training 70b models is expensive. 

For these reasons, I am currently not recommending acceptance. I suggest authors to either focus on the generalization aspect and show that the agents can solve newer tasks than where they were trained. Or, add more experiments to show stronger and more reliable gains or include more domains. Ideally, both should be done. I do like the choice of WebArena and I think this paper is in an important direction.