Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation
Authors: Lajanugen Logeswaran
Jaekyeom Kim
Sungryull Sohn
Creighton Glasscock
Honglak Lee
Venue: COLM 2025 (2025)
Keywords: web agent
evaluation
distillation
TL;DR: Scaling Web Agent Training through Fine-grained Constraint Evaluation

Abstract: We present a scalable pipeline for automatically generating high-quality training data for web agents. In particular, a major challenge in identifying high-quality training instances is trajectory evaluation - quantifying how much progress was made towards task completion.
We introduce a novel constraint-based evaluation framework that provides fine-grained assessment of progress towards task completion.
This enables us to leverage partially successful trajectories, which significantly expands the amount of usable training data. We evaluate our method on a new benchmark we propose called BookingArena, which consists of complex booking tasks across 20 popular websites, and demonstrate that our distilled student model outperforms open-source approaches and matches or exceeds commercial systems, while being a significantly smaller model. Our work addresses the challenge of efficiently creating diverse, realistic web interaction datasets and provides a systematic evaluation methodology for complex structured web tasks.

Published as a conference paper at COLM 2025
Scaling Web Agent Training through Automatic Data Generation and Fine-grained Evaluation
Lajanugen Logeswaran, Jaekyeom Kim, Sungryull Sohn, Creighton Glasscock, Honglak Lee
(ilajan,jaekyeee creiglas, Jong.aa)8lggreseerch... sradmmn...ngggggsggggggaggggerrr LG AI Research
Abstract
We present a scalable pipeline for automatically generating high-quality training data for web agents. In particular, a major challenge in identifying high-quality training instances is trajectory evaluation quantifying how much progress was made towards task completion. We introduce a novel constraint-based evaluation framework that provides fine-grained assess- ment of progress towards task completion. This enables us to leverage partially successful trajectories, which significantly expands the amount of usable training data. We evaluate our method on a new benchmark we pro- pose called Booking which consists of complex booking tasks across 20 popular websites, and demonstrate that our distilled student model outperforms open-source approaches and matches or exceeds commercial systems, while being significantly smaller model. Our work addresses the challenge of efficiently creating diverse, realistic web interaction datasets and provides a systematic evaluation methodology for complex structured web tasks.
1 Introduction
The rise of large language models (LLMs) has spurred significant interest in web agents-systems capable of navigating and interacting with complex, dynamic websites to accomplish real-world tasks. These agents face substantial challenges: each web page can contain hundreds of interactive elements, and successful task completion requires making sequential decisions across multiple pages.
Advances in agents capable of fluently performing tasks in virtual environments such as the web rely on two critical, interrelated ingredients: data and automatic evaluation. Large-scale agent trajectory data is essential for training competent models. Due to the scale required, synthetic data generation becomes crucial. However, this introduces the challenge of verifying that the generated samples are meaningful and of high quality. The sequential decision-making nature of the tasks, combined with the inherent complexity of the web environment, makes this a particularly difficult problem.
In this work, we aim to address both challenges by proposing an automatic data generation pipeline, along with novel automatic evaluation metrics capable of assessing agent trajec- tories with high fidelity. Our approach focuses on scalable trajectory data generation- relatively underexplored but crucial area for web agent development. We propose a fully automatic data generation pipeline that leverages publicly available large language models to produce large scale synthetic trajectories. Our approach targets complex, real-world web navigation tasks that involve multiple constraints such travel booking scenarios.
The complexity of these multi-constraint tasks renders traditional evaluation approaches insufficient. Existing trajectory evaluation frameworks that rely vision-language mod- els (VLMs) or large language models (LLMs) as judges (He et al.) 2024 Pan et al. 2024a Trabucco et al. 2025 Pahuja et al., 2025) primarily depend on the models' ability to inter- pret requirements described in natural language. This reliance limits their effectiveness when applied to complex tasks with nuanced or multi-faceted constraints. To overcome

---

Published as a conference paper at COLM 2025
Seed tasks
Trajectory
On southwedt.com com, find one way fight ...
2. On yalp.com find a swimming pool
Target website
WWWw varaso com
loccaion
Teacher Agent
click [58 wi wheeee (66 (Beciloo)) [Chwck in
Task Generator
click 13] whore (S [generi] [8]
Task
Instances
Aspen, CD April 68, April 2025, that has skHin/skiouut access.
Dataset
Fine-grained Constraint based Evaluation
Task find vacation rental Aspen, CO April 2025, to April 12, 2025, that has skrin/ski-out access.
Observation
Constraint Satisfaction Rate: 0.83
URL:https:/ vacasa 096547checkkIn
Screenshot
Agent's constraint state Evaluation
LLM
nade selection: true true
location: Aspen, CD true
type: Mountsin Lodge
Target constraints
_adte: 04/98/2025 true
nade. selection: true
true
Incation: Aspan, CD
ski-out access: unknown false
type: vacation rontal
start_date January 08, 2025
and, date: January 2025
sski-in/ske
VLM/LLM Judge
Figure Overview. We introduce scalable pipeline to automatically generate and evaluate web agent trajectories for training competent small language models (Section 2.1 Given trajectories generated by a few-shot prompted teacher agent, our novel constraint-based fine-grained evaluation framework (Section 2.2) extracts high-quality training instances for distillation. See text for details.
this limitation, we propose novel constraint-based evaluation framework that enables fine-grained assessment of agent trajectories. Specifically, this framework examines each individual criterion specified in the task description and verifies whether satisfied in the resulting trajectory. Compared to binary success/ failure metrics, our evaluation approach offers a more nuanced and interpretable measure of task progress while remaining robust to the complexities introduced by multi-constraint tasks.
Leveraging these innovations, we curate a large high quality training dataset. Our 24B pa- rameter model finetuned on this data outperforms prior open-source well as commercial systems, demonstrating the effectiveness of our overall pipeline. We make the following main contributions in this work:
We propose scalable automatic data generation pipeline which leverages few-shot prompted publicly available language models to synthesize large number agent trajec- tories.
We propose novel constraint-based evaluation framework which enables fine-grained assessment and filtering of trajectories, enabling the curation of a high quality training dataset for agent training.
We introduce Booking new benchmark consisting of 120 complex and structured tasks on real-world booking websites.
Evaluation complex booking tasks across 20 popular websites reveals that our distilled 24B parameter model outperforms both open-source alternatives and matches or exceeds the performance of commercial systems. Notably, our student model achieves better results than the much larger 405B parameter teacher model used for trajectory generation.
2

---

Published as a conference paper at COLM 2025
2
Approach
We present a scalable pipeline for generating high-quality data for training competent web agents. We first present our trajectory generation and distillation pipeline for generating large-scale web agent trajectory data from the web with few-shot prompted large language model agents in Section 2.1 Next, we propose new fine-grained constraint-based evalu- ation framework for automatic evaluation and curation of trajectories in Section 2.2 See Figure I for an overview of our pipeline.
2.1 Automatic Trajectory Generation and Distillation
Our data generation pipeline consists of four key components described next: Task Genera- tion, Trajectory Generation, Trajectory Evaluation and Distillation.
Task Generation. We begin by compiling list of the 1,000 most popular safe websites, removing any that are inaccessible (e.g., those returning error codes). We then prompt GPT-4 with manually curated seed tasks to generate diverse set of target tasks, following few-shot learning paradigm (See FigureBor the Appendix for the prompt). The prompt encourages the model to generate tasks that are diverse, realistic, and span varying difficulty levels.
Trajectory Generation. Given set of tasks, we employ few-shot prompted Large Language Model (LLM) agent to generate trajectories by interacting with the web. The agent's prompt template (shown Figure of the Appendix) is adapted from [Shi et al.] (2024). At each timestep, the agent receives as input the task description, current URL, page's s accessibility tree, history of previous actions and reasoning, and currently open browser tabs. The accessibility tree provides structured representation of the webpage, where each element is formatted as [id] [tagType] [text content] [properties]. Here, id is unique numerical identifier, tagType specifies the HTML element type (e.g., button, link), text content contains the element's 'ss visible text, and properties lists any additional attributes. Based on these inputs, the agent predicts the next action a; from set of possible operations including element clicks, text input, hover events, and page navigation actions. The agent also produces (chain-of-though)) reasoning for the predicted action. A trajectory is defined as sequence of action-observation we log metadata including the URL, accessibility tree, and page screenshot. This stage produces collection of task description d; and corresponding agent trajectory z; pairs D (dd T
(at, ot)) For each observation,
Trajectory Evaluation. Trajectories generated by the few-shot agent inevitably fail due to various factors including the limited capabilities of the underlying LLM, impossible tasks, and bot detection mechanisms. Therefore, reliable and efficient automatic evaluation framework is essential for identifying high-quality trajectories suitable for dataset construc- tion. Evaluating web agents presents significant challenges, particularly for tasks involving multiple constraints that must be satisfied. Prior approaches automatic evaluation largely adopt an LLM-as-judge (Or Vision Language Model (VLM) as judge) paradigm where an LLM/VLM is asked to directly judge success failure given screenshots from an agent trajectory (He et al., 2024). While effective for simple, short-horizon tasks, the scalability of this approach to more complex, multi-criteria tasks remains unclear. One of our key contributions is comprehensive evaluation framework to automatically evaluate agent trajectories (Section 2.2
Using our automatic evaluation framework, we curate high-quality training set Dirrai for distillation. The student model undergoes supervised fine-tuning to mimic the teacher model's behavior, learning to predict both the next action and its accompanying reasoning given the current observation and agent history.
Distillation.
3

---

Published as a conference paper at COLM 2025
2.2 Constraint Framework
We propose fine-grained metric to evaluate task completion. Figure 1 illustrates our constraint-based evaluation process. Given a task, we first identify set of constraints that must be satisfied in order to complete the task using an LLM. For instance, consider the task "Find a hotel in Paris for the dates Aug 2 -3, 2025". An agent that successfully completes this task must fulfill the following constraints: (location: Paris, start_dar Aug 2, 2025, end_date: Aug 3, 2025). Our evaluation metric examines how many of these individual constraints the agent satisfied during task execution, providing granular assessment of performance. Traditional binary success / failure metrics treat all failures equally, offering no distinction between agent that made no progress toward the task and one that made substantial progress but failed to fulfill all requirements. In contrast, constraint-based evaluation provides more nuanced signal that can differentiate between these scenarios, offering valuable insights into partial task completion. This approach also enhances evaluation objectivity by introducing a systematic framework that reduces subjective judgment in performance assessment.
Automatic Evaluation based on Constraints. Having defined the relevant constraints for each task, we define an evaluation metric termed Constraint Satisfuction Rate (CSR). CSR computes a binary success/failure score for each constraint representing whether the agent satisfied the corresponding constraint in task execution and computes the average score across all constraints. This metric is applicable to both LLMs and VLMs judges. For a VLM judge, we use the task description, final web page screenshot as well as the URL for evaluating constraint satisfaction. For LLM judge, we use the final web page accessibility tree instead of the screenshot. Figure 5 shows the prompt used for constraint- based automatic evaluation.
Given a task d, from which a corresponding set of constraints C = (ci, 8; Hー are identified (e.g., using an LLM), we define the Constraint Satisfaction Rate (CSR) for given observation o as follows:
1 E [[f(oloo,d) = 11 1=1
CSR
(1)
where f represents an LLM/VLM judge that predicts the observed value of the constraint
q given the observation o and task description d. Given a trajectory T = {(a,, ot))
the constraint satisfaction rate is defined as the CSR of the final observation: ( = (or) The task success rate is defined as
(R.(T) = (CSRa(T) =
(2)
These metrics are extended to collection of trajectories by macro-averaging.
Dataset Curation with Constraint-based Evaluation. In addition to providing fine- grained evaluation metric, constraints can also be helpful in evaluating progress towards the task. This provides an opportunity to leverage partial trajectories for training. This is important since teacher agents' success rates on hard tasks will be low and restricting to successful trajectories alone will significantly limit the amount of training data. While task success/ failure metrics are only meaningful at the end of task completion, constraint satisfaction can provide a meaningful evaluation for partial trajectories as well. At any stage during task execution, CSR can be evaluated to assess the number of constraints fulfilled up to that point (illustrated in Figure 2)
Given a task description and trajectory T ((at, we examine the maximum CSR value achieved at any point during the trajectory (Cmax max; (o)) If C > 0, we extract the trajectory prefix T' = o; where is the smallest time-step for which (op) = Cmax (i.e., I' = min ft: CSR = Cmax )).
In the trajectory prefix extraction process described above, note that invalid 'stop' actions (i.e., action that marks the end of task execution) could have been included. These are actions a stop where (SR2(o:) # 1 instances where agent prematurely predicted stop
4

---

Published as a conference paper at COLM 2025
Action:
Type Sydney into location input
Click Nov 23, 2024
Click on "Travelodge Holel"
Go back
c5R=02
CSR 0.4
CSR
CSR 0.6
selectencction: talso false
www.ction:
selection false false
selection
location NíA (sise
Sydney
Sydrey
Sydney|
start date: NJA felsa
-wart date: NAA
start datei 25, true
date: NiA] false
and date: NIA faise
and date: N/A talse
and dota: Nov 24, 2024 true
and date: NiA taise
booking_ typer Stayy: true
bookingg type: Stays true
booking type: Stays | true
booking type: Staye true
Task: On booking.com com. find a hotel in Sydney, Australia for one night on November 23, 2024
Figure 2: Prefix Extraction with Constraint Evaluation: For a given trajectory, we compute the constraint satisfaction rate (CSR) for each time-step and extract the smallest prefix that reaches maximum CSR. In this example, only the first two actions are retained and the third action discarded as it results in decrease of CSR (Agent should have clicked on the 'search' button instead).
action despite not all constraints of the task being met. We retain a stop action a; stop if and only (o,) = 1. For stop actions with (o;) < 1, we use a hindsight re-labeling approach to revise these for inclusion in the dataset described next.
Hindsight Re-labeling. For stop actions where the agent made partial progress toward completing the task, 0 (o) < 1, we rename the task description so that these become valid stop actions with the following hindsight re-labeling approach. Given that C' c C constraints were satisfied, we revise the original task description d to d' so that the new description is consistent with the constraints C' (i.e., constraints C' C' are removed from the task description). For example, consider the task "On hilton.com, find a hotel in New Orleans on February 08 for 3 people". Suppose the status of constraints after task execution was judged to be { sade_selection: False, location: "New Orleans", date: "February 8, 2025", guest_counts <Unnobserued> ). After hindsight re-labeling, the task revised to "On hilton.com, search for hotels in New Orleans February 8". In addition, we also revise the agent's 'ss reasoning steps for the stop action to be consistent with the new task description.
3 Experiments
Dataset Construction. We use LLAMA 3 70B and LLAMA 3.1 405B Instruct models (ou- vron et al. 2023) as teacher agents for trajectory generation, where the bigger model was necessary for complex tasks (e.g, booking). We collect 150k trajectories, which correspond to approximately 1M individual actions/ steps (note that this includes successful and unsuc- cessful trajectories). The generated trajectories are evaluated with our constraint evaluation approach using LLAMA 3.3 70B and Gemma 3 27B (Team et al. 2025) models (Prompt provided in Figure 6 of Appendíx). These evaluation outputs are used by the prefix ex- traction and hindsight relabeling steps (Section 2.2) for curating a high-quality training dataset. To counter evaluation noise, we only retain training instances that are common to both these judge models. From the 150k trajectories collected, 166k were judged to be completely successful (based on our constraint evaluation metric). On the other hand, we consider trajectories that are partially correct (i.e., CSR 0 for at least one step), ~65k trajectories have usable prefixes, which leads to a dataset of 300k actions / steps. All experiments conducted using two machines, each equipped with 16 A100 40GB GPUs.
find that
5

---

Published as a conference paper at COLM 2025
student agents. The prompt used for fine-tuning is shown in Figure of the Appendix. We use LoRA fine-tuning for distillation, where we fine-tune the query and value projections (q proj, v v_proj). We use learning rate of 1e-4 with a cosine scheduler and a batch size of 16.
Distillation. We train Mistral 3 Small 24B Team 2025
Evaluation Setup. In contrast to some prior work that use static datasets for evaluation, which have been shown to poorly correlate with task success rate in the wild (Zheng al. 2024), we focus on online tasks, where the agent is expected to execute task and success/ / failure evaluation is performed at the end of trajectory. We consider multiple evaluation sets for accurate assessment of different agents' performance on web tasks.
website
Since existing benchmarks for online evaluation lack representation of complex long-horizon tasks, we develop new benchmark termed Booking focused travel booking tasks (flights, hotels, tickets, etc.). This benchmark com comprises 6 tasks from each of 20 popular websites commonly used for real-world booking activities. We define difficulty levels based on the number of constraints specified in each task. The 6 tasks per website are distributed as 2 easy, 2 medium, and hard tasks, with average number of constraints for each category being 3,95, 5.45, and 7.30 respectively. Each task is identified by a starting url and task description. Agents are required to interact with actual websites and complete tasks. To prevent tasks from being outdated with the passing of time, we also provide scripts that automatically detect past dates and update them with future dates to keep the tasks valid. We hope this benchmark will be useful to study complex, long-horizon tasks in the future.
In addition, we also evaluate our approach on Web Voyager (He et al. 2024), which is dominated by search tasks. This benchmark comprises 643 tasks from 15 websites, with an average of 43 tasks per website. 55 tasks that are no longer possible (e.g,, due to their time-sensitive nature), are excluded from our evaluation. For booking tasks that are fixable by updating dates (e.g,, 'Find deals for a vacation in Mexico in Dec 2024'), we update these dates that they are in the future.
The rest of this section is organized as follows. We begin by presenting the evaluation on our Booking Arena benchmark in Section 3.1 Next, we provide ablation studies our pipeline's design choices (Section 3.2) and present a study on our automatic evaluation approach (Section 3.3 Finally, we present results on the WebVoyager benchmark in Section 3.4
3.1 BookingArena Evaluation
Baselines. We consider commercial and open-source state-of-the-Art VLM agents as baselines: UI-TARS QQin al.) 2025), Claude-Computtr-Usse (computer-use 2u0e--u-----222-1-1---2 2024-10-22) (Anthropic 2024), Operator
(OpenAT 2025), Browser
Use Müller & Zunić 2024.. Browser Use is set-of-marks-baseed agent where interactable screen elements are highlighted with colored bounding boxes with numerical ids and the agent predicts element ids interact with. UI-TARS, Claude Computer- Use, and Operator, on the other hand, are pixel-based approaches, which interact with screen elements based on pixel coordinates. These agents take one or more screenshols input and produce actions.
Evaluation Protocol. We use the constraint- success rates, SR and CSR, as defined in Equations (1) and (2) for this evaluation. For evaluation of the final models, use GPT-4o with visual observation space (final screenshot and URL). The prompt used for evaluation is given in Figure 5f the Appendix. Note that this is different from the models (LLAMA, Gemma) used to curate the dataset, alleviating concerns of dataset curation bias.
Results. Our approach outperforms prior methods based on closed-source models such as Claude Computer Use, Operator and Browser Use, as well as open-source alternatives such as UI-TARS on the task SR metric, and is only inferior to Operator This is particularly significant, given that our agent is a 24B parameter model. However, we also find that SR is poor across the board due to the challenging nature of the tasks, with performance several websites being zero. Task success rate is a stringent metric that equally penalizes agent that got majority of the constraints in the task correct, as well as agent that

---

Published as a conference paper at COLM 2025
Browser Use
Claude CU
UI-TARS
Operator
Ours
Website
SR
CSR
SR
CSR
SR
CSR
SR
CSR
SR
CSR
Booking.com
20.0
44.0
0.0
6.7
66.7
83.3
0.0
55.6
50.0
89.2
Vrbo
0.0
40.3
33.3
62.0
0.0
26.9
16.7
68.5
66.7
88.4
Trivago
33.3
66.5
0.0
50.5
33.3
66.3
50.0
84.2
50.0
85.9
Travelocity
0.0
50.0
0.0
56.7
50.0
68.9
20.0
77.7
50.0
85.6
Rome2rio
40.0
71.7
33.3
58.3
0.0
25.0
50.0
75.0
50.0
81.9
Orbitz
16.7
53.1
33.3
49.6
16.7
28.9
83.3
98.3
80.0
80.0
Homeaway
16.7
40.5
16.7
59.5
33.3
76.8
50.0
66.7
50.0
73.8
Airbnb
0.0
0.0
34.1
0.0
56.8
20.0
66.0
33.3
72.9
Kayak
50.0
80.0
0.0
43.0
33.3
57.2
25.0
80.4
16.7
68.0
Vacasa
33.3
43.3
33.3
56.3
0.0
37.9
50.0
69.0
33.3
66.2
Plumguide
50.0
76.4
16.7
38.9
33.3
48.6
20.0
58.3
33.3
63.9
IHG
33.3
40.0
33.3
60.7
33.3
61.9
40.0
67.0
33.3
59.8
Sonder
0.0
10.0
16.7
30.0
0.0
11.7
25.0
56.7
16.7
56.7
Redawning
16.7
47.4
16.7
36.9
33.3
56.2
16.7
49.0
20.0
50.2
Hotels.com
16.7
62.2
16.7
59.2
66.7
80.0
25.0
65.0
16.7
48.3
Momondo
16.7
38.9
16.7
78.9
33.3
80.0
50.0
90.8
0.0
41.1
Hilton
16.7
36.1
50.0
76.7
50.0
84.4
80.0
96.0
0.0
33.3
Google Flights
0.0
27.5
16.7
60.5
33.3
59.2
40.0
88.1
0.0
31.2
Radisson
20.0
36.0
0.0
17.8
16.7
26.7
0.0
47.7
0.0
12.2
Megabus
0.0
34.4
0.0
36.1
0.0
35.0
0.0
23,9
0.0
12.2
Average
18.8
45.3
16.8
48.7
26.7
53.6
33.0
68.7
29.9
60.2
Table 1: Website-level performance comparison of different approaches on our BookingArem benchmark. We compare our approach against Browser Use (uüller & Zunič 2024), Claude Computer Use (Anthropic 2024), UI-TARS (Qin et al. 2025) and Operator (OpenAl 2025). SR represents task success rate and CSR represents constraint satisfaction rate. Best performance boldfaced and second best underlined.
made progress task. For such difficult tasks, we argue that CSR provides more fine-grained signal to compare different models.
We find that Browser Use struggles with dense webpages, where the set-of-marks annotation can make the screenshot complex and difficult to parse. While Operator is generally capable with the best CSR among all approaches, we find that it often makes one-off errors with date selection (e.g., choosing 'May 14' when the provided date is 'May 15'), which explains the low SR. A key advantage of our proposed constraint-based metric is that it provides such aine-graieed insights about the deficiencies of agents, which is critical for advancing the frontier on web agents.
We further evaluate how often a specific constraint is successfully satisfied by our agetnt across tasks. Success rates for the 10 most frequent constraints appearing in the evaluation tasks are shown in Table Intuitively, location-related constraints have higher success rates, as they are often filled early the task. In contrast, constraints related to amenities, such as pet-friendliness, are handled later in the task and have low success rates as a result, due to the dependency on prior actions.
3.2 Ablations on Agent Training
We perform ablations understand the impact of different components of our approach such as the choice of teacher model, impact of our data filtering strategy, and fine-tuning design choices. For these ablations, we report SR and CSR on Booking Arena in Table 2
Teacher Performance. We compare different few-shot prompted teacher models with demonstration examples in the first section of Table We find that LLAMA models perform well, with the 405B model performing best complex tasks.
7

---

Published as a conference paper at COLM 2025
Model
Data Source
Training/ Inference
SR
CSR
Constraint
Success
%
%
Rate (%)
Qwen 2.5 72B
N/A
Few-shot
18.0
46.0
location
79
Llama 3.3 70B
N/A
Few-shot
22.5
49.8
departure loc.
73
Llama 3.1 405B
N/A
Few-shot
22.5
52.4
destination loc.
67
return date
64
Mistral 24B
Success only
LoRA
25.0
55.0
departure date
56
Mistral 3 24B
All trajectories
LoRA
29.4
57.2
start date
55
Mistral 24B
Partial success
LoRA
29.9
60.2
pet-friendly
50
Mistral 3 24B
Partial success
Full finetune
28.4
58.7
end date
47
rental type
41
Table 2: Performance comparison across model variants and prompting/ poomptin/ttaininng configurations. We evaluate teacher models using few-shot prompting and student models trained with different data filtering strategies and training approaches. Task SR and Constraint SR represent the task success rate and constraint satisfaction rate respectively.
num guests
25
Table 3: Success rates of our fine-tuned agent for the top- 10 most frequent constraints in the evaluation tasks.
Task SR (%)
Constraint SR (CSR) (%)
Model
Screenshot
+URL
+History
Screenshot
+URL
+History
Claude Computer Use
12.6
16.8
21.8
39.1
48.7
53.0
LLAMA 405B few-shot
19.0
22.5
23.3
47.9
52.4
53.3
Mistral 24B fine-tuned
25.0
29.9
30.0
50.3
60.2
58.8
Table 4: Performance comparison across different models and evaluation settings. Task SR represents task success rate, while Constraint SR represents constraint satisfaction rate. The +URL and +History columns indicate additional information provided during evaluation.
Dataset curation. We consider the following variations in the choice of data for training: only successful trajectories, all trajectories (successful + unsuccessful), and partially suc- cessful trajectories. We find that learning from partially successful trajectories is better than learning from successful complete trajectories alone, which demonstrates that we benefit from scale. We also consider a multi-stage training strategy where we train on all trajecto- ries, followed by successful trajectories only. While this approach leverages unsuccessful trajectories and benefits from data scale, it is inferior to the simpler single-stage training strategy with partially successful trajectories alone.
Training Strategy. We find that LoRA training is more effective than full finetuning, This shows that LoRA is more effective in unlocking the knowledge of the base PLM, while full finetuning is prone to overfitting, especially in our data regime. Overall, we find the simple training recipe of training relatively small LLMs with LoRA on partially successful trajectories to be an efficient and effective training recipe which significantly outperforms much larger prompted 405B model.
3.3 Reliability of Automatic Evaluation
We study the reliability of our evaluation strategy by considering the inclusion/ exclusion of information provided to the evaluator. We consider page screenshot as the minimun source of information about the agent state available to the evaluator and study the impact of including/ excluding information such as the page URL and the history of actions performed by the agent (Table 4). First, we find that inclusion of URL leads to better evaluation scores as it can encode usetul information such as location, dates, etc. that can be complementary to webpage contents. Consider the following example url: https://www. nomendo, https/uuurttyyccckkoott 20%272025 01- 13h In this case, even though the agent was eventually blocked by a captcha, clearly made progress towards the task by setting the correct rental type, location and dates, as is evident from the url.
N2Fcarst2FSann FranciscoN22CAA- 11000-c1112222222255333
8

---

Published as a conference paper at COLM 2025
Website
Wilbur
Web WebVoyager (Claude)
WebVoyager (GPT-4o)
WebVoyager (GPT-4V)
Ours
Lutz et
2024
He et al. 2024
He et al. 2024
He et al. 2024
Allrecipes
60.0
45.9+334
16.3
51.1
62.5100
Amazon
43.9
58.644.2
53.7
52.9+1.4
63.22000
Apple
60.540.0
58.1
56.641.3
62.8+2.3
63.640.0
ArXiv
51.2
55.0+700
60.5+000
52.0+1.3
59.5+00
GitHub
22.0+0.0
56.9+1.4
57.7+3.7
59.3+3.7
62.5+0.0
Booking
38.6+0.0
19.0
43,9+3.5
32.6+2.7
55.0+0.0
ESPN
59.1
46.211.3
44.0127
47.0+1.3
47.5.000
Coursera
51.1
68.2+1.
65.1128
57.912.
70.0
Cambridge Dict.
86.0+0.0
71.3436
82.2+1.3
71.3+1.
76.7
BBC News
81.0+0.0
66.7 148
54.8124
60.3+28
77.1
Google Flights
0.0
15.1
28.6+0.0
51.6+1.4
66.7
Google Map
39.0
55.3+1.4
56.9+28
64.3+28
60.5+000
Google Search
67.4.0.0
72.9+13
63.6413
77.542.7
75.0+00
Huggingface
53.5400
53.514.7
42.643.
55.8+2.3
68.6100
Wolfram Alpha
65.21000
51.5+5.4
65.212.2
60.912.2
58.7100
Overall
52.610.0
52.811.
55.5
57.1
64.5100
Table 5: Comparison of various models Web Voyager. For each automatic evaluation, run GPT evaluator three times to calculate the performance mean and standard deviation.
On the other hand, notice that inclusion of action history leads to generous evaluation due to hallucination the judge model. For instance, the action 'type [3] [New York]' is provided in the agent history, the judge often simply assumes that the action of typing 'New York' into the relevant field the page was successful, regardless of whether the action was successfully executed in the page. We further confirmed that inclusion of action history leads to evaluation outputs the correlate less with human evaluation. These ablations indicate that url and page contents provide the most unbiased signal for evaluation.
3.4 Web Voyager Evaluation
Baselines. We consider Wilbur (Lutz et al. 2024) and the agents reported in WebVoyage (He et al., 2024 baselines. For fair comparison, only compare with baselines that evaluate performance based on the standard evaluation protocol proposed in WebVoyager We use GPT-4o for evaluation since the GPT-4V model used these works is deprecated.
Evaluation Protocol. We follow the standard evaluation protocol proposed in He al. (2024). The final 15 screenshots of the agent trajectory, along with any final textual responses are provided the VLge- which required to output a success failure judgement. Performance average and standard deviation are calculated based on three runs of the GPT- 4o evaluator, similar He et al. (2024). Note that this evaluation protocol different from the evaluation metrics considered in previous sections we only consider this evaluation approach for this dataset for consistency with prior reported results.
Results. We compare our approach against prior work in Table 5 We outperform prior approaches based on large closed-source foundation models such as GPT and Claude with 24B parameter small language model. In particular, we achieve significant performance improvements on hard domains such as Booking and Google Flights, where we achieve improvements of 11.1% 111% and 15.1% 1% respectively, compared to the best baseline approaches on those domains. To our knowledge, our work is the first to achieve competitive performance
Agent-E (Abuelsaad et al. 2024 and Browser Use Müller & Zunić 2024 used human evaluation,
and other commercial Cl control systems did not reveal their evaluation setting.

---

Published as a conference paper at COLM 2025
on the WebVoyager benchmark with fine-tuned small language model. We also note that the all standard deviations are less than 0.05 due to high self-agreement of GPT-4o.
4 Related Work
Evaluating web navigation agents presents significant challenges due to complex observations, multi-step interactions, and incomplete information. To address these challenges, prior work have proposed static benchmarks such as Mind2Web (Deng et al., 2023) and WebLINX (Lü et al., 2024), which provide efficient evaluation frameworks with bernig predicted actions against reference ground-trui actions.
Benchmarks.
step-level assessment by comparing However, results from these benchmarks may not accurately reflect agent performance on real-world tasks. Alternatively, benchmarks using live websites, including WebShop (Yao et al. 20022,, WebArena (Zhou et al., 2023), and VisualWebArena (Koh et al., (002), enable controlled evaluation on fixed, simulated websites but often fail capture the complexity, stochasticity, and dynamic nature inherent in real-world web environments. Web Voyagrg (He et al. 2024) addresses this limitation by proposing evaluation tasks and automatic evaluators for fask execution on actual websites, making it one of our chosen evaluation benchmarks. To further assess web agents on highly complex tasks, we introduce our own evaluation benchmark, Booking Arena, which comprises real-world booking tasks that better reflect the challenges agents face in practical applications.
Automatic Evaluation for Web Agents. Existing web agent evaluation approaches face significant limitations in both scalability and reliability. WebCanvas (Pan et al. 20244) and VisualWeb (Koh et al. 2024) rely on human-annotated subgoal states and symbolic reward functions respectively y, which lack scalability for broader applications. Conversely, Web Voyager (He et al. 2024) and other approaches (Pan et al., 2024a) employ VLM//LLM judges that evaluate success based final screenshots and agent responses, but these methods prove unreliable and imprecise for complex, multi-criteria tasks. To address these challenges, we propose constraint-based automatic evaluation framework for structured tasks defined by specific constraint sets, offering more systematic approach than existing WMM/LLM-based methods while maintaining greater scalability than human-specified reward functions.
Automatic Trajectory Data Generation for Web Agents. Automatic generation of trajec- tory data for training web agents remains relatively underexplored area. InSTA (Trabucco) let al.) 2025) uses LLMs to generate web tasks and evaluate agent-generated trajectories, while Explorer (Pahuja et al. 2025) prompts LLMs to propose and refine tasks, performing screenshol-augmentee eval luation of agent trajectories. Although we share the scaling focus with these works, their evaluation frameworks primarily rely on the heuristic capabilities of evaluator models to assess trajectories based on natural language requirements, which limits their effectiveness for complex tasks. In contrast, we introduce a constraint-based frame- work for fine-grained evaluation, enabling our pipeline to collect high-quality trajectories for challenging tasks.
5 Conclusion
In conclusion, we presented a scalable, automatic data generation and distillation pipeline for developing proficient small language model agents. We introduce a novel automatic evaluation approach for reliably assessing agent trajectories. Using these metrics, we demonstrate how partial trajectories can be effectively leveraged to scale up high-quality training data. Our trained small language model agent, with 24B parameters, outperforms most prior approaches based on much larger closed-source foundation models, both on new benchmark we propose and the Webvoyager benchmark. We also contribute novel benchmark focused on challenging booking tasks, which we hope will be valuable in advancing the development of agents for complex web tasks. Directions for further improvement of our work include extending support to multimodal observations and broadening the scope to other platforms and environments beyond the web.
10

---

Published as a conference paper at COLM 2025
References
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXin:2407.13032 2024.
Anthropic.
URL Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024.
htww::/www.oopoc.ccccccc//////sa wttp:///....o////////////-/5 mode and computer-use
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. Advances in Neural Information Processing Systems, 11888091-21111 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. Xiviv preprint arXiv:2401.13919 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwe- barena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13644 2024.
Xing Han LU, Zdeněk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. ar preprint arXiv:2402.055) 00w,2200,.,9,,,,, 2024.
Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, and Giovanni Campagna. Wilbur: Adaptive in-context learning for robust and accurate web agents. arXiv preprint arXiv:2404.05002, 2024.
Magnus Müller and Gregor Zunič. Browser use: Enable ai to control your browser, 2024.
URL /tt wttp:////.w.oo.....ss///////seee
OpenAl, Operator: Autonomous web task automation agent, 2025. URL hnttps://w/
opwnaa..co/ htt.c////nnneeiiitnouuiinnggonee operator/ Research preview of Computer-Using Agent (CUA) model for web-based tasks.
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah. Explorer: Scaling exploration-drrven web trajectory synthesis for multimodal web agents. arviv preprint ar arXie:2502.1135) 2025.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. ar preprint ar orXio2244.00677 2024a.
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arviv preprint ar 4rXi::2406.122 12373, 2024b.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. Xiv preprint ar Xiv::25001. 12326,, 2025.
Jing Shi, Yuanbin Yao, Yiheng Jiang, Jialu Yin, Aditi Raghunathan, and Percy Liang. Visual WebArena: Evaluating large language models in web navigation and interaction. In International Conference on Learning Representations, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma technical report. arXiv preprint ar arXit:2503.199866 2025.
Mistral AI Team. Mistral small 3: A latency-optimizedd 24b aab-paraeeter model, 2025, URL https://mistral..
Accessed: 2025-03-28.
11

---

Published as a conference paper at COLM 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXio preprint arXto:2302.13977, 2023.
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards internetsccale training for agents. arXiv preprint arXiv:2502.06776, 2025.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards
scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. aiv preprint ar Xriv24401.01614 2024.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXio:2307.13854 2023.
12

---

Published as a conference paper at COLM 2025
A Appendix
I an trying to generate examples of realistic tasks that custoners night ask their assistents complete on the internet.
You nust create tasks that adhere the specific categories of tasks as expressed by these examples: (examples)
Each task instruction should fulfill the following criterla:
(1) Precision: should be sufficiently precise for the AI assistant complete the task without having to ask followu questions.
of tasks DO NOT nake every task related to adding an iten to shopping cart (at nost, cort-related tasks should be 58% tasks) searching using search har.
(2) Diversity: The
should
don't
all same. instance,
(3) Realism: The tasks should be realistic. Think about what our hunan custoner base night actually try to acconplish using our AI assistant.
(4) Complexity: The task should very simple for easy difficulty, then stale for harder difficulties by
adding addit additional constraints to the task.
instance, task regards potential purchase, it should always conclude by adding it to the cart identifying the booking rather than actually purchasing it.
(5)
spend noney in order complete
assistant no access to paynent nethods (e.g., credit cards). (6) Fresential-Free: The AI assistant should not need access our custoners' personal data complete the task. So, for instance. filling out forn that requires user's 's name or address, or logging into real online account, is off limits.
(?) Trajectory-Dentric Tasks should not require the AT ataistant to report back on results. We have a separate model that extracts the correct information from the webpages that visited. require the assistant to perforn the correct actions on the web necessary to yield the correct webpage (e-g.r visiting webpage containing the correct information) to elicit the right results (e.g., adding it to the cart) fron website.
should only
(8) HlihhLLeee:: Tasks should not include low-level details (e.g., "filter to show only refr igerators with water dispensers") because it difficult know GUI interactions filters might be laai website. Instead, tasks should be high-level, expressing a goal that can be accomplished by methods determined assistant.
particular
(9) Dated: For any task involving booking date, the date should be between one to four months after (current_dates Tasks should always be expressed using that fornat (1.e., Month DD, YYY such "December 17, 2825")
(10) Relevance: Focus on the core tasks available an each website using your specific knowledgee of that site and the functionalities available it, such UI features and filaring and search options.
job will generate (nur) instructions have already generated the following list of task instructions for this website. Please nake sure that your new task instructions are unique (i.e., they should be different from this list). [iv_tuss__istruutiioss
difficulty level for website (website).
Based the above criteria, and using the examples inspiration, please generate (nun) task instructions for each difficulty level for the website (website). Please separate each task by a single carriage return ONLY.
Figure 3: Task generation prompt template used to generate diverse tasks for each website. The template includes specific criteria for task generation and ensures consistency across different difficulty levels while maintaining website-specific rules.
13

---

Published as a conference paper at COLM 2025
You are an autonomous intelligent agent tasked with navigating a web browser. You will be gen web-based tasks. of specific actions you
These tasks will accomplished through
information you'll have:
The user's objective: This is the task you're trying complete.
The current web page's accessibility tree: This is simplified representation the webpage, providing key infornation.
the elements in the
web
the format [id] [taglype] [text content]
[(optional) properties] where tagType is the type of the element (e.E., button, link), text content is the text content
field which example, [1234] [button] [Add to Cart] neans that there is button with id 1234 and text content 'Add to Cart' page.
element. properties
properties
The current web page's URL: This is the page you're currently navigating.
The open tabs: These are the tabs you have open.
The previous reasonsings: These the reasonings for taking corresponding actions.
The previous action: These are the actions you have performed. It may be helpful to track your progress.
The actions can perforn fall into several categories:
Page Operation Actions:
"'click [id]'"; This action clicks on an element with a specific id on the webpage.
this type the pressed after typing unless press eeneeeeteereeeee _atter is set to 8, 1.e., ""type [id] [content] [0]
"' "type [id] [content]"
" "";
into the field with default, the "Enter"
.. [id]'"
over elenent
URL Navigation Actions:
" gota [url]"': Navigate specific URL.
go_back"" Navigate to the previously viewed page.
go, "oofoorard""": Navigate the page previous backback' action performed).
Completion
"ssoop [answer]""; ": Issue this action when helieve the task is complete. If the objective is to find text-based answer, provide the answer the bracket.
To be successful, it is very important to follow the following rules:
valid 2. You should only issue one action at a time.
only
observation
examples to reason stop by step Generate the action the correct fornat. Start with "In sunmary, the next action 1 will perform is" phrase. followed by action inside
For example, "In summary, the next action will perforn is "'cllck
[1234]'**..
5. Issue stop action when you think you have achieved the objective. Don't generate anything after stop. entire
scroll
page given.
[1744] [link] [HP CB782ARABA 640 Inkjet Fax Machine (Renewed)]
[1749] [StaticText] [$279.49]
[1757] [button] [Add Cart]
[1768] [button] [Add to Wish List]
[1761] [button]
Conpare]
URL: http:///onestopaarkt.ccoo cttp///ffiic--prrreee//affiaaeae
OBJECTIVE: What
price Inkjet Machine? None
PREVIOUS REASONINGS:
PREVIOUS ACTION: None
Let's think step-by-step. This page 1ist the information of HP Inkjet Fax Machine, which is the product identified the objective. price $279.49. action with the answer. In summary, the next action will porforn is """stop [$279.49]
achieved objective,
stop
Figure 4: Prompt used for few-shot agent.
14

---

Published as a conference paper at COLM 2025
You are an expert in assessing the performance web movi gaigiatin agent, whose role is to help a hunan user navigate website complete task. You will
the following information:
user's 's intent:
is
agent trying to complete
Constraints: These are the constraints the agent needs to fulfill in order complete this task The screenshot of the final webpage the agent's task execution
The URL of the final webpage the agent's task execution
evaluate agent's trajectory based the following guidelines, Please reason step by step before giving the final judgement. Take close look at the screenshot of the final webpage and URL to see whether it neets user's requi rement. For each constraint that given, agent; be considered met, the correct value constraint must be either explicitly present on the page or the URL.
Based this infornation, your task
the process described below to check whether the constraint net by the
Note that the constraint 'nade_selection' corresponds to locating one specific listing. this constraint is results page end; dedicated to one specific listing.
agent
met, the
page
the value particular constraint cannot determined based on the page screenshot, check the URL to see if the information present.
Output your ovaluation valid isan format shown the example below, where, for each constraint: "pround_truth" represents the the value of the constraint given to you
"agent_state"" observed
agent's final webpage
"natching" represents whether the constraint is net (trus/false) in the agent 's final webpage, based whether agent_state natches ground_truth
nade_selection': Fround__truth'. true. "egent_state": "true", "natching": "true",
locationn: { "ground_trutte truth": "Aspen, CD", "agent_state": "Aspen", "natching": "true".
b. "unit_type":
"ground_truth" truth": "vacation rental", "agent_state": "retreat house", "natching": "true",
start_date"
ground_truthi "January 88, 2025', "agent_state": "January 11, 2025", "natching": "false",
"end_date": (
"ground_ trundt:: "January 2825". staeet_statt:: "Janwary 12, 2025". "matching": "true",
Figure 5: Prompt used for constraint evaluation with screenshot observations and VLM judge.
15

---

Published as a conference paper at COLM 2025
You are an expert in assessing the performance web movi povigation agent, whose role is to help a hunan user the following information:
navigate website complete task. You will
is Constraints: These are the constraints the agent needs to fulfill in order complete this task
user's intent:
agent trying to complete
The accessibility tree of the final webpage the agent's task execution the agent's task execution
The URL of the final
evaluate agent's trajectory based the following guidelines. Please reason step by step before giving the final judgement. Take close look at the accessibility tree of the URL to see whether it neets user's requitenent.
Based this infornation, your task
final webpage
For each constraint that given, the process described below to check whether the constraint net by the agent; be considered met, the correct value constraint must be either explicitly present on the page or the URL.
Note that the constraint 'nade_selection' corresponds to locating one specific listing. this constraint is results page end;
agent is dedicated to one specific listing.
met, the
page
particular constraint cannot determined based on the page accessibility tree, check the URL to see if the information present.
the value
Output your ovaluation valid jsan format shown the example below, where, for each constraint: "pround_truth" represents the the value of the constraint given to you
observed "natching" represents whether the constraint is net (trus/false) in the agent 's final webpage, based whether agent_state natches ground_truth
"agent_statee"
agent's final webpage
nade_selection': Fround__truth'. true. "egent_state": "true", "natching": "true",
locationn: { "ground_trutte truth": "Aspen, CD", "agent_state": "Aspen", "natching": "true".
b. "unit_type":
"ground_truth" truth": "vacation rental", "agent_state": "retreat house", "natching": "true",
"stat_ date""
ground_truthi "January 88, 2025', "agent_state": "January 11, 2025", "natching": "false",
"end_date": (
"ground_ trunhh: "January 2825". staeet_statt:: "Janwary 12, 2025". "matching": "true",
Figure 6: Prompt used for constraint evaluation with accessibility tree text observations. and LLM judge.
16

---

Published as a conference paper at COLM 2025
You expert in assessing the performance web navi navigation agent, whose role help hunan navigate website to complete a task. You will be given the following information: intent:
agent trying complete
given intent, identify key constraints vacasa. f'nn vacation rental in Aspen, 00 for January 88, 2025, January 12, 2025, that accomoddaes 8 guests and has ski-in/ski-out access." nentions the following constraints: made_selection. location, rental end
example,
type,
number of guests, ski-in/ski-out If the intent involves locating one specific listing, and the intent does not use the word search, then create unmet
search results at the end; for it to be net, the agent MIST be on page dedicated to one specific listing. You must abolutely not forget to include made_selection constraint
constraint called
selection'
agent
cases. in doubt, include it.
Here's an exapple.
0BJECTIVE: Locate an apartnent in Tokyo, Japan, available on January 30, 2025.
CONSTRAINTS:
madn_selection True
location: Tokyo, Japan
date: January 30, 2025
locate_apartnng iment: true
Figure 7: Prompt used for constraint extraction.
You intelligent web agent. Given the following information, your task to predict the action. Task: This is the task the agent is trying to complete.
Previous
previous actions the agent perforned. webpage.
URL: The URL of the current webpage
Page accessibility
accessibility representation
Task: <task>
Previous Actions: <past_actions>
URL: <url>
Page accessibility <page_
Figure 8: Prompt used for finetuning.
https:/ https://ww...
https: www.honeaway.ccommmmmm
https:// h/apps://wM.... http://w...oleeeeeee
https:// /tw.://ww......./// com
https:// //MAM. //w.. www.romezrio.com
https:// //w/w. www.orbitz.com
https:/ //swwe/ www.hiilton.com
https:/ // htw.://////wwb.ooo.gg..o.o
www.bbgg.cc.ooooo con
https:/ www.aibrbb.coo
https: /smww. traveloci to.ciyy..coo..o.o.
https::/
plunguide.com
https: /ww.. wtt.///......... ihg..com
https: http:://wwww.hehhm www..hotels.coo con
https:/ //w0:s////.....///
https://ww... www.dddde.coo .on
https:// /ww.wredddww.nnggggggmmmmm.
https:/ //w..///......... www.rradsssorhoteoo
https:/ momondo,
https: //MDs..///MM..////// http://ww.nnggggee mwg.mus..co
https: //www. www.k.yakkcoo
Figure 9: Websites used for evaluation.
17


Reviews:
[{"review_id": "TWhGH6DQ92", "reviewer": "Reviewer_mfjm", "summary": "The paper proposes a method for synthesizing trajectories for training a web agent. Given a set of websites, prompted LLMs/VLMs are used to generate tasks, derive **target constraints** for the tasks (e.g. if the task asks to book a trip on April 8, one constraint would be “start_date: April 8”), sample trajectories by interacting with the websites, and score the trajectories against the constraints. The high-quality trajectories are used to LoRA-tune a Mistral 24B model. \n\nEvaluation is done on a new benchmark BookingArena, which focuses on the travel booking task, and [WebVoyager](https://aclanthology.org/2024.acl-long.371/), which are mostly search tasks. The generated target constraints are used to compute partial credits during evaluation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **The software, training trajectories, and the BookingArena dataset** will be beneficial to other researchers. These are high-quality resources that would be cumbersome or expensive to generate.\n\n2. **The proposed method is reasonable.** While the use of automatically generated constraints for validating model outputs is a known technique (e.g. in natural language generation), the application here is well-justified. It also allows partial credits based on intermediate steps in the trajectory that might not be present in the final task output (e.g. whether the model successfully clicked a page transition button does not appear in the final POST request).\n\n3. The method empirically **outperforms reasonable baselines** (though see Weakness 1 below).", "reasons_to_reject": "1. The comparison against the baselines might be **a bit unfair, as the proposed method is trained on a large amount of trajectories** (16k complete trajectories and 65k prefixes). This would be even more severe if **the test tasks have large overlap with the training tasks,** which seems to be the case if I am not missing something. The model would roughly know how to solve the task in the same distribution from the training trajectories. Granted, the closed-source baselines were probably also trained on a lot of web tasks (though in a different distribution), and the proposed method also works well on WebVoyager which is an existing dataset.\n\n2. There are some parts that are missing details. Please see the “Questions To Authors” section below.\n\n3. There is not a lot of **qualitative success and error analysis** of the models’ behaviors. For example, are there specific behaviors that the training trajectories help with? How well can the model generalize to tasks that are different from training tasks? The ablation runs in Table 2 could be a good source for the analysis\n\n4. Using a bot to *intensively* interact with websites may **violate their terms of service.** Other web agent works try to sidestep this by working on local websites (e.g. [WebShop](https://proceedings.neurips.cc/paper_files/paper/2022/hash/82ad13ec01f9fe44c01cb91814fd7b8c-Abstract-Conference.html)) or, for companies, working with their product partners. This work scraped the top 1000 websites and even got blocked by some.", "questions_to_authors": "1. What is the format of the BookingArena benchmark? Does it include local copies of the websites, or does it require interacting with actual websites. If actual websites are required, could the dataset become broken if the website changes?\n\n2. Could the author give some statistics about the BookingArena benchmark? For example, do the tasks require navigating to different pages? How many required steps or target constraints are there on average? How diverse are the tasks?\n\n3. Line 255 mentions human evaluation of the automatic evaluation method. Would it be possible to elaborate on this? What is the protocol, and what are the concrete statistics?\n\n4. Why does the model fail completely on Radisson (Table 1) while the baselines get some partial rewards?\n\n**Other suggestions and typos:**\n\n* Lines 27-31: These are redundant with the previous paragraph.\n* Lines 118-124: These could be compressed to something like “we average the scores” without having to write the full formulas.\n* Line 244: “leads to leads to” → “leads to”\n* Lines 257-267: These should probably be moved under Section 3.4.", "extra_scores": {}}, {"review_id": "WmLAiiLqBl", "reviewer": "Reviewer_3mSW", "summary": "This paper introduces a pipeline for training web agents by automatically generating and curating high-quality training data. Though certain parts of the method, such as generating tasks using LLMs and exploration, have been investigated in prior works, the proposed constraint-based evaluation framework to filter out low-quality trajectories and also provide a way of keeping partially successful trajectories is effective and interesting. CSR, the metric used for filtering trajectories, is also extendable for downstream evaluation. The paper also proposes a new benchmark, BookingArena, featuring complex booking tasks on 20 popular websites, and demonstrates that a relatively small model with only 24B parameters, when distilled, outperforms existing open-source models and matches or exceeds the performance of closed-source models. The approach is also validated on WebVoyager.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The constraint-based CSR metric is effective in filtering out trajectories for training using web agent data and helps in significantly increasing the size of the dataset. It is also a significant improvement over binary success/failure metrics generally used for evaluation, providing more granular insights into model performance that can also be used to improve them further.\n2. The proposed method obtains competitive performance when compared to both open-source and closed-source models on BookingArena and WebVoyager.\n3.  The BookingArena benchmark is another valuable contribution for diversifying web agent evaluation.", "reasons_to_reject": "1. I believe that there are numerous steps where the pipeline relies on LLMs. I find it necessary to conduct a corresponding human study (even on a subset) to gauge the agreement of the proposed approach with human annotations.\n2. While the method is described as scalable, the paper does not provide a detailed analysis of the computational cost associated with generating and evaluating large-scale trajectories. Per-step evaluation requires separate LLM/VLM calls that add up depending on the complexity of the task generated.", "questions_to_authors": "1. Can you provide a list of example tasks generated by the LLM for generating the training data (the prompt in Fig 12 also lacks concrete few-shot examples but only mentions them while discussing the constraints) and also the distribution of the final generated dataset (300k action/obs) across the 1000 websites used? I am curious to see the intersection with booking websites in the training data used for the student models.\n2. Can you provide more details about the task creation in BookingArena? It seems that the content is missing crucial details in this regard.", "extra_scores": {}}, {"review_id": "ORwPl6QOIv", "reviewer": "Reviewer_puzT", "summary": "This paper aims to build a high-quality data generation (more precisely, selection) pipeline to improve web agent training, by proposing fine-grained evaluations on the progress made towards task completion, instead of sparse binary rewards. Experiments on self-created BookingArena benchmark demonstrate the effectiveness of the proposed method in distilling better student models out of data synthesized by the teacher model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Valid Solution for the Important Evaluation Issue**: This paper tackles a pressing issue for web agents -- automatic evaluation, by measuring a trajectory's progress towards success instead of a binary 0/1 reward. The method (by breaking the tasks down into multiple constraints) is reasonable, and if producing reasonably valid scores, can be utilized to better train agent policy/reward models.\n\n2. **Effectiveness of the Proposed Method** Experiment results demonstrate the big improvements brought by the proposed method, demonstrating its effectiveness in yielding better student agent models.", "reasons_to_reject": "1. **Significance of Results?** As there are only 120 self-created tasks in the experimented BookingArena dataset (what makes it worse is that each website only has 6 tasks? which is far from a good coverage of different tasks and contexts), it is unclear if the performance differences between methods are statistically significant, and thus make it the validity of conclusions questionable.\n\n2. **Reliance on Multi-Faceted Queries** This work creates special tasks that involve multiple constraints in travel booking scenarios, which is a relatively limited scenario, if considering the numerous websites and varied tasks one could do on the web. It is unclear if this method (particularly constraint-based evaluation) could be similarly applied to benefit (i) simple tasks, (ii) long-horizon tasks, e.g., with long procedures but not many constraints, or other common scenarios.\n\n3. **Limitation: Performance Bounded by Teacher Model** The proposed method is essentially selecting high-quality data synthesized by an existing model, which means the to-be-trained student model is bounded by the teacher's performance. This method is more of a better data selection (for better model distillation), than advancing the upper-bound of top-performing agents.", "questions_to_authors": "1. Wrong template? At least the font is off.\n\n2. For the “Constraint Framework”, since it’s using LLM, how to make sure that the set of constraints is stable (for similar tasks ,the constraints are similar) and comprehensive (compared to humans to come up with the constraint, how does LLM perform)?\n\n3. Based on CSR, do you only discard intermediate steps that fail? How about the following steps?", "extra_scores": {}}, {"review_id": "MknECOKFqs", "reviewer": "Reviewer_3xdz", "summary": "The paper proposes a novel training strategy for fine-tuning smaller language models to be used as browser (web) agents. In particular, they propose an automatic data collection strategy using a novel evaluation regime based on assessing trajectory over multiple independent constraints, automatically extracted from the agent's goal. The automatically obtained data is then used to fine-tune an LLM, achieving performance competitive or above that of significantly larger models, without additional manually annotated training data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While the idea of decomposing LLM judges into constraints is not new (see for example the survey by Gu et al at https://arxiv.org/abs/2411.15594), it's application to web agents is both interesting and non-trivial, as it is made possible by the novel automatic constraint decomposition from the agent goal.\nAdditionally, the use constraint decomposition seems very powerful, as it allows to use partial trajectories where the agent succeeded up to a point, yet failed the overall task. This process greatly increases the amount of training data that can be obtained even with a relatively weaker teacher. The paper further improves on this with the idea of retroactively adjusting the goal based on whatever partial trajectory the agent achieves.\n\nFinally, the proposed benchmark can be useful to the community, as it uses very challenging tasks (booking tasks are among the hardest in WebVoyager) but it also considers more websites than tasks, allowing to measure generality across websites.", "reasons_to_reject": "My main concern with the paper is that the method relies on decomposing the goal into independent constraints.\nWhile this seems plausible in search and transaction tasks such as booking tasks, it is less clear how possible this is in general, especially with goals that do not provide obviously extractable slot values, that provide soft constraints, or that require alternative reasoning (\"if X is possible I want X\", \"either X or Y\", etc). These situations do not appear in WebVoyager, and I would love to know if these were explored or considered in designing BookingArena. If not, additional evaluation beyond booking tasks might be necessary.\n\nRelated, there is very little discussion or evaluation of the constraint extraction itself. It is not clear if the method relies on defining some sort of task-specific ontology (slot names), or the LLM can infer reasonable constraints itself. It would also be very helpful to see whether the LLM can extract accurate constraints, especially across different tasks in WebVoyager.\n\nSecondarily, the use of partial trajectories seem to provide only a small boost in performance (Table 2), despite a much larger training set. I would love to see additional ablations investigating which partial trajectories are useful, and also specifically whether the idea of retroactively rewriting goals to match the agent trajectory has any measurable impact.", "questions_to_authors": "1. I don't fully understand the experiment in Section 3.3. Is there a ground truth value for the evaluator? Why should we assume that an evaluator that assess the underlying model as correct more often is itself more correct than a harsher one?\n\n2. Related, I think the text in lines 257-267 should be in Section 3.4\n\n3. Did you consider using the same model to both generate trajectories and fine-tune? What would you expect in that case?\n\n4. WebVoyager uses screenshots from the last 15 steps of the agent for evaluation, while BookingArena only uses the final screenshot (line 201). Why?\n\n5. The prompt in Fig. 7 is booking specific. What did you use for other WebVoyager tasks?\n\n6. It seems the paper is using a different font than the COLM template. Not sure if this was intentional.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a constraint-driven pipeline that automatically generates and filters web-booking trajectories, then distills a 24B student that matches or outperforms a much larger teacher. All four reviewers leaned positive. They converge on strengths (clever constraint-based evaluation and a useful booking benchmark) and raise similar worries about evaluation rigor (train/test overlap, small task set, LLM reliance, compute). Author rebuttal helped clarify some of the overlap/compute doubts, but statistical-significance and “soft-constraints” remain partly open. 

Strengths
1. The idea is novel. Fine-grained constraint evaluation can unlocks partial-trajectory reuse (mfjm, 3mSW, puzT, 3xdz)
2. The paper showed convincing performance (mfjm, 3xdz)
3. BookingArena benchmark will be a valuable resource (mfjm, 3mSW, 3xdz)
4. Reviewer mentioned that the authors had clear instructions and scripts in the opensource code (mfjm)

Weaknesses
1. Possible train–test leakage and baseline fairness (mfjm). The rebuttal provides overlap statistics) and additional ablations which addressed this.
2. Transparency about compute cost (3mSW). Rebuttal addressed by giving GPU counts & throughput 
3. Limited statistical power on a 120-task benchmark (puzT) – authors doubled tasks but Reviewer suggested adding significance tests.
4. Reviewers were a bit concerned that the method is on narrow domains, with unclear transfer to simpler or “soft-constraint” tasks (puzT, 3xdz). Authors argue that method generalizes, but reviewers are still looking for concrete evidence beyond booking flows. They suggested some cross-domain experiments.
5. Reviewers asked about the reliance on LLM-extracted constraints (3mSW, 3xdz). Authors report a reasonable human-LLM agreement but reviewers encourages the authors to also add a discussion of failure modes and possible mitigations.

Overall, I think the paper makes valuable contributions, and would encourage the authors to make corresponding revisions as suggested by the reviewers.
Decision: Accept