AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Authors: Xing Han Lù
Amirhossein Kazemnejad
Nicholas Meade
Arkil Patel
Dongchan Shin
Alejandra Zambrano
Karolina Stanczak
Peter Shaw
Christopher Pal
Siva Reddy
Venue: COLM 2025 (2025)
Keywords: Agent
Web Agent
LLM Judge
LLM-as-a-Judge
Digital Agent
Benchmark
Reward Modeling
TL;DR: AgentRewardBench is a benchmark that evaluates how well Large Language Model (LLM) judges align with human preferences when assessing autonomous web agents, using over 1,200 expert-reviewed trajectories across five benchmarks

Abstract: Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determine whether the agent successfully completed the tasks. Rule-based methods are widely used for this purpose, but they are challenging to extend to new tasks and may not always recognize successful trajectories. We may achieve higher accuracy through human evaluation, but the process would be substantially slower and more expensive. Automatic evaluations with LLMs may avoid the challenges of designing new rules and manually annotating trajectories, enabling faster and cost-effective evaluation. However, it is unclear how effective they are at evaluating web agents. To this end, we propose AgentRewardBench, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AgentRewardBench contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in AgentRewardBench is reviewed by an expert, who answers questions pertaining to the success, side effects, and repetitiveness of the agent. Using our benchmark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents, highlighting a key weakness of rule-based evaluation and the need to develop more flexible automatic evaluations. We release the benchmark at: https://agent-reward-bench.github.io

Published as a conference paper at COLM 2025
AGENTREWARDBENCE Evaluating Automatic Evaluations of Web Agent Trajectories
Xing Han Lu 12 Amirhossein Kazemnejad 22
Nicholas Meade 12 Arkil Patel 12 Dongchan Shin 2 Alejandra Zambrano Karolina Stanczak
Peter Shaw Christopher J. Pa12567 Siva Reddy 1257
"Core contributor McGill University Mila Quebec Al Institute +Google DeepMind
SCanada CIFAR AI Chair 'Polytechnqque Montréal 7ServiceNow Research xing. wtw.hann..e... lu@mail. mccglll ca; siva. reddy@mi la quebec
Abstract
Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determine whether the agent successfully com- pleted the tasks. Rule-based methods are widely used for this purpose, but they are challenging to extend to new tasks and may not always recognize successful trajectories. We may achieve higher accuracy through human evaluation, but the process would be substantially slower and more ex- pensive. Automatic evaluations with LLMs may avoid the challenges of designing new rules and manually annotating trajectories, enabling faster and cost-effective evaluation. However, it is unclear how effective they are at evaluating web agents. To this end, we propose AGENT BENCH, the first benchmark to assess the effectiveness of LLM judges for evaluating web agents. AGENTREWARD contains 1302 trajectories across 5 benchmarks and LLMs. Each trajectory in AGENTREWARD- BENCH is reviewed by an expert, who answers questions pertaining to the success, side effects, and repetitiveness of the agent. Using our bench- mark, we evaluate 12 LLM judges and find that no single LLM excels across all benchmarks. We also find that the rule-based evaluation used by common benchmarks tends to underreport the success rate of web agents, highlighting key weakness of rule-based evaluation and the need de- velop more flexible automatic evaluations. We release the benchmark at: hatps:////gtttewarreeee
gi thub.
1 Introduction
Giving a Large Language Model (LLM) access to web browser unlocks entirely new capability paradigm: beyond interacting with a user through a chat interface, such models can interact with the online world to complete tasks similar how human would. The promise of a new paradigm has motivated the design of LLMs to control interfaces such as web browsers, starting from earlier foundation models such as ACT-1 (Adept, 2022) to the more recent OpenAl Operator (OpenAI, 2025) and Claude Computer use (Anthropic, 2024a), showing promising results in real-world tasks (Zhou al., 2024).
To measure the progress of web agents, well-designed benchmark should compile collection of realistic tasks across diverse websites. As illustrated in Figure 1, a user may ask the agent to locate Classifieds listing for Google Pixel phone and submit an offer via a comment. Inside a dedicated environment (e.g., a self-hosted Classifieds site), the web agent would complete the task by filling the search bar, identifying the correct listing, and writing a comment to show interest in purchasing the item. To determine the agent successfully completed the request, we need to automatically evaluate the agent's chosen actions - known as trajectories using a set of rules uniquely designed for the task of finding Pixel phone on Classifieds. As expected, rule-based evaluation is time-consuming for
1

---

Published as a conference paper at COLM 2025
Web Agent (e.g. GPT-4o)
fill"white Google Pixel")
click(Search)
PPwel phone listing)
Find the latest listing of white Google Pixel phone and post comment offering $10 less than their asking price.
fill 'I am interested in purchasing this phone.
click(Send)
Task Complsted
Successful
Successful
Thought: analysa agent's porformance .. listing shows a whita Google Fizel 3XL priced at $260 00, which matches the search crillrna.
tfaLLM Judge (e.g. Claude 3.7)
Human
Side Effect No Repetition
Example from AGENTREWARDBENCH, where an LLM judge evaluates a web agent's trajectory on VisualWebArena (Koh et al., 2024), The benchmark compares judgments against expert annotations to determine the effectiveness of the judge for evaluating web agents.
Figure/ 1:
experts to design, and may not cover every successful scenario (e.g,, what if the agent finds different but valid listing?). It is also possible for an expert to annotate the trajectories, but it would slow and expensive to scale across many web agents. This brings us to the following questions: Given web agent trajectory, can LLM decide if it is successful? If so, how do we determine which LLM the most capable at evaluating web agents?
Past works have shown that LLMs can be used judges to evaluate the output of LLM chatbots (Zheng et al., 2023). More recently, LLM judges have been used for automatically evaluating trajectories from web agents (Pan et al., 2024; Murty et al., 2025; Trabucco al., 2025). With highly accurate automatic evaluation methods, we can measure the progress of web agents on new sets of tasks, use them to synthesize trajectories for finetuning smaller models, and design reward models that can be used in a reinforcement learning (RL) setting. However, it remains unclear whether current automatic evaluators, whether rule-based or LLM-based, can predict the success of a trajectory in a way that reflects expert judgment.
To address this problem, we introduce AGENTREWARDBENCH ($3), benchmark for deter- mining the capability of an LLM at evaluating web agents (see Figure 1). It consists of 1300 trajectories produced by popular LLM agents on 5 diverse web environments, ranging from common tasks like online shopping and posting on forum, to highly specialized requests professional environments, such as updating task schedules on IT task manage- ment platforms. Each trajectory is labeled by expert annotators to determine whether the agent successfully completed the task, caused unintended side effects, or entered cycles of repetitive actions. Using this benchmark, we evaluate both existing and novel LLM judges (54) alongside rule-based evaluation. We find that rule-based methods, which are used as the official automatic evaluation by environment-based benchmarks, severely underestimate the capabilities of agents and do not reflect how experts define success ($5). We further provide in-depth analysis (56) that highlights the weaknesses of existing LLMs when used as judges. Overall, we believe AGENTREWARD can be used to enable better automatic evaluation and reward modeling for web agents.
2 Related Works
Web Agents and Environments Designing agents that can automatically navigate user interfaces has been long standing problem; earlier approaches employed program-based heuristics (St. Amant & Zettlemoyer, 2000), whereas later works on web navigation focus training reinforcement learning (RL) models (Gur et al., 2018; Humphreys et al., 2022),
2

---

Published as a conference paper at COLM 2025
language models (Nakano et al., 2021; Gur et al., 2023; Deng et al., 2023) and multimodal models (Shaw et al., 2023; Lü et al., 2024; Zheng et al., 2024). To measure the advancements in web agents, various benchmarks have been proposed, with initial works proposing simplified environments (Shi et al., 2017; Liu et al., 2018) and subsequent iterations focusing specific tasks like web shopping (Yao et al., 2022). More recent benchmarks focus designing realistic environments that cover commonly used websites (Zhou et al., 2024; Koh et al., 2024) well as specialized environments (Drouin et al., 2024; Boisvert et al., 2025).
LLM Judges Zheng et al. (2023) proposed using LLMs predict human preferences of dialogue completion for chat models. They show that a GPF-4-based judge achieves over 80% agreement with human votes on the task of selecting better completions between models pairs. Subsequently, Agent-as-a-Judge (Aal) was proposed by Zhuge et al. (2024) to extend LLM judges to coding agents, leveraging intermediate feedback from the environment. He et al. (2024) extend the idea by using LLMs to judge trajectories from web agents, allowing them to determine task completion without human annotators, resulting in a high correlation with humans on a private subset of trajectories. To determine the quality of automatic judgments, Pan et al. (2024) evaluate four LLM judges using trajectories from GPT-4 agent on WebArena tasks, and find that the best judge achieves 80.6% accuracy against the rule-based evaluator from WebArena. Unlike prior works on LLM judges, we design AGENT BENNCHHH with trajectories from several LLM agents on diverse web benchmarks, where each one is annotated by human experts on multiple dimensions. By following a human-focused approach similar to Lambert et al. (2024), we ensure that LLM judges are evaluated against expert preferences on wide range of scenarios.
Trajectory Synthesis Leveraging web environments that can be created and reset without real-world impact, recent works started to explore generating trajectories without human supervision. Leveraging LLM judges and LLM-generated tasks, trajectory synthesis can be used to bootstrap agent-judge training loops (Murty et al., 2024; 2025), to create contrastive pairs (Putta et al., 2024) for direct preference optimization (Rafailov et al., 2023), as training data to finetune a base model (Lai et al., 2024; Patel et al., 2024; Trabucco et al., 2025). Although all the methods leverage an LLM judge, they lack clear way of directly determining the quality of judgments, instead relying on the downstream performance improvement to validate their approach. To this end, AGENTREWARD researchers to choose the most appropriate LLM judge for category of web tasks based on their effectiveness at evaluating web agents.
enables
3 AGENTREWARDBENCH
In this work, we introduce ACENTREWARDBENCC, a benchmark designed to assess the capabilities of LLM judges for evaluating web agents ($3.1). We curate diverse web environments and tasks ($3.2) in order to collect trajectories from web agents based on 4 LLMs (53.3). For each trajectory, a team of expert annotators carefully reviews the screen- shots, actions, and the agent's reasoning chains before labeling them as either successful or unsuccessful, alongside other auxiliary labels (see Figure 2). Finally, we evaluate LLM judges (Table 1) by com comparing their predictions with expert annotations to determine their effectiveness for automatic evaluation.
3.1 Assessment Framework
Trajectory Definition Let o; be an observation of browser at time step i, a; be an action that can be executed on a webpage through browser navigation engine B such that 0++1 = B(0,,4)) and ri be the reasoning for choosing the action. We define web agent trajectory as the sequence T {o1, (r1 (1), (12,42)...., 0n-1 (fn-1, 4n-1) 1), 0n, where op is the final observation in the trajectory. Each observation contains screenshot of the browser s;, the Document Object Model (DOM) tree representation of the browser, and an accessibility (A11Y) tree rendered from the DOM tree. For the observation to be useful for an LLM agent, we need representation function R that produces p; R 71, ", ...,.,..,))) which can be used as an input for an LLM. If the agent is multimodal, o; would include screenshots; otherwise, it would be textual representation of the page (e-g., accessibility
3

---

Published as a conference paper at COLM 2025
Web Environment ($3.2)
Assessment Platform ($3.1)
Instructions
Human
Guidelines
Web Agent ($3.3)
Trajectories
Annotations
AgentRewardBench ($3)
Figure 2: AGENTREWARDBENCH creation process. We first collect trajectories from LLM agents inside web environments using instructions from several benchmarks. Then, the trajectories reviewed by expert annotators, who indicate if the trajectory is successful, led to side effects, and contains repetition cycles. Finally, we use the annotated trajectories to evaluate LLM judges.
tree). Then, pi is given language model to produce completion ci LM(p;), or = (LM(p;,s;) in the case of multimodal LLM. The completion is parsed by an execution function produce (a, r;) E(c:).
Annotation Design For each trajectory, an expert annotator reviews goal g and sequence
{sı, (f), Sn-1, (Tn-1,AR-1),$m in order to answer questions Q = {411..., 4m We consider the answers produced, A = (aj...., .ff I as the ground truth annotations for the trajectory, which indicate whether the agent successfully completed g. To collect A", the following Q in the annotation guidelines:
1. Success: Was the sequence of actions successful achieving the goal?
2. Side Effect: Did the agent perform unnecessary actions that could lead to unintended side effects?
3. Repetition Cycle: Did the agent loop through a sequence of actions that did not make progress towards the goal?
Agreement with respect to success is the primary criterion with which we evaluate LLM judges. The remaining can useful as auxiliary criteria for detecting issues ahead of time.
For example, if an agent purchases several irrelevant products when the user only requested one, then the trajectory would be flagged for side effects, independent of task success. A judge also indicate the presence of cycle, for example, if the agent repeatedly clicks a disabled button. Both signals can be used to penalize the agent during training or steer to another action at inference.
Annotation Setup The team of annotators consisted of 6 experts with deep understanding of the tasks and environments through their research on web agents. They used custom- built user interface that displays each trajectory with screenshots, actions, and reasoning. Rating disagreements were resolved by annotators discussing among themselves until clear annotations can be produced for ambiguous trajectories. Moreover, the annotators also have access to the environment and accessibility trees when screenshots are insufficient.
Judge Model Given goal g, trajectory T and questions Q, judge model returns judg- ment A, which an estimate of A*. We can use Á to derive reward in RL to automati- cally evaluate web agents when A is unavailable. To implement the judge, we need judge- specific function Rj that produces a representation of the trajectory, p Ry F1, #1, ..., 0n).
R,= can vary substantially, ranging from simple list of actions #j, ......,ae 0n-1 using another LLM process the observation history. We describe judges used in previous works and in- troduce simplified judge in Section 4 and provide supplementary details in Appendix A.3.
3.2 Tasks and Environments
We select benchmarks designed to evaluate web agents inside dedicated environments and real websites, including general-purpose (Zhou et al., 2024), vision-focused (Koh et al.,
4

---

Published as a conference paper at COLM 2025
2024), real-world (Yoran et al., 2024), and enterprise-oriented (Drouin et al., 2024; Boisvert al. 2025) tasks. In total, we curate 351 unique tasks across environments and 66 websites, which we separate into 51 development and 300 test tasks (details in Appendix A.1).
WebArena (WA; Zhou et al. 2024) This benchmark comprises 6 self-hosted websites covering a wide range of domains: customer relationship management, map navigation, online encyclopedia, shopping site, social forum, and software development collaboration platform. Each environment is derived from real open-source projects that develop self- hosted environments for both commercial and personal usage. Each task consists of textual goal that requires a good understanding of one or multiple environments to complete.
Aren (VWA; Koh et al. 2024) To complement WebArena's text-based goals, we also include VisualWebArena (VWA), benchmark focusing on tasks that require visual reasoning to complete. For instance, user may include an image alongside the goal, the task could be designed to only be solved if the agent selects an item with a unique visual characteristic (e.g., purchasing TV with the widest bezel). VWA also introduces new online marketplace environment (Classifieds).
AssistantBench (AB; Yoran et al. 2024) In addition to the self-hosted environments, we consider trajectories resulting from agent execution on real-world websites. This benchmark defines tasks that require navigating the internet, starting from search engine. Since the test set is private, we use the validation set, which consists of 33 unique tasks.
WorkArena (Work; Drouin et al. 2024) and WorkArena++ (Wk++; Boisvert et al. 2025) To increase the diversity of tasks relevant to professional environments, incorporate WorkArena (Boisvert et al., 2025), benchmark of 18 basic tasks on ServiceNow, software- as-a-service platform for professional workflows in the information technology (IT), human resources, and customer management domains. WorkArena++ introduces tasks with greater complexity, requiring planning and reasoning to correctly complete multiple sub-tasks, Including this alongside WorkArena allows us to evaluate judges on a wider range task difficulty. We focus on the Level 2 tasks since Level 3 is too challenging for current agents.
3.3
Web Agents Design
To collect trajectories on the 5 benchmarks, we design web agents using two models from major commercial providers and two open-weight LLMs.
LLM backbones On the commercial side, we use OpenAl's GPT-402 (Hurst et al., 2024) and Anthropic's Claude 3.7 Sonnet (Anthropic, 2024b). They are the flagship models of their respective providers, both of which offer computer-use agents powered by their LLMs, namely OpenAl Operator (OpenAI, 2025) and Anthropic Claude's Computer use (Anthropic, 2024a). We select two leading ween-wssghss LLMs to complement the commercial LLMss Llama-3.3-705 (Grattafiori et al., 2024) and Qwen2.5-VI. (Bai et al., 2025). In both cases, we choose the instruction-tuned variant, which have undergone post-training for tool-use or UI navigation. Moreover, since Llama-3.3 is text-only model, it was excluded from VisualWebArena, which requires image-based reasoning.
Agent Platform By default, each LLM backbone receives an input processed by a represen- tation function R and generates completion c;. Then, is interpreted as an action by execution function E. To implement E, we use AgentLab and BrowserGym (Chezelles et al., 2025), ecosystem for designing web agents using LLMs (details in Appendix A.1).
Trajectory Annotations and Splits We collect total of 1302 trajectories from our 4 LLM- based web agents across five benchmarks. Based on the task split ($3.2), 196 trajectories are in the development split and 1106 are in the test split (details in A.2). The annotators follow the process described in Section 3.1 to label all trajectories, producing a total of 3906 binary annotations. To assess agreement between annotators, we annotated the GPT-4o agent's
hhttps:/ httt:::///eeelope.: http://ddeoooprereevieeennn servicenww.cccmm com We the version gpt-40-2024-11-4
5

---

Published as a conference paper at COLM 2025
Overall Recall
AB
VWA
WA
Work
Wk++
Category
Judge
Precision
Precision
Official
Rude-based"
83.8
55.9
67.1
25.0
85.2
79.0
100.0
83.3
AER-C
67.7
71.9
69.7
83.3
56.0
68.8
100.0
66.7
Existing
AER-V
67.6
71.5
69.5
83.3
61.2
67.6
96.4
59.3
NNetNav
52.5
82.4
64.1
20.8
54.5
54.3
77.3
43.2
Claude 3.7 S.
68.8
81.6
74.7
87.5
61.0
69.3
85.0
66.7
GPT-4o
69.8
83.1
75.9
77.8
63.0
70.2
94.6
63.0
Ours (A)
GPT-4o Mini
61.5
86.1
71.7
80.0
57.9
63.5
84.2
49.4
Llama 3.3
67.7
79.0
72.9
75.0
59.6
68.2
94.3
62.7
Qwen2.5-VI.
64.3
89.8
75.0
72.7
59.3
63.6
87.2
60.3
Claude 3.7
69.4
76.3
72.7
71.4
64.8
69.3
85.3
66.7
GPT-4o
68.1
80.3
73.7
77.8
60.7
69.9
93.8
59.6
Ours (S)
GPT-4o Mini
64.5
78.3
70.8
80.0
57.4
66.9
90.3
54.8
Qwen2.5-VL
64.5
86.1
73.7
70.0
58.5
62.9
93.8
64.4
Table 1: Judge performance for predicting success, measured with precision (54.2). We report recall and FI as auxiliary scores. We examine two variants of the simplified judge: one with the final accessibility tree (A), and the other with the final screenshot (S). "Rule-based evaluation included for reference.
trajectory on WebArena with a second annotator. We obtained an inter-annotator agreement of 89.3% on success, indicating high level of consistency among annotators.
4 LLM judges for web tasks
4.1 Judge implementations
We consider two existing implementations of LLM judges for web agents, Agent Eval Refine (AER; Pan et al. 2024) and NNetNav (Murty et al., 2025), and introduce simplified judge that simultaneously predicts success, side effects, and repetition. In Agent-as-a-Iude (Zhuge et al., 2024), the method assumes the judge can interact with the environment after the agent finishes executing actions, which isn't feasible when the environment state cannot be preserved or shareable across agents. Other LLM judge variants were proposed (He et al., 2024; Putta et al., 2024; Lai et al., 2024; Trabucco et al., 2025), but our three judge implementations cover major strategies for representing trajectories.
AER (Pan et al., 2024) The judge in this framework takes as input the sequence of agent thoughts and actions alongside the final browser state, which is either passed vision- enabled model as screenshot (AER-V) or as a caption generated by captioner model (AER-C). Then, the judge outputs its reasoning before predicting success or failure. For both the judge and captioner, we implement this method using GPTT-4,, which is an overall stronger model than the GPT-4 (Achiam al., 2023) model originally used.
NNetNav (Murty et al., 2025) In this work, a Llama 3.1 70B judge receives a summary of changes across all observations and has to give rating between (worst) and 5 (best) after providing the thought process; the rating is binarized by thresholding at 4, based on the original implementation. To generate summaries,
LLM is used to describe the change between two observations based on the accessibility trees instead of screenshots. We use Llama 3.3 70B (Al-Dahle, 2024), an improved version of the original backbone.
Simplified judge (ours) We propose a simplified design for our judge. First, directly answers the three questions asked to the annotators. This allows it to return multiple labels within a single completion. Then, we decouple the system prompt and reasoning chain from the final state representation, allowing the judge to receive either the accessibility tree the screenshot. This differs from AER, which requires vision-enabled model, and NNetNav, which requires long-context model capable of receiving multiple accessibility trees. Our

---

Published as a conference paper at COLM 2025
Success R
Side Effect
Repetition
A11Y Screen
P
P
F1
F1
/
62.1
81.7
70.6
6.5
31.9
10.8
92.5
16.8
28.4
x
61.5
86.1
71.7
7.2
70.8
13.0
78.6
46.4
58.3
X
/
64.5
78.3
70.8
6.6
31.9
11.0
92.3
18.5
30.8
X
X
60.7
73.9
66.7
7.2
76.4
13.2
78.1
59.1
67.3
Table 2: Ablation study of our GPT-4o mini judge, measured in precision (P), recall (R), and F1. We judg, consider how including accessibility trees and screenshots in the input affects the predictions.
method is compatible with both multimodal and text-only LLMs and does not require separate LLM caption the screenshot or summarize changes across observations.
4.2 Evaluation
To evaluate LLM judges, we the precision score, which is the ratio of true positives over all predicted positives (true + false positives). The metric is good fit for rejection finetuning (RFT), where we interested in increasing the number of true positives (actual successful trajectories) while reducing the number of false positives (failed trajectories added to the dataset due to poor LLM judgments). For reward modeling, we also want prioritize true positives since they are the primary signals for many RL algorithms, while false positives should be minimized to avoid introducing noise to the loss function. Moreover, recall and F1 benefit from minimizing false negatives, which is useful for improving sample efficiency by reducing the number of valid trajectories removed; we report them as auxiliary metrics.
4.3 Judge Performance
In Table 1, we provide an overview of the performance of judges across benchmarks using the metrics defined in Section 4.2. We find that GPT-4o and Claude 3.7 Sonnet-based simplified judges achieve higher precision compared to prior approaches, indicating that removing the internal LLMs for captioning or summarizing changes does not hinder their capabilities. Notably, no judge consistently stands out across benchmarks, highlighting the importance of selecting an appropriate LLM backbone based on the nature of the task.
Low precision limits existing judges notice that judge achieves above 70% precision, which means that 30% of trajectories are erroneously marked as successful. This severely limits the usefulness of the judges for downstream applications, such using the filtered trajectories for finetuning an agent, as the agent will learn to generate incorrect trajectories for a substantial portion of the tasks. This indicates LLM judges are currently not reliable way of assessing the true capabilities of agents. Consequently, judges will need to achieve higher precision before they are useful for automatic evaluation, which also affects their downstream utility for methods like RFT and RI.
Official rule-based evaluation underestimates success Similar to LLM judges, the rule- based evaluation used by benchmarks can be compared with expert annotations. Since they use task-specific configurations to determine success, they may reject successful trajectories due to inconsequential differences. For instance, in WebArena, if a user asks "What's the closest national park to the largest city in Maine?", the agent may reply: "The closest national park to Portland [...] is Acadia National Park". Rule-based evaluation considers it unsuccessful since the configuration requires it to exactly match "Acadia National Park". As result, the rule-based approach achieves recall of 55.9%, indicating higher rate of false negatives compared to LLM judges. Overall, a substantial precision gap exists between rule- based methods and LLM judges, but rule-based methods severely underestimate the true performance of web agents, highlighting the need for more flexible automatic evaluation.
Impact of Input Representation Browser screenshots represent an intuitive state for hu- mans, but LLMs may need more than vision alone, as screenshots miss page structure and hidden attributes found in accessibility trees. To investigate the impact of different representations, we ablate our GPT-4o-mini simplified judge in Table 2. We observe that
7

---

Published as a conference paper at COLM 2025
Human
GPT-4o Judge
Rule-based
Agent
VWA
WA
Wk++
VWA
WA
Wk++
VWA
WA
Wk++
Claude 3.7 S.
28.3
55.1
18.4
34.8
64.1
20.7
23.9
30.8
8.1
GPT-4o
35.9
42.3
18.4
47.8
50.0
11.5
17.4
25.6
4.6
Llama 3.3 Qwen2.5-VL
0.0
22.4
9.2
0.0
27.6
5.8
0.0
18.4
3.5
21.7
33.3
13.8
34.8
52.6
14.9
17.4
29.5
11.5
Table 3: Success Rate of web agents measured by expert annotators, GPT-4o Judge (with accessibility tree) and rule-based evaluation on various benchmarks ($3.2). Results by environment are in Table 6.
only including screenshots achieves high precision for success and repetition, whereas only including accessibility trees allows higher recall. Surprisingly, including both accessi- bility trees and screenshots yields lower performance than including only the screenshot, indicating that more information distracts rather than assists the judge.
Revisiting how we evaluate task success rate
One of the core applications of LLM judges is to estimate the success rate on web nav- igation benchmark, which is useful in scenarios where there are no dedicated functions to calculate the rule-based success rate, which is the standard evaluation for many web agent benchmarks. However, rule-based approaches may not always agree with experts. In Table we compare the success rate calculated from expert annotations, rule-based evaluation, and a GPT-4o judge with accessibility trees.
Rule-based evaluation does not reflect expert-defined success rates We notice stark difference between the judge and rule-based approach: whereas the LLM judge tends to overestimate the success rate of every agent (with two exceptions in WorkArena++)) rule-based methods consistently underestimate it. Moreover, the underestimation varies substantially, with the performance of GPT-4o being 16.7% lower on WebArena and 18.5% lower on VWA compared expert annotations. This highlights a major discrepancy between the official task success rate reported by rule-based methods and the success rate according to expert annotators. For instance, rule-based evaluation ranks Qwen2.5-VL above GPT-4o WebArena and WorkArena++ (and equally VWA), whereas expert annotators prefer GPT-4o over Qwen2.5-VI on all benchmarks, with over 14% higher success rate on VWA. Overall, this stresses the need to develop new methods to calculate task success rate that more precisely reflect expert judgments.
Error Analysis
In this section, we qualitatively examine failure cases of LLM judges. Following a GPT-4o judge, we focus on the common error categories to understand how LLM judges fail.
Grounding mismatch When an agent misunderstands what is happening on the screen, its thought process may not reflect the reality of the webpage. In such cases, judge without access to the screenshots may produce an erroneous judgment due to the agent misunderstanding what is happening show "the product page for the item in the second row, second column," the Qwen2.5-VL agent ends up selecting the second item in the first row. However, writes in its reasoning chain that "Based on the layout of the page, the second row, second column item is the Jenergy Drink)." The judge accepts the agent's thought process without realizing it opened the wrong page: "The agent's 's goal was to navigate to the product page for the item in the second row, second column. The agent successfully reached the correct product page."
the screen. For example, when user requests to
Misleading agent reasoning The agent may have misleading elements in its reasoning, leading the judge to reason that the agent completed the task correctly. In multi-step WorkArena++ task, the user requested the agent to apply a search filter include a unique ID. A After several unsuccessful attempts, the agent ended up stating it succeeded in its
8

---

Published as a conference paper at COLM 2025
Goal (#843)
Can you buy the cheapest cereal with graphic character on the in the "Cereals" category?
Grooyy
Agent (Claude 3.7 Sonnet)
Reply: "I've found the cheepest cereal with a graphic character on the box. It's Cocoa Puffs I've already added it to your cart." X
LIKGRO
was vour
YUAKER
Judge (GPT-4o)
Thought: The agent successfully identified and purchased the cheapest cereal with a graphic character on the box...
(980)
Figure 3: Example of judge enror (discussed in Section 6). In this example, the user requests the agent to buy cereal in Arena,, but the agent stops after adding it to the cart. The judge erroneously identified the trajectory successful, even though the agent missed an important detail in the instruction.
reasoning chain, even though no filter was applied. The judge was misled by the agent and wrote in its own reasoning chain that "The agent successfully [...] applied the filter to extract entries..." " before incorrectly indicating that the trajectory was successful.
Missed instruction details In some cases, the agent does not complete the task completely, missing crucial details from the instruction (see Figure 3). For example, when the user re- quests to "buy the cheapest cereal with graphic character on the box in the Cereals category," the agent finds the correct product and informs the customer: "I've found the cheapest cereal with a graphic character on the box. It's Cocoa Puffs, 10.4 Oz Box..." missed a crucial detail: the user requested that they buy the product. Unfortunately, the judge mistakenly believes that the agent completed purchase: "The agent successfully identified and purchased the cheapest cereal with graphic character on the box..." "
" However, it
Misunderstanding action intents We found that the agent may misuse certain actions, leading to the trajectory to fail very close to completion. In such cases, the LLM judge may incorrectly decide that the trajectory is successful and ignore the misused action. In one instance where the goal was to answer "What the minimum travel time by car from Animal Rescue League of Pittsburgh to Schenley park?," the Qwen2.5-VE agent completes all required actions, but ends up reporting the task as unfeasible instead of replying to the user. The GPT--o judge (with screenshot) correctly reasons that the travel time was shown on the screen, but does not point out that reporting the task as unfeasible is incorrect, instead asserting that "all actions were confined to the task of finding the travel time."
Overall, current LLM judges are limited by their capability to detect nuanced issues within trajectories, as shown by the judge missing details and misunderstanding over, they will easily agree with the agent's reasoning even when it is wrong, which has been previously observed in LLMs (Sharma et al., 2023). Future research should aim to address these issues to improve the performance of LLM judges for evaluating web agents.
action. More-
Conclusion
We introduce AGENTREWARDDENCCH a benchmark for evaluating LLM judges for web agent trajectories. The benchmark consists of over 1300 trajectories, each annotated by experts across three dimensions: whether the agent succeeded, whether it caused unin- tended side effects, and whether repeated unnecessary actions. We evaluate 12 LLM judges
and find that simpler input representation can achieve higher agreement with expert annotators compared to prior approaches. Moreover, we find that rule-based evaluation, often used by environment-based benchmarks, does not

---

Published as a conference paper at COLM 2025
achieve lower-than-expecte agreement with experts. Instead, it tends to reject many valid trajectories, which results in the success rate of certain web agents being lower than what an expert would perceive. Overall, we believe our benchmark will help researchers design better LLM judges for web agents trajectories, which will enable the design of automatic evaluators and reward models that better reflect expert judgments.
Acknowledgments
Xing Han Lü acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) [funding reference no. 579403]. The project is supported by the Google-Mila grant. We thank Alexandre Lacoste, Shikhar Murty, and the McGill NLP group members for helpful discussions.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint orX4:42303.8877, 2023. URL http:::t////xx 0bs/2333.088774
Adept. Act-1: Transformer for actions, 2022. URL https: htttps::/www.ast http:://www.aeeet.../bb/////ee
Ahmad Al-Dahle. The future of ai: Built with Ilama, December 2024. URL https://aaa........ meta. -it
Anthropic. Introducing computer use, new claude 3.5 sonnet, and claude 3.5 haiku, 2024a. m-odoloolsssse s- and computer-use.
URL https://www wwt.www.....iiccctotcmmm 535
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024b. URL https://api. wwwp:cchholaar.orrroruuuouue : 268232499.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL http::///axxi..//aa/////222 13923.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarena++: Towards compositional planning and reasoning-based common 05291.
knowledge work tasks, 2025. URL
Greg Brockman, Vicki Cheung, Ludwig Jones Schhhnhn and Wojciech Zaremba. Openai gym, 2016. https:://rxxiv.rr//bbb///
John Schulman, Jie Tang, 01540.
Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Ke- unho Jang, Xing Han Lü, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The browsergym ecosystem for web agent research, 2025. URL https: //arxiv. wwtiiv..rg//bbsee wttp//////////....////// 05467.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systemss, 36:28091-28111 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.
http://wwwx...o/oab//////// 07718.
10

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint, 2024. URL
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur: Learning to navigate the web. Xrv preprint ar xrXie:1812.09195, 2018.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint ar arXii:2307.1220 12856, 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. ArXiv, 267211622.
2024. URL httpps////pppp semanticscholar wwwpollarrrrg///ooeoe org/CCrpusII:
Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Tim- othy Lillicrap. A data-driven approach for learning to control computers, 2022. URL https://arxiv. wtgp:////////..oo.////// httts://aarxv.orrr//////222000. 08137.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint ar Xit:2440.22176, 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWeb Arenaa Evaluating Multimodal Agents on Realistic Visual Web Tasks, June 2024. URL http: //arxiv.org/abs/ 2401. 13649. arXiv:2401.13649 [cs].
Hanyu Lai, Xiao Liu, Iat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model- web navigating agent, 2024. URL http://:////axxoooo//////////// 03648.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Han- naneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL http:://aaxxvv..rgaa/pp 23777.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Re- inforcement Learning on Web Interfaces using Workflow-guided Exploration. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL /tttp:://opereeiier wwwperrewwwieeweefrrreee id=ryTp3f-@-.
Xing Han Lù, Zdenëk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint ar arXio:2402.05930, 2024.
Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. BAGEL: Bootstrapping Agents by Guiding Exploration with Language, June 2024. URL http: //arxiv.org/abs//2/. /203. 08140. 4rXiv:2403.08140 [cs].
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D. Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild, 2025. URL https: //axxiv. org/abs/2411 02907.
Reiichiro Nakano, Jacob Hilton,Suchir Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. arXiv preprint ar xrxi0:21122009332, 2021.
OpenAl. Introducing operator, January 2025.
URL https:://ppee
index/
introducing-operrtorr
11

---

Published as a conference paper at COLM 2025
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents, 2024. URL https://arxiv..org abs/2404.0644 06474.
Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei Marius-Constantin Dinu, Chris Callison-Burch, and Sepp Hochreiter. Large language models can self-improve at web agent tasks, 2024. URL https: //axii://aaxiv...o
http:///aaaaii...oaabb////s///// wwg..b//////./5 20309.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024. URL https: //axxiv http:////////////////// 0ttps/228...777 07199.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-553 53741,, 2023.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023. URL httpt:t/taaxiiii http:://aaaiiv.ogaa///////ss wwt.orr/abbbb/gga/e 13548.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to UI actions: Learning to follow instructions via graphical user interfaces. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevieee www.eeeewwweeeeennnneeee net/forumlid= 3PjCt4kmRX.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Robert St. Amant and Luke S. Zettlemoyer. The user interface agent environment. In Proceedings of the Fourth International Conference on Autonomous Agents, AGENTS '00, pp. 483-490, New York, NY, USA, 2000. Association for Computing Machinery. ISBN 1145/336595 337575.
1581132301. doi: 10.1145 3355555.5555577
URL
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. Towards internet-scale training for agents, 2025. URL 06776.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards Scalable Real-world Web Interaction with Grounded Language Agents. In NeurlPS, 2022. URL htt.httt.//arrv.oorraa/////////
.0..0.12200 006.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL
15711.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded, 2024. URL
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging lmm-as---judge with mt-bench and chatbot arena, 2023. URL https://arxiv. org/abs/2300 05685.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. URL http:
//arxiv. serxiv.orgg/bss/2ee0/ 13854. rrXvv:2307.33854 [cs].
12

---

Published as a conference paper at COLM 2025
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judgee Evaluate 10934.
agents with agents, 2024. URL https:/
13

---

Published as a conference paper at COLM 2025
A Benchmark
A.1 Environment and Experiments Details
AssistantBench Although an unlimited number of websites can be visited, we observed that the agents visited a total of 66 unique domains between and 129 times across all trajectories we collected. The number of times a domain was visited can be found in Table 4. Additionally, we replace the default search engine with an alternative search engine (https: /twp//uckkkkk.oooo////////// com) as the original homepage blocks browser automation, which renders the tasks unachievable.
Tasks Subgroups We define the subgroup for WebArena and VisualWebArena as the combination of web domain and evaluation method from the original works. The evaluation methods consist of string matching, HTML-based programs, webpage image querying, and final URL matching. We randomly sample up to 8 tasks from each domain-evaluation group for WebArena, and up to 9 for VisualWebArena since certain domain-evaluation groups have a very small number of tasks. For WorkArena, we attempt to evenly distribute the task categories. As a result, we have the following task distributions:
WebArena: Wikipedia (8), Map (18), Reddit (18), Shopping Admin Shopping (19), Gitlab (19)
VisualWebArena: Wikipedia (17), Reddit (27), Classifieds Shopping (28)
WorkArena: Sophisticated memory (15), Information retrieval (20), Contextual understanding infeasible tasks (21), Planning and problem solving (22), Data driven decision making and reasoning (22)
Agent Hyperparameters The binary flags used in AgentLab (Chezelles et al., 2025) are shown in Table 5. We set a maximum limit of 40K input tokens and 8192 output tokens.
Agent Platform Implementation In addition to abstracting websites and browser engines Inpoemmemm ennneoompstlleeeeevrnnnttts (Brockman et al., 2016), BrowserGym (Drouin et al., 2024; into Cym-compatible environments
Chezelles et al., 2025) offers advanced preprocessing of complex web inputs (i.e., DOM and accessibility trees) and can automatically parse LLM output and execute them as browser actions like clicks, form inputs, tab actions, etc. Additionally, the BrowserGym ecosystem includes AgentLab, framework for processing input representation and managing web agent experiments. We use AgentLab to design our representation function R, ensuring unified hyperparameters and inputs. As result, we can avoid unintended differences that may arise from customizing prompts and representations for each LLM.
A.2 Annotations
Trajectory filtering In total, 351 tasks were considered across 5 benchmarks (33 in As- sistantBench, 100 in VisualWebArena, 100 in WebArena, 18 in WorkArena, and 100 in WorkArena++). We collect trajectories from agents built from each of three multimodal models: Claude 3.7 Sonnet, GPTT-4,, Qwen2.5-VL. Moreover, since Llama 3.3 is not multi- modal, we only collect trajectories on 251 tasks (excluding Webuaa))bArena).. Additionally, Llama 3.3 did not complete two WebArena tasks (nos. 735 and 805) due to timeout issues that consistently occurred in the environment, despite multiple attempts to restart. Thus, obtain total of 1302 trajectories, where 196 are stored in the development split and 1106 in the test split.
Interface To annotate the trajectories, we designed fully customized annotation interface using Gradio (see Figure 4). For a selected agent and task, we displayed the goal and each of the steps of the trajectory taken by the model. It shows the model's reasoning and action, as well as screenshot with the action element on overlay. Then, the annotators are prompted to answer a series of questions regarding the success, side effects, and repetitiveness of the agent, using the same questions that we ask the LLM judges.
Shared Knowledge Given that the annotators are experts, it is possible that the annotators share knowledge of web agents that non-expert may not possess; we identify several shared
14

---

Published as a conference paper at COLM 2025
knowledge facts. (1) web agent design and capabilities: the annotators are aware that the agents are designed with LLMs, some of which have multimodal capabilities, and that they are capable of generating reasoning traces to support actions, and that the LLMs may be subject to hallucination or may product repetitive sequences of text. (2) dedicated web environments: the annotators know that several the websites used in the project come from prior publications in the domain, including WebArena (Zhou et al., 2024), VisualWeb (Koh et al., 2024), WorkArena (Drouin et al., 2024; Boisvert al., 2025). They are aware that some of the websites are designed specific for the task, whereas others come from real-world websites. (3) Automatic Evaluation: the annotators know that the web environments employ automatic evaluation methods, such as string matching and URL matching, to evaluate the agents. Thus, a task that is successful or unsuccessful may terminate earlier, but the agent will not be guaranteed to receive positive reward for that task.
Annotator agreements and disagreements resolution For most tasks, binary annotations can be produced. However, some cases, the annotator may not be certain of their annotation, and are allowed to mark a trajectory reviewed by the other annotators. In some cases, annotators may disagree with their judgments. In general, a salient reason for mismatch the ambiguity of the instructions. For example, a task instruction might mention "buy black t-shirt", but may not specify fully black or can have other graphics. In such cases, annotators are advised go for the most lenient option. More generally, to ensure that annotators can easily voice their uncertainty and disagreement, the first half of the annotation was conducted in person with all annotators present concurrently. Thus, when an annotator was uncertain about the annotation for trajectory, they can ask other annotators, who can deliberate about the correct annotation until consensus is reached. This approach further allows other annotators to align to the consensus for the remaining annotations.
uncertain, which was subsequently
A.3 LLM Judges
Prompts We use simple system prompt (Figure 5) and user message (Figure 6) templates without model-specific commands, allowing our prompt to be transferred to any LLM. We use distinct tags, such as <success> and <reasoning?, facilitate parsing the model output.
Results We report extended results for 10 10 judges, with the overall results in Table and the finegrained results in Table 8 over all agents; the unaggregated results are presented in Tables to 12.
15

---

Published as a conference paper at COLM 2025
AUndEas Nome
Aerobiiiors Doenoon
Telimu the namber von owo the slow reutnd by mentin wr "wet"
Step
Jowene
Dashboard
Fraae Soa
Overloy
Sole
$0.00
Peorage Ordar
Go Button
Step
lentons
Eine Insclatin
to
Serv Mert
Figure 4: User Interface used by annotators for answering questions
Domain
Domain
Domain
duckduckgo.com
129
google.com
wizards com
blackboudhosting.com
21
femex..cm
migguldissh.com
usps.com
12
fidelity.ca
weathergguv
yelp.com
linkedin.com
rottentomatoes.com
nih gov
8
logplayer.com com
imdb.com
yahwo.com
cagreatamerica.com
thedrinknationn.c
tripadvisor.com
6
expressdhl
californagrestameric..
seatlechaldennsmuseen..gg
5
monday.com
fubo.tv
philamuseum.org
5
weatherspark.com
bing.com
ensembl.org
4
wellhuh.comm
hubbion.com
wholefoodemarket.comm
alltrails
target.com
andersonsmarialatts.com
2
wikipedia.ogg
sfyimby.com
currentresults.com
stuckanalysin.com
speakrj.com
x.com
2
apple.com
extremeweatherwatchccom
tmplelubs.co com
2
sixflags.com com
eff.com
netflixneleases.com
weather-and -ndiiirr--dddddmeccmm
wunderground.com
com
redfin com
inleofimmuutaiimmaa.coom
themeparkeenter.com
seattleweatherblog.ccm
chromewebdata
peacefoodnyc.com
sec.gov
calicolabs.com com
casyship.com
valinnssipppingaalcalttr..cm
tripadvisor.ca
nyunews.com
fandango-com
aimojo.ão
anytuts.com
morningstar.com
visitphilly.com
Table AssistantBench Website Visit Counts
16

---

Published as a conference paper at COLM 2025
Value
Flags
True
se__screnshoo, use_som, use_ sse_thinking, mee_concrette_exmpee, use__bstractt_exmple use_hints, be_cautious
False
use_html, use_past Beee_past_errr______ filter_ www.sssleeeeeeetss.oooo
use_think_history use_diff,
long_descriptionn individual__exappees use_plan, criticiee use_memory, enable_ chat
Table 5: Agentlab Hyperparameters
Benchmark
Agent
Expert
LLM Judge
Rule-based
Claude 3.7 S.
11.1
11.1
0.8
GPT-4o Llama 3.3
14.8
14.8
3.7
AssistantBench
3.7
5.3
Qwen2.5-VI.
0.0
0.0
2.2
Claude 3.7 GPT-4o
55.1
64.1
30,8
42.3
50.0
25.6
WebArena
Llama 3.3
22.4
27.6
18.4
Qwen2.5-VL
33.3
52.6
29.5
Claude 3.7 S.
28.3
34.8
23,9
GPT-4o Qwen2.5-VL
35.9
47.8
17.4
VisualWebArena
21.7
34.8
17.4
Claude 3.7
68.8
68.8
50.0
GPT--o Llama 3.3
50.0
56.2
50.0
WorkArena
56.2
50.0
56.2
Qwen2.5-VI.
56.2
56.2
56.2
Claude 3.7 S.
18.4
20.7
8.1
GPT-4o Llama 3.3
18.4
11.5
4.6
WorkArena++
9.2
5.8
3.5
Qwen2.5-VL
13.8
14.9
11.5
Claude 3.7
33.0
38.0
20.4
GPT-4o
31.3
35.3
16.3
Overall
Llama 3.3
17.0
17.5
13.3
Qwen2.5-VI.
22.3
31.7
19.5
Table 6: Success Rate by evaluation type. For the LLM judge, we use GPT-4o with accessibility trees.
Success R
Side Effect
Repetition R
Judge
P
F1
P
R
F1
P
F1
AER-C
67.7
71.9
69.7
AER-V
67.6
71.5
69.5
Claude 3.7 (A)
68.8
81.6
74.7
14.0
34.7
20.0
82.8
94.9
88.4
Claude 3.7 (S)
69.4
76.3
72.7
14.1
44.4
21.4
82.0
94.5
87.8
Functional
83.8
55.9
67.1
GPT-4o (A)
69.8
83.1
75.9
7.7
91.7
14.2
80.4
96.9
87.9
GPT-4o (S)
68.1
80.3
73.7
7.5
90.3
13.8
79.2
96.2
86.9
GPT-4o Mini (A)
86.1
71.7
7.2
70.8
13.0
78.6
46.4
58.3
GPT-4o Mini (S)
64.5
78.3
70.8
6.6
31.9
11.0
92.3
18.5
30.8
Llama 3.3 (A)
67.7
79.0
72.9
6.9
79.2
12.7
80.1
91.6
85.5
NNetNav
52.5
82.4
64.1
Qwen2.5-VL. (A)
64.3
89.8
75.0
9.0
55.6
15.4
88.1
72.6
79.6
Qwen2.5-VL.C (S)
64.5
86.1
73.7
58.3
15.2
88.7
64.6
74.7
Table 7: Results over all benchmarks by judge. We report the precision (P) as the primary metric, and F1 and recall (R) as the auxiliary metrics.
17

---

Published as a conference paper at COLM 2025
Benchmark
Judge
Success
Side Effect
Repetition
P
R
R
AER-C
83.3
62.5
714
AER-V
83.3
62.5
Claude 3.7 S. (A)
87.5
87.5
87.5
7.1
33.3
11.8
70.1
94.0
80.3
(S)
71.4
33.3
Functional
25.0
12.5
16.7
GPT-44 (A)
77.8
87.5
82.4
3.2
100.0
6.3
68.6
96.0
80.0
AssistantBench
GPT-4o (S)
77.8
87.5
92.4
100.0
67.1
94.0
78.3
GPT-4o Mini (A)
80.0
100.0
88.9
2.7
66.7
5.2
59.3
32.0
41.6
GPT-4oo Mini
33.3
Ilama 3.3 (A)
75.0
75.0
75.0
3.4
100.0
6.6
65.8
96.0
78.0
NNetNav
20.8
31.2
Owen2.5-VI. (A)
72.7
100.0
84.2
0.0
73.8
62.0
67.4
Qwen2.5-V1.
70.0
87.5
77.8
3.7
66.7
10.5
72.2
52.0
60.5
AER-C
56.0
70.9
62.6
79.7
Claude 3,7 (A)
61.0
77.2
68.2
16.7
35.7
22.7
78.7
96.8
86.8
Claude 3.7
64.8
74.7
69.4
42,9
24.5
77.8
97.6
Functional
85.2
58.2
69.2
GPT-4o (A)
63.0
86.1
72.7
12.4
89.3
21.8
75.2
98.4
85.2
VisualWebAAeriaa
GPT-4u (S)
60.7
GPT-44 Mini (A)
57.9
83.5
68.4
10.7
64.3
18.4
74.4
46.0
56.9
GPT-4o
57.4
73.4
11.2
35.7
87.1
Llama 3.3 (A)
59.6
74.7
66.3
12.0
82.1
21.0
74.4
92.1
82.3
NNetNav
54.5
69.6
61.1
Qwen2.5VVI.
59.3
88.6
71.1
17.2
71.4
27.8
86.9
73.8
79.8
Qwen2.5-VL.
58.5
87.3
70.0
15.4
64.3
24.8
83.3
59.5
69.4
AER-C
68.8
83.2
75.3
AER-V
67.6
84.0
74.9
Claude 3.75 (A)
69.3
89.1
77.9
11.8
28.6
16.7
84.9
88.6
86.7
Clande 3.7 (S)
69.3
89.1
77.9
14.8
38.1
21.3
80.8
85.1
82.9
Functional
79.0
53.8
GPT-4o (A)
70.2
89.1
78.5
9.6
90.5
17.4
82.9
93.9
88.1
WebArena
GPT-4u (S)
76.2
GPT-44 Mini (A)
63.5
90.8
74.7
10.1
71.4
17.7
70.3
39.5
50.6
GPT-4o Mini (S)
66.9
86.6
75.5
28.6
11.9
86.7
11.4
20.1
Ilama 3.5
68.2
86.6
76.3
76.2
15.4
79.7
86.0
82.7
NNetNav
54.3
90.8
67.9
Qwen2.5-VL
63.6
28.6
63.2
Owen2,5-VL
62.9
V2.4
74.8
9.4
38.1
15.1
90.5
50.0
64.4
AER-C
100,0
81.1
89.6
AER-V
96.4
73.0
83.1
Claude 3.7 (A)
85.0
91.9
88.3
50.0
14.3
76.0
86.4
80.8
Claude 3.7 (S)
85.3
78.4
81.7
0.0
0.0
0.0
77.3
77.3
77.3
Functional
100.0
GPT-44 (A)
94.6
94.6
94.6
2.6
50.0
5.0
70.4
86.4
77.5
WorkArena
GPT-4o
93.8
81.1
100.0
72.0
GPT-4o Mini (A)
84.2
86.5
85.3
3.3
50.0
6.2
66.7
36.4
47.1
GPT-44 Mini (S)
90.3
75.7
82.4
0.0
0.0
100.0
18.2
30.8
Illama 3.3
94.3
89.2
91.7
0.0
61.8
81.8
81.8
NNetNav
77.3
91.9
83.9
Qwen2.5-VL. (A)
87.2
0.0
Qwen2.5-VL.9 (8)
93.8
81.1
87.0
0.0
0.0
0.0
86.7
59.1
70.3
AER-C
66.7
42.3
51.8
AER-V
59.3
30.8
40.5
Claude 3.75 (A)
66.7
62.7
38.9
87.5
97.4
Claude 3.7 (S)
66.7
50.0
57.1
13.6
61.1
22.2
67.3
98.9
92.8
Functional
83.3
GPT-40 (A)
63.0
55.8
39.2
5.5
100.0
10.3
83.6
98.5
91.6
WorkArena+
GPT--4 (S)
59.6
53.8
36.6
100.0
10.4
84.5
98.2
90.8
GPT-44 Mini (A)
49.4
76.9
60.1
5.2
83.3
9.7
67.8
52.9
66.1
GPT-4o Mini (S)
54.8
65.4
39.6
4.6
33,3
8.1
96.5
20.2
33.4
Llama 3.3
62.7
61.5
83.3
NNetNav
43.2
78.8
55,8
Qwen2.5-VL. (A)
60.3
77.8
Qwen2.5-VL (5)
64.4
73.1
68.5
77.8
11.8
93.2
75.7
83.6
Table 8: Finegrained results by benchmark and judge for all agents. We report the precision (P) the primary metric, and Fl and recall (R) as the auxiliary metrics.
18

---

Published as a conference paper at COLM 2025
Benchmark
Judge
Success
Side Effect
Repetition
P
R
R
AER-C
83.3
62.5
714
AER-V
83.3
62.5
Claude 3.7 S. (A)
87.5
87.5
87.5
7.1
33.3
11.8
70.1
94.0
80.3
(S)
71.4
33.3
Functional
25.0
12.5
16.7
GPT-44 (A)
77.8
87.5
82.4
3.2
100.0
6.3
68.6
96.0
80.0
AssistantBench
GPT-4o (S)
77.8
87.5
92.4
100.0
67.1
94.0
78.3
GPT-4o Mini (A)
80.0
100.0
88.9
2.7
66.7
5.2
59.3
32.0
41.6
GPT-4oo Mini
33.3
Ilama 3.3 (A)
75.0
75.0
75.0
3.4
100.0
6.6
65.8
96.0
78.0
NNetNav
20.8
31.2
Owen2.5-VI. (A)
72.7
100.0
84.2
0.0
73.8
62.0
67.4
Qwen2.5-V1.
70.0
87.5
77.8
3.7
66.7
10.5
72.2
52.0
60.5
AER-C
56.0
70.9
62.6
79.7
Claude 3,7 (A)
61.0
77.2
68.2
16.7
35.7
22.7
78.7
96.8
86.8
Claude 3.7
64.8
74.7
69.4
42,9
24.5
77.8
97.6
Functional
85.2
58.2
69.2
GPT-4o (A)
63.0
86.1
72.7
12.4
89.3
21.8
75.2
98.4
85.2
VisualWebAAeriaa
GPT-4u (S)
60.7
GPT-44 Mini (A)
57.9
83.5
68.4
10.7
64.3
18.4
74.4
46.0
56.9
GPT-4o
57.4
73.4
11.2
35.7
87.1
Llama 3.3 (A)
59.6
74.7
66.3
12.0
82.1
21.0
74.4
92.1
82.3
NNetNav
54.5
69.6
61.1
Qwen2.5VVI.
59.3
88.6
71.1
17.2
71.4
27.8
86.9
73.8
79.8
Qwen2.5-VL.
58.5
87.3
70.0
15.4
64.3
24.8
83.3
59.5
69.4
AER-C
68.8
83.2
75.3
AER-V
67.6
84.0
74.9
Claude 3.75 (A)
69.3
89.1
77.9
11.8
28.6
16.7
84.9
88.6
86.7
Clande 3.7 (S)
69.3
89.1
77.9
14.8
38.1
21.3
80.8
85.1
82.9
Functional
79.0
53.8
GPT-4o (A)
70.2
89.1
78.5
9.6
90.5
17.4
82.9
93.9
88.1
WebArena
GPT-4u (S)
76.2
GPT-44 Mini (A)
63.5
90.8
74.7
10.1
71.4
17.7
70.3
39.5
50.6
GPT-4o Mini (S)
66.9
86.6
75.5
28.6
11.9
86.7
11.4
20.1
Ilama 3.5
68.2
86.6
76.3
76.2
15.4
79.7
86.0
82.7
NNetNav
54.3
90.8
67.9
Qwen2.5-VL
63.6
28.6
63.2
Owen2,5-VL
62.9
V2.4
74.8
9.4
38.1
15.1
90.5
50.0
64.4
AER-C
100,0
81.1
89.6
AER-V
96.4
73.0
83.1
Claude 3.7 (A)
85.0
91.9
88.3
50.0
14.3
76.0
86.4
80.8
Claude 3.7 (S)
85.3
78.4
81.7
0.0
0.0
0.0
77.3
77.3
77.3
Functional
100.0
GPT-44 (A)
94.6
94.6
94.6
2.6
50.0
5.0
70.4
86.4
77.5
WorkArena
GPT-4o
93.8
81.1
100.0
72.0
GPT-4o Mini (A)
84.2
86.5
85.3
3.3
50.0
6.2
66.7
36.4
47.1
GPT--4 Mini (S)
90.3
75.7
82.4
0.0
0.0
100.0
18.2
30.8
Illama 3.3
94.3
89.2
91.7
0.0
61.8
81.8
81.8
NNetNav
77.3
91.9
83.9
Qwen2.5-VL. (A)
87.2
0.0
Qwen2.5-VL.9 (8)
93.8
81.1
87.0
0.0
0.0
0.0
86.7
59.1
70.3
AER-C
66.7
42.3
51.8
AER-V
59.3
30.8
40.5
Claude 3.75 (A)
66.7
62.7
38.9
87.5
97.4
Claude 3.7 (S)
66.7
50.0
57.1
13.6
61.1
22.2
67.3
98.9
92.8
Functional
GPT-40 (A)
63.0
55.8
39.2
5.5
100.0
10.3
83.6
98.5
91.6
WorkArena+
GPT--4 (S)
59.6
53.8
36.6
100.0
10.4
84.5
98.2
90.8
GPT-44 Mini (A)
49.4
76.9
60.1
5.2
83.3
9.7
67.8
52.9
66.1
GPT-4o Mini (S)
54.8
65.4
39.6
4.6
33,3
8.1
96.5
20.2
33.4
Llama 3.3
62.7
61.5
83.3
NNetNav
43.2
78.8
55,8
Qwen2.5-VL. (A)
77.8
Qwen2.5-VL (5)
64.4
73.1
68.5
77.8
11.8
93.2
75.7
83.6
Table 9; Finegrained results by benchmark and judge for Qwen2.5-VL agent. We report the precision (P) as the primary metric, and Fl and recall (R) as the auxiliary metrics.
19

---

Published as a conference paper at COLM 2025
Benchmark
Judge
Success
Side Effect
Repetition
P
R
R
AER-C
83.3
62.5
714
AER-V
83.3
62.5
Claude 3.7 S. (A)
87.5
87.5
87.5
7.1
33.3
11.8
70.1
94.0
80.3
(S)
71.4
33.3
Functional
25.0
12.5
16.7
GPT-44 (A)
77.8
87.5
82.4
3.2
100.0
6.3
68.6
96.0
80.0
AssistantBench
GPT-4o (S)
77.8
87.5
92.4
100.0
67.1
94.0
78.3
GPT-4o Mini (A)
80.0
100.0
88.9
2.7
66.7
5.2
59.3
32.0
41.6
GPT-4oo Mini
33.3
Ilama 3.3 (A)
75.0
75.0
75.0
3.4
100.0
6.6
65.8
96.0
78.0
NNetNav
20.8
31.2
Owen2.5-VI. (A)
72.7
100.0
84.2
0.0
73.8
62.0
67.4
Qwen2.5-V1.
70.0
87.5
77.8
3.7
66.7
10.5
72.2
52.0
60.5
AER-C
56.0
70.9
62.6
79.7
Claude 3,7 (A)
61.0
77.2
68.2
16.7
35.7
22.7
78.7
96.8
86.8
Claude 3.7
64.8
74.7
69.4
42,9
24.5
77.8
97.6
Functional
85.2
58.2
69.2
GPT-4o (A)
63.0
86.1
72.7
12.4
89.3
21.8
75.2
98.4
85.2
VisualWebAAeriaa
GPT-4u (S)
60.7
GPT-44 Mini (A)
57.9
83.5
68.4
10.7
64.3
18.4
74.4
46.0
56.9
GPT-4o
57.4
73.4
11.2
35.7
87.1
Llama 3.3 (A)
59.6
74.7
66.3
12.0
82.1
21.0
74.4
92.1
82.3
NNetNav
54.5
69.6
61.1
Qwen2.5VVI.
59.3
88.6
71.1
17.2
71.4
27.8
86.9
73.8
79.8
Qwen2.5-VL.
58.5
87.3
70.0
15.4
64.3
24.8
83.3
59.5
69.4
AER-C
68.8
83.2
75.3
AER-V
67.6
84.0
74.9
Claude 3.75 (A)
69.3
89.1
77.9
11.8
28.6
16.7
84.9
88.6
86.7
Clande 3.7 (S)
69.3
89.1
77.9
14.8
38.1
21.3
80.8
85.1
82.9
Functional
79.0
53.8
GPT-4o (A)
70.2
89.1
78.5
9.6
90.5
17.4
82.9
93.9
88.1
WebArena
GPT-4u (S)
76.2
GPT-44 Mini (A)
63.5
90.8
74.7
10.1
71.4
17.7
70.3
39.5
50.6
GPT-4o Mini (S)
66.9
86.6
75.5
28.6
11.9
86.7
11.4
20.1
Ilama 3.5
68.2
86.6
76.3
76.2
15.4
79.7
86.0
82.7
NNetNav
54.3
90.8
67.9
Qwen2.5-VL
63.6
28.6
63.2
Owen2,5-VL
62.9
V2.4
74.8
9.4
38.1
15.1
90.5
50.0
64.4
AER-C
100,0
81.1
89.6
AER-V
96.4
73.0
83.1
Claude 3.7 (A)
85.0
91.9
88.3
50.0
14.3
76.0
86.4
80.8
Claude 3.7 (S)
85.3
78.4
81.7
0.0
0.0
0.0
77.3
77.3
77.3
Functional
100.0
GPT-44 (A)
94.6
94.6
94.6
2.6
50.0
5.0
70.4
86.4
77.5
WorkArena
GPT-4o
93.8
81.1
100.0
72.0
GPT-4o Mini (A)
84.2
86.5
85.3
3.3
50.0
6.2
66.7
36.4
47.1
GPT--4 Mini (S)
90.3
75.7
82.4
0.0
0.0
100.0
18.2
30.8
Illama 3.3
94.3
89.2
91.7
0.0
61.8
81.8
81.8
NNetNav
77.3
91.9
83.9
Qwen2.5-VL. (A)
87.2
0.0
Qwen2.5-VL.9 (8)
93.8
81.1
87.0
0.0
0.0
0.0
86.7
59.1
70.3
AER-C
66.7
42.3
51.8
AER-V
59.3
30.8
40.5
Claude 3.75 (A)
66.7
62.7
64.7
38.9
87.5
97.4
Claude 3.7 (S)
66.7
50.0
57.1
13.6
61.1
22.2
67.3
98.9
92.8
Functional
GPT-40 (A)
63.0
55.8
39.2
5.5
100.0
10.3
83.6
98.5
91.6
WorkArena+
GPT--4 (S)
59.6
53.8
36.6
100.0
10.4
84.5
98.2
90.8
GPT-44 Mini (A)
49.4
76.9
60.1
5.2
83.3
9.7
67.8
52.9
66.1
GPT-4o Mini (S)
54.8
65.4
39.6
4.6
33,3
8.1
96.5
20.2
33.4
Llama 3.3
62.7
61.5
83.3
NNetNav
43.2
78.8
55,8
Qwen2.5-VL. (A)
77.8
Qwen2.5-VL (5)
64.4
73.1
68.5
77.8
11.8
93.2
75.7
83.6
Table 10: Finegrained results by benchmark and judge for Llama 3.3 agent. report the precision (P) as the primary metric, and Fl and recall (R) as the auxiliary metrics.
20

---

Published as a conference paper at COLM 2025
Benchmark
Judge
Success
Side Effect
Repetition
P
R
R
AER-C
83.3
62.5
714
AER-V
83.3
62.5
Claude 3.7 S. (A)
87.5
87.5
87.5
7.1
33.3
11.8
70.1
94.0
80.3
(S)
71.4
33.3
Functional
25.0
12.5
16.7
GPT-44 (A)
77.8
87.5
82.4
3.2
100.0
6.3
68.6
96.0
80.0
AssistantBench
GPT-4o (S)
77.8
87.5
92.4
100.0
67.1
94.0
78.3
GPT-4o Mini (A)
80.0
100.0
88.9
2.7
66.7
5.2
59.3
32.0
41.6
GPT-4oo Mini
33.3
Ilama 3.3 (A)
75.0
75.0
75.0
3.4
100.0
6.6
65.8
96.0
78.0
NNetNav
20.8
31.2
Owen2.5-VI. (A)
72.7
100.0
84.2
0.0
73.8
62.0
67.4
Qwen2.5-V1.
70.0
87.5
77.8
3.7
66.7
10.5
72.2
52.0
60.5
AER-C
56.0
70.9
62.6
79.7
Claude 3,7 (A)
61.0
77.2
68.2
16.7
35.7
22.7
78.7
96.8
86.8
Claude 3.7
64.8
74.7
69.4
42,9
24.5
77.8
97.6
Functional
85.2
58.2
69.2
GPT-4o (A)
63.0
86.1
72.7
12.4
89.3
21.8
75.2
98.4
85.2
VisualWebAAeriaa
GPT-4u (S)
60.7
GPT-44 Mini (A)
57.9
83.5
68.4
10.7
64.3
18.4
74.4
46.0
56.9
GPT-4o
57.4
73.4
11.2
35.7
87.1
Llama 3.3 (A)
59.6
74.7
66.3
12.0
82.1
21.0
74.4
92.1
82.3
NNetNav
54.5
69.6
61.1
Qwen2.5VVI.
59.3
88.6
71.1
17.2
71.4
27.8
86.9
73.8
79.8
Qwen2.5-VL.
58.5
87.3
70.0
15.4
64.3
24.8
83.3
59.5
69.4
AER-C
68.8
83.2
75.3
AER-V
67.6
84.0
74.9
Claude 3.75 (A)
69.3
89.1
77.9
11.8
28.6
16.7
84.9
88.6
86.7
Clande 3.7 (S)
69.3
89.1
77.9
14.8
38.1
21.3
80.8
85.1
82.9
Functional
79.0
53.8
GPT-4o (A)
70.2
89.1
78.5
9.6
90.5
17.4
82.9
93.9
88.1
WebArena
GPT-4u (S)
76.2
GPT-44 Mini (A)
63.5
90.8
74.7
10.1
71.4
17.7
70.3
39.5
50.6
GPT-4o Mini (S)
66.9
86.6
75.5
28.6
11.9
86.7
11.4
20.1
Ilama 3.5
68.2
86.6
76.3
76.2
15.4
79.7
86.0
82.7
NNetNav
54.3
90.8
67.9
Qwen2.5-VL
63.6
28.6
63.2
Owen2,5-VL
62.9
V2.4
74.8
9.4
38.1
15.1
90.5
50.0
64.4
AER-C
100,0
81.1
89.6
AER-V
96.4
73.0
83.1
Claude 3.7 (A)
85.0
91.9
88.3
50.0
14.3
76.0
86.4
80.8
Claude 3.7 (S)
85.3
78.4
81.7
0.0
0.0
0.0
77.3
77.3
77.3
Functional
100.0
GPT-44 (A)
94.6
94.6
94.6
2.6
50.0
5.0
70.4
86.4
77.5
WorkArena
GPT-4o
93.8
81.1
100.0
72.0
GPT-4o Mini (A)
84.2
86.5
85.3
3.3
50.0
6.2
66.7
36.4
47.1
GPT-44 Mini (S)
90.3
75.7
82.4
0.0
0.0
100.0
18.2
30.8
Illama 3.3
94.3
89.2
91.7
0.0
61.8
81.8
81.8
NNetNav
77.3
91.9
83.9
Qwen2.5-VL. (A)
87.2
0.0
Qwen2.5-VL.9 (8)
93.8
81.1
87.0
0.0
0.0
0.0
86.7
59.1
70.3
AER-C
66.7
42.3
51.8
AER-V
59.3
30.8
40.5
Claude 3.75 (A)
66.7
62.7
38.9
87.5
97.4
Claude 3.7 (S)
66.7
50.0
57.1
13.6
61.1
22.2
67.3
98.9
92.8
Functional
83.3
GPT-40 (A)
63.0
55.8
39.2
5.5
100.0
10.3
83.6
98.5
91.6
WorkArena+
GPT--4 (S)
59.6
53.8
36.6
100.0
10.4
84.5
98.2
90.8
GPT-44 Mini (A)
49.4
76.9
60.1
5.2
83.3
9.7
67.8
52.9
66.1
GPT-4o Mini (S)
54.8
65.4
39.6
4.6
33,3
8.1
96.5
20.2
33.4
Llama 3.3 (A)
62.7
61.5
83.3
NNetNav
43.2
78.8
55,8
Qwen2.5-VL. (A)
60.3
77.8
Qwen2.5-VL (5)
64.4
73.1
68.5
77.8
11.8
93.2
75.7
83.6
Table 11: Finegrained results by benchmark and judge for Claude 3.7 Sonnet agent. We report the precision (P) as the primary metric, and Fl and recall (R) as the auxiliary metrics.
21

---

Published as a conference paper at COLM 2025
Benchmark
Judge
Success
Side Effect
Repetition
P
R
R
AER-C
83.3
62.5
714
AER-V
83.3
62.5
Claude 3.7 S. (A)
87.5
87.5
87.5
7.1
33.3
11.8
70.1
94.0
80.3
(S)
71.4
33.3
Functional
25.0
12.5
16.7
GPT-44 (A)
77.8
87.5
82.4
3.2
100.0
6.3
68.6
96.0
80.0
AssistantBench
GPT-4o (S)
77.8
87.5
92.4
100.0
67.1
94.0
78.3
GPT-4o Mini (A)
80.0
100.0
88.9
2.7
66.7
5.2
59.3
32.0
41.6
GPT-4oo Mini
33.3
Ilama 3.3 (A)
75.0
75.0
75.0
3.4
100.0
6.6
65.8
96.0
78.0
NNetNav
20.8
31.2
Owen2.5-VI. (A)
72.7
100.0
84.2
0.0
73.8
62.0
67.4
Qwen2.5-V1.
70.0
87.5
77.8
3.7
66.7
10.5
72.2
52.0
60.5
AER-C
56.0
70.9
62.6
79.7
Claude 3,7 (A)
61.0
77.2
68.2
16.7
35.7
22.7
78.7
96.8
86.8
Claude 3.7
64.8
74.7
69.4
42,9
24.5
77.8
97.6
Functional
85.2
58.2
69.2
GPT-4o (A)
63.0
86.1
72.7
12.4
89.3
21.8
75.2
98.4
85.2
VisualWebAAeriaa
GPT-4u (S)
60.7
GPT-44 Mini (A)
57.9
83.5
68.4
10.7
64.3
18.4
74.4
46.0
56.9
GPT-4o
57.4
73.4
11.2
35.7
87.1
Llama 3.3 (A)
59.6
74.7
66.3
12.0
82.1
21.0
74.4
92.1
82.3
NNetNav
54.5
69.6
61.1
Qwen2.5VVI.
59.3
88.6
71.1
17.2
71.4
27.8
86.9
73.8
79.8
Qwen2.5-VL.
58.5
87.3
70.0
15.4
64.3
24.8
83.3
59.5
69.4
AER-C
68.8
83.2
75.3
AER-V
67.6
84.0
74.9
Claude 3.75 (A)
69.3
89.1
77.9
11.8
28.6
16.7
84.9
88.6
86.7
Clande 3.7 (S)
69.3
89.1
77.9
14.8
38.1
21.3
80.8
85.1
82.9
Functional
79.0
53.8
GPT-4o (A)
70.2
89.1
78.5
9.6
90.5
17.4
82.9
93.9
88.1
WebArena
GPT-4u (S)
76.2
GPT-44 Mini (A)
63.5
90.8
74.7
10.1
71.4
17.7
70.3
39.5
50.6
GPT-4o Mini (S)
66.9
86.6
75.5
28.6
11.9
86.7
11.4
20.1
Ilama 3.5
68.2
86.6
76.3
76.2
15.4
79.7
86.0
82.7
NNetNav
54.3
90.8
67.9
Qwen2.5-VL
63.6
28.6
63.2
Owen2,5-VL
62.9
V2.4
74.8
9.4
38.1
15.1
90.5
50.0
64.4
AER-C
100,0
81.1
89.6
AER-V
96.4
73.0
83.1
Claude 3.7 (A)
85.0
91.9
88.3
50.0
14.3
76.0
86.4
80.8
Claude 3.7 (S)
85.3
78.4
81.7
0.0
0.0
0.0
77.3
77.3
77.3
Functional
100.0
GPT-44 (A)
94.6
94.6
94.6
2.6
50.0
5.0
70.4
86.4
77.5
WorkArena
GPT-4o
93.8
81.1
100.0
72.0
GPT-4o Mini (A)
84.2
86.5
85.3
3.3
50.0
6.2
66.7
36.4
47.1
GPT-44 Mini (S)
90.3
75.7
82.4
0.0
0.0
100.0
18.2
30.8
Illama 3.3
94.3
89.2
91.7
0.0
61.8
81.8
81.8
NNetNav
77.3
91.9
83.9
Qwen2.5-VL. (A)
87.2
0.0
Qwen2.5-VL.9 (8)
93.8
81.1
87.0
0.0
0.0
0.0
86.7
59.1
70.3
AER-C
66.7
42.3
51.8
AER-V
59.3
30.8
40.5
Claude 3.75 (A)
66.7
62.7
38.9
87.5
Claude 3.7 (S)
66.7
50.0
57.1
13.6
61.1
22.2
67.3
98.9
92.8
Functional
83.3
GPT-40 (A)
63.0
55.8
39.2
5.5
100.0
10.3
83.6
98.5
91.6
WorkArena+
GPT--4 (S)
59.6
53.8
36.6
100.0
10.4
84.5
98.2
90.8
GPT-44 Mini (A)
49.4
76.9
60.1
5.2
83.3
9.7
67.8
52.9
66.1
GPT-4o Mini (S)
54.8
65.4
39.6
4.6
33,3
8.1
96.5
20.2
33.4
Llama 3.3 (A)
62.7
61.5
83.3
NNetNav
43.2
78.8
55,8
Qwen2.5-VL. (A)
77.8
Qwen2.5-VL (5)
64.4
73.1
68.5
77.8
11.8
93.2
75.7
83.6
Table 12: Finegrained results by benchmark and judge for GPT-4o agent. We report the precision (P) as the primary metric, and and recall (R) as the auxiliary metrics.
22

---

Published as a conference paper at COLM 2025
System Prompt Template
are Language Model specialized in judging the performance of web agents web navigation tasks. For a certain website, you are given the goal a navigation task, the current URL of the webpage, the actions taken by the agent, and the thought process of the agent. Additionally, you will have access to the first and last screenshots, well the last accessibility tree. Your task is answer several questions about the agent's performance in the task.
Question 1: Was the sequence of actions below successful in achieving the goal? ssucess/Unsucessfuu///uucce
Choices: <sucessSSucessfull//cccee
Question the agent perform unnecessary actions that could to unintended
side effects?
Choices: <sid>>Ys<<sside: <side>No</side>
Question Did the agent perform the task optimally, by only performing necessary actions and avoiding unnecessary ones?
Choices: <optimal>1. Complete Failure</optimal>, <optimal>2. Suboptimal<<opttim>>>>> <optimal>3. Somewhat 0ptimal</optimal>, <optimal>4. Completely Optimal</optimal>
Question 4: Did the agent loop through a sequence of actions that did not make progress towards the goal? Choices: <loop>Yes</loop> <loop>No</loop>
You should provide your reasoning process for each question, and you can refer the screenshots and the accessibility tree to support your answers. Your answer must follow the following format:
<reasoning-your reasoning here</reasoning>
<succes>>anserr<succcs>> <side>answer</side> <gotimal>annserr</ppttmm>>>>>>>> <loop>answer<</oopp>
Figure 5: System Prompt Template used for the simplified judge
User Prompt Template
user goal is: (goal}
The agent performed the following actions:
Step: (step_number)
URL: [url]
Action: (action)
Reasoning: (reasoning)
The last accessibility tree is: {axtree)
Here is the screenshot of the last step. (screenshot)
Provide your reasoning and answer the four questions from the system prompt, using the specified format.
Figure 6: User Prompt Template used for the simplified judge
23


Reviews:
[{"review_id": "hZZFIFi9cD", "reviewer": "Reviewer_NzzY", "summary": "This paper introduces a benchmark, AGENTREWARDBENCH, to assess the effectiveness of LLM judges for evaluating web agents. LLM judges evaluate web agent in terms of success, side effects, and repetitiveness of the agent. In this paper, 12 LLM judges are evaluated on this benchmark. It is found that no single LLM excels across all tasks and rule-based evaluation tend to underestimate the capability of web agents.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset introduced by this paper can be of interest to the research community, e.g. in terms of its scale and diversity: it consists of 1300 trajectories produced by 4 popular LLM agents on 5 diverse web environments, with expert annotations of the ground truth judgement.\n\n2. Multiple aspects are used to evaluate LLM agents comprehensively, including success, side effects and repetition cycle. Not only is the execution success, but also the effectiveness of LLM agents, is taken into consideration.\n\n3. The paper is clearly structured and easy to read. The multiple diagrams or illustrations help reads understand the paper better. The mathematical description of the assessment framework (including trajectory definition, annotation design, judge model, etc.) is rigorous.", "reasons_to_reject": "1. In result reporting, the paper uses precision as the primary metrics, and recall and F1 as the secondary metrics. However, the recall can be as important as precision in many RL applications. For example, if an LLM judge is employed as a reward model, it is important that incorrect trajectories are not noted as correct to give LLM agents wrong signals.\n\n2. The \"error analysis\" section is not very complete: it only includes only failure cases on the \"success / completion\" dimension of LLM agents, missing the other two dimensions (side effect, repetition) included in the benchmark.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "notUsmqIJ9", "reviewer": "Reviewer_NwGC", "summary": "The authors introduce AgentRewardBench, which is a new benchmark for evaluating the effectiveness of LLM judges that assess the success of web agents. While there are existing benchmark datasets that evaluate LLM-based judges for other tasks, this seems to be the first one that evaluates LLM judges for web agents. The dataset contains 1302 trajectories generated by 4 different LLMs across 5 popular web agent benchmarks and annotated by human experts on three dimensions: success/failure, side effects and repetitiveness.\n\nThese are some of their findings:\n- The official rule-based evaluations of these 5 benchmarks can be inflexible and tend to under report the success rate\n- None of the existing judges they evaluated exceeded 70% precision compared to expert annotations\n- Existing LLM judges fail due to some common error categories such as misleading web-agent reasoning, failing to report missed instruction details, grounding mismatch (judge does not have access to screenshots, only the web agent's thought process) etc.\n\nThe paper is well-written and structured. The authors clearly describe the problem statement, proposed benchmark, experimental details, results etc. along with informative figures and tables. They also include an in-depth evaluation of the results and qualitative error analysis. The impact of this paper is potentially high, given that it's based on some existing popular web agent benchmarks. I believe the paper should be accepted.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and structured\n- Useful first-of-its-kind benchmark for evaluating LLM judges focused on web agent evaluation. Tackles the under-explored area of evaluating the evaluators themselves.\n- The tasks and trajectories are based on five popular web agent benchmarks, so will prove valuable to many users.\n- Useful analysis of the current limitations of existing LLM judges as well as official rule-based evaluations.\n- Qualitative error analysis highlighting common error categories where existing LLM judges fail.\n- The authors provide substantial details regarding their methodology, including agent platform, annotation processes, example prompts etc. which helps with reproducibility and verifiability of their findings.\n- This benchmark will help improve automatic evaluation methods, which in turn will be useful for synthesizing trajectories for fine-tuning, reward models for reinforcement learning etc.", "reasons_to_reject": "Some minor points:\n\nRelated works should have focused more on LLM judges and existing benchmarks that evaluate them rather than web agents / environments / trajectory synthesis.\n\nThe authors mention that \"The team of annotators consisted of 6 experts who work on web agents research\" ... which suggests that the annotators were probably the authors themselves or close colleagues from the same research group. This does have a few disadvantages: \n --(1) these researchers might interpret success or failure in a way that's somewhat different from the more typical user \n --(2) A small group of closely working experts might develop a very specific, shared understanding of the annotation criteria that isn't explicitly documented and might not be easily replicable by an external group. The authors state that annotators resolved disagreements/ambiguous situations through discussion. The resulting consensus should be compiled and included in the appendix as annotator guidelines.\n  On the other hand, having experts with deep domain knowledge as annotators is also beneficial (focusing on things that matter rather than quirks of web agent trajectories), since questions related to trajectory optimality and side effects of unnecessary actions might be harder for the average user to answer.\n\nOverall, the strengths of this paper outweigh these minor points.", "questions_to_authors": "- Nitpick: In the Appendix, Line 512 mentions AgentLab, before you actually describe what it is in the next paragraph (Line 519-522).\n- Was the code/dataset not included as supplementary data?\n- Did you consider annotating for more granular success/failure signals, such as 'partially successful,' or did you find it unnecessary?", "extra_scores": {}}, {"review_id": "cL42T0TG8w", "reviewer": "Reviewer_9MNf", "summary": "This paper presents AgentRewardBench, a new benchmark for evaluating reward models for web agent trajectories. This benchmark consists of over 1300 trajectories generated by multiple agents across five web agent benchmarks, with each trajectory manually labeled by human experts. This work systematically evaluates 12 LLM-based judges and existing rule-based systems, providing detailed comparative analysis, ablation studies, and error cases. This work is timely for the community to understand the current status of reward model for web agents and further develop and evaluate these reward models. The paper is well-written.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper studies a very important problem for the agent community: building and evaluating reward models for agent trajectories, which can potentially benefit a lot of relevant research, like reinforcement learning, inference time search, etc.\n\n* This work presents a thorough and systematic study regarding the performance of different judges across five web agent benchmarks under a unified setting, along with a details analysis and insights.  \n\n* The annotation protocols and the released dataset can provide valuable resources for the community to enable further research, especially on scaling up the reward label scale and further evaluating the reward model with training.", "reasons_to_reject": "* The evaluation of judgment methods appears to be missing an important baseline of the Agent-as-Judge approach. This method is recognized as a powerful method for the automatic evaluation of agent trajectories. Including results for this baseline would strengthen this study and provide a more comprehensive comparison.", "questions_to_authors": "* Training models towards a reward model is an exciting application of the annotated reward labels provided in this work. Have the authors explored training a model using the collected dataset? If so, how does this influence model or agent performance? Any results or insights from such experiments would be valuable to share.", "extra_scores": {}}, {"review_id": "g6KMDYaNrq", "reviewer": "Reviewer_2xan", "summary": "The paper propose the AgentRewardBench a benchmark to assess the effectiveness of LLM judges for evaluating web agents. The benchmark has1300 trajectories collected based on popular LLMs across 5 diverse web environments. The benchmark uses a simplified judge design that can receive either the accessibility tree or a screenshot as the representation of the final state. The author also shared several important findings from the evaluation.\n\n## Quality\nThe paper constructs a benchmark dataset comprising 1300 annotated trajectories from multiple web agents across diverse benchmarks. It also leverages human annotations to provide high-quality ground-truth labels. The paper also evaluates several large language models as judges, comparing them with both expert annotations and conventional rule-based evaluations. In the error analysis, it also provides comprehensive analysis and insights.\n\nHowever, it does not cover the information related to the computational overhead or scalability.\n\n## Clarity\nThe paper is well written with clear structure and motivation.\n\n## Originality\nThe introduces the first benchmark about evaluating the reliability and accuracy of LLM judges for web agent trajectories. It provides the limitations of rule-based evaluation methods compared to LLM-based evaluation, contributing original empirical findings.\n\nHowever, this paper only has incremental innovation.  Even the benchmark itself is novel, the underlying concept is not new.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "## High quality\nThe paper constructs a benchmark dataset comprising 1300 annotated trajectories from multiple web agents across diverse benchmarks. It also leverages human annotations to provide high-quality ground-truth labels. The paper also evaluates several large language models as judges, comparing them with both expert annotations and conventional rule-based evaluations. In the error analysis, it also provides comprehensive analysis and insights.\n\n\n## Clear presentation\nThe paper is well written with clear structure and motivation.", "reasons_to_reject": "## Limited innovation\nThis paper has incremental innovation. The benchmark itself is novel, however the underlying concept is an incremental advancement.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: AgentRewardBench introduces a 1.3 k-trajectory benchmark to see how well LLM “judges” agree with humans on web-agent outcomes. Reviewers largely align on the same strengths (useful dataset + clear study) and note similar weaknesses (metric emphasis, missing baselines/docs):

Strengths
1. The dataset is large and diverse (NzzY, NwGC, 9MNf, 2xan)
2. The paper is the first to target LLM judges for web agents (NwGC, 9MNf)
3. Evaluation is thorough: 12 judges vs rule-based baselines + error analysis (NwGC, 9MNf, 2xan)
4. The paper provided enough detailed methodology & prompts that supports reproducibility (NwGC)

Weaknesses
1. The evaluation is a bit biased towards certain metrics and cases (NzzY), e.g., metrics favors precision over recall, error analysis focuses on success demisions. Authors explained the reason and expand appendices with more analysis.
2. The paper missed some related work on evlauation of judges (NwGC, 9MNf). Authors mentioned they would add papers Agent-as-Judge, RewardBench, etc. & also mentioned AsJ is hard to be adapted to this setting.
3. Data distribution warrants more discussion -- e.g. Annotator pool seems small and homogeneous (NwGC). Authors mentioned they would create new appendices for this.
4. There were also some concerns on the conceptual novelty (2xan), though the reviewer also agreed that the dataset itself is a novel contribution. 

Overall most reviewers find this to be a valuable paper and a useful benchmark for the community. We hope that author could make the promised changes to the revised paper and release the code and the data!
Decision: Accept