The BrowserGym Ecosystem for Web Agent Research
Authors: Thibault Le Sellier de Chezelles
Maxime Gasse
Alexandre Lacoste
Massimo Caccia
Alexandre Drouin
Léo Boisvert
Megh Thakkar
Tom Marty
Rim Assouel
Sahar Omidi Shayegan
Lawrence Keunho Jang
Xing Han Lù
Ori Yoran
Dehan Kong
Frank F. Xu
Siva Reddy
Graham Neubig
Quentin Cappart
Russ Salakhutdinov
Nicolas Chapados
Venue: Accepted by TMLR ()

Abstract: The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs) for web interaction tasks. Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (2024) introduced BrowserGym which aims to solve this by providing a unified, gym-like environment with well-defined observation and actionspaces, facilitating standardized evaluation across diverse benchmarks. We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, a complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. This standardized approach seeks to reduce the time and complexity of developing web agents, supporting more reliable comparisons and facilitating in-depth analysis of agent behaviors, and could result in more adaptable, capable agents, ultimately accelerating innovation in LLM-driven automation. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic’s latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.

Published in Transactions on Machine Learning Research (02/2025)
The BrowserGym Ecosystem for Web Agent Research
Thibault Le Sellier De Chezelles 123
Maxime Gasse 123
Alexandre Lacoste"
Core contributors:
Alexandre Drouin 12
Massimo Caccia
Léo Boisvert (23 Megh Thakkar
Tom Marty
Rim Assouc!?? Sahar Omidi Shayegan' !25
Benchmark contributors:
Lawrence Keunho Jang
Xing Han L Ori Yoran Dehan Kong Frank F. Xu"
Affiliated advisors:
Siva Reddy! 125
Quentin Cappart
Graham Neubig* Ruslan Salakhutdinov Nicolas Chapados 123
Lead:
Maxime Gasse 123 Alexandre Lacoste
ServiceNow Research 2 Mila Polytechnique Montréal
Carnegie Mellon Universily McGill Universily Tel Aviv Universily Universilé de Montrénl MiMenn AT
Reviewed on OpenReview:
Abstract
The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchnarking of web agents, particularly those leveraging automation and Large Language Models (LLMs) for web interaction tasks. Many existing benchmarks suffer from fragmentation and inconsistent evalu- ation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (024)) introduced BrowserGym which aims to solve this by pro- viding a unified, gyk-like environment with well-defined observation and action spaces, facilitating standardizod evaluation across diverse benchmarks. We propose an extended BrowserGym- ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. This standardized approach seeks to reduce the time and complexity of developing web agents, supporting more reliable comparisons and facilitating in-depth analysis of agent behaviors, and could result in more adaptable, capable agents, ultimately accelerating innovation in LLM-driven automation. As a supporting evidence, we conduct the first large-scale, multi- benchmark web agent experiment and compare the per- formance of 6 statc-of-the-art L.LMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepaneyy between OpenAl and Anthropic's latests models, with Claude-3.5-Sonme Sonnet leading the way almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains significant challenge, due to the inherent complexity of real-world web enviromments and the limital ions of current models.
1 Introduction
In recent. years, the emergence of powerful large language models (LLMs) and vasion-language models (VLMs)
OpenAl 2024al
Authropic
2024a
Metal 2024) has led to revolution in the field of conversational assistants,
"Fqual contribution.

---

Published in Transactions on Machine Learning Research (02/2025)
otherwise known chatbots. Since then, the capabilitics of these traditional, chat-only assistants have been expanded to include rumning search queries on the web, reading user documents, executing sand-boxed code, and generating images. Of particular interest in this work is the capability for conversational assistants to perform actions in a web browser on behalf of the user, by directly manipulating its human-facing graphical user interface (UI).
Many everyday tasks we perform on the web are simple, yet repetitive and time-consuming. Filling out redundant administrative forms, visiting e-commerce websites in search of the best deal, or manually synchronizing different calendars are all tedious tasks that involve multi-step exccution. Having autonomous web assistants that follow instructions and execute such tasks our behalf would empower users to focus higher-level, higher-value decisions instead. They also represent great opportunity to improve digital accessibility, by helping visually or otherwise impaired users who may find certain websites difficult to navigate. Lastly, assistants acting through a UI offer interpretability, since users can visually verify what the agent does on the screen, along with wide compatibility, bypassing the need to develop specific application programming interfaces (APIs) adapt existing software for the agentic cra. We also note that web agent research conveniently accessible LLMs due the textual modality of the UT, although most research planning and reasoning with web agents is likely to translate to pixel-based UT agents.
AgentLab
blue components are part of AgentLab
Efficient Parallel Experiments
Your Agent
Dynamic Prompting
Unified LLM API
Reproducibility
Unified LeaderBoard
BrowserGym
Xray Anslyse your agent custom Ul
WebArena
MiniWoB
WorkAreno
WebLinx
Dataset
Share agent traces for collective dataset
Mind2Web-live
VisualWebArena
AssistantBench
Figure 1: Overview of the ecosystem, including AgentLab, BrowserGym, and the web agent benchmarks.
The lield of web agents has been particularly active in the last few years. Most recent works focus on evaluating the capabilities of state-of-the-ar LLMs V VLMs at solving tasks of varying difficulty, which has led to abundance
of new web agent benchmarks (Zhou et al., 2024b Koh et al. 2024a Lü et al., [2024] Drouin et al.) 2024 He et al.
2024 Boisvert et al. 2024 Yoran et al. 2024 Pan et al. 2024.. Unfortunately, with this multitude of contributions
also comes a multitude of diverging codes for evaluating web agents, which yields research silos with inconsistent. practices and incompatible implementations. This fragmentation detrimental to the fair comparison of existing web agent implementations across benchmarks, which compounds the cumbersome process of installing diverse setups, adapting data formats, and manually incorporating agents into each separate benchmark.
In this paper, introduce the BrowserGym ecosystem, highlighted in Figure which is meant to provide open, easy-to-use, and extensible framework to accelerate the field of web agent research] It. consists of AgentLab, which offers the tooling for building web agents and running large-scale experiments, and unified collection of
It important that this work provide research framework, not consumer product.
2

---

Published in Transactions on Machine Learning Research (02/2025)
web agent benchmarks exposed through BrowserGym (Drouin et al. 2024). which provides the standard interface for web tasks and web agents. In its philosophy, the BrowserGym ecosystem intends to address the three following use cases: a) the design of new web agents, and their straightforward evaluation on wide range of existing benchmarks; b) the dosign of new benchmarks, and the straightforward evaluation of cxisting web agents on them; c) the comparison of different models (LLMs/ / VLMs) in their ability to solve web agent lasks, by switching the backbone model in a state-of-the-art web agent implementation.
We summarize contributions as follows:
We expand the existing BrowserGym library from Drouin et al.
(0024) (Section 3). and we provide a
4 proposed by the scientific community in BrowserGym, ranging from MiniWoB(++)) SShi ct al. 2017 Liu al. 2018) to more recent benchmarks such as AssittantBench (Yoran al. 2024) and
unification of existing benchmarks (Section
(Koh al., 2024ap. Despite their differences, all benchmarks are made available through the same, unified BrowserGym interface.
We introduce Agt (Section 5) set of tools to simplify parallel large-scale experimentation with agents over BrowserGym in a reproducible manner. It also comes with AgentXRay, a visual tool to introspect the behavior of agents individual tasks. Finally, it provides reusable building blocks to accelerate the development of new agents.
We conduct the first, large-scale web agent experiment (Section 6 showeasing the BrowscrGym ecosystem. We compare the performance of six stale-oftheeart LLMs / VLMs from the GPT-4 (OpenAl 2024a), Claude 3.5 (Anthropic 2024a) and Llama 3.1 families (Meta 2024 across all web agent benchmarks available in BrowserGym. Among others, these results highlight an impressive performance for the new Claude-3.5-Somme model, which reaches an umprecedented success rate of 39.1% on the WorkArena T.2 benchmark, to be compared with 8.5% for GPT-4o (2nd place). These results serve as first entries to the BrowserGym leaderboard
2 Background and Related Works
The field of autonomous agents has long been a major research interest both in academia and industry, with a recent surge in LLM-based agents (Wang et al. 2024a), motivated by the emergence of reasoning and planning
capabilities in L.MSs (Huang et al., 2024b Wang al.
2024cp- While large body of work focuses on software Hao al. 2024 Du ct al. 2024), another line of research focuses
control via APIs Yang et al. 2023a
automating human-like interactions, by directly manipulating Uls mobile devices (Li et al. 2020 Rawles el al.
2023 2024), desktops (Xie et al. 2024 Bonatti et al. 2024), and websites (Furuta et al. 2023 Gur et al. 2023a
Kim et al. 2023 Drouin ct al. 2024 Là ct al. 2024). This last category encompasscs the field of web agents, which can automate software interaction even in environments without APIs, improve human productivity, and provide accessibility for users with disabilities. Next, we provide an overview of the recent. literature related to web agents, which we divide into two parts: web agent benchmarks and web agent implementations.
2.1 Web Agent Benchmarks
Given the rapid pace of improvement in the field of Artificial Intelligence (AI), web agent benchmarks have been consistently evolving to evaluate more and more complex capabilities. Early benchmarks such MiniWoB and MinWWo+++ (Shi et al., 2017 Liu et al. 2018) established foundational metrics for evaluating basic web interactions like form filling and button clicking in controlled environments. WebShop (Yao et al. 2022a) later introduced complex e-commerce scenarios, requiring agents to navigate product catalogs while considering user preferences and constraints. Mind2Web (Deng et al. 2023) further expanded this scope and released a dataset of over 2,000 human traces collected from open-ended tasks across 137 real-world websites, along with a structured approach to evaluate multi-step interactions to develop and evaluate generalist web agents. Web Voyager (He) et al. 2024) proposed a live, online benchmark for evaluating web agents on world tasks from 15 popular
1 httpa://github.cc htwps/hwwwwwcccmeeeeeeee/////tte
3

---

Published in Transactions on Machine Learning Research (02/2025)
websites. WebCanvas (Pan et al. 2024 later extended Mind2Web from static dataset to online benchmark called Mind2Web-Tive, by proposing key-node evaluation mechanism for web agents.
WebArena. (Zhou et al. 2024b) along with its counterparts Visual WebArena (Koh et al. 2024a and VideoWeb- Arena (Jang et al. 2024. brought innovation by testing agents on live websites rather than simulated environments. This approach evaluated agents' abilities to handle real-workd variability in website design and the dynamic nature of the content. JisualWWeAAenn specifically addressed the role of visual understanding in modern web interfaces by introducing evaluation tasks that require interpreting and manipulating visual content across e-commeroe websites, including product image comparison, visual search, and spatial reasoning about interface layouts. WebLINX (Li ct al. 2024) focuses on the instruciionfollowing abilitics of agents and provides datasct of over 100k human inter- actions across 155 real-world websites. Building in the same direction, AssistantBench (Yoran et al. 2024) presents 214 realistic tasks requiring agents to navigate multiple websites to gather and synthesize information. MMInA (Zhang et al., 2024b) extends this further with 1,050 multimodal tasks that require agents to navigate between multiple websites, spocifically evaluating their ability to handle both visual and textual information across naturally compositional (asks. In the same vein, GATA (Mialon al. 2023) proposes another question-answering benchmark that requires agents to navigate and combine information from diverse sources of the web to find the correct answer.
While these benchmarks focus on everyday websites, WorkArena and +orkArena++ (Drouin et al.
2024 Boisvert
et al. 2024 rodueed the first benchmark for web agents in the enterprise software setting, eva evaluating on common knowledge work tasks following realistic workflow. CRMArena (Huang et al. 2024a) proposes nine customer service tasks distributed across three personas: service agent, analyst, and manager, designed to evaluate the abilities of AI agents in CRM systems.
Broader evaluation frameworks like AgentBench (Liu ct al. 2023b). AndroidWorld (Rawles ct al. 2024 OS World (Xie et al. 2024), and Windows Agent Arena (Bonatti et al.) 2024g incorporate web tasks as part of more general agentic environment, providing insights into more general agent capabilities. These benchmarks explore the intersection of web, desktop, and mobile interfaces, recognizing the incrcasingly blurred boundaries between user interfaces. Addressing the need for safety evaluation, ST- Weneeneggeneecch (Levy et al.) 2024) introduces the first benchmark specifically focused on assessing web agents' compliance with organizational policies and safely requirements in enterprise settings.
2.2 Web Agent Implementations
Due to their immense potential, there has been a substantial effort in developing web agents. in how they interact with the web environment, the backbone models, and the goal they aim to achieve, among
These agents differ
others. Popular approaches either interact directly with HTML elements (Nakano et al.
2022
Deng et al.
2023
Gur et al. 2023b 2024) or leverage vision-language models (Liu et al. 2023a OpenAl
2024a
Anthropic
2024a
Meta| 2024 ; by grounding screenshots to web elements (He et al. 2024 Furula el al. 2024
Zheng et al.
2024a
Koh et al. 2024a Kil et al. (2024) or directly at the pixel-level (Shaw et al. 2023 Cheng et al. [024]] Gou et al.
2024p. Beyond interaction methods, rescarchers have also explored ways to enhance agent reasoning capabilities
through prompting or tuning. Previous works suggested specialized prompting (Sridhar et al.
2023 Zheng et al.
2024c Sarch et al. 2024 Chi et al. 2024), planning and search architectures (Zhou et al.] 2024a Yoran et al. 2024
Zhang et al. 2024a] Gu et al.
2024 Koh et al. 2024b), and training methods (Gur et al. 2019 Nakano et al.
2022 Humphreys et al. 2022
Chen et al. 2023 Putta et al., 2024 Sodhi et al., 2024) to improve performance.
Due to the recent success T.LM-based agents and the challenging reasoning required, agents often use prompting
tochniques such as chain-of-thoughtt (Wei ot al. 2023), troo-of-thought (Yao ct al. 2023). ReAct (Yao et al. 2022b
and RCI (Kim et al. 2023) Wang al. (2024a) provides a survey of LM-based agents.
Other works focus on introducing frameworks for web agent development, by simplifying access to evaluation Ma et al.) 2024), agent development (Xic et al. 2023), and interaction with live websites (Pan el al. 2024 Zheng et al. (224b).. It is thus important to focus on standardizing web agents research, aiming to simplify future work, and introducing research best evaluation with statistical power across benchmarks, simple ecosystem-growe
datascts (Lhoest et al., 2021 UK AI Safety Instituto. 2024
enabling open-source research, and reprodueible experiments at scale. Such a framework is needed not only for the development of the next generation of open-source web agents but also for general tasks that require web access,
4

---

Published in Transactions on Machine Learning Research (02/2025)
including the development of autonomous coding agents (Wang al. 2024b). To address those important points, we developed BrowserGym and AgentLab, which we describe in detail in the subsequent sections.
3 BrowserGym
BrowserGym provides a unified interface for web agents, Lee,, conversational assistants that can use a web browser as tool. As showcased in Figure 2 both the user and the agent have access to a chat where they can exchange messages, and a browser in which they can perform actions, such as lyping lext, clicking buttons, or opening tabs. The user typically writes instructions in the chat, and the agent tries to follow them by navigating across pages, performing UI actions, extracting information, and writing back messages in the chat.
appears partally circle above the body.
head
2he Drawing the Hoad craw circle form
Let's add the head now:
mouso, drag and dropi210,, 180, 280,
Hill your uooistert, perform
wit??
W
aticmman
Figure rendered BrowserGym environment, with the chat on the left and the web browser on the right.
Implementation From the agent's perspective, this interaction loop can be conceptualized a Partially Observable Markov Decision Process (POMDP), where the environment (chat and browser) provides the current observation and reward, and the agent produces the next action. BrowscrGym environments follow the standard OpenAl Gym API [Brockman et al. 2016 2016)/Towers et al. 2024 and are made available in Python through the gymnasium 5 interface (Figure 3 Internally, BrowserGym relies on the Chromium [0) browser and the Playwright library to automate browser interactions.
import gymaasiun gyn
action (at)
environnend creation
eav make take("bar http://ggw/.niiiee.ccoooeeeeeeee
Agent
Environment
interactson loop
ohs, info anv, resat()
while action - # agent reasoning és performed here
observation (ot), reward (rr)
obs, revard, terminated, truncated, (tv.step(action) if terminated or truncated: break
Figure The BrowserGym interaction loop: (left) theoretical model, (right) concrete Python API.
3.1 Extensive Observation Space
BrowserGym provides a rich observation space, with its main components being the task's goal (or chat history), the list. open tabs, and the current page's content.
https: httpp://gymmasium farama. org/ https://wweg http:///wwwccccc..uo.. chromium. org/ Home/
https://playrrigghe dev/ hyth///

---

Published in Transactions on Machine Learning Research (02/2025)
Submissions
Comments
Hot
Submissions
Comments
Hot
[homemade] Obligatory Halloween Pumpkin Loaf!
Homeeade H
Halloween L
17705
Submitted Insechuliee [15, yidiil 2 years ago
17705
B8
Submmed by tod ta 2 years ago
comments
Il atel Maple Pecan Croissant
PWY
Patel Mãolé Pečan Croissant
Raw screenshot
(b) Set-of-Marks prompting Yang al.
2023b
<form action=/sx//100r hid="167" class="vote">..
<button bid="169" "Iplotttpvota" type="aubnit"
value="1">... "rfp...//button
<apan bid-"174" class- "voco___ntt-score"
data-rote-target 17705
</span?...
[167] visible
Chutton bid="179" title="Dounota"" typa="aubnit"
[169] button 'Upvote', clickable, visible
value="-1122 12... </button?
StaticText '17705'
</forn>
[179] button 'Dounvote', elickable, visible
(c) HTML
(d) AXTre Zhou et al. 2024b
Figure Different visual and text representations of the current page from BrowserGym.
Structured page description BrowserGym provides the raw DOM (Document Object Model; Web Hypertext
Application Tochnology Working Group 2024) and the AXTree (accessibility tree; World Wide Web Consortium 2017) structured representations of the current page, which are extracted using the Chrome Developer Protocol (CDP). They are both provided as-is (dom_ coom__jectt, axtree_ob _bbject with only minimal alterations such as the injection of unique identifier attribute (bid, see next paragraph). It is then left for the web agent implementation to decide how to further process and format. these observations, for example by filtering irrelevant elements or attributes and converting the DOM or AXTree to a suitable text. format for LLM-based text agents (soe Figures 4e and Adh. Utility functions that convert these objects to basic text representation are also provided, for implementing simple agents quickly.
Extra element properties BrowserGym enriches each HTML element in the current webpage with extra information to facilitate web agent interactions. One key property is the BrowserGym ID (bid), a uniquely-generatel element-wise identifier injected into the page's DOM and AXTree, allowing for unambiguous interaction with the elements of the page. To add information about the visual rendering of the page, BrowserGym also provides for each element its bounding box coordinates (bbox) in the form of (left, top, width, height) 4-tuple, as well as its visibility ratio (visibility) in the form of scalar between 0 and 1. Finally, BrowserGym also provides Set-of-Marks boolean indicator (set_of of marks) reproduced from He et al. (2024), which indicates if an element's bounding box should be overlayed on the screenshot to enable Set-of-Marks prompting (Yang
ct al.
2023b
He ct al. 2024).
(sce Figures
4a and 4b
Screenshot While the DOM, AXTree, and extra element propertics provide rich description of the page's content, they do not give the full picture of how the page is being rendered in the browser, i.e., overlapping elements, back- ground colors, image contents, fonts, etc. Therefore, BrowserGym also provides the raw screenshot of the current page as part of the observation, in the form of an RGB image. Again, it is up to the agent implementation to decide how to process and use this visual information. Note that the bbox coordinates in the extra element propertics map exactly to pixel coordinates in the screenshot, which makes extremely easy for agents to overlay the bounding boxes and identifiers of relevant elements on the screenshot in Set-of-Marts prompting (Yang al. 2023b He 2024).
Open tabs In BrowserGym, tasks can use multiple tabs if needed. While the main focus is on the current page (screenshot and page doscription), agents are also made aware of all the open browser pages, which inchude tabs and pop-up windows. BrowserGym's observations therefore include list of all the pages' URLs and
htttp://developerr.r waa. www.iilaa.oorreegeeeee http///aw////ddda/aaa////tttttte hatp://.........o.o/o/////////ae Ctw.rtreeereeeeerarteeee

---

Published in Transactions on Machine Learning Research (02/2025)
titles (open_pages_ pegeepaas__rss,, urls, jegn_pagesttitess] as well as index to distinguish the currently active page factive_page. _a_____adexee......p
Goal and chat messages In BrowserGym web agents can extract the user instructions in two ways, either explicitly from the lask goal [goal_ object) or implicilly from the chat (chat messages). Both consist of a list of text and/or image messages. This duality is offered to facilitate compatibility with legacy web agent and web task implementations, which do not account for interactive chat in their dosign. Note however that BrowscrGym will always inject the goal of non-interactive task in the chat to mimic user interaction, but will not inject further chat messages received during task execution back into the goal. While, to this date, all web agent benchmarks in the literature are non-interactive, it is preferable to implement web agents using that_messages, at the very least to be able to record live, interactive demos.
Error feedback Finally, BrowserGym observations always include the error message of the last executed action, if any (last_ ceatt_cciion_ereee or deactivated, which results in a Playwright exception with an error message. Such exceptions do not break the interaction loop but are instead captured, and are sent back to the agent as part of the next observation. This immediate negative feedback gives web agents a chance to quickly self-correct in the next step (See Figure 5
Typical errors include trying to interact with an element that is hidden
Google
FinsoutBrror: secator.click Tincout 500ms exceeded.
Call log:
waiting get_by_ boi_by_test_iie test_id(("411)
locator reselved to <iaput bid-"261" nane""btaX' tabindex="O" role="button" type= clasa="g10890" valoe="Google Search" />
attempting click action
waiting for alanent visible, enablad and stable
C Westher miniiiel today
<div "ass=""mue_il>>.. intercepts pointer events
Figure 5: Example of an agent trying to click on the "Google Search' button, hidden behind the auto-complete memu. This action results in Playwright that is returned in the next observation's last_actte aetiaacttttetaee field.
Locate and click the upvate button weth Playuright sige..ocator( locator("forre filleer (haa_texttaaaaeeeeeeeeeeaeaatele text="17705" -got_ "rr_rreebbbttt"" "arton").firat
button page.
click using bid click("169")
button. click()
on, focus und press enter ("169", "Enter")
do 65 Javascript pago. www.uaaaaaaaaca.. d'dacunent... quw.pyaloccoorlll ('fomm)) (forn fom. www.tenteet.entnt.o.o includess (*17705")))) -querySelector( 'button') ) click(;")
click using coordinates nouee_olick(x-122 y-197)
send chat meseage
send clat message sead__sg_to_uee ("_Te \'Uprote)' button has been clicked")
send gead_nessage_Re user (Te \'Upoote\" button has been clicked")
(a) Raw Python code
(b) High-level primitives
Figure 6: Different BrowserGym actions for clicking the "Upvote" button of the first post in Figure 4 While raw Python code enables expressive actions using Playwright or Javascript, often relies on some knowledge of the page's HTML structure. Using BrowserGym's high-level action primitives (b) is more restrictive but offers more control over what the agent can do and how it should interact with the browser.
3.2 Expandable Action Space
The fundamental action space of BrowserGym is raw executable Python code (Figure 6a accoss to a Playwright page object along with two functions send message to ueeo_uuerree (text) and re- ceinfeasibleeressssose
Agents have
These respectively allow agents to interact with the browser, and the chat, and terminate the task. This design choice offers a powerful, unrestricted action space for researchers to play with, but also exposes web agent users to vulnerabilities and safety risks due to the possibility of exccuting arbitrary, untrusted code. To mitigate this, BrowserGym offers a control mechanism to further restrict the action space available to web agents, through the use of an action Sosooonnnannnnngaa function.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Action mapping In BrowserGym, users can instantiate environments with optional action__apping function, which the environment uses to convert input actions into executable Python code. It is then easy to define and experiment with new, custom action spaces in BrowserGym, for example by allowing a pre-defined list of actions in JSON or in command-line format, which are parsed and converted to safe, trusted Python code. In case of a conversion error (e-g, parsing errons), any Exception message raised by action (asoonggmmpnn) caught. and fod back the agent as an error foedback.
High-level action set By default, BrowserGym environments use a pre-defined action mapping (Figure üb which converts a of high-level action primitives in the form of function calls into actual Playwright code be executed. An exhaustive list of these primitives can be found in Appendix A Table 3 The resulting action set is encapsulated in the HighlevelActionSet class, which provides to po_python__e (rcthoo__oee((( () method for the environment (the action mapping), and a describe () method for the agent, which builds a textual description
of the available high-level primitives and how to use them (Appendix Figure 10
3.3 Extensibility: create your own task
class MyyKer/tankkkaatrrrrrssossttttt
setup(self, page):
page (tttttttttp/o////oo httpootttt:///gggge.coo com"))
Google
return "Which year was sinstain born?"
def velidate page, nestanees)))e):
stop after fürst ansuer
done chat massagon ["role"] "ansistant"
revard "1879" in chat nesssneee [-1] ("neasage '] nestage "That's correct" if revard elee return reward, done, nessage
Hill 18 sdurt, lave form vin?? Wh Phidyye wou ohoton ban??
ongister_task ("nyneutaaa"""" MyNowTask)
# awetlable le as a gyn envéroseent eav gyn nake "brooeenggm/ayuuvsseee"""
Figure Pseudo-code for creating simple web lask in BrowserGym, and the correspouding rendering.
BrowserGym minimizes the effort needed for practitioners to implement new web tasks. Doing so only requires writing the logic of the task, which takes the form of a setup() and a validate method.
Sctup Starting from blank page object, this method must bring the browser the starting point of the task, and return the goal the web agent must achieve. This will typically involve logging into a system, creating data entries if needed, and navigating to a task-specific starting URL. Then, the method must return a goal which can either be raw string as in Figure 7 or a list of OpenAl-style message that can include text and images, to allow for vision-conditioned goals (c.g., "Find a pair of shocs that look like this imago").
Validate Called after each agent action is executed, this method must check whether the agent has completed the task. It has access to the current state of the browser through the active page, as well as the history of chat chat_ messages which contains all previous user and assistant messages. Typical validation steps involve looking at the content of the current page, searching for database entries and updates, or reading assistant messages in the chat. in search of a correct answer. At the end, this method must produce a scalar reward for the agent, a boolean done flag that indicates task termination, and an optional message to be added to the chat, to simulate user interaction.
4 Unification of Web Agent Benchmarks
We bring 3 new web agent benchmarks to Browser namely WebLINX (Lü et al. 2024). VisuaWWebArena (Koh)
et al. 2024a) and AssistantBench (Yoan et al. 2024h. With these, GrowscrGym currently supports six popular
https://pllttfon http://tw..ap.ateeee.//eeeeeeeea

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Overview the Web Agent benchmarks currently available in BrowserGym. To quantify the task diversity, we report the number of task templates and their seed diversity, egg. in WorkArema, some task templates can have thousands of different configurations. On the other hand tasks in, eig., WebArena are deterministic.
Task Templates
Socd Diversity
Benchmark
Max Steps
Multi-Tab
Web Backnnd
Reference
Shi et al.
2017
MiniWoB(++))
125
medium
10
110
self-hosted pages
Liu et al.
2018
WebArena
812
30
yes
self-hosted docker
Zhou et
20244b
WebAArea
910
none
30
yes
self-hosted docker
Koh et al.
2122
WorkArena L1
33
high
30
ServiceNow demo instance
Drouin
2024
WorkArena L2
341
high
50
yes
ServiceNow denoo instance
Boisvert et al.
12024
WorkArena L3
341
high
yes
ServiceNow demo instance
Boisvert al.
2024
WebLINX
31586
none
no
none (static dataset)
Lu et a..
2024
Assistant Bench
214
none
yes
world wide web
Yoran et
20224
web agent benchmarks, listed in Table 1 Each benchmark consists set BrowserGym tasks, accessible as
BrowserGym environment through the gymnasium interface (Figure 8 Any task from any these benchmarks can then be executed using the same code base, through the same observation/acee /action API, and any web agent implemented for one benchmark readily be evaluated another The advantages the following:
Breaking silos. Unifying benchmarks encourages the development of general-purpose web agents. While benchnark-specific solutions have their own merit (Kim et al. 2023 Sodhi al., 2024) and correspond to real use cases (e.g., company developing web agent for internal website). we believe that the pursuit of generic solutions is more likely to bring long-term, impactful innovation.
Accelcrating rescarch. Proposing unified interface for web agents will also encourage the development of new, compatible benchmarks, stimulate progress in the field, and accelerate research. For example, a new BrowserGym-compatin security benchmark could easily evaluate all existing web agent implementations without the need to spend time and effort adapting agent codes.
Reducing noise. By evaluating across benchmarks, practitioners can collect stronger statistical signals for testing dosign choices and scientific hypotheses. Docs using screenshots improve the performance of web agents? LLM-A better web agent than LLM-B? Evaluating single benchmark with its own biases and specificities only provides a partial answer, while cross- benchmark evaluation gives wider picture.
Daralars
oee your escurtort, laon
You mrsne Youereeeeatbbee srem from Elevvtee festred foums reradd
minor can1 leoo
lon sour US assertnt, can
Drss this procar it the the Aphi and place Subei
Whtt call hald you with!
Featured forume
Chance my weddl be to aae
Suberit
help
help
gyn. maise wyw.maaeee"brorroo ("hroveergyn/niinivoo number www.er- cbeckbozes")
make wwwpoo/wwrggy//bbbbbeeer
Figure 8; Two laskes from the MiniWoB and WebArena benchmarks, accussible Uhrough the same APL
Brroowser Gym also suggests unified API agent creation which streamlines implementation efforts (Section 5.5

---

Published in Transactions on Machine Learning Research (02/2025)
Task coverage Benchmarks currently available in BrowserGym cover broad range of tasks that involve manipulating synthetic Uls (MiniWoB), performing everyday tasks on replicas of real-world shopping, social media, and software development platforms (WebArena, VisualWebArena). executing realistic workflows on an enterprise platform (WorkArena), running time-consuming scarches on the open web (AssistantBenchh) and replicating human interaction traces (WebLINX). For a detailed presentation of each of these benchmarks, we refer the reader Additional benchmarks currently considered for integration include GAIA (Mialon et al. 2023). Web Voyagge He al. 2024,, WebShop (Yao et al., 2022a) and Mind2Web-live (Pan al. 2024h.
to Appendix B
Metadata Each BrowserGym benchmark comes with its own metadata, which lists all the available tasks along with their benchmarkkspeccffee attributes. For example, the metadata of WorkArena (Drouin et al. 2024 Boisvert ct al. 2024) contains the level and the category of each task, which is useful for analyzing the performance of web agents on specific subsets of tasks. The metadata also proposes a default train/test split. for each benchmark, and an optional dependency graph between tasks, which indicates a (partial) order in which tasks should be exccuted to avoid inconsistencies when evaluating agents (e.g, for the WebArena (Zhou el al. 2024b) and VisualWehs WebAAra (Koh al. 2024a) benchmarks).
Suggested evaluation parameters On lop of the metadata, each BrowserGym benchmark has its own suggested action set, suggested number of seeds per task, and suggested maximum mumber of steps to be used for evaluation. These suggosted paramcters are not mandatory but provide a preferred way to evaluate web agents on cach benchmark, following the recommendations of the benchmark authors. For example, the MiniWoB(++) benchmark (Shi et al. 2017; Liu et al., 2018) is not intended to be used in a multiple tab setting, hence does not need the new_tab or close_ tab() action primitives. The Besch benchmark (Yoran et al.) 2024) on the other hand does not implement task randomization, hence does not need be evaluated with more than one seed per task.
Backend preparation Some benchmarks require resetting the backend web server to specific state before the next agent can be evaluated on it. It is the case for WebArena and VisnalWebArena, which require restoring dooker containers to their initial state so that any alteration performed by a previous agent (e.g., changing the user's bio description) is wiped off for the next agent to start in a clean, fresh server state. BrowserGym automates this step with a convenience prepare_ ba routine, which checks the backend configuration (server URL, credentials, etc.) and runs any required reset or setup antomatically.
5 AgentLab
As depicted in Figure D AgentLab is a set of tools to simplify experimenting with agents over BrowserGym. In the high-level API, we define the Study object to organize the experiments across multiple agents on a benchmark. The study is materialized on disk and can be launched to run in parallel across multiple processes (Section 5.2). Once a study is completed can be examined using Agay (Section 5.3), a custom UI made to introspect the behavior of agents individual steps of each task in the study. Due to the challenges of reproducing experiments in dynamic environments, we also implement capabilities to help reproducibility (Section 5.4]. Finally, we also provide building blocks to simplify creating new agents (Section 5.5
5.1 Launching experiments
Once agent is implemented (See section 5.5) it can be evaluated by creating Study object and running it: study Liike__syyy(
benchmark=" "iinwwob", # or "webarena", "workarena l1"
[TrES=[AGETT___00 40 [B_TT____M_N_I_]_____________ MINI], agent_are args= [AGENT_4]
study run www (n_jjob==5)
Study objects are in charge of setting up the experiment and running and saving the reproducibility checks. They also manage the relaunching of failed experiments. In such a broad range of environments, is frequent to experience system failures such a server not responding, edge-case bugs in third party libraries, or rate
10

---

Published in Transactions on Machine Learning Research (02/2025)
limits from LLM servers. Thus, it is crucial to be able to relaunch failed experiments. The method Study run is designed to relaunch up to 3 times the failed tasks, but manual relaunch can also be done as follows:
study Study 1oadd haapdd/"////yyoooiiii /otooo/rrruu//rrrr" study find_ incomplete (include_ errors-True) study run ()
5.2 Parallel experiments
Evaluating a single agent on a benchmark with a reasonable mumber of seeds per task yields several hundred episodes. When exploring multiple agent configurations this can lead to several thousand episodes in a single study. Hence, it is crucial parallelize efficiently the execution of these episodes.
We use multiprocess approach with the possibility to ray (Moritz et al. 2018 jobli most of the compute requirements outsourced to servers (LLM API, TGI server, web server), the actual com- pule requirement of the job is relatively small and we can launch up to 20 tasks in parallel on single laptop or even 10-100 tasks on a server machine with many CPU cores and large memory. With such parallelization, the rate limits of API-based LLMs are usually the bottlenock. more parallelism is required, ray can connect multiple machines into single pool of workers. We reler the reader to Appendix G for further details regarding hardware and runtime.
backend. Since
Task dependencics Benchmarks like (Visual)WebArne have task dependencics to ensure that the exccution of lask will not corrupt the instance and prevent it from properly solving other lasks. The ray backend is capable of handling such dependencies in parallel for the (Visual)WchArent) benchmarks. Also, if the study is created through make_ _akke_stdy, AgentLab ensures the sequential execution of all agents in the study with a proper instance reset between each evaluation.
I but unfortunately this also greatly limits the potential parallelism to 2 tasks
5.3 AgentXRay
To facilitate deeper analysis of agent behavior, AgentLab includes AgentXRay (not be confused with the ray parallel backend), a Gradio-based [3 interface for diving deep into the logged traces. A visual example of this interface shown Figure g After selecting the trace interest AgentXRay displays the profiling, the goal, the different parts the observation, the action taken, and the prompts, providing a comprehensive view of the agent's 's decisions and interactions. This interface allows users to visualize the step-by-step decision-making process of an agent, helping them gain valuable insights and refine their agent for better performance. This is particularly useful for identifying areas where the agent may be struggling, as it allows rescarchers to pinpoint the exact moment, when an incorrect suboptinal decision is made.
5.4 Reproducibility
Several factors can influence the reproducibility of results in the context of evaluating agents on dynamic benchmarks. Having framework robust to the noisy nature of web tasks is necessary. We first discuss core factors that can lead to discrepancies in web agent evaluations and then present the corresponding crucial features that Lab implements to mitigate these discrepancies.
5.4.1 Factors affecting reproducibility
Software version Different versions of Playwright any package in the software stack could influence the behavior of the benchmark the agent. Fixing package versions can help, but il would prevent bug fixes, security updates, and new features.
API-based LLMs silently changing Even for a fixed version, a commercial LLM may be updated, for example, to incorporate the latest. web knowledge.
httpp: httpghhhbb.cc///oll//j///i11 joblib cannot. handle dependency graphs.
https://www.gradio.aap
11

---

Published in Transactions on Machine Learning Research (02/2025)
Agent
Tonk Seed
Comstonts
Global Report
Task Selector (click for help)
Seed Selector (click for help)
name
avg cemazd
avg, stepa
d.compeeter ted
P_
cam, cont
swmadd
n_attppa
126
0.0084
anbazena, 127
3/3
Seed Selection
mebazena.332 129
2/2
00.01144
Some benchmarks have several scods available for their tasks, specitic one can be selected this table
aebazera...
3/2
0.1031
Hobazena.12 131
1/1
0.4439
Task Selection
The task table allows to overview the benchmark's tasks, and select one by clicking on it. success, steps, errors,
1/1
2. 0478
displays
webarena.135 (seed: 31)
Actions
Step emrrr after action:
Step 1/7 (Reward: 1.0)
Click( 2091)
Sccos
Goat:
Think:
Exic killian
Step Info
1/037
countt need resigate the "seyes toery" wer section This should contain couett histeey where Find the mode by
The action and chain of thought reasonning are fetched Agentinfo object and displayed
Task info:
1/2/3023.
Episode Here the task successful, observing step 7, and the
Information goal and task/associated seed
denesicagent- get aekarena.
actAn
Profiling Steps are clickable, to tabs
Screenshot
Screenshot Pajr
Screenshot
Dom
HYML
AXfree
Info
Information
Tabs
acc.com
Once selected, each
explored details.
Screenshot, prompt. any information can be logged for display
Figure Visual rendering of AgentTabbs XRay interface. AgentXRay serves as a powerfull analysis tool to inspect work on BrowserGym agents.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Live websites Agents that interact with live web content subject to frequent changes in website design, content availability, and default language settings, which can vary across countries or regions. Websites may update layouts, navigation, or features without notice, disrupting an agent's performance. Additionally, reliance on dynamic or frequently updated sites introduces unpredictabliity, as content and structure can shift over time, making consistent evaluation more challenging.
Stochastic Agents LLMs are non-deterministic, but this is less concerning since it is independent and identically distributed (IID) noise, and setting the temperature of the LLM to 0 can reduce most stochasticity.
Non-deterministic tasks Similarly, many tasks contain some level of stochasticity. However, changes should be minimal for a fixed seed.
5.4.2 Reproducibility Features
Standard observation and action space One core feature of BrowserGym is to provide slandardized observation and action space, While it allows for customization, each benchmark is defined with its own default action space. Similarly, the observation space of agents in AgentLab comes with a default way to represent the DOM, the AXTree, and other components.
Package versions The Study class contains a dictionary of information about reproducibility, including benchmark version, package version, commit hash, os version, and time stamp. By keeping track of these variables, one can isolate package versions, interactions, or system effects that can impact the performance.
Reproducibility Journal A call to Study..apeent to_journal() () antomatically uploads your results to The goal of this journal is to keep a detailed track of previous results. As mentioned in Section 5.4.1| because API changes or instance changes, results can vary overtime. This makes it casier populate a large number of reference points. The journal aims to track changes and log enough information allow anyone to reproduce as closely as possible an experiment. It stores information such agent name, BrowserGym and AgentLab version, benchmark name and metadata, experiment. performances, etc.
itnbouucbbilitty_oounnn...
Reproduced results in the leaderboard For experiments that are reproducible, we encourage users to try to reproduce the results and upload them to the leaderboard. There is special column containing information about all reproduced results of an agent on a benchmark. Tt is displayed as a range of scores alongside the original score on the leaderboard. The leaderboard, together with the Reproducibility Journal, helps monitor changes in benchmarks over time and serves as a source of motivation for the community.
The base agent Generi cAgent (detailed in section f comes along with agent called Reproducibillityygeet. Running the latter on existing study generated from GenericAgent will re-execute the same action sequence on the same lask seeds. A visual diff of the two prompts is then displayed in the AgentInfo HTML tab of AgentXray. The user can inspect what changes occured between the two executions. While the two exocutions may diverge rapidly there are some differences at the start the opisode, the diffs between the first few steps are typically insightful on what has changed since the last execution. This allows one to monitor the modifications live benchmarks that may be subject to changes over time.
Reproduuibiiltyy
5.5 Extensibility: create your own agent
At the core, BrowserGym and AgentLab are designed have a minimalist API for implementing new agents. Only few key components need to be implemented in the Agent class:
class Mydgant (ogyn. (bgyn.Agant))
def jiiit__ _=iit___aelf):
# define on ect to decode actions
action_set bgya. (
def get_action(self, obe):
# the of the egent reasoning goes here return action, agent, into
13

---

Published in Transactions on Machine Learning Research (02/2025)
The main part is the pet_action method, where implements the agent's logic. It returns action string to be interpreted by BrowserGym, based on the chosen action_set set.. The get_ get_action method can also return agent_ing info, an object of type AgentInfo containing various informations for later inspection of the agent's 's behavior.
Args On top of coding an agent, the user is also responsible for coding AgentArgs. This is essentially a data structure which the _aaeet function uses for instantiating an agent. Doing so makes it easy and reliable to pickle experiment information to disk or send it across processes. AgentLab also provides a few other methods for better integration with the framework. For example:
edataclass
clasa ArggennAAga (agentIab..geentang)))
uee_ obain_of thought: bool True
use_htal; bool Palse
def nake_agenn agent -> bgyn Agent return MyAgent use, chain, thougtt, solé, use htmll)
del aet, benchnark(seIf, benchmark);
Optional thed to set benclnark-epecifee flags
if "miniveb" in benchmark. nane:
self. use_html True
Once this is implemented, the user can evaluate their agent as presented in Section 5.1 Note: the class definition of e.g., MyAgentArge needs to be saved in a module that can be imported from the SPYTHONPATH for proper unpickling.
Dynamic Prompting While most modern LLMs can digest more than 100,000 tokens, AXTree can still be larger than 100,000 tokens, with the raw HTMLL/DOM counting more than 1,000,000 tokens. Additionally, research projects fine-tuning open-source LLMs are often limited to very small context lengths, e.g. 8,000. To manage this, our prompting tools are designed to fit the prompt to the desired length without having to naively truncate the prompt from the end, which would almost always cut crucial information such examples. This is done using the cmmomiiprngpiin class, where the user can casily configure which part of the BrowserGym observation space should be included in the prompt. Next, the length of the prompt is dynamically shrunk to fit a desired size. The fit token method iteratively calls the shrink method recursively on each component of the prompt. Default shrinking strategies are implemented, such as reducing the number of previous steps included in the history, or truncating the page from the bottom. Users can also implement their own shrinking strategies. See Figure 11 for simple example agent using dynamic prompting.
5.6 Extensibility: Plug your own LLM / VLM
AgentLab also offers a base class for LLMs. This allows users to plug their models into any existing agent by just sub-classing our objects. More precisely, we provide a BaseModelArge class, to configure and log LLM information, as well as instantiating the model, and AbstractChatModel class, which just needs __lll____ method defined to query the LLM. Lab comes with different model objects readily available, such as a class for OpenAI models, Azure OpenAl models, OpenRouted[" models. Using chat model classes allows to add some useful tools such as cost and token tracker that can automatically be added to the AgentInfo.
6 Experiments
This section showcases largo-scale experimentation that leverages the BrowscrGym ecosystem and evaluates the capabilities of several modern LLMs and VLMs in performing web lasks. We focus on two objectives:
First, showcasing how AgentLab, integrated with BrowserGym, facilitates the creation and management of experiments with web agents.
Second, assessing the current performance levels of various state-of-the-art models in tackling these benchmarks.
14 /hetpe::/opennrouter.aa/
14

---

Published in Transactions on Machine Learning Research (02/2025)
The results of this experiment provide interesting overview of the strengths and limitations of the evaluated models, offering insights into axes where they require improvements. The results are consolidated into an online leaderboard, allowing for comparisons and updates as new models and benchmarks emerge.
6.1 Setup
We evaluate Laent is default agent, GenericAgent, on all available benchmarks in the BrowserGym ecosystem. SemericAgent employs modular prompling mechanism that allows it lo utilize various tools, such Chain-of- Thought (CoT) reasoning, screenshots with Set-of-Marks (SoM), a memory mechanism, self-criticism, and in-context examples to enhance its responses. The configuration determines both the agent's 's behavior and the observation format, including options like using HTML or AXTree representations, relaining a history observations and past. actions, incorporating screenshots or Met-of-Markss images, or displaying additional HTML information from the page. The ExampleAgent shown in Figure 11 the Appendix gives a simplified overview of our GenericAgent,
Along with this dynamic prompting feature, GenericAgent implements a relry funetionality to overcome LLM side issues parsing errors. In the case of a parsing error, the LLM is re-prompted and gets 4 attempts to produce a parsable answer. After 4 consecutive parsing errors, the task is considered a failure.
Our experiments use the same agent configuration as the WorkArena++ benchmark (Boisvert et al. 2024 with the addition of the use_ think _history setting which gives the agent access to its entire chain-of-thought history throughout its execution, similarly to Putta al. (2024). Intuitively, this allows the agent to remember previous interactions and outcomes, and make more informed and consistent decisions. An example of resulting prompt is showcased in Appendix
We evaluate AgentLab's GenericAgent using a representative subset of state-of-the-art large language models, which includes:
GPT-40 and GPT-4o Mini (OpenAl 2024a) using the Azure OpenAl API, as some of the best models available on the market. We use respectively chockpoints 2024-08-06 and 2024-07-18.
o1 Mini OpenAl 2024b using the OpenRouter API, to test out the performances of test-time rcasoning- focused models. We use the 2024-09-12 chockpoint for this model.
Claude 3.5 Sonnet (Anthropic 2024a) using the OpenRouter API, as the best best-performing model on our benchmarks. We use the 2024 10-22 checkpoint, most recent to this day.
Llama-3.1 70B and 405B (Meta 2024 using the OpenRouter API, as reprosentatives of the open-weights LLM community, assess their performance relative to their proprietary counterparts and identify improvement axes.
As our evaluation metric, we use the task success rate, i.e. the percentage of tasks that were successfullyy completed by the agent, and compute the standard error (a /V). We evaluate our agent with all the previously mentioned LLMs, on all benchmarks, with some exceptions. On WorkArena L.3, except. for Claude, in Boisvert et al. (2244) as they are similar agents and we have strong expectations that the score would not change, loading to significant resouree savings. On VisualWebArena, we only run the LLMs that support multi-modal inputs. Out of fairness between models, we do not use visual inputs for the other benchmarks, as they do not require it.
report results obtained
Table 5 in the Appendix shows the settings used for each benchmark and the mumber of evaluation tasks for each. Indeed the number of tasks (or episodes) depends on the nunber of seeds (MiniWob, AorkArena L1), or the split used. We use the test splits for WebLINX and AssistantBench Finally, WorkArena L2 and L3 offer their own curricula, amounting to 235 tasks each.
6.2 Results
We present the results obtained using various open-source and closed-source LLMs as GenericAgents across standard web agent benchmarks in Table 2 We observe that Clande obtains the best performance across the majority of the benchmarks, with ol-Mini falling second. Claude surprisingly improves substantially over the
15

---

Published in Transactions on Machine Learning Research (02/2025)
previously unsolved WorkArena-L2 benchmark and obtains an average task success rate of 39.1%. One explanation for this impressive result could be Claude's specific training for computer use (Anthropic 2024b) which bears a close resemblance to using a web browser. We also observe that Llama-3.1 70B obtains very close performance to GPT-4o Mini, and Llama-3.1-405B surpasses GPT-4o-Mini significantly on mumerous benchmarks. While beaten on most complex reasoning benchmarks, 405B still obtains more than decent performances overall, which comes as a promising omen for the open-source community.
We also observe improvement in performance for GPT-4o compared to the performance measured by Boisvert et al. 2024), which goes from 23.5% to 31.4% WebArena, and from 3.8% to 8,5% WorkArena L2. Since they were using a very similar web agent implementation but an older model checkpoint, this suggests that the reasoning capabilities of GPT-4o have greatly increased thanks to the additional training performed for this new checkpoint. Another option to explain this improvement might be the public availability of those benchmarks and their support, meaning parts of their content might now be integrated into some form in the LLM pro-training/ / fine-tuning datasetss.
One interesting point is that our agent seems to underperform on Assistant a web-based question-anssweinng benchmark. Our best agent attains score of 6.9%, which is very low score compared to the current high score of 27% on the Bench leaderboard. Our agent is not specifically optimized for information retrieval tasks, which are the primary focus of the Bssch benchmark. Instead, our API and agent are designed to be broadly applicable across a wide range of web-based tasks, prioritizing generality over specialization.
6.3 Errors
When evaluating AI agents, errors can generally be classified into a few broad categories. Navigation Errors occur when the agent struggles to locate the correct page or information, often due poor search queries or navigation mis- takes. Form Handling Errors involve incorrect data entry, such as formatting issues or failure to recognize submis- sion failures. Task Understanding Errors arise when an agent misinterprels unclear instructions, leading to irrele- vant or incorrect actions. Stuck Behavior happens when an agent repeats the same failing action without adapting to the situation. Information Extraction Failures occur when the agent successfully navigates to the right location but fails to correctly extract, interpret, or return the necessary data. Finally, External Errors stem from technical issues such as APT failures, network errors, or system crashes that hinder the agent's performance. While these cate- gories are not definitive, they offer a useful framework for diagnosing weaknesses and improving Al agent reliability.
Given the scale of our experiments, we only provide qualitative overview of the errors encountered by the agent, However, we release the full traces of our experiments and let to the reader the possibility to skim through the data using AgentXRay. Appendix H showcases a detailed description of the trace from two models on the same task. This succinct analysis suggests that Claude exhibits a degree robustness, as it is able to recover from past errors. Future work will focus on conducting a deeper analysis of Web Agent errors and developing automated mothods for such evaluations.
Table 2: Results of a full round of experimentation on the unified benchmark ecosystem. The # Ep, columnn indicates the
number of evaluation episodes per benchmark, which depends the mumber of tasks and seeds used (See Appendix E for details). For budget reasons, grayed-out values (0.0) indicate skipped evaluations for the least-performingg models on the hardest. benchmark (WorkArena L3). The table also shows the total cost (USD) and average slep per episode for the overall aalong with API cost details.
best-performing model, Claude 3.5 Sonnet. Costs for each LLM are given in Appendix F
Claude-3.5 -Sonnet
GPT-4o Mini
Lama-3.1 70B
Lama-3.1 405B
Clavde- 5-Sonne Cast (S)
Benchmark
# Ep.
GPT-4o
ol Mini
Steps
MiniWoB
625
69.8
63,8+1.9
56.6+2.0
57.6 57.6-2.0
64.6+1.9
67.8+1.
23.11
3.7
WorkArena L1
330
56.4+2.7
45.5+2.7
27.0+2.4
27.9+2.
43.3+2.7
56.7+2.7
100.03
9.0
WorkArena L.2
235
39.1 13a
8.5.1
1.31o.7
2.12n9
7.2
14.9
299.44
33.8
WorkArena L3
235
0..4
0.0 40.0
0.0 10.0
0.0 20.
0.0 20.0
0.0 1:..0
191.50
25.0
WebLINX
2,650
13.7 60.6
12.340.6
11.6
8.9
7.9
12.5
104.63 62
1.0
WebArena
812
36.2 +.7
31.4+1.
17.4+1.
18.4=1.4
24.0=1.6
28.6+1..
138.76
6.8
Visual WebArena
910
21.0
21.7
16.9
134.69
4.7
Bench
181
5.211.5
4.819.4
2.1
2.8
3.92
6.9 12.
37.33
7.5
16

---

Published in Transactions on Machine Learning Research (02/2025)
7 Discussion
In this work, we introducod the BrowserGym ocosystem, which contributes to the standardization of web agent research by providing a unified enviromment for evaluating web agents. We make our contribution available two easy-to-use Python packages, BrowserGym and AgentLab. The first provides unified platform exposing all existing web agent benchmarks under the same interface, and is designed to facilitate the addition of new benchmarks. AgentLab, on the other hand, provides a complementary interface and set of tools for the implementation and evaluation of web agents, with a set of flexible web agent implementations, an expandable LLM/ / VLM API, reproducibility features, trace analysis tools, as well as an online leaderboard.
To showcase our main contribution, we conducted the first largo-scale empirical evaluation of a representative set of stale-of-the-art LLM/ / VLM models six popular web agent. benchmarks, using AgentLab and BrowserGym. While rumning such experiment using the original, fragmented benchmark codes would be extremely cunbersome, BrowscrGym makes all of them accossible through the same interface and allows for large-scale, unified experiments in single code base with AgentLab. By fostering reproducibility and facilitating consistent benchmarks, the BrowserGym ecosystem lays the groundwork for more dependable and insightful evaluations of web agent capabilities, thus accelerating the advancement of LLM-driven automation.
7.1 Limitations
Reproducibility Despite our efforts to improve reproducibility, stochasticity, and external factors remain significant challenges. Localization differences, such as time zones, default languages, and geographic settings, can lead to inconsistent agent behaviors when interacting with live websites. Additionally, variations in operating systems and browsers can affect how web pages are rendered, impacting the uniformity of results. The presence of dynamic elements, such as isemens,,,, adds to non-determinism, leading agents to face different enviromments during repeated tasks, which affects their consistency. Addressing these challenges essential for reliable and repeatable benchmarks in web agent research.
Letting an autonomous agent act on the world-wide--eb on a human's behalf can have consequences, and prove to be problematic ic on benchmarks that require an open-web access (Assistan(Bench, soon GATA). This is currently mitigated in BrowserGym via URL protection (MiniWoB, WorkArena, WebArena / VissulWebAArnaa), plus warning notice on BrowserGym and AgentLab's landing pages.
Risks
Robot detection Robot detection mochanisms significantly limit web agents, especially for open-web tasks. Websites use techniques like CAPTCHA, IP rate limiting, and behavior analysis to block automated agents, preventing them from completing tasks effectively. This limits the ability of agents to interact freely with web environments and reduces the scope of tasks that can be automated. AssistantBench is the only benchmark impacted by those issues so far in BrowserGym, as it uses the open web as its information retrieval tool.
Agent collisions The widespread use of databases the web means that many tasks involve updating records and modifying entries. When multiple agents operate simultaneously, they can interfere with each other, especially when executing tasks that alter the same database entries. For example, if two agents add an item to their cart concurrently, they may unintentionally update each other's cart, leading to inconsistent outcomes. This issue is particularly problematic in environments like WebArena and WebAree, where a single virtual machine is used for each agent. As a result, only one agent can be run at a time on these benchmarks, significantly slowing down the evaluation process.
Synchronous interactions Another limitation is the syne loop mechanism used to manage interactions within the browser environment. For example, when agents neod to perform multiple actions in quick succession, the sync loop may not keep up, leading to delays and reduced efficiency. This can result in performance bottlenocks, especially for complex tasks requiring rapid, sequential actions.
17

---

Published in Transactions on Machine Learning Research (02/2025)
7.2 Avenue for Future Work
To further develop the field, several areas require attention. One promising direction is improving safety mochanisms for web agents. Currently, safety evaluation in web agent benchmarks, including cnsuring agents' compliance with data privacy and safety policies, as well as their resilience to malicious interactions, is underexplored. The development of secure environments is crucial for adopting web agents in enterprise or sensitive applications.
Another avemue for future work involves the creation of rcal-time web agents capable of interacting with live enviromments al speeds comparable to human users. This necessitates improvements in both latency reduction and real-time decision-nee decision-making, which would enhance the applicability of web agents in dynamic settings, such as customer service or rapid data retricval.
The potential for smaller, yet efficient LLMs is also an important focus. Developing smaller models that retain the ability to understand and reason through complex web interactions would lower computational demands, making web automation more accessible and sustainable. This aligus with efforts to increase the efficiency of LLM-based agents, particularly for real-world deployment where computational resources may be constrained.
In addition, enhancing reactivity through the use of computer-level vision-language models (VLMs) can provide agents with a more nuanced understanding of the visual context on web pages. Current models often lack the capability to fully interpret complex visual layouts or graphical information, limiting their interaction fidelity with certain websites. Integrated models like Anthropic's computer use model could make web agents more versatile and capable of handling tasks that require both visual reasoning and textual inputs.
Another direction is to leverage inference time scaling, optimizing it specifically for web tasks and web agents. This could involve making better use of the model's self-reflectonn capabilities to further improve task performance, adapting dynamically to various complexities encountered during web navigation, and efficiently managing resources to reduce latency. By tailoring these improvements, web agents can become more effective at real-time, adaptive decision-making ultimately enhancing user experiences and broadening their applicability.
As BrowserGym saves and logs every piece of information seen in its experiments, it could serve as a valuable collection tool for finetuning models. This collected data could be utilized to further refine agent behaviors, improve model performance on specific web tasks, and create efficient agents capable of handling complex scenarios more efficiently.
8 Broader Impact
The development of more general Ul-based agents holds the potential to significantly transform how work is performed on computers. These agents promise to automate tasks at an umprecedented pace, with reduced costs, and with a broader and doeper scope of expertise. While current benchmarks demonstrate that this level of sophistication has not yet been achieved, the rapid pace of advancements in the field suggests that such a future may not be far off. This progress could unlock immense economic value but also likely to result in substantial and rapid job displacement, presenting critical societal challenge.
AI safety must be central to this emerging agentic revolution. Presently, jailbreaking a large language model (LLM) might only expose knowledge already available on the web. However, agents become more sophisticated, vulnerabilities such as prompt-injection attacks could be exploited to leak sensitive company information, authorize unwanted transactions, or cause other unintended harm. The widespread deployment of agents at scale will create enormous financial incentives for malicious actors to develop targeted attacks that uncover and exploit weaknesses in these systems.
The rise of Ul-based agents also introduces new challenges and considerations regarding digital advertising. As these agents navigate the web autonomously, their interactions with dynamic ad content could disrupt existing advertising models, impacting both advertisors and publishers. Unlike human users, agents are unlikely to engage with ads in traditional ways, which may lead to shifts in how ad impressions, clicks, and conversions are measured and monetized. Additionally, the ability of agents to filter, block, or bypass advertisements could create conflicts with current web monetization stratogics. To maintain a fair and functional digital ocosystem, actors from both sides should explore standardized approaches for handling ad-relatod content way that balances automation efficiency with the sustainability of online advertisement. business models.
18

---

Published in Transactions on Machine Learning Research (02/2025)
As these technologies evolve, society must proactively address the accompanying challenges by establishing robust policies and frameworks. Rapid action will be essential to ensure that the benefits of these advancements are maximized while minimizing potential risks and harm.
References
Anthropic. Claude 3.5 Somnet model card addendum, 2024a. URL https://www-cdd http:s//wwwwcco.aaatoooc....omo anthropic.com Baar/ Claude 3 Addenduu. pdf
Anthropic. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku.
https:
www.wanttrooic.coom wttp:///..c.oo/////////. httmheews///-mmmmelleeeeeeeeeeee and- compoocotpttuuuse 2024b. [Online; accessed 22-October-20241
Lóo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. +orkArena++: Towards compositional planning and reasoning-baane based common knowledge work tasks, 2024. URL https://aaxiv org/ httg//b//44000 05291
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal OS agents at scale, 2024. URL
08264
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym, 2016. URL httpp::/aarxv..org////o wwtp:/oo//////////// 01540
Qi Chen, Dileepa Pitawela, Chongvang Zhao, Gengze Zhou, Hsiang- Ting Chen, and Qi Wu. WebVLN: Vision-aud-languaae navigation on websites. In AAAT Conference on Artificial Intelligence, 2023. URL
attpp://api..smmnniic. htt.:ppp...ematttacccaaaao..o//e wrt.:laaarrrg/oroousti.... 266551596,
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Anmual Meeting of the Association for Computational Linguistics, 2024. URL https://api..e http:///ap...mmaticccc.o...rrr wtt.iiiccchoaaaaorrrruuuruuu ID: 26:0066988
Wayne Chi, Amoct Talwalkar, and Chris Donahue. The impact of element ordering on LM agent performance, 2024. URL
org/ abs///bb//2222222 12089
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web, 2023. URL https: org/ www./rgg/////oooooooo/..o/..ee/t abs/2306. 06070
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Téo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre La- coste. WorkArena: How capable are web agents at solving common knowledge work tasks?, 2024. URL
https://arxiv http:://warxvvv..aabass/essa /t////ab/24030077155
Yu Du, Fangyun Wei, and Hongyang Zhang. Tool: Sclf-reflective, hicrarchical agents for largo-scale API calls, 2024. URL httph: /tapps://arxvv..oo www.aaawww....ra/aaaaaaaaaaae wrgg/bss//400/0 04253
Hiroki Furuta, Ofir Nachum, Kuang-Huci Lec, Yutaka Matsuo, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetumed foundation models. arXiv, abs/23055.18883 2023. URL
https://arxiv.. http/aavvvv...o/aaiae 2ttg//bb/2300..111 11854
Hiroki Furuta, Kuang-Huce Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instractionn-inetumee foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents, 2024. URL
https://arxiv. wrwg hb///bb//244 abs/2410.05223
Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Chong Chang, Sanjari Srivastava, Yanan Xic, Peng Qi, Huan Sun, and Yu Su. Is your LLM secretly a world model of the internet? model-based planning for web agents, 2024. URL https:/ //tarivv...vv wttp:///////....o/eeee
19

---

Published in Transactions on Machine Learning Research (02/2025)
Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilck Hakkani-Tun gate the web. In International Conference on Learning Representations, 2019. URL. https:
Learning to navi-
Tzzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world WebAgent with planning, long context understanding, and program synthesis. arXiv. 12886
als//2307.12856, 2023a. URL httpp:///axxvv.org//aaeaa
Izzeddin Gur, Ofrr Nachum, Yingjic Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML, with large language models. In Houda Bouamor, Juan Pino, and Kalika. Bali (eds.), Findings of the Association for Computational Linguistios: EMNLP 2023, pp. 2803 2821, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/vr /000:///////22..mm///// emmlp.185. URL https://aclanthhology) wrg.g/23.gg/2.... Sindings-emnn 185
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis, 2024. URL
fttps://arxiv.orgge
2307. 12856
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings, 2024. URL https://arxiv wttp:/www.rrg/bb////000 11554
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. VoyVoge:::: Building an end-to-end web agent with large multimodal models, 2024. URL
https://arzee org/ httgabb//440..133
Kung- Boing-siing Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. CRMArena: Understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments, 2024a. URL https://arxxx: wrw.:.rgggaao..ao 2411. 02305
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of LLM agents: A survey, ArXiv, abs/2402.02716, 2024b. URL
semanticscholarr hwwpchooo.r.rrroooooooooooo Argg/CorpsIDD:. 267441188
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillierap. A data-driven approach for learning to control computers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1466-9482 PMLR, 17 17-23 Jul 2022. URL. html
https://proceeddngg mmlr
Lawrence Jang, Yinheng Li, Charles Ding, Justin Lin, Paul Pu Liang, Dan Zhao, Rogerio Bonatti, and Kazuhito Koishida. VidcoWebArena: Evaluating long context multimodal agents with vidco understanding web tasks, 2024. URL htttps://arziv.orrge 24bbb///21....1100
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei Chao. Dual-view visual contextualization for web navigation, 2024. URL https://axxx http:///axxvvoorg/aabb
org/abs abs/ 2442./4477447
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023. URL
https: /ttp:://axxvvvvv abs//b////////a attpaabb/b33332// 2303. 17491
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks, 2024a. URL htttss:t/tssiiiiii org/abs//2e 13649
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024b. URL /ttps://arxiv... www.xiv..rgg/aeo hbtp//bbs22407 01476
Ido Levy, Ben Wiesel, Sami Marrod,, Alon Oved, Avi Yacli, and Segev Shlomov. WeWWhhggetttannnn A benchmark for evaluating safety and wortheess in web agents, 2024. URL. /uttps://arxiv http://///aaaiivorrr/////////// wwtpw//wwwww.a....oooo/o
20

---

Published in Transactions on Machine Learning Research (02/2025)
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariana Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhav- itvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Th6o Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Woll. Datasets: A community library for natural language processing, 2021. URL ltteps://arxiv org/ abt./1/9....
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Annual Conference of the Association for Computational Linguistics (ACL
2020), 2020. URL https://wwa htttp:://ww..clwweb.o aclweb. org/ http//oooooogo/o...oooo aellmmain.72. 729. pdf
Evan Zheran Liu, Kelvin Gun, Pamponn Pasupat, Tianlin Shi, and Percy Liang. Reinforcement Learning on Web Interfaces using Workllow-guided Exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL
Haotian Liu, Chunyuan Li, Qingyang Wu,, and Yong Jae Lee. Visual instruction tuning. In seirty- Conference http://pperrrreeee..eeerrrrrrr//
on Neural Information Processing Systems, 2023a. URL
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Tei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang. Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents, 2023b. URL https: //wrx/vv.vvvv//////a/ org/ www./a///////////////////eeee/e/ atsp///b./////..////// 2208 03688
Xing Han Lå, Zdeněk Kasner, and Siva Reddy. WebLINX: Real-world website navigation with multi-turn dialogue. ArXiv, abs/ abs/2402.059330, 2024. URL https://api semanti :rg/:CorppsII::25555555
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Board: An analytical evaluation board of multi-turn LLM agents, 2024. URL
https://arrxe //tp:iv///iv../// wwwpxivvorr///ba
Meta. The Llama 3 herd of models, 2024. URL
21783
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GATA: a benchmark for general Al assistants, 2023. URL https://arxiv org/ ha////b///// a31. 12983
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging Al applications, 2018. URL http::////xxivvoog/aaaa 1ttpe///11.............
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Browser-assisted question-answering with human foodback, 2022. URL htttps wtt..///aabb//.......... 09332
OpenAL. GPT-4 technical report, 2024a. URL https://arxiv atgg/bs////// 2307778777777
OpenAI. OpenAl ol-mini: Advancing cost-efficient reasoning. 2024b. [Online; accessed 12 12-September-200 2024].
https: https://openaat cot.index//
openai-ol-mini -amaii-o--innn wtwpaa/wwcccccciiioooooooooaaaaa
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui. Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. WebCanvas: Benchmarking web agents in online environments, /206 12373
2024. URL
Pranav Putta, Edmund Mills, Naman Garg, Suneet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: Advanced reasoning and learning for autonomous Al agents, 2024. URL
https://arxiv.orgg wwwpaa//..... 02408007199
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicapp. Android in the wild: A large-scale dataset for android device control, 2023. URL WTR...tttss////.. httpt:t//atx....ooooo wwtp:/sw.rrr/////////a/oo..ossss http///bs/330.... 10088
21

---

Published in Transactions on Machine Learning Research (02/2025)
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Tá, William Bishop, Wei Li, Folawiyo Campbell-Ajal, Daniel Toyama, Robert Berry, Divya Tyanagundlu, Timothy Lillicrap, and Oriana Riva. Worlo A dynamic benchmarking environment for autonomous agents, 2024. URL https://arxiv.. org/
org/abs/ abs/ abs/2405. 14573
Gabriel Sarch, Lawrence Jang, Michael J. Tarr, William W. Cohen, Kenneth Marino, and Katerina Fragkiadaki.
VLM agents generate their own memories: Distilling experience into embodied programs of thought, 2024. URL
https://arxiv. www.aaawwwrggaaaa/aaaaaaaaaaaaaa atsp///.//222...../ 14596
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to UI actions: Learning to follow instruc- tions via graphical user interfaces. In Advances in Neural Information Processing Systems, 2023. URL
https://arxiv. www.riii..rggbbb... wwwgg//gggggg....///////
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Procoedings of Machine Learning Research, pp. 1135-3144- PMLR, 06-11 11 Aug 2017. URL hattps://proceedings mlr wwtpstper//s//a//aa..ttt..
Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. SteP: Stacked LLM policies for web actions, 2024. URL bttp:::/aaxxv..rg//// 2a100.22331.oom 2370.03720
Abishek Sridhar, Robert Lo, Frank F. Xu, Hao Zhu, and Shuyan Zhou. Hicrarchical prompting assists large language model on web navigation, 2023. URL https://arxii: otgg 14257
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Delen, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments, 2024. URL pattps://arxiit www.xi..oo//bbbee. arg/abs/2407. 17032
UK AI Safoty Institute. Inspect AI: Framcwork for Large Language Model Evaluations, May 2024. URL https:://githuu..comm /otoo/////./oooeeeeeeeee//////ee
ai
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Fronticrs of Computer Science, 18(6), March 2024a. ISSN 2095-2236. doi: 10.0007/111044-0244423111 URL http: doi. org/10. 1000000011000000044003333
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingshagg Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig, OpenHands: An open platform for Al software developers as generalist agents, 2024b. URL https://arxiv. org/ abs babbs/2407.1165 16741
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (MLLMs): A comprehensive survey on emerging trends in multimodal rcasoning. ArXiv, abs/2401.06905,, 2024c.
URL https: htt.:ppppp.mmmatcscchccc..o.seee org orgg/orppsID: 266999977
Web Hypertext Application Technology Working Group. DOM standard. httpps://dmm spec. http:///wwm.pc...aaatttt.. whwtwg. org/ 2024.
Accessed: 2025-01-02.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL
https://arxiv. wwwaaaaww.oggab/////aa.a wrg//bs////...oo.. httaaa/////2............... 11903
World Wide Web Consortium. Core accessibility API mappings 1.1. h/wp::://wwwww. wtt./o///////eeeeeeeeee waa.:/oorraa...... 2017. Accessed: 2025-01-02.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Teo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Open An open platform for language agents in the wild, 2023. URL
2310.10623
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorkd: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https:///axiii www.xi..org/aae www.aa//........o
Hui Yang, Sifu Yue, and Yunzhong He. Auto-GPT for online decision making: Benchmarks and additional opinions, 2023a. URL https://arxiv http:aawwwi.rrrrraaaaraaa 2tt.//2230022222
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-Mark prompting unleashes extraordinary visual grounding in GPT-4V, 2023b. URL
11441
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interac- tion with grounded language agents. Advances in Neural Information Processing Systems, 35:20744 20757, 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Syncrgizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022b. URL https:///pperreiie: http:///wwaaaooooooo.oooo//oo//a
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL
https://arxiv. http://axvvv...raaabae hts///bb/220000 10601
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Be- rant. AssistantBBenh: Can web agents solve realistic and time-consuming tasks?, 2024. URL
https: //tp///xxvvvv///// www..oogggaaaaaee //////44...177
Yao Zhang, Zijian Ma, Yumpu Ma, Zhen Han, Yu Wu, and Volker Tresp. WebPilot: versatile and autonomous multi-agent system for web task execution with strategic exploration, 2024a. URL
https://arxiv.org 2abbs/400 2598.15978
Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. MMInA: Benchmarking multihop multimodal internet agents, 2024b. URL /tttps///arxi... httgg/bbs/244..99900 2404 09992
Boyuan Zhong, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is generalist web agent, if grounded, 2024a. URL
Boyuan Zheng, Boyu Gou, Scott Salisbury, Zheng Du, Huan Sum, and Yu Su. WebOlympus: An open platform for web agents on live websites. In Delia Trazu Hernandez Farias, Tom Hope, and Manling Li (eds.), Proceodings of the 2024 Conference Empirical Methods in Natural Language Procossing: System Demonstrations, pp. 187 197, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi: 10.18653/v1 eempp-d URL http:::/accantthoogg.oorggg
emnlp-demo 20
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajeclory-as--xeenppl prompting with memory for computer control, 2024c. URL (ttpttttss:/ org/ wttp:o/r/abb///..00055555
Andy Zhou, Kai Yan, Michal Shappnnook-Roohhmae Haohan Wang, and Yu-Xiong Wang. guage agent tree search unifies reasoning acting and planning in language models, 2024a. URL
Lan-
https://arxiv.orr http://xxxxxv.o//abbaaa httgg/ab//31.00eeee/
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: realistic web environment for building autonomous agents, 2024b. URI htttps///axii: cerrxivv.orggaane wwwgg/bb//337.... 13854
23

---

Published in Transactions on Machine Learning Research (02/2025)
A
BrowserGym's High-Level Action Set
Table Complete list of high-level action primitives available when using BrowserGym's HighleevelActionSet feature.
Category
Primitive
Description
fbil, text)
Fill an input field with text.
click(bid, button)
Click an element.
dblclick(bid, button)
Double-click an element.
hover(bid)
Hover the mouse over an element.
press (bid, key_camb)
Focus element and press combination of keys.
focus
Pocus element..
clear(bid)
Clear input field.
select_ (bid,,,, options)
Select one multiple options in drop-down element.
drag_and_ dttpdddrrooooabbaa bid, to_bid) upload_ffieebid, file)
Drag and drop one element to another.
Click 'filcchoser'' element, then select multiple input files for upload.
mouse_ mouse_nove (x, y)
Move the mouse to location. Move the mouse then press and hold button. Move the mouse then release button. Move the monse and click button. Move the mouse and double-click button. Drag and drup from location to loration. Click 'filechooser' location, then select multiple input files for upload.
nouse_down, (x, y, button) mouse_up(x, button)
clise__clck(x, button)
mouse dblclick(x,, y, button)
coord
mouse_drag,
__d__draaddddd__da and aanddddppffro____ from_y, to_x, to_y)
casiee_plladdaaee file(x, y, file)
keyboardddoun(key) (ken)
Press and holds keyboard key.
reyboard_upp (keya))))
Release a keyboard key. Press combination of keys.
keyboard_ppa perooardddpddadeee (key_comb)
TTpes string text through keyboard. Insert a string of text the currently focused element.
keyboard, textertteetteeeeeeeeeee (text)
new_tab()
Open tab.
tab_ (ab_closee))
Close the
tab.
sub_focus
Bring
front (activate tab).
vo_back()
Navigate to the previous page in history.
()_
Navigate to the next page in history.
gurll
Navigate to url.
send send_nag_ mag__ng___t to_ user (message)
Send a message to the user in the chat.
report_infeasibe srr.inifesiibbeeerrrre
Send a special message in the chat and terminate.
scroll(dx, dy) (sccose)))
Scroll pixels in X and/or Y direction.
Wait and do nothing.
24

---

Published in Transactions on Machine Learning Research (02/2025)
20 different types actions are available.
click(bid: str, button: L'taral 'middle', 'right/] 'left', modifiers: list Deacription: Click elenent.
D)
Examples:
click("a51')
click("b22' button='rightt)
click( '8',, burtea='nidãle', modifiers- ("Shift")
fil1 fill (hid: atr, value: str)
Description:
field.
triggere input with
<input>, <textarea> and [contanteditahla] elements.
Examplea:
(123(',,7', 'example value')
fi31("45', "malti-Linelnxxarp""
fil1('a12', 'exampla with "quotes"!)
(delta_x: flost, delta_y:
Deacription: Scroll horizontally and vertically. Anounts in pixela, poastive for right or down acrolling, negativa for left or up scrolling. Dispatches vheel event.
Examples:
scroll 200)
scroll (-80.2, -100.5)
send_neg_00 meg_ttg_______ (tex:er(text: stx)
Deacription: Senda neasage the user. Examples: tend_agg_to "Based the results of ny search, the built 1751.')
report_ innfeasiblerraasooe
Descr iption: Notifies the user that their instructions are infessible. Examples: report_inne infensibbee'II cannot follos these instructions because there is no anail field in this form.
Only single
provided Example:
fill ('s12',
'exanple with "quotes"!)
Figure 10: Example description of high-level action set in BrowserGym (figaalleeeAccssnneeeeeeeeelll))
25

---

Published in Transactions on Machine Learning Research (02/2025)
B Detailed Benchmark Descriptions
MiniWoB
1+
Shi et al. 2017 Liu ct al. 2018) Each task only requires accessing a single HTML page, the
entire logic of the task and its validation is done in JavaScript. As such, MiniWoB natively supports parallelization since there are no collisions by design.
16 Zhou et al. 2024b) Contains 6 domains mimicking real-world websites, self-hosted via Docker containers. Docker containers must. be reset to their initial state before each new agent evaluation. Task validation is performed via logie rules (HTML content, URL match, message content), using either exact matching or semantic matching using GPT-3.5 model judge (hence implies some costs).
WebArena
Koh et al. 2024a Based on the WebArena implementation, VisualWebArena retains 2 of the original domains, plus includes a new one. Task goals inchude images for vision-based tasks, such as "Find a pair of shoes that looks like this image". Docker containers must be reset to their initial state before each new agent evaluation, and some tasks require an explicit reset of specific Docker. Task validation similar to WebArena, and also relies on visual matching using an open-source VLM (blip2-flan-t5-xx)
Visual Web.Area
WorkArena Drouin et al. 2024 Boisvert al. 2024 The backend platform requires Personal Developer Instance (PDI), available demand with a free ServiceNow account. Validation is done via logic rules, using database queries to check for specific content in the backend, and JavaScript code injected on the platform's pages to check for specific actions in the frontend. WorkArena natively supports large-scale parallel agent evaluation, as it prevents trajectory collisions by design.
19 Yoran et al. 2024) Contains 214 realistic and time-consuming open-domain tasks. Tasks require browsing more than 525 web pages from 258 different websites to answer correctly. The dataset covers a diverse range of topics and composed of sub-set of general assistant tasks and sub-set of dorain-specifie tasks created by domain experts. The benchmark features small development set, consisting of 33 tasks and larger test set of 181 tasks. The gold answers are hidden for the test, but releasod for the development set in addition to the URLs from which the answers can be inferred and and answering straeggees. Additional task-level meladata includes difficulty and time-dependency, indicating whether the answer is static or includes stable information that could change in the future, c.g., a business's opening hours. Since the tost set hidden, predictions nood to be submitted for evaluation. This is supported via an API enable easy evaluations at scale.
Assistant.ench
WebLINX Là
2024) Static, supervisod benchmark. Interaction traces the original dataset are converted to the BrowserGym duservation/actiin format, and each task consists of predicting an action that must. match recorded action taken by a human during demonstration. As result, it differs from auvirommen--aasee benchmarks by ending after one step, while formulating each recorded action different task. Consequently, validation produces scalar rewards via partial matching, using the original metrics selected by Lu et al. (0024
hctps://niniuobb.eeee www.ma.aa.a..mmmm /arrma.org/
https::/uebaren: dev/
htgps//sasistatbeench.hhhmn
26

---

Published in Transactions on Machine Learning Research (02/2025)
C
Example Prompt
In Figure 11
we show the basic usage of AgentLab's dynamic prompting tools.
agentlab. ageate import dynanic_ proopiiig as
ObeFlage:
näe_htal: bool
clase ExanpleAgent (Ageet))
use_ax, tree: bool
def ...init... (self, flags, chat_nodel_arge argss)
use_tabs: bool
self salg.flags flaga
use focusad elenant: hool
self.action_pet HighieevelActtionseeeSSiss
arror_ use hee_histor:: bool
unsfsed LLV/VX for using different provider #f eig.: OpendI, OpenRouter, Azure, or setf hosted ssing TEI. self chat_1lm. chat_model_argg make make__odel()
use use_past_eerroc logs:
action_hietory: bool ust_ecreeuchot: bool
bool axtract axtract_vistble tag: hool
def get action oba): instructions dp. GealInsttructionsadde (' ['geal_object"") observation dp. Obbervation self flage. obe)
extract_clickable
extract_coorde: bool
dp ActionPronpt action_set,, action, flagorself wwags.lattion.....
falter_visible elementssonnii
filter_with, bid_only: bool filter_soa_call bool
pronpt "la" [oin([
instructions, proupt, the goal the task observation proupt, htet and/or es, tree dependsing on flags.oba proupt, the space descrided based flage. actson action. concrete_ex, concrete exapple of the expected answer
class ActionFlaga (Flagss))
action
G bgyn Highleeeelcriinneettee long, deggddeccritionn bool
andvvduul_exaaplae bool
# Progressively shrink some part of the prorpt until it fets the lamst # Each cosponent has
skrinking strategy
proupt dp.fit_tokene
nax_proept_ tokenn-00o0)))000))
action_str action. parse_ carree_asverreee (self. chat, 1Im(prorpt) action_strr str, bgyn. (gentInfa (chaaaceenaaaeerttteeeeeeee nessages- [prompt]]
def preprocessor (self, oba) obe procereeed htnl"] procese_ httntttallll (obe (ddoeo"
obe
Example of how implement agent using dynamic prompting and the unified LLM APT. For simplicity, only subset of the prompt components and flags are depicted. Each prompt component has its own shrinking mechanism which allows for reducing the size the prompt dynamically when exceeds the maximum mumber allowed tokens.
Figure 11:
Concretely, here is an example prompt from GenericAgent WorkArena L2, at step 2:
GenericAgent example prompt
Instructiore
Bectau the currecs stats of the paze and all other informatios to slad, the best poostble neit action to soceapt your goxl. Yoor annoor be Interpreted
anvested by gragre,, sore to foller forswering instmuctione.
Ousceetoly, Expesses
Linse" sodila *Out"* xpplication.
Creats stitee the
all entriar ubsce:
Sbart deserapoion" contains 220K029-12200-00--
3. Delete espense Lines with deplicated doses dostription, leeping onl the nott exjensive deplacate.
Otsnsvatioz of curvent stop:
Cerversly oper tabe:
7kb (actten tab): Serricator hitpo
Soss: You interact with viatbla elanests. the "vtethie" elamet visthle the
KootoWhiree maratt Weirieed Bootgettion App Servicalion
(07) peerto, ttose 11aue sesertsoe", atoeic, relersnce adttictore tact
[45] postic, live live='polite polite', slomdo, cvlevant- adcitioe beot
Histury inturuction with
27

---

Published in Transactions on Machine Learning Research (02/2025)
stop
Coss appll ccatioo.
"A13" vialill '79".
<rtinio
Soct
V/acties>
stop
musu cu alemady expuaded) to "Coul" upplication upplications, Looking Uhrosgh Afliow, sw appl icttico listed, sourch "Oust" since inmolaately viuitle.
top witch
caettiono
#111 1922, (Gre
</ustiuu>
sT step
Cthisko
aen swarch, sirwady rillaed unth nens sbosting resnite. cct dan "Drpazze Linee" 1tnk hid which smettly need narigatt 1'13 ctsice this 1izk to the tipparze modtla.
11442(115461)
Action
Seoe: actios allows
with envizenmet, Beat are pythen revetion soccniig slgarright nade, The prinaey my of raferring stvnnrce in the page io thringh hid which ane spacifted to your shoertation..
20 difforent typeo of actions are available.
svoptvuit, au: floms
sg.
report,
delts_y:
Gillibid:
lististr))
siickibod: button: Liveral ('lett', "midate". 'rigat"] 'ledt", aoditiera: ftapitg Literall I'aze", "Coezzaa', "Contoolbelttta" "Seaa" [Stszt ())
dbbilack(tad: betten: Liverall beft' madtiee. "zight 'lede', aodidivem: 11st (syping Literal ('Al',, 'Oustrol', Oostcolorkete' 'Mtt''. "Stidt]]
hnverbbid: uts?
pewre(hid: stt, bey_ zoyb: str)
tocus(b54: sts?
(loartbud) ute?
drag and dropttron 3odi ebr, to, bid etr)
uplasd, (oTW(hd: wtr, fitee vbr Eieefeer)) :loae()
Locasiindos:
gesatert:
singte action be precided
fill'/ut2", 'ooppl will. "quries"!)
Sooe:
Paze tante may be gata 1iks and may requize to internct with the couse posteico
sn 2, cocrdinates.
Bine tert riald eight hore anto complatiion To ane 16, you hove type fno sta..
tharassers and usit
you harn to ett parce, don't torges led's curase.
the
Goscdisaie imsiee adeatify
Make
ssing
dropdoma complete you
aslect_eption,
Abetract
Time in shetract voraion of the ansoer oitt descrigeton of the content of such tug. Hake suze you follow this stcuctscv, but rusluce the cosvest with your assves:
stop by atep. If you to maze calcutationa noch as coorcinates, write then bers. affect your previace
the curment
pago-
soccuted.
</actiono
Commett Tonegin
Buce is caucrete saumlle of how to fecmul your szsees. Yakee sare to Jollow the vapplte with poogec Lago:
peuvious tzied aggaer your "2022",
veing option, synente tropdoen,
may
page.
28

---

Published in Transactions on Machine Learning Research (02/2025)
</tatnlo
(ltek(att01)
D GenericAgent in depth
Table 4: List of GenericAgent's prompting options. 'The setting column gives the flags used for experimentations, with the exception of the agents ran on VienalWebArena that have uae_screenshot-Tru
Category
Flag
Setting Description
use_html
Adds the HTML content to the prompt. Adds the AXTree content to the prompt. Specifics which element focused the moment.
use_axtree
feee_focused_eeeee element use use_error_log
Show the previous action out...).
(e.g. action not possible, timed
faee_pst____t past__eroo error Logs use ise_action_ history
Show all previous
Display a list of all previous actions.
use_ history use scereseeetsoot use_som
Display a of all previous Chain-of-thought reasonings. Switch vision mode.
observation
Agent usees Set-of-Marks instead screenshots. For each HTTML/ATTrs element, add tag that if it is visible. For each HTML/AXTE element, add a tag that it clickable. For each HTML//XTES element, add its coordinates. Only show int the HTML/AXTies elements that are visible.
sierracttiisible...m ortract_clickable__ tag axtract_ccorda filter_visible__d)
bectiieeeeeeeteeeee only
long_ descriptionn
Adds full docstring of each action. Adds usage example each action function. Allows performing multiple actions in one step.
action
sidividual_examplee.
miltiaacions
use thinking
Use Chain-of-thoone thtoughtt reasoning.
use_plan use use_criticize
The agents provides and refines plan each step. The L.M draft, then criticize action before outputing the real action.
reasoning
use_ cee___occrteeerm wow.rrete_exmmplee example use per__abstrat__exxppp
Whether or not to use a example for in-context learning.
Whether or not use an abstract example with descriptive in- structions.
extra_ings iitra_instruutions
N/A
A string to be passod to the prompt as an extra goal.
E
Benchmark setting
In Table we give some information about the benchmark settings during our experiments. For Miniwob and respectively 5 and 10 seeds per task. WebArena and its visual counterpart meant to be used as whole, as they do not provide task seeding. WorkArena L2 and L3 are shipped with preset curricula to balance out the types of tasks in an experiment. Finally, Assistant Bench and WebLINX provide test sets, which are used to evaluate the models.
WorkArena L1,
Table 5: Summary of experiment settings for each benchmark.
Benchmark
# Tasks
# Seeds
Max Steps
Resulting # runs
MiniWoB
125
5
10
625
WebArena
30
812
Visua/WebArena
910
/
30
910
WorkArena
33
10
30
330
WorkArena L2
341
curriculum curriculum
50
235
WorkArena L3
341
50
235
WebLINX
31586
test split
1
2650
Assistant Bench
214
30
181
29

---

Published in Transactions on Machine Learning Research (02/2025)
F Costs and API Usage
In Table [ we provide more detailed data on the cost of our experiments. We also show token counts. We highlight that the Llama models were ran using the OpenRouter API, which relies on different providers to give users access to those models. Prices for those two models may have changed since running our experiments.
Total
Input Tokens
Output Tokens
API Input Cost
API Output Cost ($/M tokens)
Agent
Cost ($)
(M tokens)
(M tokens)
(&/M tokens)
Claude
894.80
276.20
4.41
3.00
15.00
GPT-40
720.72
277.25
2.76
2.50
10.00
GPT-4o-mini
75.73
479.72
6.29
0.15
0.60
1Jama-311 405B
849.63
359.89
8.13
2.30
2.30
Llamaa3311 70B
78.79
221.23
5.02
0.35
0.35
Mini
971.12
171.25
38.11
3.00
12.00
Table Comparison of experiment costa between models. Values summed all benchmarks, except VisualWebArena.
G Experiment runtime and hardware
We provide detailed information on total runtimes and benchmark durations (Table 7j. These metrics are presented for our best- persoperfoniing model, Claude 3.5 Somnet. For each benchmark, we report the following key metries:
Cumulative Experiment Duration: The overall time taken for the benchmark experiment to complete. This value does not account for parallelization.
Average Step Duration: The mean time taken per step in the experiment.
Cumulative Environment Runtime: The total time spent running the environment, excluding agent
processing time.
Average Environment Runtime per Step: The average time spent by the environment per step.
Cumulative Number of Environment Steps: The total count of steps executed within the environment during the experiment.
Discrepancies between the total runtime and environment runtime can be observed, primarily due to the API used. Slower or overcrowded APIs may introduce significant variations.
It is important note that the reported durations do not account for parallelization. practice, experiments were conducted using up to 20 parallel jobs depending on the benchmark. This means that while the reported total experiment duration reflects the absolute runtime of the benchmark, the actual time required to complete experiment may be significantly lower due to parallel execution. The level of paralleizzatin was adjusted based on the computational requirements of each benchmark. Lighter benchmarks, such as MiniWoB, could support to 20 parallel runs without issue, whereas more resource- resource-intensive benchmarks, like WebArena or WorkArena, required limiting parallel jobs to prevent server overload and cnsure stable exccution.
Our experiments were conducted on large-scale compute clusters equipped with Intel(R) Xeon(R) Gold 6126 CPUs @ 2.60GHz and effectively unlimited RAM. Given the relatively low computational demands of our benchmarks, experiments could also be locally on standard laptops without significant performance degradation. However, the primary constraints arise from the benchmark environments themselves. For example, WebArena experiments were executed on Azure VMs with 8 CPUs and 32GB RAM, which imposed limitations on execution speed and parallelization capabilities.
Results analysis
In this scction, we compare the exocution of a task by our GenericAgent, backed by two different LLMs, Claude 3.5 Sonnet which exocuted the task correctly, and GPT-4o which failed the task. We will be looking at task from the very challenging WorkArena L22, which Claude achieves groundbreaking performances.
30

---

Published in Transactions on Machine Learning Research (02/2025)
Cumulative
Avg. Step Duration (seconds)
Cumulative Environment Duration
Avg. Environment Step Duration (seconds)
Cumulative Steps
Benchmark
Experiment Duration
(hours)
(hours)
AssistantBench
4.0
2.0
4.5
1590
MiniWoB
3.1
5.0
0.7
1.1
2282
WebArena
18.6
12.2
11.6
7.6
5493
WebLINX
4.5
6.1
0.1
0.2
26499
WorkArena
8.2
9.9
4.4
5.3
2985
WorkArena L.
23.6
10.7
10.9
4.9
7947
WorkArena L3
16.1
9.8
7.4
4.5
5884
VisualWebAreun
16.9
14.1
10.5
8.8
4319
Table 7: Summary of runtime metrics for Clande 3.5 Sonnet across various benchmarks. Cumulative durations indicate total execution time wilhout accounting for parallelization. Enviroumentt-rllleed durations reflect only the time spent in the environment, excluding agent processing overhead.
We focus on the task workarena. navigate- andee--nn---- and-order- order- staaddar -tandard-laptop 12, which has the following goal:
standard laptop
catalog
Concrotaly, you
complata following steps:
1. Navigate to the "Service Catalog" module of the "Self-Service" application.
2. Go to the hardware store and order "Standard Laptop" with configuration ('Additienal software requirenente", "Salesforce, Microsoft Office 365, Asana, HubSppt'', 'Adobe Acrobat's True, 'Adobe Photoshop's True)
Figure 12 displays the trajectory taken by Claude for this task. Interestingly, in step 12e the agent manages to overcome enviromment limitation. Clicking one of the element of the chockbox docs not trigger the checkbox, and the agent gives the following reasoning before correctly checking the checkbax:
see clicking directly on see that the label for Adobe Acrobat has hid "a184". I'll clicking on the label instead, which should toggle the chackbox.
intercepts pointer events. Looking
The agent then proceeds to complete the task.
On the other hand, Figure 13 displays the last two steps from the same task, exccuted by GPT-4o. The agent does not notice it only ordering one laptop instead of the required 6, failing the lask.
More precisely, the agent attempts click on element a243 instead of using the select_opt action in the following AXTree:
[a237]
conbobox 'Quantity' value='11 '1', visible, "esPPoppp''eeu' expanded-Falee, controls- 'qantityyty..... Lareeittty_lbbb___ee _ppnn"
selected-True memuitan '2', selectod-Palse
[a239]
[a240]
'3', selected=False
[x241]
nequiten '4', selected-Falee
[a242]
nanmitan '5', selectod=False
[+243] nequiten '6', selected-Falee
[+244]
memuiten '7', selected-Falee nemuiten
selected-Palse
[2246]
nanuitan '9", selectad-Palse
[<247]
nequiten '10', selected-False
This causes the action fail. However, soeing the previous Chain-of-thought reasoning in its prompt, GPT-4o continues with the following:
need to proceed with configuring the "Standard Laptop" order. The quantity has bean set to 6. Mos, I should check the boxes for order.
"Adoba Acrobat" "Adoba Photoshop" to
The agent fools itself and does not understand that the previous action was in fact failure. It then goes on to complete the task without correcting this mistake.
31

---

Published in Transactions on Machine Learning Research (02/2025)
Jerry!
Step 1: Clickss on the "All" menu
(b) Step 2: Opens the catalog
Hardware
(c) Step 3: Go in the hardware section
(d) Step 4: Clicks on 'Standard Laptop' product
GOOOOODO
(c) Step 5: Fill in the required Gields (maltiple steps)
(n Step 6: Complete order
Figure 12: Successful execution of the task workarena, www.eiccaeecccceeoeeoe navigate-ann and wwwerrrreddddereeeeeeeeeeeeeeeae stwnsaad-dddddadae -apptopp1,, seed 99, by Clande 3.5 Sonnet. The Steps given in the caption are not the same the actual task steps, for the sake of conciseness.
32

---

Published in Transactions on Machine Learning Research (02/2025)
BOCOOODC
Code
(a) The agent unsucossfully attempts set amount
(b) Considering the order correct, agent completes wrong order.
Figure 13: Failed steps during the execution of the task workarena servicenoo navigate davigate-and-order -apttoop-- 12, seed 99, by GPT-40
33


Reviews:
[{"review_id": "v5wpBIEfhJ", "reviewer": "Reviewer_EDqS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe main contribution of this paper is the BrowserGym framework, which provides a unified test-bed for evaluating web agents based on LLMs/VLMs. The key point is that BrowserGym includes several state-of-the-art benchmarks designed to evaluate web agents' performance in different ways. This allows researchers developing new web agents to easily evaluate their performance on various tasks using this framework.\n\nThe idea of having a leaderboard is very interesting. It provides the community with a unique reference for web agents' performance and makes it easier to compare different web agents fairly.\nAlso the code will be released to give to chance to the other researchers to use this framework.\n\n\nWEAKNESSES\nThe work appears to be an engineering effort that unifies several existing benchmarks. Additionally, it is unclear if AgentLab was developed by the authors. I ask the authors to clarify this point, as there is no evidence in the paper.\n\nAnother point to clarify is how new benchmarks can be easily included in this framework. As new well-designed benchmarks emerge, it would be useful if researchers could add them to this framework.\n\nFinally, I ask the authors to discuss the computational time related to the experiments, including the hardware characteristics used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors present the BrowserGym framework, which unifies several state-of-the-art benchmarks to provide a unique platform for evaluating developed web agents. Using the proposed BrowserGym, six state-of-the-art (SOTA) language models (LLMs) and vision-language models (VLMs) have been tested, evaluating their performance on all the included web agent tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following my suggestions, I ask the authors to clarify the issues I raised in the previous section. I believe this could be useful in improving the overall quality of the paper.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kSqGTwDIuK", "reviewer": "Reviewer_i4RC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "From my perspective, this paper is about publishing the BrowserGym environment and AgentLab framework, introducing their design, and providing some reports. It is clear that the topic aligns with the current hot trends in LLM/VLM-based web agent research and is worth reading for researchers in this area.\n\nMy main concern lies with the claim of introducing BrowserGym. An ICML 2024 paper, *\"WorkArena: How capable are web agents at solving common knowledge work tasks?\"*, also claims to introduce BrowserGym and is cited in the references. Given TMLR’s policy that submissions must be original and should not reuse text, figures, or results from prior work published in archival venues, it is unclear if the claim regarding BrowserGym in this paper meets the journal’s requirements for originality and evidence.\n\nRegarding the benchmark results, I suggest refining Sections 4 or 6 to include explanations of the abilities reflected by the benchmark scores, rather than just reporting the numbers. It is important to provide users with a clear understanding of the levels of ability demonstrated on the unified evaluation platform, saving them the effort of interpreting the details of external benchmarks. If possible, categorizing the tasks across benchmarks and summarizing the high-level performance of web agents would provide better insights into their capabilities, rather than leaving readers to compare general scores without understanding which model excels at which specific abilities.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an ecosystem for web agents driven by Large Language Models (LLMs). The ecosystem is designed to standardize, simplify, and accelerate research and development in this domain. The claimed contributions are structured as follows:\n\n1. BrowserGym Environment:  \n   A gym-like environment that generalizes interaction scenarios for web agents, providing a standardized interface for web task automation and evaluation.\n\n2. Integration of Benchmarks:  \n   Unification of several existing web agent benchmarks under the BrowserGym framework, facilitating consistent evaluation, reproducibility, and cross-benchmark comparisons.\n\n3. AgentLab Framework:  \n   A complementary framework to support the development, testing, and analysis of web agents. AgentLab includes tools such as AgentXRay for behavior analysis and dynamic prompting for advanced agent implementations.\n\n4. Benchmarking Results:  \n   A comprehensive benchmark report comparing the performance of six LLMs/VLMs across all tasks in the BrowserGym ecosystem, with results published on the BrowserGym leaderboard.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Verify or revise the claim regarding BrowserGym, ensuring it adheres to TMLR's requirements.  \n2. Provide deeper analysis and explanation of the benchmark scores. At a minimum, include information that helps readers understand how the web agents perform and the specific abilities demonstrated by the reported results.\n\n--------\n\nAccording to the latest revision, I have no further requests. The Claims and Evidence criterion is now met, particularly regarding the claim of BrowserGym and the improved transparency in evaluating agent capabilities.", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g1MHkrLUZS", "reviewer": "Reviewer_dn9K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper presents comprehensive documentation of ecosystem features, providing details on customization and demonstrating minimum configuration requirements for evaluation purposes.\n- The framework demonstrates considerable flexibility due to its modular architecture, specifically addressing key aspects:\n  1. Forward compatibility with emerging evaluation tasks\n  2. Customization of agent configuration \n  3. Extensibility for integration of new LLMs and benchmarks.\n  4. Efforts in relation to reproducibility concerns\n- The web agent evaluation presented encompasses a number of different models, including proprietary, open-source, and “reasoning”-focused variants.\n\nWeaknesses:\n- The manuscript raises significant concerns about its eligibility under TMLR [Guidelines and Editorial Policies](https://jmlr.org/tmlr/editorial-policies.html#:~:text=In%20particular%2C%20we%20do%20not%20accept%20submissions%20that%20are%20expanded%20versions%20of%20conference%20papers.) \"TMLR only accepts original contributions that don’t reuse the authors’ own prior work. In particular, we do not accept submissions that are expanded versions of conference papers.\". There appears to be substantial content overlap with an earlier work [Drouin et al., ICML 2024] which also introduces the BrowserGym ecosystem.\n- The manuscript often emphasizes implementation-specific details and code structure at the expense of higher-level feature descriptions. While the technical details are valuable, this approach causes the focus to shift from the broader conceptual framework and core utilities to implementation details. A more balanced presentation that prioritizes functional descriptions over implementation specifics would make the text more accessible to a wider audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a comprehensive ecosystem designed to standardize and streamline web agent research. The main contribution is BrowserGym, a unified framework that enables researchers to evaluate and compare different web agents across multiple benchmarks using a consistent interface and methodology. In addition, the AgentLab framework provides tools for creating, testing, analyzing, and managing experiments, including features for parallel execution and reproducibility monitoring. The authors demonstrate the utility of the ecosystem by conducting a comparative study of six state-of-the-art language models on different web agent benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n- Avoid unnecessary code implementation details and function/class nomenclature as they reduce readability and can be easily avoided (especially in Sections 3.2, 5.1, 5.5, 5.6).\n- Instead, where code and variable/function/class names are considered necessary or more effective than a textual description, be sure to:\n    - Avoid the use of `var_name` as a name in the sentence, e.g. Section 3.3 \"Starting from a blank `page`\" -to-> \"Starting from a blank `page` object\"  or \"Starting from a blank page\"\n    - Follow consistent nomenclature throughout the text: `ClassName`, `variable_name`, `function_name()` seems to be the standard adopted, but not adhered to everywhere (e.g. `action_mapping` is used as both function and object, `Study` object is capitalized but not everywhere, ...)\n- Section 5.2: \"[...] the actual compute requirement of the job is _relatively small_ and we can launch up to 20 tasks in parallel on a _single laptop_ or even 50-100 on a server machine _with many CPU cores and large memory_\" is too general. Replace vague descriptions of hardware and, if possible, provide some metrics to quantify computing time/resources.\n- Section 5.4.2 - Reproducible Journal: Provide a clearer explanation of the feature and its role.\n- Section 6.2: Is it correct to say that the Llama models and o1 Mini have an unfair disadvantage in the comparison presented, because the other methods are vision-enabled and are also fed the web page screenshot? If so, this should be emphasized more in the text.\n- Section 6.2: The last paragraph states that: \"This result suggests that the BrowserGym API for browser interaction is more adapted for action-oriented tasks than for pure question-answering tasks.\", but it is not clear what problem/limitation is causing the low performance and thus this conclusion.\n\nMinor changes:\n- Missing citation to [Gymnasium](https://arxiv.org/abs/2407.17032) framework.\n- Section 3.1 - Extra element properties: Define and explain the SoM indicator and its purpose.\n- Section 5.4.2 - Reproduced results in the leaderboard: Clarify what qualifies an agent as \"reproducible\" and better explain the leaderboard's role (up to this point is only barely mentioned in the contributions of Section 1 therefore it should be explained what is it).\n- Section 5.4.2 - ReproducibilityAgent: Better define what the ReproducibilityAgent is (function? class?).\n- Section 6.2: Similarly to the request for Section 5.2, if possible, it would be helpful to have metrics that relate performance to time.\n- Table 2: The results table includes a cost column that, from what I can conclude, refers to the monetary cost of the model's API for evaluation. This is not exactly clear, plus I don't see the value added in reporting this information (neither the currency nor the cost per token of the API is specified, so this only provides a sense of scale and not much else).\n\nNon-essential changes (minor suggestions on style and form):\n- Section 2.2: \"Due to their immense potential, there has been a substantial effort in developing web agents, _and_ agents differ in how they interact with the web environment, the backbone models, and the goal they aim to achieve, among others.\", the _and_ is improperly used as a conjunction of two unrelated clauses.\n- Section 2.2: \"[...] enhance agent reasonning capabilities\" -to-> \"... reasoning ...\".\n- Section 2.2: \"To address those important points, we developed BrowserGym and Agentlab, which we describe in detail in the subsequent sections.\" -to-> \"... AgentLab ...\".\n- Section 3.1 - Screenshot: Add missing references to Figure 4.a and 4.b.\n- Section 3.1 - Open tabs: \"In BrowserGym, both _agents_ and _tasks_ have the option to use multiple tabs if needed\". Distinguish between agent _choice_ and task _requirement_ for multiple tabs.\n- Section 5: Consider moving agent creation description to beginning of section.\n- Figure 9: `AgentXRay` -to-> AgentXRay (for consistency with other occurrences in the manuscript).\n- Section 5.4.1 - Live websites: Restructure the discussion to focus on general \"core factors that can lead to discrepancies in web agent evaluation\" and not on the characteristics of the individual benchmark, to improve fluidity and consistency with the other paragraphs.\n- Sec 5.4.1: In the first paragraph \"Software version\" is used as title while in the last paragraph \"Non-deterministic tasks\" is referenced as being part of the sentence.\n- Section 5.5: Add missing leading space in first comment of `MyAgent` class figure.\n- Sec 6.1: The first paragraph describes the details of two configurations: agent and observation. From what I can conclude the two configurations are very distinct; if not, the description is misleading.\n- Sec 6.2 \"Llama-3.1-405B surpasses GPT-4o-mini\"  -> \"... GPT-4o-Mini\"", "broader_impact_concerns": "This work raises an interesting consideration regarding web agents' interaction with dynamic advertising content. Although the authors appropriately address the reproducibility challenges associated with such content, future ecosystem development could benefit from considering how to handle advertisement-related content in a standardized manner. Currently, network-related information (normally inspected against blacklists to identify and block ad sources) is not included in agent observations, and while this may not have a significant impact on current live benchmarks, it could become relevant as web agents become more sophisticated and spreaded. As the ecosystem aims to serve as a standard for the evaluation of web agents, a consideration of how to handle such content could help prevent potential future challenges in the interaction between web agents and dynamic advertising systems.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend accepting this paper since all three reviewers lean toward acceptance, and the paper makes valuable contributions through BrowserGym - a standardized ecosystem for web agent evaluation with compelling cross-benchmark LLM comparisons - subject to incorporating the reviewers' suggested changes.
Decision: Accept (poster)