WebArena: A Realistic Web Environment for Building Autonomous Agents
Authors: Shuyan Zhou
Frank F. Xu
Hao Zhu
Xuhui Zhou
Robert Lo
Abishek Sridhar
Xianyi Cheng
Tianyue Ou
Yonatan Bisk
Daniel Fried
Uri Alon
Graham Neubig
Venue: ICLR 2024 poster (2024)
Keywords: language-guided agents; web automation; benchmark
Primary area: datasets and benchmarks
TL;DR: WebArena is a highly realistic and reproducible web environment designed for building useful agents for everyday life. Strong language models like GPT-4 perform significantly worse than humans on the realistic tasks in WebArena.

Abstract: With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting.  The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that \ours can be used to measure such progress.\footnote{Code, data, environment reproduction instructions, video demonstrations are available in the supplementary.}

Published as a conference paper at ICLR 2024
WEBARENA: A REALISTIC WEB ENVIRONMENT FOR BUILDING AUTONOMOUS AGENTS
Shuyan Zhou" Frank Xu"
Hao Zhu i
Xuhui Zhou' Robert Lo
Abishek Sridhar!
Xianyi Cheng Graham Neubig
Tianyue Ou
Yonatan Bisk
Daniel Fried
Uri Alon
Carnegie Mellon University
(shuyanzh, fangzhex, gneubig @cs. cmu, cmu.edu
ABSTRACT
With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading a disconnect with real-world scenarios. In this paper, we build an environment for torguage-guide's agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g.. user manuals) to encourage human-like task-solving. Building upon our environment, we release set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-howe morizon,,,,, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, %, significantly lower than the human performance 78.24% These results highlight the need for further development of robust agents, that current state-of oftt----rtt large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress. Our code, data, environment reproduction resources, and video demonstrations are publicly available at https: httt::://webbreaa... /ww.awebbbaaeee///
INTRODUCTION
Autonomous agents that perform everyday tasks via human natural language commands could significantly augment human capabilities, improve efficiency, and increase accessibility. Nonetheless, to fully leverage the power of autonomous agents, it is crucial to understand their behavior within an environment that is both authentic and reproducible. This will allow measurement of the ability of agents on tasks that human users care about in fair and consistent manner.
Current environments for evaluate agents tend to over-simplify real-world situations. As a result, the functionality of many environments is limited version of their real-world counterparts, leading to lack of task diversity (Shi et al.] 2017: Anderson et al., 2018 Gordon et al. 2018 Misra et al.) 2016 Shridhar et al. 2020;]2021 | Yao 2022aa. In addition, these simplifccations often lower the complexity of tasks as compared to their execution in the real world (Puig et al., 2018: Shridhar al. 2020 Yao et al. 2022a). Finally, some environments are presented as static resource (Shi al. 2017; Deng et al. (023) where agents are confined to accessing only those states that were previously cached during data collection, thus limiting the breadth and diversity of exploration. For evaluation, many environments focus comparing the textual surface form of the predicted
"Lead contributors.
I Equal contribution.

---

Published as a conference paper at ICLR 2024
Action
Tell me how much spent food purchase in March 2023
CMS
O Al Agent
Web applications from popular domains
Create a 'NolauTans' repo, listing Nolan's 's Oscar-wvinning filmss in a README file
Feedback
X Functional Failure
Tool Sites
Knowledge resources
chack rasitn Clec onnser
WebArena Environment
Functional Success
Figure I: WebArena is standalone, self-hostable web environment for building autonomous agents. WebArena creates websites from four popular categories with functionality and data mimicking their world equivalents. To emulate human problem-solving, WebArena also embeds tools and knowledge resources as independent websites. WebArena introduces benchmark on interpreting high-level realistic natural language command to concrete web-based interactions. We provide validators to programmatically validate the functional correctness of each task.
action sequences with reference action sequences, disregarding the functional correctness of the executions and possible alternative solutions (Puig et al. 2018, Jernite et al. 2019;) Xu et al. 2021] Li
et al. 2020; Deng et al., 2023). These limitations often result in discrepancy between simulated
environments and the real world, and can potentially impact the generalizability of AI agents to
successfully understand, adapt, and operate within complex real-world situations.
We introduce WebArena, a realistic and reproducible web environment designed to facilitate the development of autonomous agents capable of executing tasks ($2). An overview of WebArena is in Figure I Our environment comprises four fully operational, self-hosted web applications, each representing a distinct domain prevalent on the internet: online shopping, discussion forums, collaborative development, and business content management. Furthermore, WebArena incorporates several utility tools, such as map, calculator, and scratchpad, to best support possible human-lik, task executions. Lastly, WebArena is complemented by an extensive collection of documentation and knowledge bases that vary from general resources like English Wikipedia to more cemmain-speciffe references, such as manuals for using the integrated development tool (Fan et al.) 2022). The content populating these websites is extracted from their real-world counterparts, preserving the authenticity of the content served on each platform. We deliver the hosting services using Docker containers with gym-APIs (Brockman et al., 2016), ensuring both the usability and the reproducibility of WebArena.
Along with WebArena, release ready-to tasks ($ Each task is described as a high-level natural language intent, emulating the abstract language usage patterns typically employed by humans (Bisk et al. 2019). Two example intents are shown in the upper left of Figure 1] We focus on evaluating the functional correctness of these tasks, i.e., does the result of the execution actually achieve the desired goal ($3.2). For instance, to evaluate the example in Figure 2) our evaluation method verifies the concrete contents in the designated repository. This evaluation is not only more reliable (Zhog et al.) 2017 Chen et 202T Wang et al. 2022) than comparing the textual surface-fon action sequences (Puig et al. [2018] Deng al.) 2023) also accommodate a range of potential valid paths achieve the same goal, which
benchmark with 812 long- web-base web-based
ubiquitous phenomenon in sufficiently complex tasks.
We use this benchmark to evaluate several agents that can follow NL command and perform web- based tasks ($4). These agents are implemented in a few-shot in-context learning fashion with powerful large language models (LLMs) such as GPT-4 and PALM-2. Experiment results show that the best GPT-4 agent performance is somewhat limited, with end-to-end task success rate of only 14.41%. %, while the human performance is 78.24%. We hypothesize that the limited performance of current LLMs stems from lack of crucial capabilities such active exploration and failure recovery to successfully perform complex tasks ($5.I). These outcomes underscore the necessity for further development towards robust and effective agents (LeCun 2022) in WebArens
WEBARENA: WEBSITES AS AN ENVIRONMENT FOR AUTONOMOUS AGENTS
Our goal is to create a realistic and reproducible web environment. We achieve reproducibility by making the environment standalone, without relying live websites. This circumvents technical
2

---

Published as a conference paper at ICLR 2024
Create an efficient itinerary to visit all of Pittsburgh's art museums with minimal driving distance starting from Schenley Park. Log the order my sawesome-nrthhess-ee travel" repository
wechenzne wik grali
web.reeez.oooeeo www.ooowweseeeee..oooo com
weleeemn gillsh.om
R dilemia
Middorgh mussure
Clanr Eiccstbles
List of museums Pittsburgh
The far avouma Prabbunk Pennahvane ancorpozies
rutl andend unine pock sing aopuola
Edit
Sepace
Aleo indede soad
Travel in Northeast US
artited
oci ony oyeergg o i. vetal musarrt ar iecuree
Directions
Pittsburgh
Pitter detters Carnecde Melten Inderstiy:
Museemm
Justan hane
Search for museums in Pittsburgh
Search for each art museum the Map
U
Record the optimized results the repo
O
Figure 2: A high-level task that can be fully executed in WebArena. Success requires sophisticated, long-term planning and reasoning. To accomplish the goal (top), an agent needs to (1) find Pittsburgh art museums on Wikipedia, (2) identify their locations map (while optimizing the itinerary), and (3) update the README file in the appropriate repository with the planned route.
challenges such as bots being subject to CAPTCHAs, unpredictable content modifications, and configuration changes, which obstruct fair comparison across different systems over time. We achieve realism by using open-source libraries that underlie many in-use sites from several popular categories and importing data to our environment from their real-world counterparts.
2.1 CONTROLLING AGENTS THROUGH HIGH-LEVEL NATURAL LANGUAGE
The RebArena environment denoted as E- A, with state space S, action space A 2.2.4 and observation space O 2.3 The transition function T S A-> S is deterministic, and it is defined by the underlying implementation of each website in the environment. Given task described natural language intent i, an agent issues an action a, A based on intent i, the current observation o,E O the action history a and the observation history oj 1 Consequently, the action results in new state S411 S and its corresponding observation 0++1 O. We propose a reward function r(af" $s') to measure the success of a task execution, where a represents the sequence of actions from start to the end time step T,, and sf denotes all intermediate states. This reward function assesses if state transitions align with the expectations of the intents. For example, with an intent to place an order, it verifies whether an order has been placed, Additionally, it evaluates the accuracy of the agent's 's actions, such as checking the correctness of the predicted answer.
2.2 WEBSITE SELECTION
To decide which categories of websites to use, we first analyzed approximately 200 examples from the authors' actual web browser histories. Each author delved into their browsing histories, summarizing the goal of particular segments of their browser session. Based on this, we classified the visited websites into abstract categories. We then identified the four most salient categories and implemented one instance per category based on this analysis: (1) E-commerce platforms supporting online shopping activities (e.g., Amazon, eBay), (2) social forum platforms for opinion exchanges (e.g.. Reddit, StackExchange), (3) collaborative development platforms for software development (e.g., GitLab), and (4) content management systems (CMS) that manage the creation and revision of the digital content (e.g., online store management).
In addition to these platforms, we selected three utility-style tools that are frequently used in web- based tasks: (1) map for navigation and searching for information about points of interest (POIs) such as institutions locations (2) a calculator, and (3) scratchpad for taking notes. As information- seeking and knowledge acquisition are critical in web-based tasks, we also incorporated various knowledge resources into WebArena. These resources range from general information hubs, such as the English Wikipedia, to more specialized knowledge bases, such as the website user manuals.
Implementation We leveraged open-source libraries relevant to each category to build our own versions of an E-commerce website (OneStopShop), GitLab, Reddit, an online store content manage- ment system (CMS), a map, and English Wikipedia. Then imported sampled data from their
3

---

Published as a conference paper at ICLR 2024
Hereen www.opplllgo..oooo
websrenn onesopphhp com
webarene oneskophhp
Pot.o, Lown Gerden
<IE>
<div>
RoorWebAren "Patio, Lawn
<a href-" "><ing sre-"..." "><la
link 'Image'
<div clas?
img 'Image'
brof=" "> Outdoor Patio
link 'Outdoor Patio.."
Layout Table
<div>
Static Text 'Rating:'
spaan Rating: </spa>>
1 '82%
<div>
'12 Reviews'
sspan 82%-/span>>>>>
Static Text '549.99"
<id
button 'Add to Cart' focusable: True
<a href "revveiewss
button "Wish List' focusable:
Sspan Reviews-/span
button 'Compare' focusable:
We design the observation to the URL and the content of a web page, with options to represent the content a screenshot (left), HTML DOM tree (middle), and accessibility tree (right). The content of the middle and right figures are trimmed to save space.
Figure 3:
real-world counterparts. As an example, our version of GitLab was developed based on the actual GitLab project We carefully emulated the features of a typical code repository by including both popular projects with many issues and pull requests and smaller, personal projects. Details of all websites in WebArena can be found in Appendix [A.]] We deliver the environment as dockers and provide scripts reset the environment to a deterministic initial state (See Appendix (A.2).
2.3 OBSERVATION SPACE
We design the observation space to roughly mimic the web browser experience: web page URL, the opened tabs, and the web page content of the focused tab. WebArena the first web environment to consider multi-tab web-based tasks to promote tool usage, direct comparisons and references across tabs, and other functionalities. The multi-tab functionality offers more authentic replication of human web browsing habits compared to maintaining everything in single tab. We provide flexible configuration to render the page content in many modes: (see (see Figure 3 for an example): (I) the raw web page HTML, composed of Document Object Model (DOM) tree, commonly used in past work (Shi et al. 2017 Deng et al., 2023 Li et al. 2020); (2) screenshot, a pixel-based representation that represents the current web page as RGB array and (3) the accessibility tree of the web page Fl The accessibility tree is a subset of the DOM tree with elements that are relevant and useful for displaying the contents of web page. Every element is represented as its role (e.g. link), its text content, and its properties (e.g.. whether it is focusable). Accessibility trees largely retain the structured information of web page while being more compact than the DOM representation.
We provide an option to limit the content to the contents within a viewport for all modes. This ensures that the observation can input into a text-based model with limited context length or an based model with image size or resolution requirements.
2.4 ACTION SPACE
Following previous work on navigation and operation in web and embodied environments (Shi et al, 2017|Lin et al., 2018), we design compound action space that emulates the keyboard and mouse operations available on web pages. FFigure 4 lists all the available actions categorized into three distinct groups The first group includes element operations such clicking, hovering, typing, and key combination pressing. The second comprises tab-related actions such as opening, closing, and switching between tabs. The third category consists of URL navigation actions, such as visiting specific URL or navigating forward and backward in the browsing history.
Building on these actions, WebArena provides agents with the flexibility refer to elements for operation in different ways. An element can be selected by its on-screen coordinates, (z, y). or by a unique element ID that is prepended to each element. This ID is generated when traversing the Document Object Model (DOM) or accessibility tree. With element IDs, the element selection is transformed into an n- n-way classification problem, thereby eliminating any disambiguation efforts required from the agent or the underlying implementation. For example, issuing the action cl i ick [1582] clicks the button given the observation of [1582] Add to Cart. This flexible element selection allows WebArens support agents designed in various ways (e.g., accepting input from different modalities) without compromising fair comparison metrics such as step count.
https://devellpprr mozi http://www....aaoooooo//o//////a
4

---

Published as a conference paper at ICLR 2024
Action Type
Deseription
Category
Example
noop
Do nothing
click (elem)
Click at an element
Information Sceking
When was the last time bought shampoo
hover (elem)
Hover on an element
Compare walking and driving time from AMC Waterfront to Randyland
type (elem, text)
Type to an element Press key comb Scroll up and down
press (key_comb)
scroll (air)
Site Navigation
Checkout merge requests assigned
tab_focus (index)
focus i-th tab Open new tab
Show me the ergonomic chair with the best rating
new_tab
tab_close
Close current tab
go_back
Visit the URL
Content & Config
Post to ask "whether I need a car in NYC"
go_ forward
Undo go back
Delete reviews from the scammer Yoke
goto (URL)
Go to URL
Figure 5: Example intents from three categories.
Figure 4: Action Space of WebArena
User Role Simulation Users of the same website often have disparate experiences due to their distinct roles, permissions, and interaction histories. We emulate this scenario by generating unique user profiles on each platform. The details can be found in Appendix A.3
BENCHMARK SUITE OF WEB-BASED TASKS
We provide a benchmark with 812 test examples on grounding high-level natural language instructions to interactions in WebArena. Each example has metric to evaluate the functional correctness of the task execution. this section, we first formally define the task of controlling an autonomous agent through natural language. Then introduce the annotation process of our benchmark.
3.1 INTENT COLLECTION
We focus curating realistic intents to carry out complex and creative tasks within WebArena. To start with, our annotators were guided to spend a few minutes exploring the websites to familiarize themselves with the websites' content and functionalities. As most of our websites virtually identical to their open-web counterparts, despite having sampled data, most annotators can quickly comprehend the websites.
Next, we instructed the annotators to formulate intents based on the following criteria:
(1) The intent should be abstract and high-level, implying that the task cannot be fulfilled with two actions. As example, instead of "click the science subreddit". encouraged annotators to come up with something more complex like "post a greeting message merely
on sci ieenee subreddif", which involves performing multiple actions.
(2) The intent should be creative. Common tasks such as account creation can be easily thought of. We encouraged the annotators add constraints (e.g., "create Reddit account identical to my GitLab one") to make the intents more unique.
(3) The intent should be formulated as a template by making replaceable elements as variables. The annotators were also responsible for developing several instantiations for each variable. For example, the intent "create a Reddit account identical to my GitLab one" can be converted into "create [[sitel]] account identical to my [[site2]] one", with an instantiation like "[sitel: Reddit, site2: GitLab]" and another like "[sitel; GitLab, site2: OneStopShopping/" Notably, tasks derived from the same template can have distinct execution traces. The similarity resides primarily in the high-level semantics rather than the specific implementation.
We also provided a prompt for the annotators to use with ChatGPT for inspiration, that contains an overview of each website and instructs the model to describe potential tasks to be performed these sites. Furthermore, we offered a curated list of examples for annotators to reference.
Intent Analysis In total, we curated 241 templates and 812 instantiated intents. On average, each template is instantiated to 3.3 examples. The intent distribution is shown in Figure 6 Furthermore, we classify the intents into three primary categories with examples shown in Figure
https://chat. opwnaai...pppncc.cccmmm
5

---

Published as a conference paper at ICLR 2024
(1) Information-seekingg tasks expect textual response. Importantly, these tasks in WebArena often require navigation across multiple pages or focus on user-centric content. This makes them distinct from open ween-dmainn question-answerne 2019), which focuses on querying general knowledge with a simple retrieval step. For instance, to answer "When was the last time bought the shampoo",
(Yang et al. 2018 Kwiatkowski et al.)
agent traverses the user's purchase history, checking order details to identify the most recent shampoo purchase.
(2) Site navigation: This calegory is composed of tasks that require navigating through web pages using variety of interactive elements such as search functions and links. The objective is often to locate specific information or navigate to particular section of site.
(3) Content and configuration operation: This category encapsulates tasks that require operating in the web environment to create, revise, or configure content or settings. This includes adjusting settings, managing accounts, performing online transactions, generating new web content, and modifying existing content. Examples range from updating a social media status or README file to conducting online purchases and configuring privacy settings.
3.2 EVALUATION ANNOTATION
Evaluating Information Seeking Tasks To measure the correctness of information-seeeknng tasks where a textual answer is expected, we provide the annotated answer a' for each intent. The a" is further compared with the predicted answer à with one of the following scoring functions Tinto (ā, a*).
First, we define exact aaaact_matcchhhhh where only à that is identical with a² receives a score of one. This function is primarily applicable to intent types whose responses follow more standardized format, similar to the evaluation on question answering literature (Rajpurkar et al. 2016 Yang et al. 2018).
Second, we create must must_incll pel.include where any à containing a" receives a score of one. This function is primarily used in when an unordered list of text is expected or where the emphasis of evaluation is on certain key concepts. In the second example (Table we expect both the correct name and the email address to be presented, irrespective of the precise wording used to convey the answer.
Finally, we introduce Juzzy_maton where we utilize a language model to assess whether is semantically equivalent to a". Specifically, in this work, use gpt 00613 to perform this evaluation. The corresponding prompt details are provided in Appendix A.7 The fuzzy_ s_r_mattooe ch function applies to situations where the format of the answer is diverse. For instance, in responding to "Compare the time for walking and driving route from AMC Waterfront to Randyland", it is essential to ensure that driving time and walking time accurately linked with the correct terms. The Fuzzy_mat ch function could also flexibly match the time "2h58min" with different forms such as "2 hour 58 minutes". "2:58" and others. We demonstrate a language model can achieve nearly perfect performance on this task in A.8
Evaluating Site Navigation and Content & Config Tasks The tasks in these categories require accessing web pages that meet certain conditions or performing operations that modify the underlying data storage of the respective websites. To assess these, we establish reward functions Tpoo (s) that programmatically examine the intermediate states within execution trajectory to ascertain whether the outcome aligns with the intended result. These intermediate states are often the underlying databases of the websites, the status, and the content of a web page at each step of the execution.
Evaluating each instance involves two components. First, we provide a locator, tasked with retrieving the critical content pertinent to each intent. The implementation of this locator varies from a database query, wehsite-supported API call, to JavaScript element selection on the relevant web page, depending on implementation feasibility. For example, the evaluation process for the intent of the fifth example in Table I first obtains the URL of the latest post by examining the last state in the state sequence s. Then it navigates to the corresponding post page and obtains the post's content by running the Javascript "document... querySelectorrr
inner' Texer"
Subsequently, we annotate keywords that need to exist within the located content, For example, the evaluation verifies if the post is correctly posted in the "nyc" subreddit by examining the URL of the post and if the post contains the requested content by examining the post content. We reuse the exa aott_mateo and must incl ude functions from information-ssekinng tasks for this purpose.

---

Published as a conference paper at ICLR 2024
Function
ID
Intent
Eval Implementation
Tell me the name of the customer who has the most cancellations in the history
exact_mat.chi "Samantha Jones")
"a" "0","" ,a)
Find the customer name and email with phone number 801555 1212
must include(d. "Sean Miller")
must int__ineldee(i, "sean gmail.com")
Compare walking and driving time from AMC Waterfront to Randyland
match(ā, "walking: 2h58min") fuzzy_matchin "driving: 21min")
Checkout merge requests assigned to me
url=locate_curr ctrrree_uuuee___eeeee uur (s)
exact_matcch (URL, "gitlab.comm/ergg.
url=Locate__atcet__osst___ (s)
Fpngg (s)
Post to ask "whether 1 need a car NYC"
(s)
must must_include (UBL, "/f/nyc") must inst_include (body, "a car in NYC")
Table I: We introduce two evaluation approaches. Tinfo (top) measures the correctness of performing information seeaatiio--eeiinggg tasks. It compares the predicted answer à with the annotated reference a" with three implementations. 'poo (bottom) programmatically checks whether the intermediate states during the executions possess the anticipated properties specified by the intent.
Unachievable Tasks Due to constraints such as inadequate evidence, user permissions ($A.3) or the absence of necessary functional support on the website, humans may ask for tasks that are not possible complete. Inspired by previous work evaluating question-answering models on unanswerable questions (Rajpurkar et al. 2018), we design unachievable tasks in WebArena. For instance, fulfilling an intent like "Tell the contact number of OneStopShop" is impracticable in WebArena, given that the website does not provide such contact information. We label such instances "N/A" " and expect agent produce an equivalent response. These examples allow us to assess an agent's ' ability to avoid making unfounded claims and its adherence to factual accuracy.
The intents were contributed by the authors following the annotation guideline in $3.1 Every author has extensive experience with web-based tasks. The reference answers to the information-seeking tasks were curated by the authors and an external annotator. To ensure consistency and accuracy, each question was annotated twice. If the two annotators disagreed, a third annotator finalized the annotation. The programs to evaluate the remaining examples were contributed by three of the authors who are proficient in JavaScript programming. Difficult tasks were often discussed collectively to ensure the correctness of the annotation. The annotation required the annotator to undertake the full execution and scrutinize the intermediate states.
Annotation Process
Human Performance We sample one task from each of the 170 tem- plates and ask five computer science graduate students to perform these tasks. The human performance is on the right. Overall, the human annotators complete 78.24% of the tasks, with lower performance on information-seeking tasks. Through examining the recorded trajectories, we found that 50% of the failures are due to misinterpreting the intent (e.g., providing travel distance when asked for travel time), incomplete answers (e.g., providing only name when asked for name and email), and incomplete executions (e.g., partially filling the product information), while the remaining instances have more severe failures, where the executions off- off-target. More discussions human annotations can be found in $A.5
Avg. Time Success Ratewn 74.68%
110s
Success Rateaterss 81.32%
Success Rate 78.24%
BASELINE WEB AGENTS
We experiment with three LLMs using two prompting strategies, both with two examples in the context. In the first setting, we ask the LLM to directly predict the next action given the current observation, the intent and the previously performed action. In the second setting, with the same information, the model first performs chain-of-thought reasoning steps in the text before the action prediction (CoT, Wei et al. (2022): Yao et al. (2022b)). Before the examples, we provide a detailed overview of the browser environment, the allowed actions, and many rules. To make the model aware of the unachievable tasks, the instruction explicitly asks the agent to stop if it believes the task is impossible to perform. We refer to this directive as Unachievable hint, or UA hint. This introduction
7

---

Published as a conference paper at ICLR 2024
is largely identical the guidelines presented to human annotators to ensure a fair comparison. We use an accessibility tree with element IDs as the observation space. The agent can identify which element interact with by the ID of the element. For instance, the agent can issue click [1582] to click the "Add to Cart" button with the ID of 1582. The full prompts can be found in Appendix The detailed configurations of each model can be found in Appendix A.6
A.9
RESULTS
The main results are shown on the top of Table GPT-4 (OpenAI 2023) with CoT
CoT UA Hint
Model
SR
SRAC
SRua
prompting achieves modest end-to-end task success rate of 11.70%, which is signifi- cantly lower than the human performance of 78.24%. GPT-3.5 (OpenAl 2022) with CoT prompting is only able to successfully perform 8.75% the tasks. The explicit reasoning procedure is somewhat helpful, it brings 2.34% improvement over the version without it. Further, TEXT- NAXT-BISON-O01 (Anil et al. 2023) underperforms GPT-3.5, with success rate of 5.05%. These results under- line the inherent challenges and complexities of executing tasks that span long horizons, particularly in realistic environments such as WebArena:
TEXT-BISON-001
5.05
4.00
27.78
X
GPT-3.5
6.41
4.90
38.89
/
GPT-3.5
8.75
6.44
58.33
GPT-4
11.70
8.63
77.78
X
GPT-3.5
5.10
4.90
8.33
x
GPT-3.5
6.16
6.06
8.33
x GPT-4
14.41
13.02
44.44
Human 78.24
77.30
100.00
Table 2:
The end-to-end task success (SR %) on RenArena with different prompting strategies. CoT: the model performs step-by-step reasoning before is- suing the action. UA hint: ask the model to stop when encountering unachievable questions.
5.1 ANALYSIS
models know when to stop? In our error analysis the execution trajectories, we observe prevalent error pattern of carly stopping due to the model's conclusion of unachievability. For instance, GPT-4 erroneously identifies 54.9% of feasible tasks as impossible. This issue primarily stems from the UA hint in the instruction, while this hint allows models to identify unachievable tasks, also hinders performance on achievable tasks. To address this, we conduct an ablation study where remove this hint. We then break down the success rate for both achievable and unachievable tasks. As shown in Table 2, eliminating this instruction led to a performance boost in achievable tasks, enhancing the overall task success rate of GPT-4 to 14.41% %. Despite an overall decline in identifying unachievable tasks, 44 retains the capacity to recognize 44.44% of such tasks. It does so by generating reasons of non-achievabiiltyy even without explicit instructions. On the other hand, GPT-3.5 rarely exhibits this level of reasoning. Instead, it tends to follow problematic patterns such as hallucinating incorrect answers, repeating invalid actions, or exceeding the step limits. This result suggests that even subtle differences in instruction design can significantly influence the behavior of a model in performing interactive tasks in complex environments.
gpt S-ddrcect
Can model maintain consistent performance across similar tasks? Tasks that originate from the same template usually follow similar reasoning and planning processes, even though their observa- tions and executions will differ. We plot a histogram of per-template success rates for our models in Table Of the 61 templates, GPT-4 manages to achieve a 100% task success rate on only four templates, while GPT-3.5 fails achieve full task completion for any of the templates. In many cases, the models are only able to complete one task variation with a template. These observations indicate that even when tasks are derived from the same template, they can present distinct challenges. For instance, while "Fork metaseq" can be a straightforward task, "Fork all repos from Facebook" derived from the same template requires more repetitive operations, hence increasing its complexity. Therefore, RebArena provide a testbed to evaluate more sophisticated methods. In partieular, those that incorporate memory components,
gpt 3.5-ott opt-4icot
Temoliate 1
10
0.1 0.2 0.3 Success rate (9) within atemplate
0.4
0.5
0.6 0.9 1.0
Table Distribution of suc- cess rates on templates with successful executions on GPT models (no UA hint).
8

---

Published as a conference paper at ICLR 2024
Dynamic Interaction?
Realistic Environment?
Diverse Human Tasks?
Functional Correctness?
Benchmark
Mind2Web
Deng et
2023
Forn/QAWoB
Shi et al.
2017
x
MiniWoB-s
Liu ct al.
2018)
/
X
X
X
Wehshop
Yao et al.
20222
/
X
ALFRED
Shridhar
2020
V
X
VirtualHome
Pung et al.
2018)
X
X
AndroidEnv
Toyama et al.
2021
WebArena
Table 4: The comparison between our benchmark and existing benchmarks grounding natural language instructions to concrete executions. Our benchmark is implemented in our fully interactable highly-realistic environment. It features diverse tasks humans may encounter in their daily routines. We design evaluation metrics to assess the functional correctness of task executions.
enabling the reuse of successful strategies from past experiments Zhou et al. (022a); Wang et al. (2023).
More error analysis with examples can be found in Appendix A.10
RELATED WORK
Benchmarks for Controlling Agents through Natural Language Controlling agents via natural
language in the digital world have been studied in the literature (Branavan et al. 2009 Shi et al. 2017
Liu et al.] 2018 Toyama et al. 2021 Deng et al.) 2023 Li et al. 2020 Xu et al.) 2021). However, the balance between functionality. authenticity, and support for environmental dynamics remains challenge. Existing benchmarks often compromise these aspects, as shown in Table 4 Some works
rely on static states, limiting agents' explorations and functional correctness evaluation (Shi et al.
2017: Deng et af. (023). while others simplify real-world complexities, restricting task variety (Yao al., 2022a] Liu et 2018). While AndroidEnv (Toyama et (021) replicates Android setup, it does not guarantee the reproducibility since live Android applications are used. (Kolve et al. 2017
Shridhar al. 2020 Puig al. 2018) and extends to gaming environments (Fan et al. [022]] Küttler
et al. 2020), where the environment mechanisms often diverge from human objectives.
Interactive Decision-Making Agents Nakano al. (2021 introduce WebGPT which searches the web and reads the search results to answer questions. Gur et al. (2023) propose web agent that synthesizes Javascript code for the task executions. Adding a multi-modal dimension, [ee et al. (2023) and Shaw et al.) (2023) develop agents that predict actions based on sereenshots of web pages rather than relying on the text-based DOM trees. Performing tasks in interactive environments requires the agents exhibit several capabilities including hierarchical planning, state tracking, and error recovery. Existing works (Huang et al.) 2022 Madaan et al., 2022: Li al. 2023) observe LLMS could break task into more manageable sub-tasks (Zhou et al. 2022b). This process can be further refined by representing task executions as programs, a technique that aids sub-task management and skill reuse (Zhou al. 2022a| Liang et al. [023 Wang et al. 2023] Gao al. 2023). Mcanwhile, search and backtracking methods introduce a more structured approach to planning while also allowing for decision reconsideration (Yao al. 2023 Long 2023). Existing works also incorporate failure recovery, self-correction (Shinn et al.) 2023 Kim et al. 2023). observation summarization (Sridhar jet al., (223)) to improve execution robustness. The complexity of WebArena presents a unique challenge and opportunity for further testing and improvement of these methods.
7 CONCLUSION
We present WebArena, a highly-realistic, standalone, and reproducible web environment designed for the development and testing of autonomous agents. WebArena includes fully functional web applications and organic data from popular domains. Additionally, we curate a comprehensive benchmark consisting of 812 examples that focus on mapping high-level natural language intents into concrete web interactions. We also offer outcome-based evaluation that programmatically validate the tasks success. Our experiments show that even GPT-4 only achieves a limited end-to-end task success rate of 14.41%, significantly lagging behind the human performance of 78.24%. These findings underscore the need for future research to focus on enhancing the robustness and efficacy of autonomous agents within WebA rena environment.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to thank Emmy Liu, Zhiruo Wang, Zhitong Guo for examining our annotations, Shunyu Yao for providing the raw Amazon product data in Webshop, Pengfei Liu, Zaid Sheikh and Aman Madaan for the helpful discussions. We are also grateful to the Center for AI Safety for providing computational resources. This material is partly based on research sponsored in part by the Air Foree Research Laboratory under agreement number FA8750-19 is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of the Air Force Research Laboratory or the U.S. Government. This project was also partially supported by a gift from AWS Al.
The U.S. Government
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpret- ing visually-groudded navigation instructions in real environments.
2018 IEFE Conference 2018, Salt Lake City, UT; USA, June 18- 22, 2018, pp. 3674-3683. IEEE Computer Society, 2018. doi: 10.1109/CV 201.00087.7 /R//sttpp//ppenccesss.sbee
on Computer Vision and Pattern Recognition, CVPR
com content cvpr_ 201820112220001 Vision- http::owwwoo..uauuu.eooeaaaaaan. htt.aiigetion__ntepprree.... ctteeii_ncc_____________2 Baw.ee_pppp____ html
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellstern Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Bric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Yonatan Bisk, Jan Buys. Karl Pichotta, and Yejin Choi. Benchmarking hierarchical script knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, Volume I (Long and Short Papers), pp. 4077-4085, Minncapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/5/3//1//111110 1412. URL httpt:://///.. /tapps://acleee http:///aaaaatthoooo.rrr////////
1412
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. In Proceedings the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pp. 82-90 90, Suntec. Singapore 2009. Association for Computational Linguistics. URL
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. URL htt.ps://erxivv.or///a/e 01.0066...5444 51540
10

---

Published as a conference paper at ICLR 2024
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained code. ArXiv preprins, abs/2107.03374, 2021. URL https: /atpiii///axiivoo//a wwt.:/oggg/aaaaaaa
03374]
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web, 2023.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. StLhttts::// hhttps://openrevl.ss net. form??
ad=rc8o_]8I30 88X
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aides aided language models. International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: visual question answering in interactive environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4089-4098... IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00430. URL /ttt::://peeaccesss. wwwewh..eeeeeee.eee wttp///w...ooooootnnceooomc Gordon TQA_ Vi sual _uwest ceppttiin_________ SDVPR_20181_0 2at___221__pp_r_______
catpenttt___________e_2 800tm/
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and
Aleksandra Faust. A real-world wehagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856 2023.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL .ww.smm...ammm press/vl 62 ht/hhangg2aa... htmmlmml2
Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. CraftAssist Instruction Parsing: Semantic Parsing for Minecraft Assistant. ArXiv preprint, abs/1905.01978, 2019. URL https
wttp://o.r..aaaa1.....11 01978
Geunwoo Kim, Pierr Baldi, and Stephen McAleer. Language models can solve computer tasks. ArXiv preprint, abs/2303. 17491, 2023. URL htlLaatts///axiiaa http://aaxwwv...////////...0. 1749
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL//ttpps:/pproceeding neurii http::/sppppcccppppe//////0//0// 169ff99976643bbbee5eeeeeeee htt.:e/eeee.tt.ttaaaaaaeeaaaaaaa Abst
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Bing-WWi Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slay Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics. 7:452-466 2019. doi: 10. 16.1/al/ a 00276. URL / https://acl antho www.anhhwoo..ogg.ooooeooo wgw..ogggl.........

---

Published as a conference paper at ICLR 2024
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.. Open Review, 62, 2022.
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pp. 18893-18912. PMLR, 2023.
Xinze Li, Yixin Cao, Muhao Chen, and Aixin Sun. Take break in the middle: Investigating subgoals towards hierarchical script generation. ArXiv preprint, abs/2305.10907, 2023. URL
https::/aaxiis org/ haarg//b///ae
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural lan- guage instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meel- ing of the Association for Computational Linguistics, pp. $198-8210, Online, 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2020.co 10..865//1/2000.ddmmmi.. mainm...22. URL https
//aclanthology. wwwpoogy.rgg//0oooooooo acl wwwaacainnnn....aaaa.a main. 729
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL wwwpt:ttt//t/c.tt /ttpss//ppeeevvieeee ew net forummimmnoo
ryIp3f-01 0-
Jieyi Long. Large language model guided tree of ArXiv preprint, abs/2305.082911 2023. URL https::/arxiv.... wwwpaiivorrrraaee httg://b//3300..0 08291
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1384-1403, Abu Dhabi, United Arab Emirates. 2022. Association for Computational Linguistics. URL https: http////ccaaattooocc/oe org/ 2022. emn www.eenmmmmme pa-mllpmann... 90.
Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, (111-3)221-3300 2016.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answeringg with human feedback. arXiv preprint arXiv:2112.09332, 2021.
OpenAl. Chatgpt: Optimizing language models for dialogue. 2022.
OpenAl. Gpt-4 technical report. arXiv,, pp. 2303-08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. httpa//////////eeeceeeeeeeeeeeee
IEEE Computer Society, 2018. doi: 10.1109/CVPR. cvpr ceppr_20180 1018/html Pui g Vi rttal Home at or paper. htm
2018.00886. URL http: htw.:peeccccceee.eeeeeeeeec.oooe
12

---

Published as a conference paper at ICLR 2024
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10. 10.18653/v 6- 1264. URL wtt.::tpps......m haa.ttppo./allaattooooooooooooo www.annnooooooooooooooo
Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10. 1/1333//P11-21224 URL https: http///aaa/tth/////// logy
org/P18-2124
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. arXiv preprint arXiv:2306. 330::20226000 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Perey Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW; Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3135- 3144. PMLR, 2017. URLAtttp://proceed.no
mlr. wrw.s/rsep....... / htt///00//stas/ 7a html
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and reffereftectinn. ArXiv preprint, abs/2303.11336, 2023. URL atttps://arxe
org/ abs/ 2303. 11366
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle WA, USA, June 13-19. 2020, pp. 10737-10746. IEEE, 2020. doi: 10. 1109/CVPR4220002220000000 01675
URL http://///aloooro/.0000/////////
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpps://openrertt net forumormmmoo
0I0X0YcCaIn.
Abishek Sridhar, Robert Lo, Frank F Xu, Hao Zhu, and Shuyan Zhou. Hierarchical prompting assists large language model on web navigation. arXiv preprint 12X2 2023.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: reinforcement learning platform for android. ArXiv preprint, abs/2105.13231. 2021. URL attpps://axxiv.cr///att
13231
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv preprint, abs/2305.16291, 2023. URL httpttts://aarttt/
org/ abs abs/2305. 1629
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution- evaluation for open-domain code generation. ArXiv preprint, abs/2212.10481,, 2022. URL
org/aba/ 2ttp:/222..ce 10481
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. -hain-of-thouuht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24888 2022.
Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1022-1032. Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 1 Inacc -naacl-main.800 URL https: / ////aalanntttt www.aatthhoogy.oogg...
2021. naac -main.
13

---

Published as a conference paper at ICLR 2024
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2388 Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/11/D18-122
antho logy. -0g.110/000///11125551
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. volume abs/2207.01206, 2022a. URL
http://wwwv.oo.aaaaoaoa als ht.tps:/arxxiv.orgg
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03622 2022b. URL ltttps://arxii. org/ abs/2210. 03629
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv preprint, 1ass22305510601,, 2023. URL httttt::///axxi
org/ abs 200.///3355000011
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arxiv 2017. ArXiv preprint, 0bs/1709.00103, 2017. URL
org/ abs 00103
Shuyan Zhou, Pengcheng Yin, and Graham Neubig. Hierarchical control of situated agents through natural language. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pp. 67-84, Seattle, USA, 2022a. Association for Computational Linguistics. URL http:t/s///alarrrtoooooo.rr/////
doi: 10.18653/v1/2222.suu
1..8
Shuyan Zhou, Li Zhang, Yue Yang. Qing Lyu, Pengcheng Yin, Chris Callison- Buroch, and Graham Neubig. Show me more details: Discovering hierarchies of procedures from semi-structured web data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2998-3012, Dublin, Ireland, 2022b. Association for Computational Linguistics. doi:
URL https: // ac anthology........ ogy.
2022. acl-long..111
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 WEBSITE IMPLEMENTATION
Given the selected websites described in 2.2 we make the best attempt to reproduce the functionality of commonly used sites in reproducible way. To achieve this, we utilized open-source frameworks for the development of the websites across various categories and imported data from their worldworld counterparts. For the E-commerce category, we constructed shopping website with approximately 90k products, including the prices, options, detailed product descriptions, images, and reviews, spanning over 300 product categories. This website is developed using Adobe Magento, an open- source e-commerce platform fd as that included in the Webshop data dump Yao et al. (2022a). As for the social forum platform, we deployed an open-source software Postmil the open-sourced counterpart of Reddi We sampled from the top 50 subreddit=" We then manually selected many subreddit for northeast US cities as well as subreddit for machine learning and deep learning-related topics. This manual selection encourages cross- cross-website tasks such as seeking information related to the northeast US on both Reddit and the map. In total, we have 95 subreddits, 127390 posts, and 661781 users. For the collaborative software development platform, we choose GitLat We heuristically simulate the code repository characteristics by sampling least ten repositories for every programming language: 80% of them are sampled from the set of top 90 percentile wrt stars repos using a discrete probability distribution weighted proportional to their number of stars; the remaining are sampled from the bottom ten percentile set using similar weighted distribution. This is done to ensure fair representation of repos of all kinds, from popular projects with many issues and pull requests to small personal projects. In total, we have 300 repositories and more than 1000 accounts with at least one commit to a repository. For the content management system, we adapted Adobe Magento's admin portal, deploying the sample data provided in the official guide. We employ OpenStreetMar" 0 for map service implementation, confining our focus to the northeast US region due to data storage constraints. We implement calculator and scratchpad ourselves.
Data resources were obtained from data from actual online sites, such
Lastly, we configure the knowledge resources as individual websites, complemented with search functionality for efficient information retrieval. Specifically, we utilize Kiwix to host offline version of English Wikipedia with a knowledge cutoff of May 2023. The user manuals for GitLab and Adobe Commerce Merchant documentation scraped from the official websites.
A.2 ENVIRONMENT DELIVERY AND RESET
One goal for our evaluation environment is ease of use and reproducibility. As result, deploy websites in separate Docker images [] one per website. The Docker images are fully self-contained with all the code of the website, database, as well as any other software dependencies. They also do not rely on external volume mounts to function, as the data of the websites are also part of the docker image. This way, the image easy distribution containing all the pre-populated websites for reproducible evaluation. End users can download our packaged Docker images and run them on their systems and re-deploy the exact websites together with the data used in our benchmarks for their local benchmarking.
Since some evaluation cases may require the agent to modify the data contained the website, e,g, creating new user, deleting post, etc., it is crucial to be able to easily reset the website environment to its initial state. With Docker images, the users could stop and delete the currently running containers for that website and start the container from reset the environment to the initial state. Depending on the website, this process may take from few seconds to minute. However, not all evaluation cases would require an environment reset, as
original image again to fully
https://postmil] XY2/
htttss//www.redddt.co
htm.
httpsss//ww.ddocee..
15

---

Published as a conference paper at ICLR 2024
Map
22.4%
E-commerce 23.0%
5.9%
Cress Site
13.1%
22.2%
Reddit
Gitlab
Figure 6: The intent distribution across different websites. Cross- site intents necessitate interacting with multiple websites. Notably, regardless of the website, all user intents require interactions with multiple web pages.
many of the intents are information gathering and are read-only for the website data. Also, combined with the inference time cost for the agent LLMs, we argue that this environment reset method, through restarting Docker containers from the original images, will have non-negligible but small impact evaluation time.
A.3 USER ROLES SIMULATION
Users of the same website often have disparate experiences due to their distinct roles, permissions, and interaction histories. For instance, within an E-commerce CMS, shop owner might possess full read and write permissions across all content, whereas an employee might only be granted write permissions for products but for customer data. We aim to emulate this scenario by generating unique user profiles on each platform.
On the shopping site, we created a customer profile that has over 35 orders within span of two years. On GitLab, we selected user who maintains several popular open-source projects with numerous merge requests and issues. This user also manages handful of personal projects privately. Reddit, our chosen profile was user who actively participates in discussions, with many posts and comments. Lastly, on our E-commerce CMS, we set up a user profile for shop owner who has full read-and-wrid write access all system contents.
All users are automatically logged into their accounts using pre-cached cookie. To our best knowledge, this is the first publicly available agent evaluation environment implement such characteristic. Existing literature typically operates under the assumption of universally identical user
roles Shi al. (2017) Liu al. (2018) Deng al. (2023).
A.4 INTENT DISTRIBUTION
The distribution of intents across the websites are shown in Figure
A.5 HUMAN PERFORMANCE
We acknowledge that there may difference human performance when annotators with different demographics are involved. In fact, many tasks in our dataset require domain-specific knowledge. For instance, an average user may not know what a git merge request is; or how to create a product in complex content management system. We aim to design tasks that have -o-iigiie outcomes (e.g., e a new product page is created) rather than those that are easily performed by an average user without significant domain knowledge.
16

---

Published as a conference paper at ICLR 2024
CoT UA Hint Model SR
x
GPT-3.5
6.28
Table 5: The task success rate (SR %) of GPT-3.5-TURBO-I6K-00 with temperature 0.0.
Dataset
gpt-4-0613
gpt-4-1106 -oee-iierevvwv
Date (900 examples)
100
100
Time duration (900 examples)
100
100
Table 6: The accuracy (%) of two versions of GPT-4 on judging dates and time duration of different formats are equivalent.
A.6 EXPERIMENT CONFIGURATIONS
We experiment with OPT-3.5-TURBBE 1e8---66--000 -0113, GPT-4-0613, and TEXT-BISON-00 with temperature of 1.0 and top-p parameter of 0.9. The maximum number of state transitions is set to 30. We halt execution the same action repeated more than three times on the same observation or if the agent generates three consecutive invalid actions. These situations typically indicate high likelihood of execution failure and hence warrant early termination. For TEXT- BISON-001 we additionally allow retries until it generates valid action.
Primarily, we use high temperature of 1.0 to encourage the exploration. To aid replicating the results, provide the results of T-33B-TTTR---666 the execution trajectories in our code repository.
with temperature 0.0 in Table 5 and
A.7 PROMPT FOR FUZZY_MATCH
Help a teacher to grade the answer of a student given a question. Keep in mind that the student may different phrasing wording to answer the question. The goal is to evaluate whether the answer semantically equivalent to the reference answer. question: ([intent))
reference answer: [[reference answer))
all the string 'N/A' that you see special sequence that means "not achievable'
student answer: { Conclude the judgement by correc/innorrecc/prtially correct.
Predictions that are judged as "correct" will receive a score of one, while all other predictions will receive score of zero.
A.8 THE ACCURACY OF FUZZY MATCH FUNCTION
To evaluate this, we manually checked examples and found that of them are identical to our human judgment. In addition, among the 82 examples that require using GPT-4 for evaluation, the answer 49 (60%) examples is date (e.g.. 10/23/2022) or time duration (e.g.. 15 minutes). these cases, GPT-4 is only used to judge the different format of the answers. We quantitatively evaluate the correctness GPT-4 in this case by generating different formats of a date and time duration programmatically. We randomly sample negative examples. For instance, Nov 2022, November 3, 2022, 3rd November 2022, Nov 2022, 2022-1 1-03, and 3rd of November, 2022 are all correct variances of 2022/1 1/03. The accuracy GPT-4 shown Table 6 We can see that versions of GPT-4 are extremely accurate, both achieving 100% accuracy.
A.9 THE PROMPTS
BASELINE WEB AGENTS
The system message of the reasoning agent for both GPT-3.5 and GPT-4 is in Figure and two examples are in Figure The system message of the direct agent for GPT-3.5 is in Figure 9 and the two examples are in Figure 10. UA hint refers the instruction of If you believe the task is
17

---

Published as a conference paper at ICLR 2024
You autonomous intelligent agent tasked with navigating web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions you can issue.
Here's the information you'll have:
The user's objective: This is the task you're trying to complete.
The current web page's accessibility tree: This is simplified representation of the webpage, providing key information.
The current web page's URL:
This is the page you 're currently navigating.
The open tabs: These are the tabs you have open. The previous action:
This is the action you just performed. It may be helpful to track your progress.
The actions you can perform fall into several categories:
Page Operation Actions
element with specific id on the webpage. type [id] [content] [press_ eeaeeeeeeeteeeeeeeeeeel afterr001]] : Use this to type the content into the field with id. By default, the "Enter" key pressed after typing unless press_enteraatte hover [id]": Hover over an element with id.
click [id]': : This action clicks
to O.
press [key_comb]] Simulates the pressing of a key combination on the keyboard (e.g., Ctrl+v).
scroll [direction=dowalupp Scroll the page up or down.
Tab Management Actions:
new_tab Open new, empty browser tab. tab focus [tab bobb_index]": Switch the browser's focus to specific tab using its index.
close_tab tab Close the currently active tab.
URL Navigation Actions:
goto [url]" Navigate a specific URL.
go go_back Navigate to the previously viewed page.
go forward Navigate to the page (if previous
go_ bockback action was performed).
Completion Action:
stop [answer]": Issue this action when you believe the task is complete. If the objective is find text-based answer, provide the answer in the bracket. If you believe the task is impossible to complete, provide answer as "N/A" in the bracket.
Homepage:
If you want visit other websites, check out the homepage at http://hommepgee.cm. It has list of websites you can visit. http://hmmepage. com/password..hng lists all the account names and passwords for the websites. You can use them to log in to the websites.
To be successful, it is very important to follow the following rules:
You should only issue an action that is valid given the current observation
2. You should only issue one action at a time.
You should follow the examples to reason step by step and then issue the next action.
4. Generate the action in the correct format. Start with "In summary, the next action I will perform
is" phrase, followed by action inside
For example, "In summary, the next action I will
perform is click [1234]*
5. Issue stop action when you think you have achieved the objective. Don't generate anything after stop."
Figure 7: The system message of the reasoning agent. This message has a general explanation of the task, the available actions, and some notes avoiding common failures.
impossible to complete, provide the answer as "N/A" in the bracket.". We remove this sentence in our ablation studies.
18

---

Published as a conference paper at ICLR 2024
example
OBSERVATION:
[1744] link "HP CB782A#ABA 640 Inkjet Fax Machine (Renewed)"
[1749] StaticTeext '$279.49'
[1757] button AAdd Cart'
[1760] button "Add to Wish List"
[1761] button 'Add Compare'
URL: http://onestommrrket
wwtpopppsooococ-eeeeeeeeee html
OBJECTIVE: What is the price HP Inkjet Fax Machine PREVIOUS ACTION: None
example assistant tsasttc_assistant
Let's think step-by-step. This page lists the information of HP Inkjet Fax Machine, which is the product identified in the objective. Its price is $279.49. think have achieved the objective. I will issue the stop action with the answer. In summary, the next action will perform is stop [$279.49]
example user
OBSERVATION:
[164] textbox 'Search' focused: True required: False
[171] button 'Go"
[174] link 'Find directions between two points'
[212] heading 'Search Results'
[216] button 'Close"
URL: http://openstrretmap..o
OBJECTIVE: Show me the restaurants near ABC
PREVIOUS ACTION: None
example_assistan assistant
Let's think step-by-step. This page has search box whose ID is [164]. According to the nominatim rule of openstreetmap, I can search for the restaurants near a location by restaurants near. can submit my typing by pressing the Enter afterwards. In summary, the next action will perform is type [164] [restaurants near ABC] [1]
Figure 8: The two examples provided example_use _use and example_assisteee for the reasoning agent. Before issuing the action, the agent first perform reasoning.
19

---

Published as a conference paper at ICLR 2024
You are an autonomous intelligent agent tasked with navigating a web browser. You will be given web-based tasks. These tasks will be accomplished through the use of specific actions you can issue.
Here's the information you'l II have:
The user's objective: This is the task you're re trying to complete.
The current web page's accessibility tree: This is simplified representation of the webpage, providing key information.
The current web page's ' URL: This is the page you currently navigating.
The open tabs: These are the tabs you have open. The previous action: This the action you just performed. may helpful to track your progress.
The actions you can perform fall into several categories:
Page Operation Actions
*click [id]": : This action clicks on an element with a specific id on the webpage.
'type [id] [content] [press enter after=0l1]" Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless element with id.
enter after set to O.
hhvvr [id] Hover
press [key_comb]] Simulates the pressing of a key combination on the keyboard (e.g.. Ctrl+v).
scroll [direction=downlup] Scroll the page up or down.
Tab Management Actions:
new new_tab" Open new, empty browser tab. tab focus [tab_ index]" Switch the browser's focus to a specific tab using its index.
close_tab's Close the currently active tab.
URL Navigation Actions:
goto [url] Navigate to a specific URL.
go backack Navigate to the previously viewed page.
go forforrrrd" Navigate to the next page (if previous
go back action performed).
Completion Action:
stop [answer]": : Issue this action when you believe the task is complete. If the objective is to find text-based answer, provide the answer in the bracket. you believe the task impossible to complete, provide the answer "N/A" in the bracket.
Homepage:
If you want to visit other websites, check out the homepage at http///homepage.com It has list of websites you can visit. htwp://aawag..oopaasaaaeeee html lists all the account name and password for the websites. You use them to log in to the websites.
To be successful, it is very important to follow the following rules:
To be successful, it is very important to follow the following rules:
You should only issue an action that is valid given the current observation
2. You should only issue one action at a time.
Generate the action in the correct format. Always put the action inside a pair of >* For example,
click [1234]*
Issue stop action when you think you have achieved the objective. Don't generate anything after stop."'
em em message of the direct agent. This message has the general explanation the task, the available actions and some notes avoiding common failures.
Figure 9: The syst
20

---

Published as a conference paper at ICLR 2024
example user
OBSERVATION:
[1744] link 'HPP CB782A#ABA 640 Inkjet Fax Machine (Renewed)*
[1749] StaticText $$7
[1757] button Add to Cart'
[1760] button "Add Wish List"
[1761] button 'Add to Compare'
URL: http://onestopmarkkt wwtpessoommarrtt.ooofffiiiicc propuucttoofccc-cccctoiiissssssi
OBJECTIVE: What is the price of HP Inkjet Fax Machine
PREVIOUS ACTION: None
example assistant stop [$279.49]
example user
OBSERVATION:
[164] textbox 'Search' focused: True required: False
[171] button 'Go'
[174] link 'Find directions between two points"
[212] heading 'Search Results'
[216] button 'Close'
URL: http://openstreetmaap.om
OBJECTIVE: Show me the restaurants near ABC
PREVIOUS ACTION: None
example assistant
for the type [164] [restaurants near ABCJ [1]
Figure 10: The two examples provided as example seample_user user and direct agent. The agent directly emits the next action given the observation.
21

---

Published as a conference paper at ICLR 2024
Manu
Log in
Sign up
Search
Search
Search query
What are searching for?
DMV area
Facebook
Q
Search
Projects
50 results for area:
area
991
[2430] searchbox 'Search query' [5172] StaticText 'DMV area'
Merge requests
99
Milestones
[2361] link 'Projects O'
Users
[2365] link 'Users 1'
[2070] heading We couldn't
Q We couldn't find any projects matching
find any projects matching
Facebook
Facebook"
Figure 11: Two examples where the GPT-4 agent failed, along with their screenshot and the accessibility tree of the relevant sections (grey), On the left, the agent fails to proceed to the "Users" section accomplish the task of "Fork all Facebook repos"; on the right, the agent repeats entering the same search query even though the observation indicates the input box is filled.
A.10 ADDITIONAL ERROR ANALYSIS
Observation Bias Realistic websites frequently present information on similar topics across various sections ensure optimal user accessibility. However, a GPT-4 agent often demonstrates tendency to latch onto the first related piece of information it encounters without sufficiently verifying its relevance or accuracy. For instance, the homepage of the E-Commerce CMS displays the best-selling items based on recent purchases. while historical best-seller data is typically accessed via separate report. Presented with the task of "What is the top-l best-sell selling product in 2022", , the GPT-4 agent defaults to leveraging the readily available information the homepage, bypassing the necessary step of generating the report to obtain the accurate data.
Failures in Observation Interpretation Interestingly, while GPT-4 capable of summarizing the observations. occasionally overlooks more granular information, such as the previously entered input. As the right-hand example of Figure search term "DMV area" has already been entered. However, the agent disregards this detail and continuously issues the command type [2430] [DMV area] until reaches the maximum step limit. Furthermore, the agent often neglects the previous action information that is provided alongside the observation.
[5172] StaticTes
indicates that the
We hypothesize that these observed failures are related to the current pretraining and supervised fine-tuning on dialogues employed in GPT models Ouyang et al. (2022). These models are primarily trained to execute instructions given immediate observations (i.e.. the dialogue history); thereby, they may exhibit a lack explorations. Furthermore, in dialogue scenarios, subtle differences in NI expressions often have less impact the overall conversation. As result, models may tend to overlook minor variations in their observations.
22


Reviews:
[{"review_id": "3QOqMYMGeo", "reviewer": "Reviewer_JxdP", "summary": "The paper emphasizes the potential of generative AI in creating autonomous agents that can handle daily tasks using natural language commands. Recognizing the limitations of current synthetic environments, the authors introduce \"WebArena,\" a realistic and reproducible web environment. This environment hosts websites from four key domains: e-commerce, social forums, collaborative software development, and content management, and is equipped with tools and knowledge bases to support human-like task performance. The authors also provide a benchmark of diverse tasks that mimic human internet activities and prioritize evaluating functional correctness over mere textual similarity. Testing with agents, including a GPT-4-based one, showed a significant performance gap, with the agent achieving only a 14.41% success rate compared to humans at 78.24%. This underscores the need for enhanced agent development and the value of WebArena as a testing ground for future advancements.", "questions": "See the weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "**Originality**: I am truly delighted to have the opportunity to review this work. I had the privilege of reading this manuscript a few months ago, and its significance resonated with me. The issues addressed in this paper are both critical and captivating. The work notably bridges a substantial gap, laying a pivotal foundation for future industrial applications of web agents. Over the last six months, I've come across numerous works on agent benchmarks. However, this particular study stands out, primarily due to its compelling motivation and remarkable originality. It has quickly become one of my favored works in this domain.\n\n**Quality**: After personally setting up the environment, running the provided code, and assessing the dataset, I can attest to the high caliber of this work. The construction of the benchmark is solid and robust, testifying to the meticulous efforts behind it.\n\n**Clarity**: The paper is lucidly crafted with a coherent structure and logical flow, making it accessible and comprehensible.\n\nIn conclusion, this is a high-quality, original, clear, and significantly impactful piece of scholarship.", "weaknesses": "While the work presented is undeniably valuable, from an academic perspective, I believe there are several weaknesses, primarily related to experimental evaluations and the choice of baselines. Here are the specific areas of concern:\n\n1. **Lack of Evaluation with the Latest Intelligent Agents**: \n\nThe paper seems to miss out on evaluating some of the latest intelligent agents, especially those grounded in modern reasoning and planning methods. Works like the \"Tree of Thought\" and the new \"Reflection\" architecture have been in the public domain for a while. It would have greatly enhanced the paper's comprehensiveness if these contemporary agents were included in the evaluations.\n\n2. **API Call Methodology and Ablation Experiments**: \n\nThe manner in which API calls are presented in the paper, particularly as web pages, does not seem to align with the current prevalent paradigms where APIs are usually invoked within context. It raises the question of whether an agent can effectively utilize this format. Additionally, it would have been illuminating if the authors had included ablation studies in their experiments. Specifically, it would be insightful to discern the efficacy of these tools and whether they genuinely aid the agent in realizing the desired goal of \"encouraging human-like task-solving\".\n\n3. **Html or Accessibility Tree?** \n   - Many language models (LLMs) are pre-trained with an abundance of HTML content, but they might not necessarily contain the Accessibility tree. Hence, it might be more natural for these LLMs to understand and parse HTML.\n   - Both DOM and the Accessibility tree adopt a tree-like structure. The seemingly \"redundant\" symbols in HTML could potentially assist LLMs in better understanding the hierarchical nature of the content.\n   - It is vital to conduct empirical tests to validate the advantages of either approach. Given that the Accessibility tree is not commonly adopted in other benchmarks, using it here could also be viewed as one of the paper's core contributions, setting it apart from the current landscape of research in this area.\n\n4. **Gold Trajectories**: \n\nThe paper would benefit significantly from the inclusion of \"Gold\" trajectories. These trajectories can offer a benchmark for the best possible action sequences, making them an invaluable asset for future research in this domain. The absence of these trajectories is a noticeable gap in the paper.\n\n5. **Evaluator Demographics**: The choice of computer science graduate students as evaluators raises certain concerns regarding the generalizability of the results.\n   - Computer science graduate students typically possess an advanced understanding of web page interactions, which might not be representative of the average user. Their performance might be notably better than what we'd observe with a more diverse group, especially when considering common tasks like online shopping that even non-technical users frequently engage in.\n   - Furthermore, it's essential to address potential biases that might arise if any of the evaluators were involved in the dataset's creation. This could compromise the validity of the scores. From a personal standpoint, I, along with several colleagues, have engaged in case studies with this dataset. Interestingly, our accuracy rates didn't match the high scores reported in the paper, which adds a touch of humor to this serious concern.\n\nTo ensure the paper's robustness and generalizability, it's crucial to address these points, preferably with empirical evidence and further discussions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZPwCKRjIIL", "reviewer": "Reviewer_uu7U", "summary": "The paper proposes a new, realistic RL envionrment for Web tasks named WebArena as well as a first evaluation of GPT-based agents performing the defined tasks. The framework includes environments for e-commerce, social forums, collaborative software development similar to Gitlab and content management and therefore provides additional tools, including maps or Wikis. The obervation space can be screenshots of web pages, HTML DOM trees or accessibility trees. The authors proposes a Partially Observable Markov Decision Process modelling of tasks, where the action space comprises keyboard and mouse. The authors present 812 tasks in their benchmark and evaluate GPT-based agents for the tasks, which yield sub-par performance against human baselines from a user study.", "questions": "Can new tasks be easily added to the benchmark within the available environments?\n\nWould it it have been possible to include the benchmark tasks into another, existing benchmark system from the related work?\n\nWhat could be future works wrt to (RL-) agents for solving the benchmark tasks? \n\nAre the presented tasks on par wrt difficulty or even superior to other benchmarks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors propose an Independent platform, implementing a large variety of realistic end-user tasks on the Web. The framework provides provides realistical, challenging tasks for Web agents. The quality of the benchmark is sufficiently high. To this end, a good choice of task variety was made, which is backed up by a user study. This is very nice to see, as the taken design decisions then are probabily matching with user needs.  \n\nThe paper includes a preliminary evaluation of agents based on closed-source LLMs (ChatGPT / Text-Bison), which gives first insights.\n\nThe code is made available for review, and is usable and documented, which makes the paper's contributions quite clear. The available tasks are sufficiently challenging for evaluating (LLM) agents, which makes the contributions significant for more research advances in the field.\n\nWhile there are other related benchmarks in the field, it is quite clear from the paper content what is being improved / what is original.", "weaknesses": "The related work advantage not completely clear. The related work states functional correctness as advantage over AndroidEnv, but no further explanation is given. It might hint to the diffeence between the used evaluation metrics, but it would be interesting/important to clarify this. Also, it mentions the lack of diverse or complex task availability, but new tasks can be defined within the framework.  \n\nThe agent evaluation is performaned with standard GPT variants only, not pointing to stronger alternatives. Also, little to no details about how the agent was implemented/tested are given in the main paper. Only the appendix shows examples, which impedes understanding the paper.\n\nAs the used LLMs for the evaluation are closed-source, this impedes reproduciability. As the evaluation can been seen as first validation of the benchmark, this might still be fine, but it would be good to have open-source agents integrated.\n\nLastly, the POMDP model is not argued for in the paper, but it would be important to justify the modelling choice. This is not to say that a POMDP model is not sensible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eyGMvQ6YDy", "reviewer": "Reviewer_Y9es", "summary": "This paper proposes a web environment designed for the development and testing of autonomous agents. The proposed environment WebArena includes fully functional web applications and genuine data from four major categories, providing a realistic platform for agent interaction. The authors also proposes a benchmark consisting of 812 examples, as well as an evaluation method. The experiments show that GPT-4 only achieves a task success rate of 14.41%, which is much lower than human performance of 78.24%.", "questions": "1. Is there any analysis or discussion on the performance of GPT4 evaluation?\n\n2. Why the success rate of human is only 78%?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a highly-realistic and complicated web environment compared with the previous simplified environment;\n\n2. The proposed environment includes four common and real domains;\n\n3. The paper is well written and easy to follow.", "weaknesses": "1. The major weakness of this paper is the lack of technical novelty. Though the contribution on simulated environment/datasets/resources are welcomed and very important to the research community, such papers may not match the general style of ICLR papers.\n\n2. For evaluation, the proposed framework uses GPT4 to evaluate the answer or the execution paths, which potentially has two issues: 1. GPT4 is a commercial tool, which may limit the potential use of this environment; 2. GPT4 is not guaranteed to be 100% right, which may make the evaluation results not convincing.\n\n3. The success rate of human on the designed tasks are only 78%, which is a little surprising since it seems that these tasks are not that difficult for human to complete. It is better that the authors provide more analysis on these tasks and evaluations to show that why human fails and if these tasks are too difficult for agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)