Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems

## Abstract

Abstract AI Agents are changing the way work gets done, both in consumer and enterprise domains. However, the design patterns and architectures to build highly capable agents or multi-agent systems are still developing, and the understanding of the implication of various design choices and algorithms is still evolving.
In this paper, we present our work on building a novel web agent, Agent-E 1 1 1 Our code is available at https://github.com/EmergenceAI/Agent-E . Agent-E introduces numerous architectural improvements over prior state-of-the-art web agents such
as hierarchical architecture, flexible DOM distillation and denoising method, and the concept of change observation to guide the agent towards more accurate performance.
We first present the results of an evaluation of Agent-E on WebVoyager benchmark dataset and show that Agent-E beats other SOTA text and multi-modal web agents on this benchmark in most categories by 10-30%. We then synthesize our learnings from the development of Agent-E into general design principles for developing agentic systems. These include the use of domain-specific primitive skills, the importance of distillation and de-noising of environmental observations, the advantages of a hierarchical architecture, and the role of agentic self-improvement to enhance agent efficiency and efficacy as the agent gathers experience.

## 1 Introduction

Recent advancements in large language models have led to significant interest in the development of autonomous agents that can execute complex tasks on the web [, , ] and on device [, ]. Automation of complex and repetitive tasks presents an invaluable opportunity to increase individual and organizational efficiency. In addition, robust automation systems will unlock new use cases and experiences by enabling collaboration between users and AI agents working together to accomplish sophisticated tasks.

Figure 1 shows a simplified anatomy of a web agent. A typical web-agent, at its most basic, has two key capabilities: the capability to sense the web page and the capability to act on the web page.

- •
Sensing: Sensing the state of the web-page, for a web agent, typically involves encoding the Document Object Model (DOM) of the page [, ], using the accessibility tree of the page [] and/or using screenshots of the page [].
- •
Acting: The action space can be comprised of simple actions such as navigating to URLs, clicking on elements, and entering text in a field, or composite actions comprising of several simple actions. An example of composite action is used by Lutz et al. [], which they call ‘Input’. ‘Input’ selects a text box, deletes any existing content, inputs text and presses submit button.

Based on the user’s task and the current state of the page, the agent can create a plan i.e., it determines the subsequent next action or series of actions required to complete the task.

Figure: Figure 1: Simplified anatomy of web agents
Refer to caption: extracted/5738750/General_Web_automation_diagram.png

Developing a robust web agent that can autonomously perform tasks on the browser presents multiple challenges. Firstly, HyperText Markup Language (HTML) Document Object Models (DOMs) can be noisy and expansive, so that they often exceed an LLM’s context window. This renders them unusable without intelligent simplification or de-noising steps. Further, even with simplification, information may exceed the context window limit of modern LLMs after a few interactions. This is an important challenge that restricts their use for complex tasks []. Secondly, websites are primarily designed for human visual consumption, wherein information is organized to prevent information overload for a user; thus, websites utilize established human-computer interaction patterns that may not be optimal for agent interaction. Complex widgets, such as date selectors, are easy for humans to operate but pose difficulties for agents []. Thirdly, while humans can naturally execute complex web tasks (e.g., finding the cheapest flight between two destinations), agents require detailed multi-step planning to execute such tasks. Further, the plan may need multiple revisions if the initial approach proves unfruitful.

Despite these challenges, recent work has demonstrated promising outcomes with web agents executing diverse tasks on the internet. Yet, state-of-the-art web agents leave much to be desired in terms of practical usefulness []. Most previous studies simply report success rates. Task success rates show that web agents are more error-prone compared to a human performing the same task and are not yet ready for effective mainstream use [, , ].

Task success rates, while important, may not tell the whole story when it comes to evaluating web-agents. For example Kapoor et al. [] note that since that the underlying LLMs are stochastic systems, simple retry mechanisms often improve success rates. Thus, simply measuring task success rates is misleading. There are other important factors that influence the utility and user experience of using agentic systems, such as:

- •
Task completion times: The duration required for the system to complete a task, which impacts overall efficiency and productivity. Task completion times are especially relevant for online tasks or tasks with a human in the loop. The task completion times can be influenced by extrinsic factors (network speed, computing environment influencing loading times of web pages) or intrinsic factors (token generation speed of the LLMs, the path taken by the agent to perform the task, errors made during execution etc). Despite the variability, task completion times is an important measure since it can be critical in many real world applications.
- •
Cost: The total expenditure involved in executing the task. The cost is a variable measure and depends on the underlying model and the per token input/output cost of different model providers. This is also a ‘point in time’ measure, since the pricing models set by LLM providers change frequently. While not fully encompassing, a good proxy for cost is number of LLM calls involved.
- •
Error-awareness: The capability of the system to recognize and report its own errors or failures, ensuring reliability and ability to provide a seamless user experience by highlighting and recommending fallback options. Ability of system to detect that it has not completed the task appropriately will also open up avenues to collect human demonstration (e.g. I could not complete this task, can you demonstrate how you would do it?) and a path towards continuous improvement.

In this paper, we introduce Agent-E, a state-of-the-art web agent capable of performing complex web-based tasks. Central to Agent-E are two LLM-powered components: the planner agent and the browser navigation agent. The planner agent is responsible for task planning and task management. It breaks down the user task into a sequence of sub tasks and delegates them one at a time to the browser navigation agent. The browser navigation agent is tasked with executing the individual sub tasks by sensing the page using different DOM distillation capabilities available to it, finding the next actions to execute and reporting its task success or failure back to the planner. This loop is executed iteratively until the task is completed. This tiered architecture ensures that the planner agent is insulated from the overwhelming and noisy details of the website and DOM, and the browser navigation agent is freed from the complexities of the overall task planning and orchestration.

Agent-E can run in two modes: autonomous mode and human-in-the-loop mode. In the autonomous mode, Agent-E performs the task end to end as best as it can, and in the human-in-the-loop mode, it falls back to the user when it encounters steps that it cannot accomplish (e.g. logging to a web page, solve a captcha triggered by the website) or to ask for clarifications when the task itself is ambiguous. We believe human-in-the-loop workflows are critical for adoption of agentic systems. However, since there are no web automation benchmarks for human-in-the-loop agents [], in this paper we will focus on Agent-E as an autonomous web agent.

We evaluate Agent-E in autonomous mode on the WebVoyager benchmark [], where we achieve a new state-of-the-art result of 73.2%, 20% higher than previous state-of-the-art for text-only [] and 16% higher than previous state-of-the-art multi-modal web agent []. For specific sites like Wolfram Alpha, it achieves upto 30% improvement.

### 1.1 Contributions

- •
We introduce a novel hierarchical architecture for web agents that enables the execution of more complex tasks through a clear separation of roles and responsibilities between a planner agent and a browser navigation agent. We illustrate the utility of the hierarchical architecture for capabilities such as task verification, backtracking, and error recovery, with examples where these capabilities are invoked.
- •
We propose a flexible DOM distillation approach whereby, given a task, the browser navigation agent can choose the most suitable DOM representation from three different implementations. We demonstrate its effectiveness with examples.
- •
We highlight the benefits of ‘Change observation’, a paradigm conceptually similar to Reflexion []. After the execution of each action, the outcome (change in state) is monitored and used to provide verbal feedback to the browser navigation agent. This guides the agent towards better awareness of the current state and more accurate performance.
- •
We report detailed end-to-end evaluations of Agent-E on the WebVoyager benchmark and show that it achieves a new state-of-the-art results with a 73.2% success rate. This marks a 20% improvement over the previous text-only web agent and a 16% increase over the former state-of-the-art multi-modal web agent.
- •
Beyond task success rates, we are the first to report additional metrics such as error awareness, task completion times, and the number of LLM calls, offering a baseline for comprehensive evaluation of future web agents.
- •
Agent-E is a result of numerous experimentation and exploration into the space of AI Agents. We synthesize our learning from the development of Agent-E into design principles for development of agentic systems that we believe generalize beyond Agent-E and web automation, and provide useful insights to practitioners.

## 2 Agent-E: System Description

Figure 2 shows the high level architecture of Agent-E. It comprises of two LLM-powered agents: Planner Agent and Browser Navigation Agent, and two executor components: Planner Skills executor and Browser Navigation Skills executor. Each LLM-powered agent has skills associated with it, which are python functions that are described to the LLM for function calling. We do not make any distinction between skills associated with sensing (e.g. *Get DOM*) and skills associated with acting on the page (e.g. *click on element*). The executor components execute the function suggested by the LLM and relays the response back to the LLM.

Agent-E is built using Autogen, the open-source programming framework for building multi-agent collaborative systems [] and Playwright(^2^22[https://playwright.dev/](https://playwright.dev/)) for browser control. Its architecture leverages the interplay between skills and agents as shown in Figure 2.

Figure 3 shows a conceptual flow diagram of Agent-E. Given a new user task, the planner decomposes the task into a sequence of steps that need to be executed. The next step is then delegated to the browser navigation agent for execution. Browser navigation agent is implemented as a nested chat that is freshly instantiated by the Planner for each run (i.e., it does not contain previous steps in its chat history). Browser navigation agent has a set of predefined ‘primitive’ or foundational skills for observing denoised browser state and controlling the browser instance using Playwright. Figure 4 shows the skills available to the browser navigation agent. The browser navigation agent uses the skills available to it to perform the sub task and return a summary of actions it took to perform the task and/or answer the planner if the task was a question. Figure 5 shows an example communication between planner and the browser navigation agent and Figure 6 shows the browser navigation agent using the primitive skills available to it to perform a related sub task.

Figure: Figure 2: A high level architecture of Agent-E
Refer to caption: extracted/5738750/Agent-E.png

Figure: Figure 3: Conceptual flow diagram of Agent-E. Individual blocks represent functions; In Agent-E, a single LLM call is used to perform multiple functions.
Refer to caption: x1.png

Figure: Figure 4: Skills registered to the Browser Navigation Agent for sensing and acting on the web page.
Refer to caption: extracted/5738750/Skills.png

Figure: Figure 5: An example of Agent-E execution highlighting communication between the planner and browser navigation agent for the user task Find the price of Canva Teams subscription and minimum number of users required for it
Refer to caption: x2.png

### 2.1 Skills Design for Browser Navigation Agent

There are two key differences between Agent-E and previous web agent implementations such as Wilbur [] and WebVoyager [] in terms of skills .

- •
Sensing Skills: Agent-E supports multiple DOM synthesis techniques that allows the browser navigation agent to choose the approach best suited for the task (see Figure 4). If the task is to summarise information on a page, it can simply use Get DOM with text_only content type. If the task is to identify and execute a search on a page, it can use the content type input_fields. If the task is to list all the interactive elements on a page, it can use all_fields. This optimizes the information available to the agent and prevents the problems associated with noisy DOM. Another key difference is our DOM de-noising techniques for *all_fields* and *input_fields* attempts to preserve the parent child relationship of elements wherever possible and relevant. This is unlike some of the other previous implementations which uses a flat DOM encoding (e.g. []). Further, to make identifying and interacting with HTML elements easier, Agent-E injects a custom identifier attribute (mmid) to each element as part of sensing, similar to [] and [].
- •
Action Skills: All the action skills are designed to not only execute an action but also report on any change in state as an outcome of the action, a concept we call ‘Change observation’. This is conceptually similar to Reflexion paradigm [] which uses verbal reinforcement to help agents learn from prior failings. However, a key difference is that change observation is not directly associated or limited to a prior failure. The observation returned can be any type of outcome of the action. For example, a click action may return a response Clicked the element with mmid 25. As a consequence, a popup has appeared with following elements. Such verbal responses nudges the agent towards the correct next step.
Change Observation capability is implemented by observing changes in selected attributes of elements (e.g aria-expanded) and Mutation Observer Web API (^3^33[https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver](https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver)) that allow observing changes in DOM immediately following an action execution. This is especially useful for extremely dynamic pages such as a flight booking website and nudges the LLM to take appropriate next step given the task. (see Figure 6 for an example)

Figure: Figure 6: An example of Agent-E nested chat execution loop for the sub task “Navigate to the plans and pricing section” which is part of the larger task introduced earlier “Find the price of Canva Teams subscription and minimum number of users required for it”
Refer to caption: x3.png

## 3 Evaluation

### 3.1 Introduction to WebVoyager

WebVoyager [] is a recent web agent benchmark that consists of web navigation tasks across 15 real websites as shown Figure 7. Each website has about 40-46 tasks resulting in a benchmark dataset of 643 tasks (see Table 1 for example tasks). These tasks could be completed through DOM manipulation (textual) as well as augmenting with image understanding (multi-modal). We chose WebVoyager since it covers a diverse range of tasks across different live, dynamic and representative websites. Other alternative benchmarks either focused solely on single task domain [], used custom created websites that are significantly simpler than real-world versions in terms of DOM complexity [], or used cached versions of real-websites only supporting a fixed route for the agent and not allowing free form exploration [].
One of the issues with WebVoyager though is that it uses static dates for some of the tasks (e.g. Search a hotel with free WiFi and air conditioning in Bali from Jan 1 to Jan 4, 2024.). The static dates makes some of these tasks unachievable since the dates have passed. We manually went through the tasks to identify ones with fixed dates in the tasks that make the task unachievable, and added 8 months to the dates (i.e., Search a hotel with free WiFi and air conditioning in Bali from Aug 1 to Aug 4, 2024.).

Figure: Figure 7: WebVoyager Websites
Refer to caption: extracted/5738750/WebVoyager.png

**Table 1: Example tasks from WebVoyager benchmark.**
| Website | Task |
| --- | --- |
| Allrecipes | Find a recipe for Baked Salmon that takes less than 30 minutes to prepare and has at least 4-star rating based on user reviews. |
| Booking | Find the cheapest available hotel room for a three-night stay from 1st Jan in Jakarta. The room is for 2 adults, just answer the cheapest hotel room and the price. |
| Github | Search for an open-source project related to ‘climate change data visualization’ on Github and the report the project with the most stars. |
| Google Flights | Search a one-way flight from Dublin to Athens Greece for 1 Adult that leaves on December 30 and analyze the price graph for the next 2 months. |

### 3.2 Evaluation Setup

The entire benchmark was divided among 5 human evaluators who ran 125-130 tasks each. For each task, the evaluators were instructed to classify the task as pass or fail along with a textual reason in case of failures. They were instructed to mark the task as pass if it was completed successfully in full (in case the task has multiple parts). The evaluators ran the task from India during IST office hours (this may have implications when you interpret or compare measures such as task completion times).
For the evaluation, Agent-E was used in autonomous mode (i.e., *ask user for input* skill was disabled) and used GPT-4-Turbo as the LLM for both planner and browser navigation agent.

### 3.3 Measures

We report four important measures that are relevant for comprehensive evaluation of web agent and understanding their practical implementation readiness.

- •
Task success rates: The percentage of tasks that Agent-E performed successfully across websites.
- •
Self-aware vs Oblivious failure rates: Detecting when the task was not completed successfully is of utmost importance, since it can be used for enabling a fallback workflow, to notify the user of failure or use as an avenue to gather human demonstration for the same task.
Self-aware failures are failures where agent is aware of its own failure in completing the task and responds with a final message explicitly stating so, e.g. I’m unable to provide a description of the first picture due to limitations in accessing or analyzing visual content. or ‘Due to repeated rate limit errors on GitHub while attempting to refine the search…’. The failures could be due to technical reasons or agent deeming the task unachievable since it could not complete the task after repeated attempts. On the other hand, oblivious failures are cases were the agent wrongly answers the question or performs the wrong action (e.g. adds the wrong product to cart or provides a wrong information). For mainstream utility, oblivious failures should be as minimal as possible.
For the current evaluation, failures were categorized to self-aware and oblivious failures by manual annotation. However, it would be trivial to employ an LLM critique to automatically do the same task, similar to [].
- •
Task completion times: The average time required to complete the task, across websites for failed and successful tasks.
- •
Total number of LLM calls: The average number of LLM calls (both planner and browser navigation agent) that was required to perform the task. This includes both successful and failure cases.

### 3.4 Result: Quantitative

**Table 2: Evaluation of Agent-E on WebVoyager.**
| Publication | Task success rates on websites |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Allrecipe | Amazon | Apple | Arxiv | Github | Booking | ESPN | Coursera |
| He et al. 2024 (text) | 57.8 | 43.1 | 36.4 | 50.4 | 63.4 | 2.3 | 28.6 | 24.6 |
| He et al. 2024 (multi) | 51.1 | 52.9 | 62.8 | 52.0 | 59.3 | 32.6 | 47.0 | 57.9 |
| Lutz et al. 2024 (text) | 60 | 43.9 | 60.5 | 51.2 | 22.0 | 38.6 | 59.1 | 51.1 |
| Agent-E (text) | 71.1 | 70.7 | 74.4 | 62.8 | 82.9 | 27.3 | 77.3 | 85.7 |
| Failure modes | Agent-E Error Analysis on Websites |  |  |  |  |  |  |  |
| Overall failures % | 28.9 | 29.3 | 25.6 | 37.2 | 17.1 | 72.7 | 22.7 | 14.3 |
| Self-aware failures % | 17.8 | 14.6 | 9.3 | 18.6 | 12.2 | 4.5 | 13.6 | 4.8 |
| Oblivious failures % | 11.1 | 14.6 | 16.3 | 18.6 | 4.9 | 68.2 | 9.1 | 9.5 |
| TCT | Agent-E Avg. Task Completion Times (seconds) |  |  |  |  |  |  |  |
| TCT (Success) | 116 | 286 | 122 | 137 | 104 | 183 | 187 | 119 |
| TCT (Failed) | 196 | 246 | 200 | 176 | 384 | 317 | 387 | 177 |
| LLM Calls | Agent-E Avg. Number of LLM calls |  |  |  |  |  |  |  |
| Total | 22 | 23.1 | 21.5 | 25.5 | 21.5 | 36.4 | 24.0 | 25.5 |
| Planner | 6.5 | 6.4 | 5.9 | 6.9 | 5.4 | 6.6 | 6.3 | 6.3 |
| Browser Nav Agent | 15.5 | 16.7 | 15.6 | 18.6 | 16.1 | 29.8 | 17.7 | 19.2 |

**Table 3: Evaluation of Agent-E on WebVoyager (Contd.)**
| Publication | Task success rates on websites |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Dictionary | BBC | Flights | Maps | Search | Hug.Face | Wolfram | Overall |
| He et al. 2024 (text) | 66.7 | 45.2 | 7.1 | 62.6 | 75.2 | 31.0 | 60.2 | 44.3 |
| He et al. 2024 (multi) | 71.3 | 60.3 | 51.6 | 64.3 | 77.5 | 55.8 | 60.9 | 57.1 |
| Lutz et al. 2024 (text) | 86.0 | 81.0 | 0.0 | 39.0 | 67.4 | 53.5 | 65.2 | 52.6 |
| Agent-E (text) | 81.4 | 73.8 | 35.7 | 87.8 | 90.7 | 81.0 | 95.7 | 73.1 |
| Failure modes | Agent-E Error Analysis on Websites |  |  |  |  |  |  |  |
| Overall failures % | 18.6 | 26.2 | 64.3 | 12.2 | 9.3 | 19.0 | 4.3 | 26.9 |
| Self-aware failures % | 16.2 | 9.6 | 57.1 | 12.0 | 4.6 | 14.3 | 2.1 | 14.1 |
| Oblivious failures % | 2.4 | 16.6 | 7.1 | 0 | 4.6 | 4.7 | 2.1 | 12.8 |
| TCT | Agent-E Avg. Task Completion Times (seconds) |  |  |  |  |  |  |  |
| TCT (Success) | 98 | 105 | 244 | 127 | 106 | 140 | 68 | 150 |
| TCT (Failed) | 136 | 110 | 234 | 177 | 135 | 167 | 94 | 220 |
| LLM Calls | Agent-E Avg. Number of LLM calls per Task |  |  |  |  |  |  |  |
| Total | 22.0 | 21.3 | 53.8 | 22.9 | 19.4 | 22.8 | 14.5 | 25.0 |
| Planner | 6.6 | 6.0 | 11.4 | 5.8 | 5.6 | 6.2 | 4.4 | 6.4 |
| Browser Nav Agent | 15.4 | 15.3 | 42.2 | 17.0 | 13.7 | 16.6 | 10.15 | 18.6 |

In this section, we will present quantitative results of how well Agent-E performs in WebVoyager benchmark. Tables 2 and 3 shows the summary of evaluation of Agent-E on WebVoyager.

#### Comparison with prior state-of-the-art agents

Agent-E overall successfully completed 73% of the tasks and outperforms prior state-of-the-art text-only web agent WILBUR [] by 21% and state-of-the-art multi-modal web agent [] by 16%. The effectiveness of Agent-E varied across websites from 27.3% (Booking) to 95.7% (WolframAlpha). Comparing by websites, Agent-E outperformed both Wilbur [] and WebVoyager multi-modal agent [] in 11 out of 15 websites. Booking.com and Google Flights continue to be websites, where all web agents, including Agent-E does not do well.

It is important to note that [] uses task and website-specific prompting and [] uses vision for observing the page. In contrast, Agent-E is a generic text-only web agent which does not employ any task or website-specific instructions. This suggests that there is likely room for further improvement in Agent-E using some of these strategies.

#### Agent-E failure modes

Overall, Agent-E was self-aware of the failures for more than 52% of the failed tasks. Typically, Self-aware failures occur when the reason for failure are technical in nature (e.g., navigation issues, inability to extract certain information from DOM elements such as Iframes, canvas or images, inability to operate a button, anti-scraping policies employed by websites, inability to find the answer despite multiple attempts etc.). The Oblivious failures are scenarios where Agent-E gives a response that was wrong. These are typically scenarios where agent overlooks certain task requirements and provides an answer that only partially meets the requirements. It could also stem from DOM observation issues (e.g., not being aware that the date got reset due to incorrect format in Google Flights) or website capability understanding issues (e.g., not using advanced search capability when needed, or assuming search functionalities are perfect and every search result will completely satisfy the search requirements). Similar error modes were also observed by [] who classify them as agent hallucinations.

#### Task Completion Times

Agent-E on an average took 150 seconds to successfully complete a task and 220 seconds for completion when the task was a failure. The longer duration for failed tasks is interesting and also expected, since given a difficult task, Agent-E may try multiple approaches to complete the task before giving up on it. The difference in task completion times across websites (e.g., 68 seconds to successfully complete a task in WolframAlpha vs. 286 seconds in Amazon) also reflects the differences in task and website complexity. Amazon.com is a feature-rich and content-heavy website (for example, the Amazon and Booking.com homepage contains 3189 and 2,952 DOM elements respectively, in contrast to 1520 in Google Flights and 590 for WolframAlpha homepages). When the website is content-heavy and has numerous DOM elements, the DOM de-noising step takes more time, and the LLM token count for browser navigation agent may increase as well.

#### LLM Calls

On an average Agent-E took 25 LLM calls to execute a task (6.4 calls by the planner and almost 3 times as much by the browser navigation agent). The average number of LLM calls per website, as expected, is consistent with task completion times. Unfortunately, no prior studies have reported an end-to-end number of LLM calls required for task completion. While [] provides the number of LLM calls required by one (out of three) of the LLM-based components in their system, this information is insufficient for a conclusive comparison.

### 3.5 Result: Qualitative

In this section, we will present qualitative results of concrete examples showing how different design choices made in Agent-E help perform complex web tasks.

#### Hierarchical Planner helps error detection and recovery

The hierarchical architecture allows easily detecting and recovering from errors. The planner agent is prompted to perform verification (by asking questions or asking confirmation) as part of the plan whenever necessary. Figure 8 shows an example instance where the planner agent asks the browser navigation agent for more information (i.e., list the search results), and from the response (i.e., there are no specific search results) identifies that it may have made an error by making the search query too focused. In the example, the planner creates a new plan of action for performing the task.
Another common pattern in the evaluation was the planner’s ability to detect errors and easily backtrack to a previous page to continue execution. Given that the planner has the URL of the page at each step available to it, it allows the planner to effortlessly backtrack to a previous page by adding it as a step in the plan (e.g., navigate to the search result page using the <url>).

Figure: Figure 8: An example instance of Agent-E detecting and recovering from errors. The conversation is truncated with ‘…’ to enhance readability in the image.
Refer to caption: x4.png

#### Support for multiple DOM observation methods

HTML DOM can be extremely large (e.g., YouTube Homepage with all the DOM elements and their attributes is about 800,000 tokens). Thus, it is important to denoise and encode the DOM such that only task relevant information is presented to the LLM. However, information relevant for a given task is very dependent on the task at the hand. Some tasks may only need a complete textual representation (e.g., summarise the current page), some tasks may only need input fields and buttons (e.g., search on google). On the other hand, more exploratory tasks, may need a complete representation of the page (e.g., what elements are on this page).
Most previous web agent work have used a single DOM representation, e.g. [] used accessibility tree, multi-modal web agent in [] used screenshots and [] used direct encoding and denoising of the HTML DOM. However, in our view, there is no single DOM observation method that suits all the tasks. Thus, Agent-E supports three different DOM representation methods text_only, input_fields, all_fields. This allows Agent-E to flexibly select the DOM representation that it feels is best suited for the task. Also, it allows for backup, that if one does not work as expected, Agent-E can fall back to a different representation.
There were numerous examples in our benchmark where this multiple DOM representation was useful. Figure 9 illustrates an example where Agent-E adaptively uses all_fields DOM representation for interaction and text_only for summarization.

Figure: Figure 9: Providing multiple options for DOM observation allows to flexibly select one fit for task. The conversation is truncated with ‘…’ to enhance readability in the image.
Refer to caption: x5.png

#### Change observation helps grounding

Change observation is a concept we introduced in which each action execution also observes for changes in the state and returns a linguistic feedback to the LLM. Some typical scenarios where this is useful is where the browser navigation agent tries to click on a navigation item (e.g., click on the soccer link on ESPN.com), however instead of navigating to the relevant section, it may instead open a popup menu that requires further selection. Another common example is when browser navigation agent attempts to set the source airport in a flight booking websites and a list of possible airports open up as a drop down. In both these cases, the interaction is not yet complete, but the browser navigation agent may assume it is (i.e., navigation to soccer section would require clicking on a new link from the popup menu, setting source airport would require selecting one of the option from the drop down). In both these cases, the click skills will return a feedback to the LLM that as a consequence of the click, a menu has appeared where you may need to make further selection. See Figure 6 for an example.
Conceptually, the purpose of the change observation is to provide linguistic feedback to the LLM whether the action has been completed without any errors and did it lead to any tangible change in the environment, in order to inform subsequent actions. We also envision efficiency improvements if the change observation can return a list of elements, so that LLM can make subsequent selection without again using Get DOM skill to observe the state of the DOM.
Change observation is adjacent to the concept of Reflexion []. However, there are nuanced differences between the two. Reflexion paradigm provides feedback of a prior failure. However, change observation is not directly associated or limited to a prior failure. The observation returned can be any outcome of the action.
Reflexion uses an LLM to analyse the scalar ‘success/failure’ signal from an action and current trajectory to produce the verbal feedback to the agent. Reflexion did not provide any tangible improvement when directly applied to web automation in the Webshop[] benchmark. On the other hand, change observation is implemented using classical code-based approaches to observe the consequence of an action and linguistic feedback of actions is implemented using heuristic-based approach to help the system to be better aware of the current state of the environment, and nudge the system towards the correct next action.

## 4 Discussion

In this section, we will synthesize our learnings from the development and evaluation of Agent-E into a series of agent design principles. We believe these principles can be generalized beyond the domain of web automation.

### 4.1 Agent Design Principles

- 1.
Well crafted set of primitive skills can enable powerful use-cases:
A well crafted ensemble of foundational skills can serve as a building block to support more complex functionalities. LLMs can effectively combine these skills to unlock a broad range of usecases. The analysis of the intended use case is crucial to arrive at the requisite primitive skills. In the case of Agent-E these primitive skills were click, enter text, get DOM, Open URL and Press Keys. These were only a subset of what a user could potentially perform on a page (e.g. we did not support drag, double click, right click, tab management, etc. We considered the primitive skills we enabled in Agent-E to be enough for vast majority of general web automation tasks. Nonetheless, if we were to specialise Agent-E to work on certain websites where right-click to select a functionality is a prominent interaction pattern, we would need to introduce that as a primitive skill.
- 2.
Consider hierarchical architecture for complex tasks:
Hierarchical architecture can be useful in agents with multiple LLM-based components. It allows execution of more complex tasks through a clear separation of roles and responsibilities. Hierarchical architecture excels in scenarios where tasks can be decomposed into sub-tasks that need to be handled at different levels of granularity. Additionally, it aids in the identification of tasks that can be executed in parallel, potentially leading to performance enhancements. It also supports the development and improvement of various components in isolation.
It is important to keep in mind that hierarchical architecture may not be suitable for all tasks. In case of Agent-E, if all we had to support were, e.g., navigate to specific URLs or perform simple web search, a hierarchical architecture may be over-complicated. In such cases, a much simpler architecture may suffice.
- 3.
Perform payload denoising when relevant:
Mitigating noise in the payload is critical in creating reliable, cost and time efficient agents. Noise could be in the form of unnecessary or irrelevant information, which could lead to incorrect or sub-optimal performance. It is also important to keep in mind that what is considered noise may be dependent on the task.
Simple techniques such as filtering irrelevant data, transforming the data to an easily consumable format, and focusing on key information can contribute to more accurate decision-making by the agent. This is especially important in environments with a high degree of uncertainty or very large input payload. In the case of Agent-E, we performed denoising on the HTML DOM. Further, we provided multiple DOM observation capabilities that the agent could adaptively use given the task requirements.
- 4.
Provide linguistic feedback of actions:
Our exploration into the space of agents capable of performing actions that has tangible consequences suggests that linguistic feedback of actions could be useful to help executor agents to be better aware of the environment and any consequence of the actions (e.g., a dialog box appeared as a consequence of the click action). Change observation helps refine the agent’s subsequent actions by providing a clear narrative of cause and effect, and also improved awareness of the environment. This could apply in a variety of usecases such as desktop automation or automations in the physical space (e.g robot control).
- 5.
Support human in the loop architecture when necessary:
Supporting human in the loop architecture is critical for agentic systems. Given that agentic systems are so new, users may not trust the system completely. Further, there will arise scenarios where the agent may need to have varying degree of user involvement for a multitude of reasons: e.g., to get more clarification on the task, when there is ambiguity how to proceed, to approve critical actions, perform some mission critical actions that the agent cannot or should not perform, audit the final output and re-invoke if needed, or take over responsibility when agent identifies that it cannot perform the task. Human in the loop architecture could also be leveraged for self-improvement where the agent identifies opportune times to gather human demonstration (e.g. Sorry, I could not do this task. Can you show me how you would do it?. Thus, incorporating mechanisms for human oversight can play a pivotal role in building that trust, and seamless handover.
Human in the loop could mean different levels of human involvement. From simple notifications of task progress, human involvement at key decision points, all the way to a handover of an incomplete task to a human.
- 6.
Analyse, reflect and aggregate past experiences routinely for self-improvement:
For Agentic systems to be adopted widely, they need to (gradually) achieve close to human-level performance. In the context of web automation, Agent-E could perform 73.1% of the tasks with an average task completion time of 150-220 second and 25 LLM calls. While very promising, this is not practical for production use-cases in terms of both effectiveness and efficiency.
A simple agentic solution to improving efficiency is to cache LLM calls. These mechanisms are supported out-of-the-box by agent development frameworks such as Autogen []. However, a naive caching may not work well in dynamic contexts (e.g., a web site will continuously change in terms of content, advertisements etc.). A better approach would be to establish offline workflows that routinely analyse, reflects on and aggregates past tasks and human demonstrations to convert them to more classical automation workflows. These classical approach could be re-triggered upon a new task if it matches a workflow that it has encountered in the past, and use the exploratory agentic approach only as a fallback. This would mean the task could be completed faster and cheaper.
- 7.
Introduce internal and external guardrails into the Agentic System: To ensure safe and effective operation of an agentic system, it’s important to introduce various guardrails into its operation. These guardrails can be in the form of rules, guidelines or constitution that the system must adhere to.
Some of the guardrails may be external to system, e.g., a task appropriateness guardrail may filter out inappropriate tasks even before agentic system is invoked.
In addition, there could be guardrails intrinsic to the system such as guardrails against prompt injection attacks, and operational guardrails that limits how the agent performs different tasks.
In the context of Agent-E, the operational guardrails include domain boundaries within which Agent-E should operate (e.g., intranet of an organisation), task specific boundaries (e.g., Agent-E should only respond to a medical question from authoritative websites) or user-configured boundaries (e.g., for shopping, use only Amazon)
- 8.
Choose between generic agent vs. task specific agent:
Generic agentic systems by definition can perform a wide range of tasks. However, in many practical implementations, a more focused set of capability may be desirable.
For example, Agent-E is a generic web agent that can perform a wide range of tasks on the internet, but not necessarily optimized for any specific task. It would be possible to optimize Agent-E for specific type of tasks (e.g., form filling) or specific websites (e.g., Atlassian Confluence pages) to achieve significantly higher performance. Depending on the use case, an optimized agent may suit better for certain workflows than a generic version.

## 5 Related Work

LLM-based Planning and Reasoning Over the last few years, large language models (LLMs) have demonstrated extraordinary abilities in text generation, code generation, and the generation of natural language multi-step reasoning traces. Spurred by these, there has been much work in the use of LLMs to solve multi-step reasoning and planning problems. The many variants of ‘chain-of-thought’ techniques [, ] encourage the LLM to produce a series of tokens with causal decoding that drive towards the solution of problems in math, common-sense reasoning and other similar tasks [, , , ]. With tool-usage for sensing and acting, LLMs have also been used to drive planning in software environments and embodied agents e.g., [, , , , , , ]. Finally, there has been related work investigating the limits of LLMs when it comes to planning and validation. For examples of negative results, see [, , , , ] among others.

Specialized Agents for Repetitive Tasks Beyond the examples above, and as described in Section 1, there has been much recent interest in building specialized agents for the web [, , ] and on device [, ]. Also related is recent work on building agentic workflows to replace robotic process automation []. Further afield, the work on building agents and training language models for API usage is also related, given that many software tasks and workflows involve the use of APIs; examples include [, , ] and many more.

Hierarchical Planning The notion of hierarchical AI planning has been around for five decades or more. Instead of planning directly in the space of low-level primitive actions, planning in a space of ‘high-level actions’ constrains the size of the plan length (and hence the size of the plan-space), which can result in more effective and efficient search. Examples from prior work include [, , ] and many more; see [] for more details. Also related is the use of temporal abstractions in planning and in reinforcement learning, for example the use of options in [, ]. In recent years, multiple papers have proposed the use of hierarchical planning for solving tasks in complex environments or with embodied agents; examples include [, ] and several others.

## 6 Conclusion

In this paper, we introduced Agent-E, a novel web agent designed to perform complex web-based tasks that makes use of numerous architectural improvements over prior state-of-the-art web agents such as hierarchical architecture, flexible DOM distillation and denoising method and concept of change observation to guide the agent towards more accurate performance.

Agent-E was evaluated on the WebVoyager benchmark, achieving a state-of-the-art success rate of 73.2%, a significant improvement over previous text-only and multi-modal web agents. Beyond task success rates, we also reported on additional metrics such as error awareness, task completion times, and the number of LLM calls, providing a more comprehensive evaluation of Agent-E’s performance.

We presented our learnings in the form of eight general design principles for developing agentic systems that can be applied beyond the realm of web automation.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 5
presentation: 3
questions: 1. Can you add an explanation of DOM distillation, with performance analysis under different conditions?
2. Can you provide an in-depth study on the self-refinement mechanism’s impact on various error types and discuss potential trade-offs?
3. Can you include computational efficiency metrics and discuss optimizations or scalability considerations?
soundness: 3
strengths: Originality
The paper demonstrates a nice degree of originality, primarily through the architectural approach in Agent-E. By introducing a hierarchical framework with distinct, specialized roles (planner agent, browser navigation agent, and validation agent), the authors effectively address several challenges in web automation. The flexible DOM distillation approach is another contribution, as it allows the browser navigation agent to dynamically tailor DOM representations to the specific needs of each task. This feature moves beyond static DOM handling methods seen in prior work, reducing cognitive load and enhancing accuracy. Furthermore, the self-refinement mechanism, inspired by a Reflexion-like paradigm, adds a unique layer of adaptability, allowing the agent to detect and correct failures in real-time. Together, these components present a good advancement over traditional web agents.

Quality
The paper is supported by an experimentation and evaluation on the WebVoyager benchmark. The authors provide a detailed comparison with both text-only and multimodal web agents, showing improvements over existing methods. 

Clarity
The paper is generally clear and well-organized, with each component of Agent-E’s architecture clearly described. The role and function of the planner agent, browser navigation agent, and validation agent are each explained in detail, providing readers with a solid understanding of how Agent-E manages complex tasks. The authors also do a nice job of explaining the novel DOM distillation and change observation mechanisms (assuming you are reading the appendix). 

Significance
Agent-E’s contribution looks significant in the field of autonomous web navigation, overcoming limitations in current web agents—particularly around handling complex, multi-step web tasks and interpreting lengthy and dynamic DOMs. The hierarchical architecture and the adaptive DOM distillation approach are likely to inspire future research on modular and adaptable agent architectures. The self-refinement mechanism also has broader implications, showcasing a feasible pathway for self-correcting agents that can enhance reliability in real-world applications. Given the increasing integration of LLM-powered agents in business and personal automation, Agent-E’s success rate and improved reliability on the WebVoyager benchmark underline its potential impact in advancing practical applications in web-based automation.
summary: The paper introduces Agent-E, a web agent designed to perform complex web-based tasks more efficiently. Agent-E employs a novel hierarchical architecture comprising three LLM-powered components: a planner agent, a browser navigation agent, and a verification agent.

The planner agent is responsible for high-level task management, breaking down user instructions into a sequence of manageable subtasks. These are delegated to the browser navigation agent, which plans and executes the necessary lower-level actions to complete each subtask. To handle the complexity of DOMs and improve interpretability, the browser agent utilizes a flexible DOM distillation approach, selecting the most suitable DOM representation for each task to highlight key elements and avoid overwhelming the LLM with unnecessary information. Additionally, the agent employs a 'change observation' mechanism, inspired by the Reflexion paradigm, where it monitors state changes after each action and receives verbal feedback to enhance situational awareness and performance.
Agent-E also incorporates a verification agent that provides feedback on incomplete or failed tasks, enabling a self-correcting system through a self-refinement mechanism. Agent-E was tested in on the WebVoyager benchmark.
weaknesses: 1. My main concern is regarding the limited benchmarking scope. While the paper presents results on the WebVoyager benchmark, the reliance on a single (one would say old) benchmark limits Agent-E’s effectiveness. Given the paper's goal to establish Agent-E as a state-of-the-art web agent, it must be evaluated on additional benchmarks: WorkArena, WebArena, ST-WebAgentBench.
This is a major weakness as I am not sure if the results will be the same on the SOTA benchmarks. I must admit that it is very hard for me to judge this agent based on the WebVoyager benchmark solely. 

2. Agent-E’s architecture, with separate planner, browser, and validation agents, potentially introduces increased complexity and computational overhead. The paper does not fully address how this architecture scales in terms of computation and memory requirements, particularly when applied to larger, real-world workflows. Including benchmarks of computational resources used by Agent-E compared to simpler, single-agent systems would provide valuable insights.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - Have you empiricially evaluated the impact of using provided API and the agent design? The impact of the validation agent was evaluated separately, so one can extract the added value.
- Does the task-specific agent design might have limitations to Web tasks or would it generically work well for any browser-based Web task? 
- What is the motivation and influence of using gpt4-o as validation agent and not sticking to gpt4-turbo? Would the results be less competitive with a gpt4-turbo validation agent?
- Have you performaned fine-tuning experiments with open-source models?
soundness: 3
strengths: - Sensible research direction, as LLMs can be of great use in automatizing various Web tasks
- Superior empirical results on existing benchmark, which includes representative Web tasks
- Key learnings are extracted from the proposed method, including the implmeneted task-specific agent design
summary: The paper proposes a novel architecture for solving Web tasks, comprising a multi agent system with a planner agent, browser navigation agent and validation agent. Next to the agent architecture, the authors propose a novel preprocessing/action formulation, where the agent gets access to a hand-made API. The latter enables to get the DOM tree in different representation or additional fine-grained filtered information. 

The new agent system is evaluated on the WebVoyager benchmark, where it is compared with the provided baselines of the benchmark (using gpt4-turbo) itself plus a recent text only approach. The results show that the new agent system is on par or better for the different sub-tasks, where an additional improvement can be seen when activating the validation agent (using gpt4-o).
weaknesses: - Unclear if added value comes from DOM API or multi-agent system. At this point, it would be of value to have ablations or a proper baseline with only one LLM which uses the API. 
- Unclear if choice of gpt4-o for the validation agent has an impact on the results.
- Related work does not concisely depict the delta to other works, but simply list other works.
- No usage of open-source models, which could additionally be fine-tuned

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: **Question**
 - Nitpick: Why did you choose the name verification agent, this confused me on the first read of the paper as I thought this agent would verify the *plan*, instead it seems that this agent is used to assess whether a task has succeeded after execution, and prompt re-attempt on failures. Perhaps something along the lines of "reviewer", "monitor" or "feedback" agent may be better.
 - "Hierarchical architecture excels in scenarios where tasks can be decomposed into sub-tasks that need to be handled at different levels of granularity"; realistically this just seems to be helping an LLM with Chain of Thought by getting it to decompose tasks at different levels of granularity giving it more time to "think". Have you run experiments to see if this hierarchical architecture still provides benefit when using models like o1-preview that are able to do this kind of chain-of-thought work out of the box.
 - Is DOM Distillation a term that the authors of this paper have coined, or is it used elsewhere?
 - What methodology, if any, was used to identify the 3 agent architecture - were there any other architectures that were tried before this?
 - Why was only the validation agent tested with vision modalities?
soundness: 2
strengths: - The paper tells a clear narrative, and does a good job of presenting the high level agent architecture, and capabilities of the agent.
 - Achieves SOTA performance on the WebVoyager benchmark, and justify use of this benchmark due to the diversity of pages available. The paper could be further strengthened by running their agent on the other benchmarks discussed in their paper.

**Originality**
The authors introduce several seemingly novel techniques that allow them to achieve SOTA performance on WebVoyager , including:
 - Use of distinct agents for high-level planning, browser navigation, and validation of success
 - Use of feedback from validation agent to re-attempt failed tasks
 - Use of DOM Distillation
 - Providing change actions to the browser navigation agent

**Quality**
The evaluation is thorough and displays the performance of different variations of the validation / refinement architecture across different websites in the benchmark.

**Clarity**
The paper clearly describes the high-level architecture of the agent, novel contributions and evaluation. However, it lacks various details that make it understand, e.g., the implementation of each agent (prompting and inputs) and does not have supplementary materials such as a codebase to facilitate this understanding.

**Significance**
The system achieves SOTA performance on the WebVoyager benchmark, beating previous models by over 16%.
summary: This paper presents Agent-E, an LLM-driven web agent designed to perform a range of web tasks including: page interaction, form filling, content summarisation, and analysis of DOM structures. Agent-E uses 3 LLM-powered agents to respectively perform high-level task planning, browser navigation to complete given tasks, and validation - in particular providing feedback on browser state when tasks are incomplete; allowing the agent to re-attempt the task and self-correct.

Further, the authors introduce 3 novel DOM Distillation strategies to pre-process the DOM that is presented to the LLM-powered agents. These are (1) text only - used in summarisation tasks (2) input fields - used in search or form-filling type interactions and (3) all fields - a complete JSON representation of all elements in the DOM. Additionally the authors provide change observations, such noting that popups appear when an LLM interacts with a button, to support the browser navigation agent in planning its next step.
weaknesses: - The authors choose to not make their code available for review. This makes it difficult to assess the accuracy with which the paper describes their codebase. Please provide anonymised repo using something like https://anonymous.4open.science/, and describe more details of your agent architecture in the appendix (i.e. prompts used for each agent). Morever, this limits the *theoretical* contributions of the paper, as various contributions of the work, are not described in great detail in the work. Such contributions include:
    - Change observation: No explanation is given of what information is given to the LLM to generate the natural language change observation; is it the DOM before and after? a diff? or some more novel algorithm that is applied?
     - What is the architecture of the validation agent / what information is it given to identify whether a task has been completed or not and give feedback
 - There are several claims that are not well quantified by the authors, including:
	 - "We consider the primitive skills we enabled in Agent-E to be enough for the vast majority of general web automation tasks": Perhaps there are statistics you can provide such as the number of tasks in the WebVoyager benchmark which require skills that are not enabled; and elaborate more on why 
- The Agent Design Principles are based soley on the authors learnings and intuition, this section could be improved by drawing upon and referencing existing works that discuss architectures / design principles for (1) agentic software (2) LLM planning (3) LLM accuracy optimisation esp. when dealing with structured data. We also comment on some specific design principles:
	- "Routinely analyze, reflect": please use more precise language than "reflect"; it seems like you have (1) batch jobs that find common tasks and turn them into reproducible workflows that can be called (2) allow for tasks to be re-run with knowledge of outcomes from past tasks - much of this seems like optimisations for production settings, but not something that is particularly insightful from a scientific standpoint. I would have expected the word reflect to likely indicate fine tuning but that does not appear to be the case here.

Meta-review: This paper introduces Agent-E, a hierarchical LLM-powered web agent with innovative mechanisms like flexible DOM distillation and self-refinement. While the authors demonstrate promising results on the WebVoyager benchmark, a key weakness lies in the limited evaluation scope. Despite the authors' arguments for WebVoyager's suitability and their willingness to incorporate additional benchmarks in the final version, the current lack of generalizability raises concerns.

Strengths:

The hierarchical architecture, flexible DOM distillation, and self-refinement mechanism carry some novel ingredients to the field of web agents (hWZC, WFFR, UT8D). Agent-E achieves state-of-the-art performance on the WebVoyager benchmark (hWZC, WFFR, UT8D). On the presentation wise, the paper is well-written and clearly presents the agent's architecture and mechanisms (hWZC, UT8D).

Weaknesses:

The evaluation solely relies on the WebVoyager benchmark, limiting the generalizability of the results and raising concerns about the agent's performance on other established benchmarks like WebArena (hWZC, WFFR, UT8D). This is a significant weakness that hinders a comprehensive assessment of Agent-E's capabilities compared to existing state-of-the-art web agents.

Key Discussion Points:

DOM Distillation: The authors provided a detailed explanation and performance analysis of their flexible DOM distillation approach in response to reviewer questions.

Self-Refinement: An in-depth study on the self-refinement mechanism's impact on various error types was conducted, addressing reviewer hWZC's concerns.

Ablation Studies: Ablations were performed to assess the individual contributions of the DOM API and the multi-agent system, as requested by reviewer WFFR.

Change Observation: Reviewer UT8D's request for clarification on the implementation of change observation was met with a detailed explanation.

Agent Design Principles: The authors strengthened the theoretical grounding of the paper by connecting their design principles to existing works in response to reviewer UT8D's feedback.

Although Agent-E demonstrates promising results on the WebVoyager benchmark and the authors made efforts to address some of the concerns raised by the reviewers, the limited evaluation scope remains significant concerns about the generalizability of the findings.  The lack of evaluation on other established benchmarks prevents a comprehensive assessment of Agent-E's capabilities and its comparison to existing state-of-the-art web agents.

Decision: Reject