Beyond Browsing: API-Based Web Agents

## Abstract

Abstract Web browsers are a portal to the internet, where much of human activity is undertaken. Thus, there has been significant research work in AI agents that interact with the internet through web browsing.
However, there is also another interface designed specifically for machine interaction with online content: application programming interfaces (APIs).
In this paper we ask – what if we were to take tasks traditionally tackled by browsing agents, and give AI agents access to APIs ?
To do so, we propose two varieties of agents: (1) an API-calling agent that attempts to perform online tasks through APIs only, similar to traditional coding agents, and (2) a Hybrid Agent that can interact with online data through both web browsing and APIs.
In experiments on WebArena, a widely-used and realistic benchmark for web navigation tasks, we find that API-based agents outperform web browsing agents.
Hybrid Agents out-perform both others nearly uniformly across tasks, resulting in a more than 20.0% absolute improvement over web browsing alone, achieving a success rate of 35.8%, achiving the SOTA performance among task-agnostic agents.
These results strongly suggest that when APIs are available, they present an attractive alternative to relying on web browsing alone. 1 1 1 Code is available at https://github.com/yueqis/API-Based-Agent ; Project Webpage is available at https://yueqis.github.io/API-Based-Agent/

## 1 Introduction

Web agents use browsers as a interface to facilitate humans in performing daily tasks such as online shopping, online planning, trip planning, and other work-related tasks (Liu et al., ; Li et al., ; Rawles et al., ; Patil et al., ; Pan et al., ; Chen et al., ; Huang et al., ; Durante et al., ).
Existing web agents typically operate within the space of graphical user interfaces (GUI) (Zhang et al., ; Zhou et al., ; Zheng et al., ), using action spaces that simulate human-like keyboard and mouse operations, such as clicking and typing.
To observe web pages, common approaches include using accessibility trees, a simplified version of the HTML DOM tree, as the input to text-based models (Zhou et al., ; Drouin et al., ), or multi-modal, screenshot-based models (Koh et al., ; Xie et al., ; You et al., ; Hong et al., ).
However, regardless of the method of interaction with web sites, there is no getting around the fact that these sites were originally designed for human consumption, and may not be the ideal interface for machines.

Figure: Figure 1: A comparison of three types of agents. The Browsing Agent performs tasks through web browsing only, utilizing the accessibility tree to interact with web pages, achieving an average performance of 14.8% on WebArena. The API-Based Agent performs tasks by making API calls and generating code without relying on web browsing, achieving an average accuracy of 29.2%. The Hybrid Agent combines both methods, dynamically switching between web browsing and API calling, depending on the task. This allows the execution of either API calls or web browsing actions, or both in combination, improving performance by more than 5 percentage points compared to the API-Based Agent.
Refer to caption: x1.png

Notably, there is another interface designed specifically for machine interaction with online content: application programming interfaces (APIs). APIs allow machines to communicate directly with the backend of a web service (Branavan et al., ), sending and receiving data in machine-friendly formats such as JSON or XML (Meng et al., ; Xu et al., ). Nonetheless, whether AI agents can effectively use APIs to tackle real-world online tasks, and the conditions under which this is possible, remain unstudied in the scientific literature.
In this work, we explore methods for tackling tasks normally framed as web-navigation tasks with an expanded action space to interact with APIs.
To do so, we develop new *API-based agents* that directly interact with web services via API calls, as depicted in Figure 1. This method bypasses the need to interact with web page GUIs through simulated clicks.

At the same time, not all websites have extensive API support, in which case web browsing actions may still be required. To address these cases, we explore a *hybrid* approach that combines API-based agents with web-browsing agents, as described in Figure 1.
By implementing an agent capable of *interleaving* API calls and web browsing, we found that agents benefit from the flexibility of this hybrid model. When APIs are available and well-documented, the agent can directly interact with the web services. For websites with limited API support, the agent seamlessly switches to web browsing mode, simulating human interaction to ensure task completion.

We evaluated our API-based and Hybrid Agents on WebArena, a benchmark for real-world web tasks (Zhou et al., ), and the results are shown in Figure 1. Our experiments revealed three key findings: (1) The API-based agent consistently outperforms browsing-based agents on WebArena tasks by around 15% on average, regardless of the comprehensiveness of APIs.
(2) The API-based agent yields a higher success rate on websites with extensive API support (e.g., Gitlab) compared to those with limited API support (e.g., Reddit).
This result underscores the importance of developing comprehensive API support for more accurate and efficient web task automation in the future.
(3) The Hybrid Agent outperforms solely browsing-based agents and solely API-based agents, further improving accuracy by more than 5% compared to the API-based agent.
By dynamically switching between approaches, the Hybrid Agent is able to provide more consistent and reliable outcomes.

In sum, our results suggest that allowing agents to interact with APIs, interfaces designed specifically for machines, is often preferable or at least complementary to direct interaction with graphical interfaces designed for humans.

## 2 Background: Web Browsing

### 2.1 The Web Browsing Task

Various benchmarks have been developed to evaluate the performance of web browsing agents. MiniWoB (Miniature World of Bits) is an early benchmark that provides simple web-based tasks such as clicking links or typing into forms, but it remains limited in complexity and realism (Shi et al., ). Mind2Web scales up these tasks, introducing more sophisticated interactions across websites, but it often lacks the dynamic, real-world scenarios found on the broader web (Deng et al., ). WebArena (Zhou et al., ) advances web browsing benchmarks by creating reproducible sandboxes of a variety of websites, such as managing repositories, posting online, performing online shopping, and planning trips using map services, while VisualWebArena extends WebArena to the vision modality (Koh et al., ).

In this paper, we focus on WebArena tasks, which simulate real-world scenarios to evaluate an agent’s ability to complete diverse web-based activities.(^2^22Notably, upon investigation of VisualWebArena we found that APIs for handling images were relatively limited, and hence we chose to experiment on text-only tasks in this paper.)
Tasks in WebArena include interacting with platforms like Gitlab (to manage projects and repositories), Reddit (to browse and post content), e-commerce websites (for shopping), and mapping services (for trip planning) (Zhou et al., ). Task success is evaluated in three ways: (1) if the task requires producing a specific output, the agent’s response is checked for correctness; (2) for tasks involving changes to a website’s state (e.g., adding an item to a shopping cart), success is measured by verifying whether the state has changed as expected, such as ensuring the correct item and quantity have been added to the cart; and (3) if the task involves navigation, success is determined by whether the agent reaches the correct URL displaying the desired content.

### 2.2 A Baseline Web Browsing Agent

Figure: Figure 2: The API-based agent can often solve problems in many fewer function calls than traditional browsing agents. In this task, web browsing failed to solve the intent "find the number of commits the user SaptakS made to the repo a11yproject" after 15 steps, while our API-based agent successfully completed the task with only three lines of code.
Refer to caption: x2.png

While there are a wide variety of agents proposed for such web navigation tasks, in this work we build upon the WebArena baseline agent (Zhou et al., ), which operates purely through web interaction by leveraging the accessibility tree(^3^33[https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree](https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree)), a structure that exposes interactive elements like buttons, input fields, and hyperlinks (Yao et al., ; Gu et al., ). Each element of the accessibility tree is characterized by its functionality such as a hyperlink, its content, and specific web attributes (Liu et al., ; He et al., ; Lù et al., ).
This exposes web page elements in a hierarchical structure that is easy for agents to navigate (Samuel et al., ; Burns et al., ).

Agents based on this framework utilize an action space that simulates human browsing behavior, incorporating actions such as simulated clicks, form input, and navigation between pages (Liu et al., ; Song et al., ; Gur et al., ). Importantly, these agents maintain a comprehensive history of their previous actions, allowing them to contextualize their decision-making in past actions.

While agents utilizing this method can navigate arbitrary web pages and often perform well on simpler layouts, challenges arise with the complexity of the accessibility tree. Many large language models (LLMs) are not familiar with this structure, leading to difficulties in completing tasks that require numerous or complex interactions. As a result, the average accuracy hovers in the low double digits (Liu et al., ; Deng et al., ; Fu et al., ). These methods also struggle with content that need to be dynamically loaded or contents not immediately visible within the tree (Abramovich et al., ; Chen et al., ; Lutz et al., ).

To give one motivating example, in Figure 2, we demonstrate a task where the agent needs to perform a task determining the number of commits made by the user SaptakS in a repository named a11yproject. Specifically, for each task, the agent is given a fixed number of steps within which it has to finish the task. Using a traditional web-browsing approach, the agent follows a complex trajectory, starting with logging into the website, navigating to the correct project, accessing the repository, and finally attempting to view the list of commits. However, due to the large number of commits made by other users, the commits by SaptakS are located much further down on the web page, requiring the agent to scroll down many times. As a result, despite completing 15 actions, the browsing agent is unable to retrieve the required information.

## 3 From Web Browsing to API Calling

In contrast to web browsing, API calling offer a direct interface for machines to communicate with web services, reducing operational complexity.
In this section, we explore an API-based approach when performing web tasks.

Figure: Figure 3: An example of API documentation showing how to get commits of a project, the API call using a Python script to retrieve commits from a project repository, and the resulting JSON response.
Refer to caption: x3.png

### 3.1 APIs and API Documentation

For websites that offer API support, pre-defined endpoints can be utilized to perform tasks efficiently.
These APIs, following standardized protocols like REST(^4^44[https://en.wikipedia.org/wiki/REST](https://en.wikipedia.org/wiki/REST)), allow interaction with web services through sending HTTP requests (e.g., GET, POST, PUT) and receiving structured data such as JSON objects(^5^55[https://www.json.org/json-en.html](https://www.json.org/json-en.html)) as responses. Websites often provide official documentation for the APIs, which can give guidance on how to utilize the APIs. Some documentation is provided in README (^6^66[https://en.wikipedia.org/wiki/README](https://en.wikipedia.org/wiki/README)) format, some are in OpenAPI YAML(^7^77[https://yaml.org/](https://yaml.org/)) format, and some are in plain text format. For instance, Figure 3 shows the official README documentation of a Gitlab API GET /api/{id}/commits. It documents the functionality, input arguments, and output types of the API. For example, one could use the Python requests library, by calling requests.get("gitlab.com/api/a11yproject/commits"), to retrieve all commits of the repository a11yproject. This would return a JSON list containing all the commits to this repo, as shown in Figure 3.

### 3.2 Obtaining APIs for Agents

One important design decision is how to obtain APIs for agents to use. The way agents interact with APIs depends heavily on the availability of APIs and quality of API documentation. In this work, we acquired APIs by manually looking up official API documentation on a website, although this process could potentially be automated in the future.
We classify the availability of APIs according to the following three scenarios:

##### Sufficient APIs and Documentation

Many websites provide comprehensive API support and well-documented API documentation in YAML or README format.
In this case, simply use the APIs/documentation as-is.
Figure 3 depicts an example of API documentation.

##### Sufficient APIs, Insufficient Documentation

There are some challenging situations where APIs exist but good documentation is not officially available. In such cases, additional steps may be required to obtain a list of accessible APIs. In this case, we inspected the frontend or backend code of the website to extract undocumented API calls that can still be utilized by the agent. Then, based on the implementation of APIs of the website, leverage an LLM (GPT-4o(^8^88[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/))) to generate these YAML or README files.
By prompting GPT-4o with the relevant implementation details of the APIs (for example, the implementation files of the APIs or example traces of API calls), we generate comprehensive documentation, including input parameters, expected outputs, and example API calls.

##### Insufficient APIs

In the more challenging cases, where only minimal APIs are available, it may be necessary to create new APIs. These custom APIs allow agents to perform tasks that otherwise would require manual web browsing steps. In our case, this was necessary for 1 of 5 web sites in the WebArena benchmark that we utilized, such as creating Reddit APIs discussed in Section 6.2.

### 3.3 Using APIs in Agents

Once we have the APIs and documentation, we then need to provide methods to utilize them in agents.
We utilize two different methods based on the size of the API documentation.

##### One-Stage Documentation for Small API Sets

For websites with a smaller number of API endpoints(^9^99Specifically, we use a threshold of 100 APIs, but this could be adjusted depending on the supported language model context size.), we directly incorporate the full documentation into the prompt provided to the agent.
This approach of directly feeding the full documentation worked well for websites with a limited number of API endpoints, as it allowed the agent to have immediate access to all the necessary information without the need for a more complex retrieval mechanism.

##### Two-Stage Documentation Retrieval for Large API Sets

For websites with a larger number of endpoints, providing the full documentation directly within the prompt was impractical due to the size limitations of agent inputs. To address this, we employ a two-stage documentation retrieval process, which allowed access to only the relevant information as needed, keeping the initial prompt concise.

In the first stage, the user prompt provide a description of the task, with a list of all available API endpoints along with a very brief description of each API. For example, {"GET /api/{id}/commits": "Get a list of commits in a project"}. This initial summary helps facilitating understanding the scope of all the available APIs while staying within the prompt size constraints.

In the second stage, if the model determines that it needs detailed information about a specific API endpoint or some API endpoints, it can use a tool called get_api_documentation. This tool maintains a dictionary that maps each API to its API documentation respectively. The dictionary is generated using pattern match in Python to retrieve substrings related to each endpoints. get_api_documentation is able to searche the dictionary and retrieve the full README or YAML documentation for any given endpoint by calling get_api_documentation with the endpoint’s identifier. This might include the input parameters, output formats, and examples of how to interact with the endpoint. For example, to retrieve the documentation for the endpoint GET /api/id/commits, the agent would call get_api_documentation("GET /api/id/commits"), and an example returned API documentation is the documentation in Figure 3.

This retrieval method allows the agent to make flexible and informed choices during the execution of tasks. If the agent finds that an API does not meet its needs or if it encounters an error, it can easily retrieve the documentation for a different API endpoint by calling the function again. This dynamic approach promotes adaptability and minimizes the risk of incorrect API usage when the number of APIs available is large.
The prompt can be found in Appendix A.2.

## 4 Hybrid Browsing+API Calling Agents

We have proposed API-based methods for handling web tasks, but the question arises: given the benefits of API calling, should we discard web browsing altogether?
The most obvious bottleneck is that not all websites offer comprehensive API support.
Some platforms offer limited or poorly documented APIs (e.g. there is no API for shopping on Amazon(^10^1010[https://www.amazon.com](https://www.amazon.com))), forcing agents to rely on traditional web browsing methods to complete tasks.

To deal with these situations, we propose a hybrid methods that integrates both browsing-based and API-based approaches, and developed a Hybrid Agent capable of interleaving API calls and web browsing, switching dynamically based on task requirements and the available resources.
Specifically, for each task, the agent is given the fixed step budget within which it has to finish the task.
In each step, the agent could either (1) communicate with humans in natural language to ask
for clarification or confirmation, or 2) generate and executes Python code which could include performing API calling, or 3) performs web browsing actions. The agent could choose freely among these three options, depending on the agent’s confidence which method could best tackle the task.

The ideal case is that for websites that offer comprehensive API support, the Hybrid Agent can utilize well-documented endpoints to perform tasks more efficiently than it could through web browsing; for websites with limited API support or poorly documented APIs, the Hybrid Agent could rely more on web browsing to fulfill certain tasks. We later find that enabling an agent to interleave API calling and web browsing boost the agent’s performance (see Section 6).

##### Prompt Construction

The Hybrid Agent’s prompt construction extends upon the API-based agent by incorporating both API and web-browsing documentation. Similar to the API-based agent, the Hybrid Agent is provided with a description of available API calls as discussed in Section 3.3. In addition, the Hybrid Agent receives a detailed specification of the web-browsing actions, which mirrors the information given to the browsing agent described in Section 2.2, including a breakdown of all potential browser interactions. It also maintains a history of all its prior steps such that the agent could make more informed actions.
The prompt can be found in Appendix A.3.

## 5 Experimental Setup

### 5.1 Dataset Description

For our experiments, we utilized the WebArena dataset (Zhou et al., ) as the primary evaluation benchmark. WebArena is a comprehensive benchmark designed for real-world web tasks, providing a diverse set of websites that simulate various online interactions. The tasks within WebArena reflect common user activities such as navigating websites, performing administrative tasks, and posting online.

The dataset mainly includes five distinct websites, each containing various intents representing different tasks:
Gitlab, Map, Shopping, Shopping Admin, Reddit, and Multi-Website Tasks. We include a more detailed descriptions of the tasks in Appendix A.1. This diverse set of websites and tasks within WebArena allows for a comprehensive evaluation of the agents, testing their ability to handle both API-based interactions and web browsing across varied web settings.

### 5.2 API Statistics for WebArena Sites

In this section, we provide a detailed analysis of the API support for various websites used in the WebArena tasks, categorized into three levels: good, medium, and poor. The availability, functionality, and documentation of APIs, as described in Table 1, play a crucial role in the efficiency and flexibility of our agents.

**Table 1: Number of endpoints, quality of API, and documentation quality for WebArena websites.**
| Websites | Gitlab | Map | Shopping | Admin | Reddit |
| --- | --- | --- | --- | --- | --- |
| Number of Endpoints | 988 | 53 | 556 | 556 | 31 |
| API/Doc Quality | Good | Good | Fair | Fair | Poor |

#### 5.2.1 Good API Support

##### Gitlab

For Gitlab, we leveraged the open Gitlab REST APIs(^11^1111Documentation of all Gitlab APIs could be found at [https://docs.gitlab.com/ee/api/rest/](https://docs.gitlab.com/ee/api/rest/).), which consist of 988 endpoints. These APIs offer extensive coverage across a wide range of functionalities, including repositories, commits, users, merge requests, and issues. This comprehensive API support allows for effective interaction with most tasks required in WebArena, making it one of the best-supported platforms in terms of API availability.

The majority of Gitlab-related tasks can be handled with the provided APIs, with only a small fraction of tasks, such as retrieving the user’s Gitlab feed token, not covered by any existing endpoints. Overall, Gitlab’s API structure provides robust support.

##### Map

The Map website offers three sets of APIs, each offering distinct functionalities, with a total of 53 endpoints. Although fewer in number compared to Gitlab and Shopping, these APIs still provide significant coverage for the tasks in WebArena.

The first set of APIs, openly available at Nominatim(^12^1212[https://nominatim.org/release-docs/develop/api/Overview/](https://nominatim.org/release-docs/develop/api/Overview/)), offers essential endpoints for geographic searches. The second set of APIs, from Project OSRM(^13^1313Openly available at [https://project-osrm.org/docs/v5.5.1/api](https://project-osrm.org/docs/v5.5.1/api)), focuses on routing and navigation functionalities. The third set of APIs, available at OpenStreetMap(^14^1414Publicly available at [https://wiki.openstreetmap.org/wiki/API_v0.6](https://wiki.openstreetmap.org/wiki/API_v0.6)), deals primarily with map database operations. This API is rarely used in WebArena tasks but offers capabilities for interacting with OSM data.

Despite the smaller number of endpoints compared to other websites, the APIs available for the Map tasks are mostly well-documented and cover most of the essential WebArena use cases.

#### 5.2.2 Medium API Support

##### Shopping and Shopping Admin

The Shopping and Shopping Admin websites share a common set of APIs from the Adobe Commerce API(^15^1515[https://developer.adobe.com/commerce/webapi/rest/quick-reference/](https://developer.adobe.com/commerce/webapi/rest/quick-reference/)), consisting of 556 endpoints. These APIs provide a reasonable level of support for common shopping tasks such as managing products, categories, and customer accounts.

However, some features are absent, such as the ability to add items to a wish list, and thus these tasks must be handled via web browsing. Despite this, the API documentation is fairly detailed and covers most core functionalities, making it a solid, though not exhaustive, solution for handling shopping-related tasks.

#### 5.2.3 Poor API Support

##### Reddit

The Reddit tasks in WebArena are based on a self-hosted limited clone of the Reddit website (^16^1616[https://codeberg.org/Postmill/Postmill](https://codeberg.org/Postmill/Postmill)), with limited functionalities as compared to the official site. As a result, all of the available APIs are self-implemented, with a best effort to mimic to official Reddit APIs. With only 31 endpoints, this website offers minimal API support and no API documentation, making it the least API-friendly website in the benchmark.

Many critical functionalities, such as searching for specific posts, are missing, leaving agents to rely heavily on web browsing to complete tasks. The limited API support significantly hampers the efficiency of task execution on Reddit, highlighting the need for a hybrid browsing+API approach.

### 5.3 API Implementation Details

In this section, we will discuss how we provided the APIs to the agents when evaluating different web applications inside WebArena, where we follow the methodologies as discussed in Section 3.3.

#### 5.3.1 One-Stage Documentation for Small API Sets

For websites with fewer than 100 API endpoints, namely the Map and Reddit websites, we directly incorporated the full documentation into the prompt provided to the agent.

In the case of the Map API, the documentation was sourced directly from the public API documentation provided for the website. The only modification made was the addition of an explanation detailing how to make HTTP requests using the requests library in Python for interacting with the Map API’s endpoints. This ensured that the agent could comprehend both the structure of the API and how to implement calls programmatically.

For Reddit, since there was no pre-existing documentation for the APIs, we leveraged GPT-4o(^17^1717[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/)) itself to generate these README files. By prompting GPT-4o with a file containing all implementations of the API endpoints, we generated a README documentation, including input parameters, expected outputs, and example API calls.

#### 5.3.2 Two-Stage Documentation Retrieval for Large API Sets

For websites with more than 100 endpoints, such as GitLab, Shopping, and Shopping Admin, we employ a two-stage documentation retrieval process.

For GitLab, we obtained the README documentation from the official GitLab REST API documentation site. For the Shopping and Shopping Admin websites, the documentation was provided in the form of an OpenAPI specification, structured in YAML format.

### 5.4 Evaluation Framework

We employed OpenHands as our primary evaluation framework to facilitate the development and testing of our agents (Wang et al., ).
OpenHands is an open-source platform designed for creating and evaluating AI agents that interact with both software and web environments, making it an appropriate infrastructure for our proposed methods.
The OpenHands architecture supports a variety of interfaces for agents to interact with. Moreover, this framework allows agents to keep a detailed record of past actions in the prompt, enabling agents to execute actions in a way that is consistent with earlier steps. For coding tasks, it implements an agent based on CodeAct (Wang et al., ) that incorporates a sandboxed bash operating system and Jupyter IPython(^18^1818[https://ipython.org](https://ipython.org)) environments, enabling the execution of Python code.
Additionally, it includes a BrowsingAgent browsing agent that focuses solely on web navigation. This agent operates within a Chromium web browser powered by Playwright(^19^1919[https://playwright.dev/](https://playwright.dev/)), utilizing a comprehensive set of browser actions defined by BrowserGym (Drouin et al., ).
However, while the browsing agent can browse websites, and the CodeActAgent make API calls and execute code, there is not an agent that can natively do both.
Given this base, we developed two varieties of agents for API-based solving of web tasks.

##### API-Based Agent

First, our API-based agent essentially uses the CodeAct architecture (Wang et al., ).
In addition to the basic CodeAct framework, we tailor the agent for API calling by adding specialized instructions and examples that guide its understanding of various API endpoints and their usage. At each step, the agent could utilize all previous actions to make informed selection of actions. The prompt of the API-Based Agent is included in the Appendix A.2.

##### Hybrid Browsing/API Calling Agent

In addition to the API-based agent, we developed a Hybrid Agent that integrates Chromium web browsing functionalities powered by Playwright into the existing framework of the API-based agent.
This Hybrid Agent is provided the prompt describing both the APIs and the browsing actions, allowing for free transitions between API calling and web browsing. At each step, the agent can utilize the current state of the browser, all previous actions taken by the agent, and the results of those actions to determine the next course of action.
The prompt of the Hybrid Agent is included in the Appendix A.3.

For the browsing, API-based, and Hybrid Agents, we utilized GPT-4o as the base LLM. However, this could be easily changed to other LLMs.

## 6 Results

### 6.1 Main Results

**Table 2: Performance of Agents across WebArena Websites. ^†Note that SteP uses prompts inspired specifically by WebArena test set tasks, while other methods are task-agnostic. We achieve the highest performance among the task-agnostic agents.**
| Agents | Gitlab | Map | Shopping | Admin | Reddit | Multi | AVG. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WebArena Base (Zhou et al., ) | 15.0 | 15.6 | 13.9 | 10.4 | 6.6 | 8.3 | 12.3 |
| AutoEval (Pan et al., ) | 25.0 | 27.5 | 39.6 | 20.9 | 20.8 | 16.7 | 26.9 |
| AWM (Wang et al., ) | 35.0 | 42.2 | 32.1 | 29.1 | 54.7 | 18.8 | 35.5 |
| SteP (Sodhi et al., )^† | 32.2 | 31.2 | 50.8 | 23.6 | 57.5 | 10.4 | 36.5 |
| Browsing Agent | 12.8 | 20.2 | 10.2 | 22.0 | 10.4 | 10.4 | 14.8 |
| API-Based Agent | 43.9 | 45.4 | 25.1 | 20.3 | 18.9 | 8.3 | 29.2 |
| Hybrid Agent | 44.4 | 45.9 | 25.7 | 41.2 | 28.3 | 16.7 | 35.8 |

The main results of our evaluation, as summarized in Table 2, demonstrate the performance of three different agents across the websites in the WebArena benchmark.

The API-Based Agent consistently performed well across most tasks, achieving higher scores in all websites compared to the Browsing agent. This agent’s strong performance is attributed to its specialized design for API calling, enabling it to efficiently interact with the websites’ APIs and complete tasks with minimal reliance on browsing capabilities.

In contrast, the Browsing Agent, which is designed solely for navigating web interfaces, demonstrated significantly lower performance across all domains. It achieved its best scores on Gitlab and Map, but struggled more on Reddit.

The Hybrid Agent, which integrates both API calling and web browsing, outperformed the other agents in all categories. The agent’s ability to dynamically switch between API calling and web browsing proved beneficial.
API calling delivers high performance for web tasks when well-supported APIs are available,
while web browsing serves as a backup when API endpoints are unavailable or incomplete. Even if the website provides comprehensive APIs, there might be corner cases where APIs are not supportive. In these cases, relying on web browsing is still needed for
tasks that would otherwise fail through API-only interactions.
Table 3 documents the percentage of actions of our Hybrid Agent. Across all websites, our Hybrid Agent chooses to do both Browsing and API in the same task at least half of the time.

Table 4 documents the accuracy of the Hybrid Agent across website when performing different choices of actions. We can see that it show consistently high accuracy when choosing API only and API+browsing.

**Table 3: Percentage of Actions (%) that our Hybrid Agent takes for each type of tasks. Each column sums up to 1.**
| Actions | Gitlab | Map | Shopping | Admin | Reddit | Multi | AVG. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Browsing only | 7.8 | 3.7 | 38.5 | 2.2 | 17.0 | 8.3 | 14.3 |
| API only | 21.1 | 4.6 | 7.5 | 1.1 | 0.9 | 10.4 | 8.0 |
| Browsing+API | 71.1 | 91.7 | 54.0 | 96.7 | 82.1 | 81.3 | 77.7 |

**Table 4: The accuracy (%) of the Hybrid Agent across choices of actions for each website, with the number of correct instances / number of total instances in parentheses.**
| Choices of Action | Gitlab | Map | Shopping | Admin | Reddit | Multi | AVG. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Browsing only | 7.1(1/14) | 50.0(2/4) | 23.6(17/72) | 50.0(2/4) | 11.1(2/18) | 25.0(1/4) | 21.6(25/116) |
| API only | 47.4(18/38) | 40.0(2/5) | 21.4(3/14) | 50.0(1/2) | 0.0(0/1) | 20.0(1/5) | 38.5(25/65) |
| Browsing+API | 47.7(61/128) | 46.0(46/100) | 27.7(28/101) | 40.9(72/176) | 32.2(28/87) | 15.4(6/39) | 38.2(241/631) |

Overall, the results indicate that the Hybrid Agent is the most effective for handling diverse tasks in WebArena, particularly in environments that require a blend of API and browsing actions. The API-Based Agent excels in tasks that are primarily API-driven, while the Browsing Agent is more suitable for simple navigation tasks but lacks the versatility needed for more complex scenarios.

**Table 5: Average number of steps and cost of agents on WebArena tasks**
| Browsing Agent | API-Based Agent | Hybrid Agent |  |  |  |
| --- | --- | --- | --- | --- | --- |
| steps | cost | steps | cost | steps | cost |
| 8.4 | $0.1 | 7.8 | $1.2 | 8.9 | $1.5 |

Additionally, we use Table 5 to demonstrate the average steps taken and the average cost for each agent to complete WebArena tasks. The breakdown of steps and cost by website is in the Appendix A.4. Figure 4 demonstrates a scatterplot of the average accuracy of each agent on WebArena over their average steps and average cost.

Figure: Figure 4: Number of steps (left) and cost (right) of agents averaged across WebArena Websites
Refer to caption: x4.png

##### Steps

The browsing agent consistently takes more steps to complete tasks compared to the API-based agent, while the Hybrid Agent takes the most steps amongst the three agents. This is likely due to the browsing agent’s reliance on navigating web interfaces and interacting with visual elements, which involves a sequential and more time consuming processes. The API-based agent is the most efficient in terms of steps, as it can directly interact with structured data via APIs, bypassing many of the steps involved in traditional web navigation. The Hybrid Agent, combining both action spaces from the browsing agent and the API-based agent, takes more steps than both agents.

##### Costs

The cost of completing tasks shows a different trend. While the browsing agent requires more steps, it is much cheaper compared to the API-based agent and the Hybrid Agent. This is primarily because the prompts needed for browsing agents are much shorter. When browsing, the agent only needs instructions on how to use the web interface and the limited action space around 14 browsing actions. In contrast, API-based and Hybrid Agents require access to a much larger set of API calls. For example, when interacting with GitLab, the agent is provided with 988 available APIs, leading to much longer prompts and significantly increasing the cost of execution. The cost goes down when the prompt for API calling is shorter. For example, the Reddit website has the least length of API documentation, where its cost is also less than other websites. However, as visualized in Figure 4, the accuracy of the API-based agent and the Hybrid Agent is much higher than the browsing agent, which makes the increase in cost justifiable due to the significantly improved task performance. The higher cost is offset by the agents’ ability to complete tasks more accurately and efficiently. In the future, this increased cost could potentially be mitigated by methods that retrieve only relevant APIs on the fly.

### 6.2 Does API Quality Matter?

The short answer is yes, API quality does significantly impact the performance of the API-based agent. High quality APIs provide comprehensive and well-documented endpoints that enable agents to interact accurately and efficiently with websites. With comprehensive API support, the API-based agent is able to tackle more tasks through API calling, while the Hybrid Agent could rely less on the browsing agent; on the other hand, clear and detailed documentation allows agents to utilize the APIs effectively, ensuring that requests are accurate, and minimizing potential errors in task execution. For example, the websites Gitlab and Map with the best API support as mentioned in Section 5.2, demonstrates the highest task completion accuracy by the API-based agent and the Hybrid Agent across all websites.

Conversely, low-quality APIs, characterized by incomplete functionality or ambiguous documentation, can significantly degrade performance. In such cases, the absence of necessary endpoints may prevent the API-based agent from completing tasks, forcing the Hybrid Agent to resort to web browsing. Moreover, poorly documented APIs can result in incorrect parameters and headers being used, further reducing the effectiveness of the agent. This highlights the importance for websites to maintain comprehensive and well-documented API support.

**Table 6: Change in performance of the API-Based Agent on Reddit upon incorporating new APIs.**
| Number of Endpoints | 18 | 31 |
| --- | --- | --- |
| Accuracy on Reddit | 9.4% | 18.9% |

An illustrative example of this is the case of Reddit, where the initial performance of the API-based agent was suboptimal due to limited API availability.
As depicted in Table 6, initially, Reddit offered only 18 APIs, lacking the major functionality that common online forums have, such as post voting.
Recognizing this limitation, we manually introduced 13 additional APIs including one API on post voting, with our best effort trying to mimic the official Reddit website. This results in a marked improvement in the API-based agent’s performance, underscoring the direct correlation between the availability of high-quality APIs and the average performance of the API-based agent.

Moreover, API quality can also correlate with the performance of browsing agents. This may be because websites with well-implemented APIs often have clean, user-friendly interfaces, which benefit machine agents when interacting with the web interface. Good API practices suggest a thoughtful design process that tends to carry over into the overall user interface and experience, allowing the browsing agent to more easily parse and interact with the website’s elements. As a result, both API-based and browsing agents are able to function more effectively in environments where high API standards are maintained.

### 6.3 Case Studies

Figure: Figure 5: The Hybrid Agent succeeds while the browsing agent and API-based agent both fail
Refer to caption: x5.png

##### Case 1

In this section, we analyze two contrasting instances as shown in Figure 5 and Figure 6, where the Hybrid Agent and API-based agent exhibited different levels of performance on WebArena tasks. These case studies highlight the strengths and weaknesses of each agent, demonstrating scenarios where hybrid browsing outperforms API-only or browsing-only approaches, as well as cases where the API-based agent excels over the hybrid method.

One example where the Hybrid Agent succeeded, while both the API-based and browsing agents failed, involved a task from the Shopping Admin domain. The query was to ”delete all negative reviews for Sybil running short,” a product listed in the shopping admin interface. In this instance, the API-based agent failed because no relevant API endpoints were available for retrieving or deleting reviews. Similarly, the browsing agent failed, as completing this task purely through web navigation required too many steps, as depicted in Figure 5. This complexity made the task challenging for an agent relying solely on web interactions.
However, the Hybrid Agent successfully completed the task by leveraging both API and browsing functionalities. An example trace of the Hybrid Agent shown in Figure 5.
This case highlights the Hybrid Agent’s ability to efficiently combine API calls with web interactions, allowing it to tackle complex multi-step tasks that would be difficult or impossible for solely browsing or solely API-based agents.

Figure: Figure 6: Case 2: the API-based agent succeeds while the browsing agent and the Hybrid Agent fails.
Refer to caption: x6.png

##### Case 2

Conversely, there are instances where the API-based agent outperforms the Hybrid Agent.
One such case occurred in the GitLab website, where the task was to "tell me the email address of the contributor who has the most commits to ai." The API-based agent successfully completed this task by utilizing the GET /api/id/contributors API endpoint to retrieve the contributor with the highest number of commits and their associated email address.
On the other hand, the Hybrid Agent attempted to solve the task through browsing but encountered significant challenges. Accessing this information through web browsing required navigating GitLab’s interface, locating the correct repository and branch, and identifying the top contributor manually, a task that might be too difficult to perform through web navigation alone. As a result, both the browsing agent and the Hybrid Agent failed to complete the task. This case demonstrates an example where API access provides a more straightforward solution than browsing in contexts requiring structured data retrieval.

## 7 Related Work

The development of AI agents that interact with the web and APIs has garnered significant research attention. Web browsers, serving as the primary interface for interacting with online content, have long been a focus for AI research. Web-based agents that can navigate websites, extract information, and perform tasks autonomously have been studied extensively, especially in the context of large language models (LLMs) and agents designed to mimic human behavior online.

##### Web Navigation Agents

Much prior work has centered around agents that perform web-based tasks using browsing actions (Lai et al., ; Koh et al., ; Pan et al., ). These agents are particularly effective in environments where human-like interaction with a user interface is necessary (Drouin et al., ). Frameworks such as WebArena have further refined the evaluation of such agents by providing complex and realistic web navigation tasks (Zhou et al., ). Our work explores the Hybrid Agent that combine web browsing with API interactions. While prior work primarily focuses on browsing-only agents, we examine how Hybrid Agents can enhance performance by integrating structured API calls with web navigation.

##### Code Generation Agents and Tool Usage

Another stream of research focuses on agents that interact with online content via application programming interfaces (APIs) (Yuan et al., ; Patil et al., ; Wang et al., ; ). In this context, works such as CodeAct have pioneered the development of agents that generate and execute code, including API calls, to perform tasks typically reserved for software engineers (Wang et al., ; Zhang et al., ; Tang et al., ). These API-based agents are optimized for tasks that involve structured data exchanges, allowing them to perform operations more efficiently than traditional web navigation agents (Shen et al., ). On the other hand, our work integrates both browsing and API interactions, demonstrating that Hybrid Agents can outperform API-only agents in tasks requiring web navigation. While existing research shows the efficiency of API-based agents, our Hybrid Agent dynamically switches between APIs and web browsing to optimize task performance.

Additionally, we are the first to explore comparative studies of API v.s. Browsing agents on the same websites. We demonstrate that API-based agents are often more efficient than browsing agents when APIs are available, leading to significant improvements in performance.
This finding is aligned with previous studies that highlight the advantages of structured interactions through APIs compared to unstructured web browsing interactions.

## 8 Conclusion and Future Work

In this paper, we propose new web agents that use APIs instead of traditionally browsers. We found that API-based agents outperform browsing-based counterparts, especially on websites with sufficient API support. Hence we further propose an agent that is capable of switching between using APIs or browsers and empirically outperforms agents that only uses one of the two interfaces.

For future work, we aim to explore methods for automatically inducing APIs using techniques such as Agent Workflow Memory (AWM) (Wang et al., ). These methods could identify and generate API calls for websites lacking formal API support, further expanding the applicability and efficiency of API-based approaches. By automating the discovery and utilization of APIs, we envision even more robust agents capable of handling diverse web tasks with minimal reliance on manual interaction through browsing.

## 9 Limitations

##### Evaluation Benchmark

In this paper, we evaluate web agents exclusively on WebArena tasks. While WebArena offers realistic and diverse challenges, the number and variety of tasks may be limited. Other benchmarks, such as MiniWoB (Shi et al., ), Mind2Web (Deng et al., ), and VisualWebArena (Koh et al., ), provide alternative evaluation platforms. However, as discussed in Section 2.1, WebArena aligns more closely with real-world scenarios, while VisualWebArena is less applicable to our study because WebArena APIs lack support for interacting with images, a core component of VisualWebArena tasks.

##### API Availability

A key limitation of API-based agents is the inconsistent availability and coverage of APIs across websites. Even platforms with extensive API ecosystems, such as GitLab, may lack support for specific functionalities (e.g., retrieving a user’s official username from a displayed name), leading to edge cases where API-based agents are unable to complete tasks due to incomplete API support. However, advancements in techniques like Automatic Web API Mining (AWM) Wang et al. () could potentially address this limitation by automatically generating APIs for unsupported tasks, reducing reliance on manual API creation.

##### Incorporating APIs

Unlike browsing agents, which can adapt to new websites without manual intervention, the API-based agent requires additional effort to integrate the necessary APIs documentation to the action space of the agent for each website. This manual integration process increases complexity, particularly when the agent must support a wide range of websites, limiting scalability compared to agents that rely solely on web browsing for interactions. However, future advancements in automated API scraping and documentation generation could eliminate this bottleneck, allowing for more scalable and flexible API-based agents.

## 10 Acknowledgement

This work was supported in part by a grant from DSTA Singapore.
The authors would like to thank CMU NeuLab colleagues for their constructive comments.

## Appendix A Appendix

### A.1 WebArena Tasks

WebArena includes the following tasks:

- •
Gitlab – 180 instances: This website simulates tasks related to project management and version control, where agents perform tasks like opening issues, handling merge requests, or creating repositories. Example query: Submit a merge request for a11yproject.com/redesign branch to be merged into markdown-figure-block branch, assign myself as the reviewer.
- •
Map – 109 instances: For this website, tasks are centered around navigation, trip planning and queries about distances, requiring the agent to retrieve and interpret map-based data, similar to using real-world map services like Google map. Example query: Tell me the full address of all international airports that are within a driving distance of 50 km to Carnegie Mellon University.
- •
Shopping – 187 instances: This dataset represents typical e-commerce tasks, such as searching for products, adding items to carts, and processing transactions. Example query: Change the delivery address for my most recent order to 77 Massachusetts Ave, Cambridge, MA.
- •
Shopping Admin – 182 instances: This setting involves managing backend administrative tasks for an online store, like managing product inventories, processing orders, or viewing sales reports. Example query: Tell me the the number of reviews that our store received by far that mention term "satisfied".
- •
Reddit – 106 instances: Tasks here are similar to interactions with the official Reddit, where agents need to post comments, upvote or down-vote posts, or retrieve information from threads. Example query: Tell me the count of comments that have received more downvotes than upvotes for the user who made the latest post on the Showerthoughts forum.
- •
Multi-Website Tasks – 48 instances: These examples involve tasks that span across two websites, requiring the agent to interact with both websites simultaneously, adding complexity to the task. Example query: Create a folder named news in gimmiethat.space repo. Within it, create a file named urls.txt that contains the URLs of the 5 most recent posts from the news related subreddits?

### A.2 API-Based Agent Prompt

### A.3 Hybrid Agent Prompt

### A.4 Breakdown of Steps and Cost by Website

Table 7 shows the breakdown of number of steps and cost by website.

**Table 7: Number of Steps and Cost (in U.S. dollars) of Agents across WebArena Websites**
| Agents | Gitlab | Map | Shopping | Shop-Admin | Reddit | Multi Sites | AVG. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| steps | cost | steps | cost | steps | cost | steps | cost | steps | cost | steps | cost | steps | cost |  |
| Browsing | 9.4 | 0.2 | 8.0 | 0.1 | 7.3 | 0.1 | 7.0 | 0.2 | 11.1 | 0.1 | 7.5 | 0.1 | 8.4 | 0.1 |
| API-Based | 7.0 | 1.7 | 6.6 | 1.1 | 8.2 | 1.0 | 8.4 | 1.1 | 8.8 | 0.6 | 7.7 | 1.6 | 7.8 | 1.2 |
| Hybrid | 8.1 | 2.0 | 9.4 | 1.7 | 8.2 | 1.3 | 9.0 | 1.4 | 10.5 | 1.0 | 8.0 | 1.9 | 8.9 | 1.5 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: Elaborate on the error modes.
soundness: 2
strengths: Overall the notion of API + Web browsing shows lot of promise. The implication of this results also presents interesting questions in terms of development of new websites in the future or existing websites in terms of making themselves more agent friendly in terms of exposing capabilities and APIs for optimal consumption by agents.
summary: The paper presents a hybrid approach that involves use of web browsing and API calling in web agents. The rationale for this approach is that APIs are generally better suited for consumption for agents than the often noisy and expansive HTML DOM structure of webpages. 

The result of the experiment on web-arena indicate having good quality API support enables the agent to make use of the APIs to perform the tasks reasonably well by itself or in combination with web browsing.
weaknesses: In the limitations the authors point that "In our paper we only evaluate web agents on WebArena tasks. The number and diversity of tasks
might be limited. However, to the best of our knowledge, no other real-world web task benchmarks are available at the moment. The tasks we used are the only ones we could find a bemchmark with.". This is not true. There are multiple web benchmarks which are more real-world such as webvoyager, GAIA (a subset of it involves web tasks) and to a limited extend webshop. This is not to say that WebArena evaluation is not valid, i think it is a perfectly reasonable evaluation, however the statement "no other real-world web task benchmarks are available at the moment" is factually not true.
Also not the spelling error in "bemchmark"

Missing analysis: I would also like to see an analysis of the error modes for API and API+Browsing. How many of the tasks could be performed exclusively using APIs and what are the minial number of API calls required for each. How many of these could the agent perform, and for those that the agent could not, what were the common error modes. The authors does mention good quality APIs influence performance ,however this is pretty obvious and generic. A detailed analysis of the error modes would present valuable insights.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How does the hybrid agent decide when to switch between API calls and GUI-based interactions? 

2. How does your agent manage situations with incomplete, undocumented, or frequently changing APIs? Discussing strategies for API discovery, error handling, or adaptation would improve the practicality of your approach.

3. What measures are in place to address security and privacy concerns, such as authentication, rate limiting, and compliance with websites' terms of service? 

Can you share detailed information about the technical implementation of your agents, including architecture specifics, error-handling mechanisms, and integration with existing systems?
soundness: 3
strengths: Originality
The paper introduces a novel approach by incorporating API interactions as an additional action space for AI web agents, which traditionally rely on GUI-based interactions like simulated clicks and typing. By proposing API-based agents and a hybrid model that seamlessly switches between API calls and web browsing, the authors creatively combine existing ideas to address the limitations of current web agents. This innovative problem formulation expands the capabilities of AI agents in interacting with web services and tackles an unstudied area in web task automation.

Quality
The authors provide a robust empirical evaluation of their proposed agents using the WebArena benchmark. The experiments are well-designed to assess the performance across websites with varying levels of API support. The key findings are clearly supported by the data, demonstrating that API-based and hybrid agents consistently outperform traditional GUI-based agents. The paper effectively analyzes the results, highlighting the conditions under which each agent excels, and offers insights into the importance of comprehensive API support.

Clarity
The paper is fairly written. It provides a concise background on the limitations of existing web agents and the motivation for incorporating API interactions. The key contributions are explicitly stated, and the progression from problem statement to conclusion is logical and coherent.

Significance
This work is significant as it addresses a critical gap in the field of AI web agents by leveraging APIs, which are inherently designed for machine interaction. The findings have practical implications for the development of more efficient and accurate web agents capable of handling real-world tasks. By demonstrating that API-based and hybrid agents outperform traditional methods, the paper provides valuable insights that could influence future research and the design of web services. The emphasis on the importance of comprehensive API support underscores a strategic direction for both AI development and web infrastructure enhancement.
summary: The paper introduces an enhanced AI web agent that incorporates API interactions as an additional action space alongside traditional GUI-based interactions. Traditional web agents often rely on simulating human-like actions on graphical user interfaces, which can be inefficient due to the complexity of web pages and limitations in accurately understanding UIs. To address these challenges, the authors develop an API-based agent that interacts directly with web services through API calls, bypassing the need for GUI interaction.

Recognizing that API support varies among websites, the authors also propose a hybrid agent capable of seamlessly switching between API calls and web browsing based on the context. This hybrid approach allows the agent to utilize APIs when available and revert to GUI-based interactions when necessary. The agents are evaluated on the WebArena benchmark, leading to three key findings:
1) API-based agents consistently outperform browsing-based agents on web tasks, regardless of the extent of API support.
2) API-based agents achieve higher success rates on websites with comprehensive API support (e.g., GitLab) compared to those with limited support (e.g., Reddit).
3) Hybrid agents outperform both solely API-based and solely browsing-based agents
weaknesses: My main concern is the insufficient details on the hybrid agent decision Mechanism: The paper does not provide a detailed explanation of how the hybrid agent decides when to switch between API calls and GUI-based interactions. Clarifying the criteria or algorithms used for this decision-making process is crucial for understanding the agent's functionality and for others to replicate or build upon the work. I personally did not understand how it works in practice. In addition, the paper provides limited technical details on the implementation of the API-based and hybrid agents. For instance, information about the architecture, error handling, and integration with existing systems is sparse. Including more implementation specifics would improve the clarity and allow for better assessment of the work's feasibility and scalability.

Evaluation Scope and Generalizability: The experiments are conducted solely on the WebArena benchmark, which may not cover a sufficiently diverse set of websites and tasks to demonstrate the agent's general applicability. Expanding the evaluation to include a wider variety of websites with different levels of API support and varying complexities would provide stronger evidence of the agent's effectiveness and robustness (See WorkArena, WorkArena++, ST-WebAgentBench, WebCanvas).

Dependence on API Availability and Quality: The proposed approach relies heavily on the availability and comprehensiveness of APIs, which can vary widely across websites. The paper does not address how the agent handles incomplete, undocumented, or changing APIs. Discussing strategies to mitigate these issues, such as API discovery or adaptation mechanisms, would enhance the practicality of the approach.

Security and Ethical Considerations: Direct interaction with web service APIs raises potential security and privacy concerns, such as authentication management, rate limiting, and compliance with terms of service. The paper lacks a discussion of these challenges and does not propose solutions to ensure that the agent operates securely and ethically. Addressing these concerns is important for real-world deployment.

Performance Metrics and Statistical Analysis: The evaluation primarily reports success rates without sufficient analysis of other important performance metrics such as execution time, resource utilization, or learning efficiency. Additionally, the paper does not mention whether the results are statistically significant.

Adaptability to Web Changes: Websites frequently update their interfaces and APIs, which can break automation scripts. The paper does not discuss how the agent adapts to such changes over time. Exploring methods for the agent to detect and adjust to updates would improve its long-term usefulness.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - A baseline web browsing agent is described in Sec 2.2. This is very little information. There is great amount of work on developing SOTA web browsing agent and full technical reports on that. One of those should be baseline or a description of such a baseline agent shouldn't be possible in 2 paragraphs. Can you please provide more information on the baseline agent OR why one of the SOTA ones is NOT used for the baseline results? Some known agents are WebArena agent, STeP agent, Agent-E, AgentOccam, to name a few.
soundness: 2
strengths: The biggest and (unfortunately) only strength of this paper is the novelty in proposing to use web and API-based agents to highlight the need to use API-based alternative when available and not ONLY rely on web browsing agents alone. This is great idea in practice since it allows for realistic use of the new wave of web agents.
summary: This paper proposes to use API-based agents to complement web-only browser based agents for completing web tasks. Using web-arena benchmark which is a simulated web environment, they evaluate how an API-calling agent would perform with focus on highlighting how it has complementary strengths and can outperform a baseline web browsing agent. Finally a hybrid agent is proposed that combines the two agents together to hopefully have the best performance of both the agents.
weaknesses: There are three major weaknesses of the paper as mentioned below along with evidence.
1. Incomprehensive literature survey
- API Agents are not new as mentioned in line 51 on page 1; there has been a lot of work on it such as the ToolBench's ToolLlama, AnyTool's hierarchical agent, etc.
- Sec 9 in page 10 claims that there are no other real-world web task benchmarks available but there are several of them: WebVoyager, WebShop, WorkBench, to name a few.


2. Poorly written and shows that it was not proof-read and submitted in haste.
- Line 047, page 1: "Nonetheless, However, ..."
- Line 083, page 2: "that combining" -> "that combines"
- Line 134, page 3: "withe"?
- Line 157, page 3: "Fig 2" -> "Fig 1"
- Line 189, page 4: "see section ??"
- Table 2 caption, page 8: "Each row columns sums up to 1."-> "Each column sums up to 100"

3. The most important one: imperfect representation/description of the results, which doesn't allow us to be confident of the findings.
- Table 1 description in Sec 6.1 line 375 mentions that the API-based Agent achieves higher scores in all websites compared to the Browsing agent, which is NOT true as can be seen in the results of Shop-Admin and Multi Sites. 
- Lines 401-4-3, page 8 say that the browsing agent achieved its best scores on Gitlab and Map, which is not true again. While the sentiment is agreed that it performs poor, it is not represented correctly.
- Line 404, page 8: Hybrid agent did NOT outperform other agents in all categories. In fact, the SteP agent outperforms the hybrid agent in Shopping, Reddit and also on the average!
- Line 423, page 8, "Steps" results description of Table 4 states that "browsing agent consistently takes more steps to complete tasks compared to boath the API-based and hybrid agents". This is false as the table shows that Browsing agent takes least steps than the other two agents for Shopping, Shop-Admin and Multi-Sites.
- Lines 478-479, page 8: This is some leakage that needs to be addressed as adding new APIs are going to obviously help the API-based agents. Also, the numbers are claimed to improved from 9.43% to 14.15% for reddit, but none of the tables show a 9.43% for reddit for any agent.
- WebArena is arguable to be a "realistic" benchmark as WebVoyager is a more realistic one where it is not simulated.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Invoking web APIs to collect data is a standard use of web APIs. Given this, what is the novelty of the paper? What are the research challenges of including the use of web APIs in a web browsing agent. 
2. The proposed two phase documentation retrieval seems to be straightforward and lacks intelligence on interacting with API endpoints. It is unclear how to provide the list of available API endpoints given the prompt of a task? What if there is no readme document or the document is not complete or accurate? Is the agent only designed for RESTful APIs? How about SOAP-based and GraphQL APIs?
3. How to deal with practical issues of using web APIs listed in the weaknesses section?
4. The paper deals with brief and incomplete API documentation by using GPT-4 to generate it. But how reliable the GPT-4  generation would be and what the benefits of using them are unclear. 
5. The paper should discuss more about the limitations of the work, in which scenarios human interference is unavoidable, and how to minimize it. 
6. The paper compares the proposed approach with StepP, AutoEval, AWM. The paper should describe each work a bit in terms of the methods they propose and why they are selected. The reference items of AutoEval and AWM do not have the journal/conference information so it is unclear whether/where they are published. The paper should also explain why the browsing agent works much less effectively than these three baselines.
soundness: 2
strengths: This paper explores the use of web APIs for web-browsing tasks and studies the performance and cost of different settings. 
This paper utilizes the current LLM tools such as ChatGPT, CodeAct, to minimize human interference.
summary: This paper proposes a study on leveraging web APIs to improve the performance of web browsing agents. It compares the performance of three settings, web-browsing only, API only, and hybrid ones on WebArena, a recent benchmark for web-based tasks.
weaknesses: The major issue of this paper is that it lacks research depth and novelty. Web APIs have been widely used to facilitate the access to online resources, especially data related resources. This paper follows the idea and makes a simple, straightforward attempt of using web APIs if they are available to collect information from websites. No research challenges are identified and the methods are not optimized.

The approach may work in a small-scoped and closed system with the assumption of prior knowledge of the API endpoints and their documentation formats. It is work-in-progress and yet to explore and address many practical problems of leveraging web APIs in an open-end environment. Those problems could be caused due to the lack of systematic supports on locating API endpoints, understanding the semantics of API description, decomposing a task into the invocations of multiple APIs, fixing potential quality issues of codes generated by CodeActAgent, fusing results from multiple APIs, etc.