OpenWebAgent: An Open Toolkit to Enable Web Agents on Large Language Models (2024)

Abstract: We introduce OpenWebAgent, an open toolkit designed to optimize web automation by integrating both large language models (LLMs) and large multimodal models (LMMs). This toolkit focuses on enhancing human-computer interactions on the web, simplifying complex tasks through an advanced HTML parser, a rapid action generation module, and an intuitive user interface. At the core of OpenWebAgent is an innovative web agent framework that uses a modular design to allow developers to seamlessly integrate a variety of models and tools to process web information and automate tasks on the web. This enables the development of powerful, task-oriented web agents, significantly enhancing user experience and operational efficiency on the web. The OpenWebAgent framework, Chrome plugin, and demo video are available at https: //github.com/THUDM/OpenWebAgent/.

heading: 1 Introduction
text: As the Internet becomes an integral part of everyday life, the complexity of tasks that users want to automate on web platforms continues to grow (Van der Aalst et al., 2018). Modern web users expect interfaces that are not only intuitive and visually appealing but also capable of intelligent (Davenport and Kirby, 2016), predictive interactions that streamline complex tasks.
Traditional web automation tools, such as Robotic Process Automation (RPA) tools, have been instrumental in reducing manual effort (Syed et al., 2020), but fall short in areas such as contextual understanding, flexibility (Hallikainen et al., 2018), and user accessibility. These tools often require complex setups and significant technical expertise, limiting their usefulness to a narrow audience. In addition, the lack of a unified system
*Equal contribution. †Work done while these authors interned at Zhipu AI. ‡Corresponding Authors: YD and JT.
architecture among existing tools poses significant challenges for developers seeking to integrate or innovate on top of these platforms, hindering the advancement of web automation technologies.
Nevertheless, the field has advanced significantly with deep learning technologies, especially after Google introduced the Transformer model (Vaswani et al., 2017). The capabilities of large-scale pre-trained models, like OpenAI’s GPT series (Achiam et al., 2023; OpenAI, 2024a), have improved text generation, semantic understanding, and logical reasoning (Brown et al., 2020; Chan et al., 2022). This has made web automation tools using LLMs and LMMs more feasible. Recent work such as AutoWebGLM (Lai et al., 2024), shows LLMs can handle various web tasks but the lack of multimodal inputs limits their capabilities, highlighting the need for multimodal web agents.
Presented System. We present OpenWebAgent, a toolkit for advanced web interactions with innovative modules that allow developers to integrate any language or multimodal model for web automation. Figure 1 shows the task execution results of OpenWebAgent System with GPT-4 (Achiam et al., 2023) and AutoWebGLM (Lai et al., 2024) as the action generation models. OpenWebAgent includes an interactive web plugin and a modularly designed server, allowing it to execute tasks directly and autonomously on webpages while providing real-time feedback. It stands out for its efficiency and user accessibility compared to other web automation tools, thanks to these features:
• High-Performance HTML Parser. This parser optimizes performance by simplifying complex HTML into a more straightforward format (§3.1), enabling OpenWebAgent to process web content with enhanced accuracy and speed. It reduces HTML length by 99% and the number of elements by 97% (§5.1), ensuring efficient operation on any website. (§5.2)
72
• Modular System Design. OpenWebAgent integrates multimodal inputs such as action history, parsed HTML and screenshots for LLMs and LMMs to create coherent action plans that match user intent. Users can modify, pause, or reset tasks at any time (§4.2), offering flexibility and ease of use. The modular design (§3.2) allows easy model integration and module replacement by developers.
• Streamlined User Interface. The plugin requires no complex setup and is ready to use immediately after download. Its simple and attractive interface (§4.1) lets users track the process and sequence of operations easily, with task execution controlled by a few simple buttons, ensuring efficiency and ease of use.
These innovations place OpenWebAgent at the forefront of web automation technology. Through
its advanced capabilities, OpenWebAgent is not just a tool but a paradigm shift in how humans interact with and harness the power of the web for automated tasks.
Contributions. (1) We implement a powerful HTML parser engine that simplifies complex webpages into a more accessible format. (2) We design a ready-to-use web plugin automation tool that enables users to perform any desired action on any webpage. (3) We create a versatile web agent framework, allowing developers to easily integrate any LLM or LMM for web automation. This framework offers unprecedented ease in developing robust, task-oriented web agents. In summary, OpenWebAgent contributes both technically and conceptually to the understanding of the boundaries of human-machine collaboration and giving an attempt to develop the future of web automation.

heading: 2 Related Work
text: Developing an efficient toolkit for web browsing agents is challenging, especially in integrating decision language models with diverse modules for processing webpage information. This section provides an overview of related research.
Language Models (LMs). Since Google proposed Transformer (Vaswani et al., 2017), Large Language Models (LLMs) have evolved rapidly. Notable models include OpenAI’s GPT-4 (Achiam et al., 2023), Google’s Gemini (Google, 2023), PaLM-2 (Chowdhery et al., 2023), and Gopher (Rae et al., 2021), Anthropic’s Claude-2 (Anthropic, 2023), Meta’s LLaMA series (Touvron et al., 2023a,b) and OPT (Zhang et al., 2022), as well as Mistral’s Mixture of Experts models (Jiang et al., 2024). Other notable contributions include GLM-130B (Zeng et al., 2022) and BLOOM (Scao et al., 2022). These models, pre-trained on vast datasets, excel in various NLP tasks.
Large Multimodal Models (LMMs) have become the primary focus of research to address a wider range of tasks. OpenAI led the way by launching high-performance models such as GPT4-Turbo (OpenAI, 2024b) and GPT-4o (OpenAI, 2024a). The open-source community has also introduced multimodal models like LLaVA (Liu et al., 2023a), CogVLM (Wang et al., 2023), and QwenVL (Bai et al., 2023). These LMMs have inspired new approaches in Agent research.
Smaller, cost-effective models are preferred due to the high deployment costs of large models, with users prioritizing task-specific effectiveness. Opensource projects like LLaMA3-8B (Meta, 2024), Vicuna-7B (Chiang et al., 2023), and ChatGLM49B (GLM et al., 2024) show comparable capabilities to larger models in some areas.
Web Automation Systems. Previous projects, such as WebGPT (Nakano et al., 2021) and WebGLM (Liu et al., 2023b), have effectively integrated language models with web environments, mainly for question-answering tasks using internet data. These models are excellent at information retrieval for QA (Rajpurkar et al., 2016; Nguyen et al., 2016; Berant et al., 2013; Kwiatkowski et al., 2019), but they cannot perform complex or interactive web-based tasks.
Recent projects like AutoGPT1 use multiple ChatGPT agents for self-prompting and web operations through a plan-execute-reflect cycle. The GPT-4V-ACT framework2 uses the Set-of-Mark method (Yang et al., 2023) to mark screenshots and then employs GPT-4V (OpenAI, 2024b) to generate operations, but it struggles with real web pages due to insufficient operation instructions. AutoWebGLM (Lai et al., 2024) is based on the fine-tuned ChatGLM3-6B (GLM et al., 2024) model. However, it lacks image input data, limiting its performance in real-world web scenarios.
Other initiatives such as MindAct (Deng et al., 2023) involve extensive interactions to select webpage elements, suggesting a need for more efficient processes. CC-Net (Mishra et al., 2019) leverages a vast visual data set and learning techniques to manipulate web components effectively. Conversely, CogAgent (Hong et al., 2023) focuses on using visual input to generate web operation methods, while WebAgent (Gur et al., 2024) utilizes HTML-T5 and the large-scale Flan-U-Plam model (Chung et al., 2022) to control webpages, though the model’s size limits its deployment.
1https://github.com/Significant-Gravitas 2https://github.com/ddupont808/GPT-4V-Act

heading: 3 The OpenWebAgent System
text: OpenWebAgent is rooted in principles of intuitive design, flexibility, and comprehensive functionality, aiming to provide a user-centric approach to web automation. The system is inherently adaptable, and designed to allow users to easily customize it for a range of complex automation tasks. Its main objective is to develop a toolkit that is more responsive and goes beyond the limitations of traditional web agents. This toolkit does more than execute user tasks with simple pre-defined commands; it is engineered to fully analyze, decompose, and process each task to ensure thorough and efficient automation.

heading: 3.1 HTML Parsing Techniques
text: The content of HTML webpages is intricate and complex. Therefore, it should be effectively simplified before being fed into the parsing model.
Simplification aims to distill the most important information while eliminating excessive or disruptive elements that could make it difficult for the model to understand. It is crucial to maintain the basic structure of HTML and its essential content information during this process. This ensures that the model can understand and use these details for efficient webpage parsing.
Using algorithm 1 can effectively transform the element tree into a more concise representation. We can judge whether an element should be retained by determining the clickability of the element, noting that nodes near the retained element are generally able to provide more useful information and therefore have a higher retention value. Therefore, we can adopt a recursive approach to obtain the ancestor nodes, child nodes and sibling nodes of the retained element part. Finally, pruning can be done according to the information content starting from the leaf nodes.
With the processing algorithm described above, the complex HTML can be simplified into a format that is easier for the model to interpret and manipulate, thus improving the model’s performance in web parsing tasks.

heading: 3.2 Interaction Workflow
text: OpenWebAgent’s design philosophy and objectives aim to achieve a harmonious balance between advanced technological capabilities and user-friendly interaction, redefining standards for human-computer interaction in web automation. To
Algorithm 1: HTML Simplifier Data: dom tree tree, neighbor coefficient n Result: pruned tree tree, kept elements kp nodes, kp← set(), list() for e in tree.element do // selector
if not (onTop(e) and onScreen(e)) then continue if isClickableTag(e.tagname) or haveJSaction(e.attrib) or e.cursor = pointer or e.classes.include(button) then
kp.push(e) nodes.push(e) nodes.push(getNbr(e, n))
end end for e in reversed(tree) do // pruner
if not e in nodes or not (e has text or attrib or e is root or len(e.children) > 1) then
tree.remove(e) end
end
improve the flexibility and usability of our toolkit, we modularize it into several key components as shown in Figure 2. The network processing module and the action generation module are deployed as unified services in the backend. Meanwhile, the process control module and the execution module are integrated into the plugin.
Web Processing Module. This module extracts useful elements of HTML, simplifies HTML input, performs OCR on screenshots, and adds element labels to screenshots. See §3.1 for details of the methods and processes.
Action Generation Module. The main purpose of this module is to predict the next action based on the user’s task and the current webpage context. At this stage, we provide a prompt for the LLM that includes the current task, the simplified HTML of the webpage, and previous command sequences, and for the LMM, we also provide labeled screenshots for the model to use. The model outputs the next action in natural language, which we match against a pre-defined action space, and returns the action name and parameters if the match is successful.
In this module, models are accessed through interfaces, which means that developers creating new web agents can effortlessly integrate any model
into our toolkit by simply setting up an API interface for accessing the model. It is important to note that we have reserved an interface to a visual processing module in the toolkit, located between the web processing module and the action generation module, to serve better the needs of developers working with multimodal agents.
To address the lack of image information in the language-based agent, the module is pre-configured with an OCR interface that takes a screenshot of a webpage as input and returns the webpage information contained in the screenshot. LMM agent developers have the option to replace the preconfigured modules and integrate their own vision modules into our toolkit, which simplifies the development of multimodal web agents.
Execution Module. This module is used to execute specific action instructions on a webpage. When the module receives an action instruction from the action generator module, it looks for the element that actually needs to be operated on and then executes the action on the webpage using a predefined script. When the action is completed, it provides a response feedback to ensure that the model is aware of the execution of the action to adjust the plan.
Process Control Module. As shown in Figure 3, this module is implemented using a finite state machine. The main purpose of this is to coordinate the execution of tasks and to facilitate the transfer of information between the above modules.
This module serves as the primary interface for user interaction. It receives various inputs, such as user task commands and control commands (e.g. start, pause, reset). The module also records user input tasks and previous action history. When the user issues a start command, the module first fetches the HTML source and screenshot of the current webpage, and information about the clickable elements, and passes them to the web processing module (including the visual processing module). Additionally, the module sends the task and action history to the action generation module.
The module sends task and action history to the action generation module and waits for a response from the action generation module. It then parses the action into various parameters. If a valid web action returns, its details are sent to the execution module. Once a response arrives from the execution module, the action history updates and the process of retrieving webpage information repeats.
When the process is complete, the user is notified by the module.
This workflow enables real-time user interaction, users can send control commands such as pause, reset, or update their task description at any time. The process control module adapts accordingly, ensuring flexible and efficient interaction.

heading: 4 Demonstration

heading: 4.1 System Interface
text: The plugin interface is simple and easy to use, as shown in Figure 4. It consists of three main parts: • Input Box: For entering tasks to be executed. • Control Buttons: For managing task execution,
starting with “ run” and “⟳ reset” . During execution, “ q pause” replaces “ run” , allowing the user to control the process.
• Feedback Panel: Shows the executed actions and the model’s responses.

heading: 4.2 Usage Example
text: As shown in Figure 5, we illustrate the interaction process and execution results of our toolkit by integrating GPT-4 into our toolkit and executing a sample task “What is the weather like today?”.
Task Execution. The task begins with Google, a standard browser homepage that does not provide weather information. First, we can put the query “What is the weather like today?” into the task box and click the “ run” button to initiate the task. A
loading icon will appear on the feedback box to indicate that our plugin has initiated the retrieval of information from the webpage. After a brief interval, the plugin generates the instruction, “#Type# 5 weather today”. The webpage displays that “weather today” has been entered into the input box, thereby suggesting that the action has been executed successfully.
Task Management. After several actions on the webpage, the user is presented with a weather forecast, as depicted in Figure 5(b). At this juncture, the user has the option to pause the task by clicking the “ q pause” button. They can then update the task instructions to “What is the weather on Sunday?” before resuming execution by clicking “ run” (which serves as “continue” here). The plugin will adapt to the modified user task and modify the execution flow accordingly.
Task Completion. Following a series of actions on the webpage, the LLM (GPT-4) can complete the user’s task based on the information available. Our plugin then responds to the user’s query by returning a message, “Answer: 57°F / 33°F”, in the feedback box and completes the process.
Figure 6 shows how OpenWebAgent with AutoWebGLM handles a restaurant reservation task. This demonstrates that our framework provides the necessary information and actions to support the
model in performing complex tasks. In addition, as shown in Figure 1, our plugin can also perform various web tasks, such as shopping, socializing and information seeking to satisfy users’ diverse web browsing needs. This proves that our plugin has the following characteristics: • Flexibility: Users can use our plugin to accom-
plish various tasks on any webpage, in any state, anytime, anywhere.
• Efficiency: Each module in our plugin is optimized for performance, and the time taken for each step of the action depends largely on the time taken to call LLM. Therefore, our plugin executes extremely efficiently.
• Robust Interactivity: Our plugin receives user interaction at any moment during execution. Users can receive feedback and take control in real time.

heading: 5 Evaluation

heading: 5.1 HTML Parser Performance
text: Experimental setup. We selected six categories of frequently visited websites from Similarweb3 and randomly tested the effectiveness of the parser by selecting dozens of pages from each website, and the results are shown in Table 1.
3https://www.similarweb.com/top-websites
Results Analysis. The HTML simplifier effectively reduces the complexity of web pages across various websites. It significantly reduces the number of actionable elements and the length of HTML text, with simplification rates exceeding 97% and 99% respectively. The tool operates quickly, even in dense environments like Facebook, with an average processing time of just 5.83 milliseconds. This rapid performance demonstrates the tool’s practicality for real-world applications, enabling quicker and more focused web interactions by emphasizing essential content.

heading: 5.2 Efficiency
text: Experimental setup. Following the HTML Parser Performance testing methodology, we selected 12 websites from SimilarWeb. The system was assigned 80 web navigation tasks across these diverse websites. Throughout these tasks, we meticulously recorded the response times of different components. The outcomes of this evaluation are presented in Table 2.
Results Analysis. The results indicate that network transmission time makes up 70% of the system’s operational time, primarily due to connections to remote servers. Additionally, the model’s predic-
tion activities, particularly with the GPT-4-turbo model, account for 23% of the runtime. To improve efficiency, future enhancements should focus on optimizing web page transmission, such as by locally simplifying web pages to reduce data volume by 99%, thus saving time and enhancing performance.

heading: 6 Conclusion
text: OpenWebAgent represents a paradigm shift in webbased human-computer interaction. It promises to improve user experience and productivity by automating a variety of web tasks efficiently and intuitively. It provides a convenient framework for the development of web agents based on large language models (LLMs) and large multimodal models (LMMs) through advanced HTML parsing capabilities, a modularly designed system, a friendly user interface, and the visualization of the task execution process.
Limitations
While OpenWebAgent boasts remarkable capabilities, it also has its limitations: • Its performance may falter on complex or un-
conventional webpages, as it depends on understanding web structures.
• The tool is intended for general purposes and might not perform optimally for tasks that require specialized knowledge.
• The capacity of OpenWebAgent to execute web page operations is substantially influenced by the capabilities of the underlying model, including the ability to comprehend web page elements and perform image recognition.
• Although OpenWebAgent is currently efficient, its backend design needs to be improved to meet the needs of large-scale applications and faster web operations response. Future developments will address these limi-
tations and improve its applicability and performance.
Ethics Statement
Intended Use. OpenWebAgent is designed to assist users in automating web browsing tasks.
Potential Misuse. OpenWebAgent can perform tasks on the World Wide Web, however, we cannot prevent users from using this tool for network attacks, such as social forum spamming, DDoS attacks, or unauthorized data extraction.

heading: Acknowledgements
text: This work is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD0118600, Natural Science Foundation of China (NSFC) 62276148 and 62425601, the New Cornerstone Science Foundation through the XPLORER PRIZE.

heading: Future Work
text: Future developments will address these limi-
tations and improve its applicability and performance