OpenHands: An Open Platform for AI Software Developers as Generalist Agents

## Abstract

Abstract Software is one of the most powerful tools that we humans have at our disposal; it allows a skilled programmer to interact with the world in complex and profound ways.
At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments.
In this paper, we introduce OpenHands ( f.k.a. OpenDevin), a platform for the development of powerful and flexible AI agents that interact with the world in similar ways to those of a human developer: by writing code, interacting with a command line, and browsing the web.
We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, coordination between multiple agents, and incorporation of evaluation benchmarks.
Based on our currently incorporated benchmarks, we perform an evaluation of agents over 15 challenging tasks, including software engineering ( e.g ., Swe-bench ) and web browsing ( e.g ., WebArena ), among others.
Released under the permissive MIT license, OpenHands is a community project spanning academia and industry with more than 2.1K contributions from over 188 contributors. Code https://github.com/All-Hands-AI/OpenHands Benchmark https://hf.co/spaces/OpenHands/evaluation Slack http://bit.ly/OpenHands-Slack

## 1 Introduction

Powered by large language models (LLMs; OpenAI ; Team et al. ; Jiang et al. ; Chang et al. ), user-facing AI systems (such as ChatGPT) have become increasingly capable of performing complex tasks such as accurately responding to user queries, solving math problems, and generating code.
In particular, AI *agents*, systems that can perceive and act upon the external environment, have recently received ever-increasing research focus.
They are moving towards performing complex tasks such as developing software (Jimenez et al., ),
navigating real-world websites (Zhou et al., ), doing household chores (Ahn et al., ), or even performing scientific research (Boiko et al., ; Tang et al., ).

As AI agents become capable of tackling complex problems, their development and evaluation have also become challenging.
There are numerous recent efforts in creating open-source frameworks that facilitate the development of agents (Hong et al., ; Chen et al., ; Wu et al., ).
These agent frameworks generally include: 1) interfaces through which agents interact with the world (such as JSON-based function calls or code execution), 2) environments in which agents operate, and 3) interaction mechanisms for human-agent or agent-agent communication.
These frameworks streamline and ease the development process in various ways (Tab. 1, §C).

When designing AI agents, we can also consider how *human* interacts with the world.
The most powerful way in which humans currently interact with the world is through *software* – software powers every aspect of our life, supporting everything from the logistics for basic needs to the advancement of science, technology, and AI itself.
Given the power of software, as well as the existing tooling around its efficient development, use, and deployment, it provides the ideal interface for AI agents to interact with the world in complex ways.
However, building agents that can effectively develop software comes with its own unique challenges.
How can we enable agents to effectively *create and modify code in complex software systems*?
How can we provide them with tools to *gather information on-the-fly* to debug problems or gather task-requisite information?
How can we ensure that development is *safe and avoids negative side effects* on the users’ systems?

In this paper, we introduce OpenHands (*f.k.a.* OpenDevin), a community-driven platform designed for the development of generalist and specialist AI agents that interact with the world through software.(^1^11While initially inspired by AI software engineer Devin (), OpenHands has quickly evolved to support much wider range of applications beyond software engineering through diverse community contributions.)
It features:

- (1)
An interaction mechanism which allows user interfaces, agents, and environments to interact through an *event stream* architecture that is powerful and flexible (§2.1).
- (2)
A runtime environment that consists of a docker-sandboxed operating system with a bash shell, a web browser, and IPython server that the agents can interact with (§2.2).
- (3)
An interface allowing the agent to interact with the environment in a manner similar to actual software engineers (§2.3). We provide the capability for agents to a) create and edit complex software, b) execute arbitrary code in the sandbox, and c) browse websites to collect information.
- (4)
Multi-agent delegation, allowing multiple specialized agents to work together (§2.4).
- (5)
Evaluation framework, facilitating the evaluation of agents across a wide range of tasks (§4).

Figure: Figure 1: OpenHands User Interface (UI, §D) allows users to view files, check executed bash commands/Python code, observe the agent’s browser activity, and directly interact with the agent.
Refer to caption: x3.png

Importantly, OpenHands is not just a conceptual framework, but it also includes a comprehensive and immediately usable implementation of agents, environments, and evaluations.
As of this writing, OpenHands includes an agent hub with over 10 implemented agents (§3), including a strong generalist agent implemented based on the CodeAct architecture (Wang et al., ), with additions for web browsing () and code editing specialists (Yang et al., ).
Interaction with users is implemented through a chat-based user interface that visualizes the agent’s current actions and allows for real-time feedback (Fig. 1, §D).
Furthermore, the evaluation framework currently supports 15 benchmarks, which we use to evaluate our agents (§4).

Released under a permissive MIT license allowing commercial use,
OpenHands is poised to support a diverse array of research and real-world applications across academia and industry.
OpenHands has gained significant traction, with 32K GitHub stars and more than 2.1K contributions from over 188 contributors.
We envision OpenHands as a catalyst for future research innovations and diverse applications driven by a broad community of practitioners.

## 2 OpenHands Architecture

Figure: Figure 2: OpenHands consists of 3 main components: 1) Agent abstraction where community can contribute different implementation of agents (§2.1) into agenthub (§3); 2) Event stream for tracking history of actions and observations; 3) Runtime to execute all actions into observations (§2.2).
Refer to caption: x4.png

We next describe using OpenHands in detail. In particular, we discuss 1) how to define and implement an agent (§2.1), 2) how each action execution leads to an observation (§2.2), 3) how to reliably manage and extend commonly used skills for agents (§2.3), and 4) how to compose multiple agents together for task solving (§2.4).
Fig. 2 provides an overview.

### 2.1 Agent Definition and Implementation

An agent can perceive the state of the environment (*e.g*., prior actions and observations) and produce an action for execution while solving a user-specified task.

The State and Event Stream. In OpenHands, the state is a data structure that encapsulates all relevant information for the agent’s execution. A key component of this state is the event stream, which is a chronological collection of past actions and observations, including the agent’s own actions and user interactions (*e.g*., instructions, feedback). In addition to the event stream, the state incorporates auxiliary information for agent’s operation, such as the accumulative cost of LLM calls, metadata to track multi-agent delegation (§2.4), and other execution-related parameters.

Actions. Inspired by CodeAct (Wang et al., ), OpenHands connects an agent with the environment through a core set of general actions.
Actions IPythonRunCellAction and CmdRunAction enable the agent to execute arbitrary Python code and bash commands inside the sandbox environment (*e.g*., a securely isolated Linux operating system).
BrowserInteractiveAction enables interaction with a web browser with a domain-specific language for browsing introduced by BrowserGym (Drouin et al., ).
These actions were chosen to provide a comprehensive yet flexible set of primitives covering most tasks performed by human software engineers and analysts.
The action space based on programming languages (PL) is powerful and flexible enough to perform any task with tools in different forms (*e.g*., Python function, REST API, *etc*.) while being reliable and easy to maintain (Wang et al., ) .

Figure: Figure 3: Minimal example of implementing an agent in OpenHands.

This design is also compatible with existing tool-calling agents that require a list of pre-defined tools (Chase, ). That is, users can easily define tools using PL supported in primitive actions (*e.g*., write a Python function for calculator) and make those tools available to the agent through JSON-style function-calling experiences (Qin et al., ).
Moreover, the framework’s powerful PL-based primitives further make it possible for the agents to create tools by themselves (*e.g*., by generating Python functions, Yuan et al. ) when API to complete the task is unavailable.
Refer to §2.3 for how these core PL-based actions can be composed into a diverse set of tools.

Observations. Observations describe the environmental changes (*e.g*., execution result of prior actions, text messages from the human user *etc*.) that the agent observes.

Implement a New Agent.
The agent abstraction is designed to be simple yet powerful, allowing users to create and customize agents for various tasks easily.
The core of the agent abstraction lies in the step function, which takes the current state as input and generates an appropriate action based on the agent’s logic. Simplified example code for the agent abstraction is illustrated in Fig. 3.
By providing this abstraction, OpenHands allows the users to focus on defining desired agent behavior and logic without worrying about the low-level details of how actions are executed (§2.2).

### 2.2 Agent Runtime: How Execution of Actions Results in Observations

Agent Runtime provides a general environment that equips the agent with an action space comparable to that of human software developers, enabling OpenHands agents to tackle a wide range of software development and web-based tasks, including complex software development workflows, data analysis projects, web browsing tasks, and more.
It allows the agent to access a bash terminal to run code and command line tools, utilize a Jupyter notebook for writing and executing code on-the-fly, and interact with a web browser for web-based tasks (*e.g*., information seeking).

Docker Sandbox. For each task session, OpenHands spins up a securely isolated docker container sandbox, where all the actions from the event stream are executed.
OpenHands connects to the sandbox through a REST API server running inside it (i.e., the OpenHands action execution API), executes arbitrary actions (e.g., bash command, python code) from the event stream, and returns the execution results as observations.
A configurable workspace directory containing files the user wants the agent to work on is mounted into that secure sandbox for OpenHands agents to access.

OpenHands Action Execution API. OpenHands maintains an API server that runs inside the docker sandbox to listen for action execution requests from the event stream. The API server maintains:

- (1)
A bash shell that connects with the operating system environment (specified by the docker image) for command execution.
- (2)
A Jupyter IPython server to handle interactive *python* () code execution requests and return the execution results back to the event stream.
- (3)
A Chromium browser based on . The provider provides a set of action primitives defined by BrowserGym (; Drouin et al., ), such as navigation, clicking, typing, and scrolling. The full set of actions is detailed in §J.
After executing these actions, the browser runtime provides a rich set of observations about the current state of the browser, including HTML, DOM, accessibility tree (), screenshot, opened tabs, *etc*.

Arbitrary Docker Image Support. OpenHands allows agents to run on arbitrary operating systems with different software environments by supporting runtime based on arbitrary docker images.
OpenHands implements a build mechanism that takes a user-provided arbitrary docker image and installs OpenHands action execution API into that image to allow for agent interactions.
We include a detailed description of OpenHands agent runtime in §F.

### 2.3 Agent Skills: The Extensible Agent-Computer Interface

SWE-Agent (Yang et al., ) highlights the importance of a carefully crafted Agent-Computer Interface (ACI, *i.e*., specialized tools for particular tasks) in successfully solving complex tasks.
However, creating, maintaining, and distributing a wide array of tools can be a daunting engineering challenge, especially when we want to make these tools available to different agent implementations (§3).
To tackle these, we build an AgentSkills library, a toolbox designed to enhance the capabilities of agents, offering utilities not readily available through basic bash commands or python code.

Easy to create and extend tools. AgentSkills is designed as a Python package consisting of different utility functions (*i.e*., tools) that are automatically imported into the Jupyter IPython environment (§2.2).
The ease of defining a Python function as a tool lowers the barrier for community members to contribute new tools to the library.
The generality of Python packages also allows different agent implementations to easily leverage these tools through one of our core action IPythonRunCellAction (§2.1).

Rigorously tested and maintained.
We follow best practices in software engineering and write extensive unit tests for tools in AgentSkills to ensure their reliability and usability.

Inclusion criteria and philosophy.
In the AgentSkills library, we do not aim to wrap every possible Python package and re-teach agents their usage (*e.g*., LLM already knows pandas library that can read CSV file, so we don’t need to re-create a tool that teaches the agent to read the same file format).
We only add a new skill when: (1) it is not readily achievable for LLM to write code directly (*e.g*., edit code and replace certain lines), and/or (2) it involves calling an external model (*e.g*., calling a speech-to-text model, or model for code editing ()).

Currently supported skills.
AgentSkills library includes file editing utilities adapted from SWE-Agent (Yang et al., ) and Aider () like edit_file, which allows modifying an existing file from a specified line; scrolling functions scroll_up and scroll_down for viewing a different part of files.
It also contains tools that support reading multi-modal documents, like parse_image and parse_pdf for extracting information from images using vision-language models (*e.g*., GPT-4V) and reading text from PDFs, respectively.
A complete list of supported skills can be found in §I.

### 2.4 Agent Delegation: Cooperative Multi-agent Interaction

OpenHands allows interactions between multiple agents as well. To this end, we use a special action type AgentDelegateAction, which enables an agent to delegate a specific subtask to another agent.
For example, the generalist CodeActAgent, with limited support for web-browsing, can use AgentDelegateAction to delegate web browsing tasks to the specialized BrowsingAgent to perform more complex browsing activity (*e.g*., navigate the web, click buttons, submit forms, *etc*.).

Table: Table 1: Comparison of different AI agent frameworks (§C). Swe refers to ‘software engineering’. Standardized tool library: if framework contains reusable tools for different agent implementations (§2.3); Built-in sandbox & code execution: if it supports sandboxed execution of arbitrary agent-generated code; Built-in web browser: if it provides agents access to a fully functioning web browser; Human-AI collaboration: if it enables multi-turn human-AI collaboration (*e.g*., human can interrupt the agent during task execution and/or provide additional feedback and instructions); AgentHub: if it hosts implementations of various agents (§3); Evaluation Framework: if it offers systematic evaluation of implemented agents on challenging benchmarks (§4); Agent QC (Quality Control): if the framework integrates tests (§E) to ensure overall framework software quality.

## 3 AgentHub: A Hub of Community-Contributed Agents

Based on our agent abstraction (§2.1), OpenHands supports a wide range of community-contributed agent implementations for end users to choose from and act as baselines for different agent tasks.

CodeAct Agent. CodeActAgent is the default generalist agent based on the CodeAct framework (Wang et al., ). At each step, the agent can (1) converse to communicate with humans in natural language to ask for clarification, confirmation, *etc*., or (2) to perform the task by executing code (a.k.a., CodeAct), including executing bash commands, Python code, or browser-specific programming language (§2.2). This general action space allows the agent (v1.5 and above) to perform various tasks, including editing files, browsing the web, running programs, etc.

Browsing Agent.
We implemented a generalist web agent called Browsing Agent, to serve as a simple yet effective baseline for web agent tasks.
The agent is similar to that in WebArena (Zhou et al., ), but with improved observations and actions, with only zero-shot prompting.
Full prompts are in §K.

GPTSwarm Agent.
GPTSwarm (Zhuge et al., ) pioneers the use of optimizable graphs to construct agent systems, unifying language agent frameworks through modularity. Each node represents a distinct operation, while edges define collaboration and communication pathways. This design allows automatic optimization of nodes and edges, driving advancements in creating multi-agent systems.

Micro Agent(s).
In addition, OpenHands enables the creation of micro agent, an agent specialized towards a particular task.
A micro agent re-uses most implementations from an existing generalist agent (e.g., CodeAct Agent).
It is designed to lower the barrier to agent development, where community members can share specialized prompts that work well for their particular use cases.

## 4 Evaluation

Table: Table 2: Evaluation benchmarks in OpenHands.

To systematically track progress in building generalist digital agents, as listed in Tab. 2, we integrate 15 established benchmarks into OpenHands. These benchmarks cover software engineering, web browsing, and miscellaneous assistance.
In this section, we compare OpenHands to open-source reproducible baselines that do not perform manual prompt engineering specifically based on the benchmark content.
Please note that we use ‘OH’ as shorthand for OpenHands for the rest of this section for brevity reasons.

### 4.1 Result Overview

In OpenHands, our goal is to develop general digital agents capable of interacting with the world through software interfaces (as exemplified by the code actions described in §2.1). We recognize that a software agent should excel not only in code editing but also in web browsing and various auxiliary tasks, such as answering questions about code repositories or conducting online research.

Tab. 3 showcases a curated set of evaluation results. While OpenHands agents may not achieve top performance in every category, they are designed with generality in mind. Notably, the same CodeAct agent, without any modifications to its system prompt, demonstrates competitive performance across three major task categories: software development, web interaction, and miscellaneous tasks. This is particularly significant when compared to the baseline agents, which are typically designed and optimized for specific task categories.

Table: Table 3: Selected evaluation results for OpenHands agents (§4). See Tab. 4 (software), Tab. 5 (web), Tab. 6 (miscellaneous assistance) for full results across benchmarks.

### 4.2 Software Engineering

Next, we report results specifically for software engineering benchmarks in Tab. 4.

SWE-Bench (Jimenez et al., ) is designed to assess agents’ abilities in solving real-world GitHub issues, such as bug reports or feature requests.
The agent interacts with the repository and attempts to fix the issue provided through file editing and code execution.
The agent-modified code repository is tested against a test suite incorporating new tests added from human developers’ fixes for the same issue.
Each test instance accompanies a piece of “hint text” that consists of natural language suggestions for how to solve the problem. Throughout this paper, we report all results without using hint text.
A canonical subset, SWE-bench Lite, is created to facilitate accessible and efficient testing. We default to use this subset for testing for cost-saving consideration.(^2^22Running the complete set of 2294 instances costs $6.9k, using a conservative estimate of $3 per instance.)
Result. As shown in Tab. 4, our most recent version of CodeActAgent v1.8, using claude-3.5-sonnet, achieves a competitive resolve rate of $26\%$ compared to other open-source software development specialists.

Table: Table 4: OpenHands Software Engineering evaluation results (§4.2).

#### 4.2.1 HumanEvalFix

HumanEvalFix (Muennighoff et al., ) tasks agents to fix a bug in a provided function with the help of provided test cases. The bugs are created to ensure one or more test cases fail. We focus on the Python subset of the benchmark and allow models to solve the bugs by self-debug over multiple turns, incorporating feedback from test execution. We follow the setup from Muennighoff et al. () using pass@k (Chen et al., ).
Results. In Tab. 4, OpenHands CodeActAgent successfully fixes $79.3\%$ of bugs in the Python split. This is significantly better than all non-agentic approaches, almost doubling the performance of StarCoder2-15B (Lozhkov et al., ; Li et al., ).
While SWE-Agent achieves $87.7\%$, Yang et al. () provides the model a full demonstration of a successful sample trajectory fixing one of the bugs in the test dataset (“1-shot”), whereas our evaluation of OpenHands is 0-shot.
As HumanEvalFix has been created by humans and all bugs carefully validated, achieving $100\%$ on this benchmark is entirely feasible, which we seek to do in future iterations of OpenHands.

ML-Bench (Tang et al., )
evaluates agents’ ability to solve machine learning tasks across 18 GitHub repositories.
The benchmark comprises 9,641 tasks spanning 169 diverse ML problems, requiring agents to generate bash scripts or Python code in response to user instructions.
In the sandbox environment, agents can iteratively execute commands and receive feedback, allowing them to understand the repository context and fulfill user requirements progressively.
Following the setup from the original paper, we perform agent evaluation on the quarter subset of ML-Bench.
Results. As shown in Table 4, OpenHands agents with GPT-4o achieve the highest success rate of $76.47\%$ on ML-Bench, outperforming SWE-Agent ($42.64\%$). Performance drops with less capable models. These results demonstrate the effectiveness of OpenHands agent in complex ML tasks. We notice that agents show potential in reducing hallucination and syntax errors compared to non-agent approaches in the ML-LLM-Bench settings (Tang et al., ).

Gorilla APIBench (Patil et al., ) evaluates agents’ abilities to use APIs. it incorporates tasks on TorchHub, TensorHub, and HuggingFace. During the evaluation, models are given a question related to API usage, such as "identify an API capable of converting spoken language in a recording to text." Correctness is evaluated based on whether the model’s API call is in the correct domain.
Results. As shown in Table 4, OpenHands using GPT-4o, with a success rate of $36.4\%$, outperforms baselines not specifically finetuned for API calling. While Gorilla shows higher performance on APIBench, Patil et al. () finetune this model for API calling in particular.

ToolQA (Zhuang et al., ) evaluates agents’ abilities to use external tools. This benchmark includes tasks on various topics like flight status, coffee price, Yelp data, and Airbnb data, requiring the use of various tools such as text tools, database tools, math tools, graph tools, code tools, and system tools.
It features two levels: easy and hard. Easy questions focus more on single-tool usage, while hard questions emphasize reasoning. We adopt the easy subset for evaluation.
Results. Compared to all baselines, OpenHands with GPT-4o shows the highest performance. We notice that agents perform better on tasks related to CSV and database tool usage but requires improvements on math and calculator tool usage.

BioCoder (Tang et al., ) is a repository-level code generation benchmark that evaluates agents’ performance on bioinformatics-related tasks, specifically the ability to retrieve and accurately utilize context. The original prompts contain the relevant context of the code; however, in this study, we have removed them to demonstrate the capability of OpenHands to perform context retrieval, self-debugging, and reasoning in multi-turn interactions. BioCoder consists of 157 Python and 50 Java functions, each targeting a specific area in bioinformatics, such as proteomics, genomics, and other specialized domains. The benchmark targets real-world code by generating code in existing repositories where the relevant code has been masked out.
Results. Table 4 shows that OpenHands, using GPT-4o, achieves a success rate of $44.0\%$. This outperforms all prompting-based non-agent baselines, with GPT-4 alone only achieving $6.4\%$.

BIRD (Li et al., ) is a benchmark for text-to-SQL tasks (*i.e*., translate natural language into executable SQL) aimed at realistic and large-scale database environments.
We select 300 samples from the dev set to integrate into OpenHands and evaluate on execution accuracy.
Additionally, we extend the setting by allowing the agent to engage in multi-turn interactions to arrive at the final SQL query, enabling it to correct historical results by observing the results of SQL execution.
Results. As shown in Table 4, OpenHands with GPT-4o achieves an execution accuracy of $47.3\%$ on a subset of BIRD, showcasing the potential of OpenHands as a SQL agent.
The result outperforms approaches utilizing prompting with code LLMs, such as CodeLlama-7B-Instruct ($18.3\%$) and CodeQwen-7B-Chat (Bai et al., ) ($31.3\%$).

### 4.3 Web Browsing

We report evaluation results for web browsing benchmarks in Tab. 5.

WebArena (Zhou et al., ) is a self-hostable, execution-based web agent benchmark that allows agents to freely choose which path to take in completing their given tasks.
WebArena comprises 812 human-curated task instructions across various domains, including shopping, forums, developer platforms, and content management systems.
Each task is paired with a handwritten test case that verifies agent success, *e.g*., by checking the status of a web page element against a reference or the textual answer returned by the agent.
Results. From Tab. 5, we can see that our BrowsingAgent achieves competitive performance among agents that use LLMs with domain-general prompting techniques.
Some agents (e.g., AutoWebGLM) require manual effort tailored to the WebArena task domain.
This showcases the performance trade-off between a generalist vs. a domain-tailored specialist web agent, and we opt for a more general browsing agent as a building block in OpenHands.

MiniWoB++ (Liu et al., ) is an interactive web benchmark, with built-in reward functions.
The tasks are synthetically initialized on 125 different minimalist web interfaces.
Unlike WebArena, tasks are easier without page changes, require fewer steps, and provide low-level step-by-step task directions.
Note that it contains a portion of environments that require vision capability to tackle successfully, and many existing work choose to focus only on a subset of the tasks (Kim et al., ; Li et al., ; Shaw et al., ). Still, we report the performance on the full set and only include baselines that are evaluated on the full set.
Results. From Tab. 5, we see that our BrowsingAgent finishes nearly half of the tasks without any adaptation to the environment. However, due to the synthetic nature of MiniWoB++, the state-of-the-art agents explicitly trained for the environments with reinforcement learning and/or human behavior cloning have almost saturated the performance.

Table: Table 5: OpenHands Web Browsing Evaluation Results (§4.3).

### 4.4 Miscellaneous Assistance

Results for miscellaneous assistance benchmarks are reported in Tab. 6.

GAIA (Mialon et al., ) evaluates agents’ general task-solving skills, covering different real-world scenarios.
It requires various agent capabilities, including reasoning, multi-modal understanding, web browsing, and coding.
GAIA consists of 466 curated tasks across three levels.
Setting up GAIA is traditionally challenging due to the complexity of integrating various tools with the agent, but
OpenHands’s infrastructure (*e.g*., runtime §2.2, tools §2.3) simplifies the integration significantly.
Results. In our experiments, we achieved a score of $32.1$ on the GAIA (level-1 val), significantly improving over the original AutoGPT (Gravitas, ). GAIA is sensitive to the support of multimodal input and web navigation skills, suggesting further score improvements as OpenHands’s infrastructure improves.

GPQA (Rein et al., ) evaluates agents’ ability for coordinated tool use when solving challenging graduate-level problems.
It consists of 448 curated and difficult multiple-choice questions in biology, physics, and chemistry.
Tool use (*e.g*., python) and web search are often useful to assist agents in answering these questions since they provide accurate calculations that LLMs are often incapable of and access to information outside of the LLM’s parametric knowledge base.
Results. Results are shown in Tab. 6 and 7. We observe that OpenHands’s integrated for supporting diverse tool use (*e.g*., python for calculations) as well as web-search (for searching relevant facts) allows the resulting agent to better solve complex multi-step problems, surpassing the prior *state-of-the-art* by 9.6% and 12.3% on the main and diamond subsets respectively on GPQA (Rein et al., ).

Table: Table 6: OpenHands miscellaneous assistance evaluation results (§4.4).

AgentBench (Liu et al., ) evaluates agents’ reasoning and decision-making abilities in a multi-turn, open-ended generation setting.
We selected the code-grounded operating system (OS) subset with 144 tasks.
Agents from OpenHands interact directly with the task-specific OS using bash commands in a multi-turn manner, combining interaction and reasoning to automate task completion.
Results.
In our experiments (Tab. 6), OpenHands CodeActAgent v1.5 achieves a score of $57.6\%$ on the AgentBench using gpt-4o, outperforming the $42.4\%$ baseline using gpt-4 from the original paper.
Interestingly, when employing weaker models such as gpt-3.5-turbo, OpenHands agents generally underperform compared to the original baseline agents. This finding suggests that generalist agents, like those implemented in OpenHands, require a certain threshold of foundation model capability - particularly instruction following - to function effectively.

MINT (Wang et al., ) is a benchmark designed to evaluate agents’ ability to solve challenging tasks through multi-turn interactions using tools and natural language feedback simulated by GPT-4. We use coding and math subsets used in Yuan et al. (). We follow the original paper and allow the agent to interact with up to five iterations with two chances to propose solutions.
Results. As shown in Tab. 6), OpenHands agents achieve comparable performance to the default agent in the original benchmark, with a performance improvement in the math subset.

ProofWriter (Tafjord et al., ) is a synthetic dataset created to assess deductive reasoning abilities of LLMs.
Same as Logic-LM (Pan et al., ), we focus on the most challenging subset, which contains 600 instances requiring 5-hop reasoning. To minimize the impact of potential errors in semantic parsing, we use the logical forms provided by Logic-LM.
Results. In Tab. 6, OpenHands agent employs a symbolic solver to solve the task, achieving performance comparable to the *state-of-the-art* neuro-symbolic model (*i.e*., Logic-LM) (Pan et al., ).

Entity Deduction Arena (EDA) (Zhang et al., ) evaluates agents’ ability to deduce unknown entities through strategic questioning, akin to the 20 Questions game.
This benchmark tests the agent’s state tracking, strategic planning, and inductive reasoning capabilities over multi-turn conversations. We evaluate two datasets “Things” and “Celebrities”, each comprising 100 instances, and report the average success rate over these two datasets.
Results. Tab. 6 shows that CodeActAgent yields comparable performance comparing with the results reported in the original paper (Zhang et al., ).

## 5 Conclusion

We introduce OpenHands, a community-driven platform that enables the development of agents that interact with the world through software interfaces.
By providing a powerful interaction mechanism, a safe sandboxed environment, essential agent skills, multi-agent collaboration capabilities, and a comprehensive evaluation framework, OpenHands accelerates research innovations and real-world applications of agentic AI systems.
Despite challenges in developing safe and reliable agents (§A), we are excited about our vibrant community and look forward to OpenHands’s continued evolution.

## Author Contributions

This work was an open-source collaborative effort across multiple institutions. We employed a point-based system to determine contributions and award authorships, with technical contributions tracked and measured in units of pull requests (PRs)(^3^33For more details, please refer to [https://github.com/All-Hands-AI/OpenHands/pull/1917](https://github.com/All-Hands-AI/OpenHands/pull/1917).). Xingyao Wang led the project, coordinating overall development and paper writing efforts. Detailed contributions were as follows:

- •
Agent Development (§3): Xingyao Wang led the implementation of CodeAct Wang et al. () and CodeActSWE agents. Frank F. Xu led the development of web browsing agents Zhou et al. (). Mingchen Zhuge orchestrated the integration of the GPTSwarm agent Zhuge et al. (). Robert Brennan and Boxuan Li lead the development of the Micro Agent.
- •
Architectural Development (Fig. 2): Robert Brennan initiated the architecture design. Boxuan Li, Frank F. Xu, Xingyao Wang, Yufan Song, and Mingzhang Zheng further refined and expanded the architecture. Boxuan Li implemented the initial version of integration tests (§E), maintained the agentskills library (§2.3), managed configurations, and resolved resource leaks in evaluation. Frank F. Xu developed the web browsing environment (§J) for both agent execution and evaluation and integrated it with both agent and front-end user interfaces. Xingyao Wang authored the initial code for the agentskills library and the Docker sandbox. Yufan Song implemented cost tracking for evaluation, while Mingzhang Zheng developed an image-agnostic docker sandbox for more stable SWE-Bench evaluation.
- •
Benchmarking, Integration, and Code Review: Boxuan Li and Yufan Song led benchmark integration efforts, including coordination, evaluation, and code review. Yufan Song also helped track PR contributions. Graham Neubig, Xingyao Wang, Mingzhang Zheng, Robert Brennan, Hoang H. Tran, Frank F. Xu, Xiangru Tang, Fuqiang Li, and Yanjun Shao provided additional support in integration and code reviews.
Specific benchmark contributions included:
–
SWE-Bench Jimenez et al. (): Bowen Li and Xingyao Wang
–
WebArena Zhou et al. () and MiniWob++ Liu et al. (): Frank F. Xu
–
GAIA Mialon et al. (): Jiayi Pan (integration) and Mingchen Zhuge (GPTSwarm evaluation)
–
API-Bench Patil et al. () and ToolQA Zhuang et al. (): Yueqi Song
–
HumanEvalFix Muennighoff et al. (): Niklas Muennighoff and Xiangru Tang
–
ProofWriter Tafjord et al. (): Ren Ma
–
MINT Wang et al. (): Hoang H. Tran
–
AgentBench Liu et al. (): Fuqiang Li
–
BIRD Li et al. (): Binyuan Hui
–
GPQA Rein et al. (): Jaskirat Singh
–
BioCoder Tang et al. (): Xiangru Tang and Bill Qian
–
ML-Bench Tang et al. (): Xiangru Tang and Yanjun Shao
–
Entity-Deduction-Arena Zhang et al. (): Yizhe Zhang
- •
Advising: Graham Neubig advised the project, providing guidance, resources, and substantial paper edits. Heng Ji and Hao Peng offered additional project advice and assisted with paper writing. Junyang Lin contributed advisory support and sponsored resources.

## Appendix A Limitations and Future Work

We are excited about the foundations our vibrant community has laid in OpenHands and look forward to its continued evolution.
We identify several directions for future work:

Enhanced multi-modality support. While our current implementation already supports a wide range of file formats through predefined agent skills, we are interested in enabling multi-modality in a principled way through standard IPython and browser integration, such as viewing images and videos using vision-language model through a browser or processing XLSX files with code.

Stronger agents. Current agents still struggle with complex tasks, and we are interested in building better agents through both training and inference time techniques.

Agent editing improvements. Current agent suffers a lot when editing long files, and we are interested in exploring different approaches to improve the file editing performance of agents.

Web browsing improvements. Due to the extensible nature of OpenHands, orthogonal components that could improve agents can be integrated easily. For example, thanks to OpenHands’s extensible architecture, Auto Eval & Refine Pan et al. (), an agent retry-on-error strategy with Reflexion Shinn et al. () prompts and task completion reward models, will be integrated as an optional component attached to our browsing agent.

Automatic workflow generation. Currently, OpenHands’s workflow still requires a substantial handcrafted workload. We believe that graph-based frameworks such as GPTSwarm Zhuge et al. () and LangGraph Chase () could serve as alternative solutions for building agents. Particularly in GPTSwarm, when agents are constructed using graphs, it becomes easier to incorporate various optimization methods (e.g., reinforcement learning, meta-prompting). OpenHands considers these methods to lay the groundwork for promising solutions in automatic workflow generation in future versions.

## Appendix B Ethics Statement

Most AI agents today are still research artifacts and lack the ability to perform complex, long-horizon tasks in the real world reliably. However, as their performance continues to improve and they are increasingly deployed in real world, they have the potential to boost productivity while also posing security risks to society significantly.
OpenHands helps mitigate risks by:

- (1)
Enabling systematic evaluation of these agents, which can identify and address risks before they are widely deployed.
- (2)
Facilitating human-agent interaction rather than allowing agents to operate autonomously without oversight.
- (3)
More importantly, we hope OpenHands allows researchers worldwide to access the best suites of agents to conduct frontier safety research towards building safe and helpful agents.

## Appendix C Related Work

The breakthroughs in large language models (LLMs) like ChatGPT OpenAI () and GPT-4 OpenAI et al. () have significantly enhanced the capabilities of autonomous agents across various domains Ye et al. (); Tang et al. (); Park et al. (); Cui et al. (). These advances have spurred a multitude of generalist agent proposals Gravitas (); Nakajima (); Wu et al. () aimed at performing diverse user tasks and have gained attention from both developers and broader audiences. Notable works such as Auto-GPT Gravitas () harness LLMs for task completion by decomposing user goals into executable steps. Multi-agent collaboration systems leverage LLMs for elements like role-playing and task-solving capabilities Zhuge et al. (); Li et al. (); Zhou et al. (); Team (), with MetaGPT Hong et al. () emphasizing standardized operating procedures, and AutoGen Wu et al. () providing a conversation framework for interactive systems. AGENTS Zhou et al. () and AutoAgents Chen et al. () offer new paradigms for customizable agent architecture, while XAgent Team () and GPTSwarm Zhuge et al. () introduce complex management systems and optimizable graphs, respectively, for enhanced agent operations.

Software development, a front-runner in applying LLM-based agents, has seen advancements in frameworks for facilitating the development processes Hong et al. (); Qian et al. (). Innovations such as ChatDev Qian et al. () automate the software development lifecycle akin to the waterfall model, and AutoCodeRover Zhang et al. () addresses GitHub issues via code search and abstract syntax tree manipulation. AgentCoder Huang et al. () iteratively refines code generation with integrated testing and feedback, while SWE-Agent Yang et al. () integrates LLMs for automated Github issue fixing, streamlining software engineering.

## Appendix D Graphical User Interface

Besides running from the command line, OpenHands features a rich graphical user interface that visualizes the agent’s current actions (*e.g*., browsing the web, executing base commands or Python code, *etc*.) and allows for real-time feedback from the user.
Screenshots of the UI are shown in Fig. 1.
The user may interrupt the agent at any moment to provide additional feedback, comments, or instruction while the agent is working.
This user interface directly connects with the event streams (§2.1) to control and visualize the agents and runtime, making it agent and runtime agnostic.

## Appendix E Quality Control: Integration Tests for Agents

Integration tests Leung & White () have long been used by software developers to ensure software quality. Unlike large language models with simple input-output schema, agents are typically complex pieces of software where minor errors can be easily introduced during the development process and hurt final task performance.
While running a full suite evaluation (§4) is the ultimate measure of performance degradation, running them for every code changes can be prohibitively slow and expensive.
(^4^44Running a SWE-Bench Lite Jimenez et al. () evaluation with gpt-4o costs around 600 USD.).
In OpenHands, we pioneer an end-to-end agent test framework that tests prompt regression, actions, and sandbox environments.
It combines integration testing from software engineering and foundation model mocking for deterministic behavior to prevent the accidental introduction of bugs during agent development.

Defining an integration test.
The integration test framework for OpenHands is structured to validate end-to-end functionality by automating task execution and result verification. Developers define tasks and expected results; for instance, a task might involve correcting typos in a document named "bad.txt". Upon task execution through OpenHands, outputs are compared against a predefined "gold file" to ensure accuracy.

Mocking LLM for deterministic behavior.
Addressing the challenge of non-determinism in large language models (LLMs) and the associated high costs, the framework intercepts all LLM calls and supplies predefined responses based on exact prompt matches. This method not only ensures consistency in test outcomes but also reduces operational costs by minimizing the reliance on real LLMs.

Regenerate LLM responses on breaking changes.
Prompt-response pairs are managed through a script that generates and stores these pairs when new tests are introduced or existing prompts are modified. For routine tests, the framework attempts to reuse existing LLM responses by slightly adjusting the prompts. Substantial changes that affect task handling require regeneration of these pairs using real LLMs.

Benefits of integration tests.
The framework offers several advantages, including 1) Prompt regression testing: Stored prompt-response pairs facilitate change tracking and provide a reference for new team members to understand LLM interactions, 2) Multi-platform support: Tests are automatically scheduled for every pull request and commit on the main branch, running across multiple platforms, environments, and agents, including Linux and Mac, and in local, SSH, and exec sandboxes, and 3) Comprehensive error detection: It captures errors in prompt generation, message passing, and sandbox execution, thereby maintaining a high test coverage.

## Appendix F How OpenHands Runtime work

### F.1 Workflow

The OpenHands Runtime system uses a client-server architecture implemented with Docker containers. See Fig. 4 for an overview of how it works.

Figure: Figure 4: OpenHands runtime workflow.
Refer to caption: figs/runtime/oh_runtime_workflow.png

- (1)
User Input: The user provides a custom base Docker image.
- (2)
Image Building: OpenHands builds a new Docker image (the "OH runtime image") based on the user-provided image. This new image includes OpenHands-specific code, primarily the "runtime client" (i.e., runtime API server described in §2.2).
- (3)
Container Launch: When OpenHands starts, it launches a Docker container using the OH runtime image.
- (4)
Communication: The OpenHands backend (runtime.py) communicates with the runtime client over RESTful API, sending actions and receiving observations
- (5)
Action Execution: The runtime client receives actions from the backend, executes them in the sandboxed environment, and sends back observations
- (6)
Observation Return: The client sends execution results back to the OpenHands backend event stream as observations.

The role of the client:

- •
It acts as an intermediary between the OpenHands backend and the sandboxed environment
- •
It executes various types of actions (shell commands, file operations, Python code, etc.) safely within the container
- •
It manages the state of the sandboxed environment, including the current working directory and loaded plugins
- •
It formats and returns observations to the backend, ensuring a consistent interface for processing results

### F.2 How OpenHands builds and maintains runtime images

OpenHands’ approach to building and managing runtime images ensures efficiency, consistency, and flexibility in creating and maintaining Docker images for both production and development environments.

#### F.2.1 Image Tagging System

OpenHands uses a dual-tagging system for its runtime images to balance reproducibility with flexibility:

- (1)
Hash-based tag: {target_image_repo}:{target_image_hash_tag}.
Example: runtime:abc123def456
–
This tag is based on the MD5 hash of the Docker build folder, which includes the source code (of runtime client and related dependencies) and Dockerfile
–
Identical hash tags guarantee that the images were built with exactly the same source code and Dockerfile
–
This ensures reproducibility; the same hash always means the same image contents
- (2)
Generic tag: {target_image_repo}:{target_image_tag}.
Example: runtime:oh_v0.9.3_ubuntu_tag_22.04
–
This tag follows the format: runtime:oh_v{VERSION}_{BASE_IMAGE}_tag_{IMAGE_TAG}
–
It represents the latest build for a particular base image and OpenHands version combination
–
This tag is updated whenever a new image is built from the same base image, even if the source code changes

The hash-based tag ensures reproducibility, while the generic tag provides a stable reference to the latest version of a particular configuration. This dual-tagging approach allows OpenHands to efficiently manage both development and production environments.

#### F.2.2 Build Process

- (1)
Image Naming Convention:
–
Hash-based tag: target_image_repo:target_image_hash_tag.
Example: runtime:abc123def456
–
Generic tag: target_image_repo:target_image_tag.
Example: runtime:oh_v0.9.3_ubuntu_tag_22.04
- (2)
Build Process:
a.
Convert the base image name to an OH runtime image name
Example: ubuntu:22.04 -> runtime:oh_v0.9.3_ubuntu_tag_22.04
b.
Generate a build context (Dockerfile and OpenHands source code) and calculate its hash
c.
Check for an existing image with the calculated hash
d.
If not found, check for a recent compatible image to use as a base
e.
If no compatible image exists, build from scratch using the original base image
f.
Tag the new image with both hash-based and generic tags
- (3)
Image Reuse and Rebuilding Logic:
The system follows these steps to determine whether to build a new image or use an existing one from a user-provided (base) image (e.g., ubuntu:22.04):
a.
If an image exists with the same hash (e.g., runtime:abc123def456), it will be reused as is
b.
If the exact hash is not found, the system will try to rebuild using the latest generic image (e.g., runtime:oh_v0.9.3_ubuntu_tag_22.04) as a base. This saves time by leveraging existing dependencies
c.
If neither the hash-tagged nor the generic-tagged image is found, the system will build the image completely from scratch

Caching and Efficiency. The system attempts to reuse existing images when possible to save build time. If an exact match (by hash) is found, it’s used without rebuilding. If a compatible image is found, it’s used as a base for rebuilding, saving time on dependency installation.

A flowchart illustrating the build process is shown in Fig. 5

Figure: Figure 5: OpenHands Runtime Image Build Workflow.
Refer to caption: figs/runtime/oh_runtime_image_build_workflow.png

## Appendix G Additional Results For GPQA Benchmark

We showcase more detailed results, including performance on other subsets for GPQA benchmark in Tab. 7.

Table: Table 7: *Full Evaluation Results on the GPQA Benchmark Rein et al. ()* (§4.4).

## Appendix H In-context Demonstration for CodeActSWEAgent

The prompt is re-adopted from the SWE-agent’s released trajectory ([https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories/demonstrations](https://github.com/princeton-nlp/SWE-agent/tree/main/trajectories/demonstrations)). The prompt can be found at [https://github.com/All-Hands-AI/OpenHands/blob/main/agenthub/codeact_swe_agent/prompt.py](https://github.com/All-Hands-AI/OpenHands/blob/main/agenthub/codeact_swe_agent/prompt.py).

## Appendix I Supported AgentSkills

As of OpenHands v0.6, we support the following list of skills.
Please refer to the source code for the most up-to-date list of skills: [https://github.com/All-Hands-AI/OpenHands/blob/main/OpenHands/runtime/plugins/agent_skills/agentskills.py](https://github.com/All-Hands-AI/OpenHands/blob/main/OpenHands/runtime/plugins/agent_skills/agentskills.py)

## Appendix J BrowserGym Actions

The following are all the supported actions defined in BrowserGym(^5^55[https://github.com/ServiceNow/BrowserGym/blob/main/core/src/browsergym/core/action/functions.py](https://github.com/ServiceNow/BrowserGym/blob/main/core/src/browsergym/core/action/functions.py)) as of v0.3.4.
The actions can be categorized into several types and can be configured to use only a subset of the functionality.
There are agent control actions, navigation actions, page element-based actions, coordinate-based actions, as well as tab-related actions.
We use these actions from the BrowserGym library as our main browsing action primitives.

## Appendix K Browsing Agent Details

The following shows an example prompt containing all the information required for the current step to make a prediction about the next browsing actions.
Note that we also instruct the agent to predict multiple actions in one turn if the agent thinks they are meant to be executed sequentially without any feedback from the page.
This could save turns for common workflows that consist of a sequence of actions on the same page without any observation change, such as filling the username and password and submit in a login page.

And an example response to the above prompt is:

For the evaluation on WebArena benchmark, since some of the tasks require checking for answer exact match on the agent’s message back to the user, we add the following instruction to let the agent reply with only a concise answer string when messaging the user to prevent the agent from failing the test due to extra text:

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: This paper introduces an open platform with the event-driven architecture to advance the AI agent development, especially focusing on coding and web browsing. It also conducts extensive evaluations of heterogeneous agents on various benchmarks. This is a solid contribution to the LLM agent community. I have some questions about the implementation details. 
- The authors mention that event streaming enables human intervention in agent tasks, which I believe is a crucial step for calibrating agent outputs. I wonder whether human intervention is enabled during the evaluation of agents and is there a mechanism to determine whether human intervention is required?
- Regarding the multi-agent delegation feature, I’m unclear on how agents collaborate. Is there an orchestrator agent responsible for task delegation, or is this managed through event streaming? If it’s the latter, how are tasks assigned and dispatched, and where do agents deliver their outputs?
soundness: 3
strengths: - The motivation to build an open-sourced agent platform is good. 
- The agent interface, with capabilities for code execution and web browsing, is impressive.
- Evaluations conducted across various agents and benchmarks are thorough and extensive.
summary: This paper introduces OpenHands, is a community-driven platform designed to advance the development of AI agents that interact with the world in terms of code writing, command-line interaction, and web browsing. OpenHands includes an evaluation framework for benchmarking agent performance on tasks like software engineering and web browsing. It supports various LLMs and sandboxed environments for running these AI agents and conducts extensive evaluation of different agents in various benchmarks.
weaknesses: N/A

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Distinction from Existing Platforms: How does OpenHands fundamentally differ from other agent development platforms like LangGraph, CrewAI and DSPy for multi-agent collaboration or BrowserGym for benchmark evaluation? Could the authors elaborate on the unique features or advantages that OpenHands offers over these frameworks? Including these comparisons in Table 1 would provide a more comprehensive overview.

Agent Performance Analysis: Can the authors provide a more detailed explanation about the data collector in the framework? Specifically, what makes the framework the best solution to develop an agent when one approaches such a task?
Another question is: how will the platform address this in future iterations?

Handling of Observations: How does the platform treat observations and their full composition? Can the authors clarify how observations are managed and utilized by agents? Providing more details on the configuration and usage of the event stream would enhance understanding, especially for developers less familiar with event-driven systems.

Security Considerations: Given that the platform allows the execution of arbitrary code in a sandboxed environment, what security measures are in place to prevent potential exploits or breaches? Has the sandboxing been tested against known vulnerabilities? Discussing security protocols would demonstrate the platform's reliability.

Web Page Understanding Techniques: Regarding the web experiments, it is not clear which page understanding techniques are used to interpret interactable elements and web pages. Does the framework provide screen/page understanding capabilities, or should a new user implement their own methods? Clarifying this would help users understand the platform's capabilities in web interaction tasks.
soundness: 3
strengths: Originality:
OpenHands uniquely consolidates multiple agent capabilities: coding, command-line interaction, web browsing, and multi-agent collaboration, within a single, open-source platform. The introduction of AgentHub allows users to contribute their own agents, promoting community collaboration and expanding the platform's versatility. This integration distinguishes OpenHands from existing frameworks, although IMO some of the main ones are absent from the comparison (see questions).

Quality: 
The methodological approach is solid, featuring a well-defined architecture that includes an event-driven state management system and a secure, sandboxed runtime environment. The use of an extensible agent skills library enhances flexibility, allowing agents to perform complex tasks and even create tools themselves. The agent abstraction is designed to be simple, enabling users to easily create and extend agents.

Clarity:
The paper is generally well-written and logically structured, making it accessible to both practitioners and researchers. Key components of the platform, such as the agent abstraction, runtime environment, and evaluation framework, are explained clearly. Figures and tables effectively illustrate concepts and results.

Significance:
By providing an open-source, MIT-licensed platform with contributions from a large community, OpenHands has the potential to significantly impact the development and evaluation of AI agents. It introduces a varied range of datasets for each general agent domain, although it could be expanded in the browser domain (see weaknesses). Evaluating multiple close-source LLMs on each task in addition to the average costs of evaluating benchmarks is a valuable addition that offers practical insights.
summary: The paper presents OpenHands, an open-source platform designed to facilitate the development of AI agents that interact with the world through software interfaces such as code execution, command-line operations, and web browsing. A key feature of OpenHands is AgentHub, a platform where users can contribute their own agents (architecture) to the community, fostering collaborative development of agent architectures. The platform provides a simple agent abstraction, making it accessible for users to create and extend agents, and provides a modular architecture for agent interactions and task execution, including an event-driven state management system, multi-agent coordination, and an extensible agent skills library. OpenHands aims to enable agents to even create tools by themselves. The authors describe the architecture, implementation details, and evaluate the platform across multiple benchmarks, including software engineering tasks and web browsing scenarios. It's kind of a combination of AutoGen and BrowserGym.
weaknesses: Limited Novelty in Certain Aspects: While OpenHands integrates various functionalities, much of the work appears to assemble existing components from the domain into one place. Some features, like code execution in a sandbox and web browsing agents, are present in other platforms. The paper could better articulate how OpenHands distinguishes itself from similar frameworks like LangChain, DSPy, or AutoGen.

Basic Implemented Agents: The currently implemented agents are relatively basic. For example, the Browsing Agent is based on WebArena's agent, which is very simple to implement and not competitive with current SOTA agents. There are more advanced multi-agent architectures available that could be implemented in AgentHub to better demonstrate the system's capabilities in facilitating the general agents' development.

Outdated Datasets: The evaluation includes MiniWoB++, which is an outdated dataset. There are newer datasets, such as WorkArena, WorkArena++, and ST-WebAgentBench, that are more relevant. The last test agents in more complex, restricted environments using policy hierarchies. Incorporating these in the framework would strengthen the evaluation.

Evaluation Depth: The evaluation, while broad, could benefit from deeper analysis. Incorporating an automated data collector to track agent success and failure would enhance insight into agent performance and make the framework more complete.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Can we further motivate why we want a generalist agent? If we have specialized agents for different tasks, can we just have a coordinator deploy specialized agents where they perform best?
soundness: 3
strengths: OpenHands is a valuable contribution to the research community. The OpenAgents platform will surely be used to develop many future agents, and be referred to as a baseline for other agent-developing research projects. Tables 3, 4, and 5 present thorough experimentation of their agent(s) compared to many other baselines on various tasks.
summary: OpenHands is an open-source platform for the development of general AI agents. This paper details the motivation behind OpenHands, the architecture of the platform, how a user might implement an agent using OpenHands, description of some existing baseline agents, and comparison of some of the OpenHands agents against other open-source agents on software, web, and other assisting tasks. Selected results show that their agents perform reasonably well across a wide range of these tasks (generally underperform the specialized agents).
weaknesses: While the value of OpenHands is apparent, the research question of this paper is unclear. If it is that they can develop a general agent that performs better across a range of tasks than any existing agent does, it is my judgement that the results and analysis do not explore this sufficiently to warrant a research contribution. To do so, discussion and ablations should be included for the design decisions of OH Browsing Agent v*, OH CodeAgent v*, OH GPTSwarm v*, etc. To then isolate the effect of those decisions, the choice of backend model should be held consistent to those of the baselines against which the OH agents are compared. Error analysis of results comparing OH to baselines would be another welcome contribution that has been omitted from this paper.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: L305 - I'm uncertain what is meant by "we compare OpenHands to open-source reproducible baselines that do not perform manual prompt engineering specifically based on the benchmark content"

It seems surprising that OH CodeActAgent w. GPT-3.5 would get 2% on ToolQA, while ReAct (perhaps the most minimal form of agent scaffolding) would get 36%. Do you have any idea what's behind this performance difference?

Also surprising is that CodeActAgent delegating to BrowsingAgent does very slightly worse than straightforwardly using BA directly. Could you discuss this?
soundness: 2
strengths: OpenHands is an excellent and significant LLM-agent platform, seeing use e.g. as one of the reference agents in OpenAI's MLE-bench https://arxiv.org/abs/2410.07095 , and there is value in having this scaffold's architecture explained and benchmark performance reported in a peer-reviewed form.


- General competence across three different domains (SWE, browsing, and 'misc') is significant

- Solve rate of 26% on SWE-bench is competitive, as are many of the results suggested by the tables (though, see below, I have questions around like-for-like comparison)

- The paper provides a great many benchmark results

- OpenHands has many useful features, such as the API server being inside the docker container, the ability to delegate subtasks to other agents, and an extensible library of tools
summary: This paper describes OpenHands, fka OpenDevin, a platform to develop LLM agents which can take meaningful actions using computers, such as browsing, coding, and interacting with a command line.

Section 2 describes the architecture of the platform, detailing the state and event streams, actions and observations, and an extensible library of tools. Agent delegation is also mentioned.

Section 3 mentions some notable agents implemented using their platform.

Section 4 presents results of many evaluatory benchmarks, showing the OpenHands has a broad and general level of competence across domains including coding and browsing.
weaknesses: The paper is an announcement of a software platform, more than it is a description of a contribution to a research area. Normally I'd expect a paper to look like a research question, with some motivation, experimental design, and results. This paper shows little, if any, scientific exploration, instead the paper gives a high-level overview of the architecture with little motivation, and then uncritically showcases performance on a broad range of existing benchmarks. I'm unsure whether this kind of project-announcement is a fit for the conference.

Some smaller points:

- Why do the (e.g.) GPQA results not compare like-for-like LLMs? I would want to see whether a CodeActAgent powered by a particular LLM was better or worse than the bare LLM, but the table compares (Llama 3 70b chat, GPT 3.5 turbo 16k, GPT 4) with (gpt-4o-mini-2024-07-18 , gpt-4o-2024-05-13 , claude-3-5-sonnet), so it's unclear how much of the difference is due to the scaffold and what is due to the choice of LLM. This is also true for many results in Tables 4 onwards, perhaps an unfortunate downside of comparing to results from the literature, but this limitation precluding rigorous comparison should at least be acknowledged in the text.

- A large fraction of the paper comprises short summaries of many different benchmarks used to evaluate the codebase. Since these summaries are not original work, I question whether so much of the paper should be dedicated to them. (As a minor point, I'd expect this exposition to be in a setup / methodology / problem-setting section, rather than in the results.) Instead, the authors could focus more on the creative decisions made during the creation of their platform.

Meta-review: After reading the reviewers' comments and reviewing the paper, we recommend acceptance - Oral presentation. 

Below a more detailed description of the paper.

The paper introduces OpenHands, a community-driven platform designed to advance the development of AI agents that interact with the world in terms of code writing, command-line interaction, and web browsing, and it constitutes a good contribution to “infrastructure, software libraries, hardware, etc.”

The key strengths (S#) of the paper are as follows:

- (S1)	The OpenHands platform may be used to develop many future agents, and be referred to as a baseline for other agent-developing research projects. Tables 3, 4, and 5 present thorough experimentation of their agent(s) compared to many other baselines on various tasks.
- (S2)	OpenHands is original: it uniquely consolidates multiple agent capabilities such as coding, command-line interaction, web browsing, and multi-agent collaboration, within a single, open-source platform. The introduction of AgentHub allows users to contribute their own agents, promoting community collaboration and expanding the platform's versatility. This integration distinguishes OpenHands from existing frameworks, although IMO some of the main ones are absent from the comparison 
- (S3)	The methodological approach is solid, featuring a well-defined architecture that includes an event-driven state management system and a secure, sandboxed runtime environment. The use of an extensible agent skills library enhances flexibility, allowing agents to perform complex tasks and even create tools themselves. The agent abstraction is designed to be simple, enabling users to easily create and extend agents.
- (S4)	By providing an open-source, MIT-licensed platform with contributions from a large community, OpenHands has the potential to significantly impact the development and evaluation of AI agents.
(S5)	General competence across three different domains (SWE, browsing, and 'misc') is significant.
(S6)	OpenHands has many useful features, such as the API server being inside the docker container, the ability to delegate subtasks to other agents, and an extensible library of tools.

Some of the key weaknesses (W#) are as follows:

- (W1)	Limited Novelty in Certain Aspects: While OpenHands integrates various functionalities, much of the work appears to assemble existing components from the domain into one place. Some features, like code execution in a sandbox and web browsing agents, are present in other platforms. The paper could better articulate how OpenHands distinguishes itself from similar frameworks like LangChain, DSPy, or AutoGen.
- (W2)	Basic Implemented Agents: The currently implemented agents are relatively basic. For example, the Browsing Agent is based on WebArena's agent, which is very simple to implement and not competitive with current SOTA agents. There are more advanced multi-agent architectures available that could be implemented in AgentHub to better demonstrate the system's capabilities in facilitating the general agents' development.
- (W3)	Outdated Datasets: The evaluation includes MiniWoB++, which is an outdated dataset. There are newer datasets, such as WorkArena, WorkArena++, and ST-WebAgentBench, that are more relevant. The last test agents in more complex, restricted environments using policy hierarchies. Incorporating these in the framework would strengthen the evaluation.
- (W4)	Evaluation Depth: The evaluation, while broad, could benefit from deeper analysis. Incorporating an automated data collector to track agent success and failure would enhance insight into agent performance and make the framework more complete.

Decision: Accept (Poster)