OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Authors: Xingyao Wang
Boxuan Li
Yufan Song
Frank F. Xu
Xiangru Tang
Mingchen Zhuge
Jiayi Pan
Yueqi Song
Bowen Li
Jaskirat Singh
Hoang H. Tran
Fuqiang Li
Ren Ma
Mingzhang Zheng
Bill Qian
Yanjun Shao
Niklas Muennighoff
Yizhe Zhang
Binyuan Hui
Junyang Lin
Robert Brennan
Hao Peng
Heng Ji
Graham Neubig
Venue: ICLR 2025 Poster (2025)
Keywords: AI agents
evaluation
infrastructure
benchmark
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Software is one of the most powerful tools that we humans have at our disposal; it allows a skilled programmer to interact with the world in complex and profound ways. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and effect change in their surrounding environments. In this paper, we introduce OpenHands, a platform for the development of powerful and flexible AI agents that interact with the world in similar ways to a human developer: by writing code, interacting with a command line, and browsing the web. We describe how the platform allows for the implementation of new agents, utilization of various LLMs, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks. Based on our currently incorporated benchmarks, we perform an evaluation of agents over 13 challenging tasks, including software engineering (e.g., SWE-Bench) and web browsing (e.g., WebArena), amongst others. Released under the permissive MIT license, OpenHands is a community project spanning academia and industry with more than 2K contributions from over 186 contributors in less than six months of development, and will improve going forward.

Published as a conference paper at ICLR 2025
OPENHANDS: AN OPEN PLATFORM FOR AI SOFTWARE DEVELOPERS AS GENERALIST AGENTS
Xingyao Wang 1.00 Boxuan Li,, Yufan Song", Frank F. Xu², Xiangru Tang", Mingchen Zhuge,, Jiayi Pan', Yueqi Song', Bowen Li, Jaskirat Singh', Hoang H. Tran Fugiang Li, Ren Ma, Mingzhang Zheng, Bill Qian Yanjun Shao2, Niklas Muennighoff Yizhe Zhang, Binyuan Hui Junyang Lin", Robert Brennan 10,
Hao Peng', Heng Ji', Graham Neubiga. UTUC ²CMU Yale 'UC Berkeley *Contextual AT KAUST TANU
HHMMUT AAlibaaa All Hands AI
xingyao6e i wwwioii...ee.o... .ddu, gneubiglcs, cmuu.du
ABSTRACT
Software is one of the most powerful tools that we humans have at our disposal; it allows skilled programmer to interact with the world in complex and pro- found ways. At the same time, thanks to improvements in large language models (LLMs), there has also been rapid development in AI agents that interact with and affect change in their surrounding environments. In this paper, we introduce OpenHands (f.k.a. OpenDevin), platform for the development of powerful and flexible AI agents that interact with the world in similar ways to those of human developer: by writing code, interacting with a command line, and browsing the web. We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, coordination between multiple agents, and incorporation of evaluation benchmarks. Based on our currently incorporated benchmarks, we perform an evaluation of agents over 15 challenging tasks, including software engineering (e.g.. SWE-BENCH) and web browsing (e.g.. WEBARENA), among others. Released under the permissive MIT license, OpenHands is community project spanning academia and industry with more than 2. contributions from over 188 contributors.
Q
Code Slack
https//giitubb.om//AllHHaaar Aooorp/
http://bi.lyyOppenHnnn---lae Slack
INTRODUCTION
Powered by large language models (LLMs; OpenAl 2024b; Team et al. 2023; Jiang et al. 2024; Chang et al. 2024), user-facing AI systems (such as ChatGPT) have become increasingly capable of performing complex tasks such as accurately responding to user queries, solving math problems, and generating code. In particular, Al agents, systems that can perceive and act upon the external environment, have recently received ever-increasing research focus. They are moving towards performing complex tasks such as developing software (Jimenez et al., 2024), navigating real- world websites (Zhou et al., 2023a), doing household chores (Ahn et al., 2022), even performing scientific research (Boiko et al., 2023; Tang et al., 2024a).
As Al agents become capable of tackling complex problems, their development and evaluation have also become challenging. There are numerous recent efforts creating open-source frameworks that facilitate the development of agents (Hong et al., 2023; Chen al., 2024; Wu et al., 2023). These agent frameworks generally include: 1) interfaces through which agents interact with the world (such JSON-base function calls or code execution). 2) environments in which agents operate, and 3) interaction mechanisms for human-agent or agent-agent communication. These frameworks streamline and ease the development process in various ways (Tab. 1, SC).
When designing AI agents, we can also consider how human interacts with the world. The most powerful way in which humans currently interact with the world is through software software

---

Published as a conference paper at ICLR 2025
Fobidt
cocine crrtee
recation mutic
onjy sunid
Figure 1: OpenHands User Interface (UI, SD) allows users to view files, check executed bash commands/Python code, observe the agent's browser activity, and directly interact with the agent.
powers every aspect of our life, supporting everything from the logistics for basic needs to the advancement of science, technology, and AI itself. Given the power of software, as well as the existing tooling around its efficient development, use, and deployment, it provides the ideal interface for Al agents to interact with the world in complex ways. However, building agents that can effectively develop software comes with its own unique challenges. How can we enable agents to effectively create and modify code in complex software systems? How can we provide them with tools to gather information on-the-fly to debug problems or gather task-requisite information? How can we ensure that development safe and avoids negative side effects on the users' systems?
In this paper, we introduce OpenHands (f.k... OpenDevin), a community--drvven platform designed for the development generalist and specialist agents that interact with the world through software." It features:
(1) An interaction mechanism which allows user interfaces, agents, and environments to interact through event stream architecture that is powerful and flexible ($2.1).
(2) A runtime environment that consists of docker-sandboxed operating system with a bash shell, a web browser, and [Python server that the agents can interact with ($2.2).
(3) An interface allowing the agent to interact with the environment in manner similar to actual software engineers ($2.3). We provide the capability for agents to a) create and edit complex software, b) execute arbitrary code in the sandbox, and c) browse websites to collect information.
(4) Multi-agent delegation, allowing multiple specialized agents to work together ($2.4).
(5) Evaluation framework, facilitating the evaluation of agents across a wide range of tasks ($4).
Importantly, OpenHands is not just a conceptual framework, but it also includes a comprehensive and immediately usable implementation of agents, environments, and evaluations. As of this writing, OpenHands includes an agent hub with over 10 implemented agents ($3), including strong generalist agent implemented based on the CodeAct architecture (Wang et al., 2024a), with additions for web browsing (ServiceNow) and code editing specialists (Yang et al., 2024). Interaction with users is implemented through a chat-based user interface that visualizes the agent's current actions and allows
'hile initially inspired by AI software engineer Devin (Cognition.ai), OpenHands has quickly evolved to support much wider range of applications heyond software engineering through diverse community contributions.
2

---

Published as a conference paper at ICLR 2025
Interface
Event Stream
Agent Runtime
Oveennad Lane
where all dction SettionSSbeee (berccaioon()) are tracked
anhermoath enriitesrotii kondl to Sitettocii
Web
M6tt turn User Agent
Message (rancemmme))
D0 Ceaffasketion bourssaggent Theee creaad Python sppuy that
Docker Sandbox
Interaction
PythostanCnkkkttt jsurcauagend twrel
cessoute, pott server Sog 2467 cer server Dog
Opentordds arvoraticcl
Event History
prodfwtstiirary dodler
Lat </osecate_.ppp Lpyt
</eaacure 2adi
Action
Agent produce new action
Wythasstanneeeleeeeeee tataal)
Openiaans maintained Action Execution APR
Serviay
dustapent winer,
tythoa
erocaction wol semer lestend
Fras sor ait
Action
Apytham eith commrnitatty
Shell
D0 Oroweeaaiteenn wacetiinn sagard) The serve unriing part 9000 wish ne browaw to page at 5000 -eaourt,
roleecatee Sppt vow
Abstraction
Browser
butli bomily ...
gotnt (eipp 11127
Faynoagl Croasdass
Olservation Action_.,-.) -.-]
Kiet bnled)
Agest: Roent Histooy Andion
retur ser vatern)
[M] www.ert.guuttteeeateeet
Observation
Buntime: Adtam - Otservation
Figure 2: OpenHands consists of main components: 1) Agent abstraction where community can contribute different implementation of agents ($2.1) into agenthub ($3); 2) Event stream for tracking history of actions and observations; 3) Runtime execute all actions into observations ($2.2).
for real-time feedback (Fig. 1, SD). Furthermore, the evaluation framework currently supports 15 benchmarks, which we use evaluate agents ($4).
Released under a permissive MIT license allowing commercial use, OpenHands is poised to support diverse array of research and real-world applications across academia and industry. OpenHands has gained significant traction, with 32K GitHub stars and more than 2.1K contributions from over 188 contributors. We envision OpenHands catalyst for future research innovations and diverse applications driven by broad community of practitioners.
OPENHANDS ARCHITECTURE
We next describe using OpenHands in detail. In particular, we discuss 1) how to define and implement an agent ($2.1), 2) how each action execution leads to an observation ($2.2). 3) how to reliably manage and extend commonly used skills for agents ($2.3), and 4) how to compose multiple agents together for task solving ($2.4). Fig. 2 provides an overview.
2.1 AGENT DEFINITION AND IMPLEMENTATION
An agent can perceive the state of the environment (e.g., prior actions and observations) and produce an action for execution while solving user-specified task.
The State and Event Stream. In OpenHands, the state is data structure that encapsulates all relevant information for the agent's execution. A key component of this state is the event stream, which chronological collection of past actions and observations, including the agent's own actions and user interactions (e.g., instructions, feedback). In addition to the event stream, the state incorporates auxiliary information for agent's operation, such as the accumulative cost of LLM calls, metadata to track multi-agent delegation ($2.4), and other execution-relater parameters.
Actions. Inspired by CodeAct (Wang et al., 2024a), OpenHands connects an agent with the environment through a core set of general actions. Actions IPythenRunCer lAct innioon and CmdRunAction enable the agent to execute arbitrary Python code and bash com- mands inside the sandbox environment (e.g., securely isolated Linux operating system). Browser wnneerrIneeaact iveectieiiooo in enables interaction with a web browser with a domain-specific language for browsing introduced by BrowserGym (Drouin et al., 2024). These actions to provide a comprehensive yet flexible set of primitives covering most tasks performed by human software engineers and analysts. The action space based programming languages (PL.) is powerful
chosen
3

---

Published as a conference paper at ICLR 2025
and flexible enough to perform any task with tools in different forms (e.g., Python function, REST API, etc.) while being reliable and easy to maintain (Wang et al., 2024a).
This design is also compatible with existing tool-calling agents that re- quire list of pre-defined tools (Chase,
Figure 3: Minimal example of implementing an agent in OpenHands.
2022). That is, users can easily de- fine tools using PL supported in primi- tive actions (e.g,, write a Python func- tion for calculator) and make those tools available to the agent through IsON-style function-calling experi- ences (Qin et al., 2023). Moreover, the framework's powerful PL.-based primitives further make it possible for the agents to create tools by them- selves (e.g., by generating Python functions, Yuan et al. 2023) when API complete the task is unavailable. Refer to $2.3 for how these core Pl.- based actions can be composed into a diverse set of tools.
class MininalAgent:
def reset incit) None:
selffaysten message "You are helpfol aestasann
def (selt state: State):
nessages: List (dictt str]l
1rzoio"s "cysten', 'coazent's solt.egetonunesoaee
for action, obs state.histery
action secionnnesage get_acttee set_aactio_nceeaag (pron_action)
messagee. spgend iottsooneeesaaeee obs ne saage = get rbservation fon mws.nnaseeeeeee (obs) messages, appsegee.oppendiaee. (pp.ndd
use 1in to generste responce fo.e., thought, action)
respance " selt. 11m. wo__ooo__ttiooe (messages)
pazse and exccute action action selt velt.parse__e gerre_orrpoosserrpppppee if seltiis Hinish conmandiaction
ruacine
return NgentFinisabtriion selt.is_ber condand
return CndRunact ion
elif self. se__python_cod (action)
return 1PythonRunCe witttinnniiiiiiiiooeioeiooooo ccoderaction.cooo code)
self. seli..is_bcoserreoo se.bcooserrecttooiaaa return arowseittaracctivoeeet.e (code action. codc)
Observations. Observations describe the environmental changes (e.g., exe-
else:
Messacsketionn wettanntttttaaiiinntoooo action. message)
cution result of prior actions, text messages from the human user etc.) that the agent observes.
Implement New Agent. The agent abstraction is designed to be simple yet powerful, allowing users to create and customize agents for various tasks easily. The core of the agent abstraction lies in the step function, which takes the current state as input and generates appropriate action based on the agent's s logic. Simplified example code for the agent abstraction illustrated in Fig. 3. By providing this abstraction, OpenHands allows the users to focus defining desired agent behavior and logic without worrying about the low-level details of how actions are executed ($2.2).
2.2 AGENT RUNTIME: How EXECUTION OF ACTIONS RESULTS IN OBSERVATIONS
Agent Runtime provides general environment that equips the agent with an action space comparable to that of human software developers, enabling OpenHands agents to tackle a wide range of software development and web-based tasks, including complex software development workflows, data analysis projects, web browsing tasks, and more. It allows the agent to access bash terminal to run code and command line tools, utilize Jupyter notebook for writing and executing code on-the-fly, and interact with web browser for web-based tasks (e.g., information seeking).
Docker Sandbox. For each task session, OpenHands spins up a securely isolated docker container sandbox, where all the actions from the event stream are executed. OpenHands connects to the sandbox through a REST API server running inside it (i.e., the OpenHands action execution API), executes arbitrary actions (e.g., bash command, python code) from the event stream, and returns the execution results as observations. A configurable workspace directory containing files the user wants the agent to work on is mounted into that secure sandbox for OpenHands agents access.
OpenHands Action Execution API. OpenHands maintains an API server that runs inside the docker sandbox to listen for action execution requests from the event stream. The API server maintains:
(1) A bash shell that connects with the operating system environment (specified by the docker image) for command execution.
(2) Jupyter IPython server to handle interactive python (IPython) code execution requests and return the execution results back to the event stream.
(3) A Chromium browser based on Playwright. The provider provides set of action primitives defined by BrowserGym (ServiceNow; Drouin et al., 2024), such as navigation, clicking, typing, and scrolling. The full set of actions is detailed in $J. After executing these actions, the browser
4

---

Published as a conference paper at ICLR 2025
runtime provides rich set of observations about the current state of the browser, including HTML, DOM, accessibility tree (Mozilla), screenshot, opened tabs, etc.
Arbitrary Docker Image Support. OpenHands allows agents to with different software environments by supporting runtime based on arbitrary docker images. OpenHands implements build mechanism that takes user-provided arbitrary docker image and installs OpenHands action execution API into that image to allow for agent interactions. We include detailed description of OpenHands agent runtime in SF.
arbitrary operating systems
2.3 AGENT SKILLS: THE EXTENSIBLE AGENT-COMPUTRY INTERFACE
SWE-Agent (Yang et al., 2024) highlights the importance of a carefully crafted Agent-Computer Interface (ACI, i.e., specialized tools for particular tasks) in successfully solving complex tasks. However, creating, maintaining, and distributing a wide array of tools can be a daunting engineering challenge, especially when want to make these tools available to different agent implementations ($3). To tackle these, we build an AgentSkills library. a toolbox designed to enhance the capabilities of agents, offering utilities not readily available through basic bash commands python code.
Easy to create and extend tools. AgentSkills is designed as a Python package consisting of different utility functions (i.e., tools) that are automatically imported into the Jupyter IPython environment ($2.2). The ease of defining a Python function as a tool lowers the barrier for com- munity members to contribute new tools to the library. The generality of Python packages also allows different agent implementations to easily leverage these tools through one of our core action IPythonRunCe 1lAction ($2.1).
Inclusion criteria and philosophy. In the AkentSkills library, we do not aim to wrap every possible Python package and re-teach agents their usage (e.g.. LLM already knows pandas library that can read CSV file, so we don't need to re-create tool that teaches the agent to read the same file format). We only add a new skill when: (1) it is not readily achievable for LLM to write code directly (e.g., edit code and replace certain lines), and/or (2) it involves calling an external model (e.g., calling speech-to-text model, or model for code editing (Sanger)).
Currently supported skills. AgentSkills library includes file editing utilities adapted from SWE- Agent (Yang et al., 2024) and Aider (Gauthier) like edit_file, which allows modifying an existing file from specified line; scrolling functions scroll_us up and scroll_ down. for viewing a different part of files. also contains tools that support reading multi-modal documents, like for extracting information from images using vision-language
parse parse_image and models (e.g., GPT-4V) and reading text from PDFs, respectively. A complete list of supported skills can be found in SI.
2.4 AGENT DELEGATION: COOPERATIVE MULTI- INTERACTION
OpenHands allows interactions between multiple agents as well. To this end, we use special action type DgentDelegateeeee ion, which enables agent to delegate specific subtask another agent. For example, the generalist Acteent,, with limited support for browsing, can use AgentDe legat entatecciiin to delegate web browsing tasks the specialized BrowsingAgent to perform more complex browsing activity (e.g., navigate the web, click buttons, submit forms, etc.).
3 AGENTHUB: A HUB OF COMMUNITY- CONNUNTTYYCONNRBUTTEED AGENTS
Based on our agent abstraction ($2.1) 1), OpenHands supports wide range of agent implementations for end users to choose from and act baselines for different agent tasks.
contributed
CodeAct Agent. CodeActAgent is the default generalist agent based on the CodeAct framework (Wang et al., 2024a). At each step, the agent can (1) converse to communicate with humans in natural language to ask for clarification, confirmation, etc., or (2) to perform the task by executing code (a.k.a,, CodeAct), including executing bash commands. Python code, browser-specifie programming language ($2.2). This general action space allows the agent (v1.5 and above) to perform various tasks, including editing files, browsing the web, running programs, etc.
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different AI agent frameworks ($C). SWE refers to "software engineering', Standardized tool library: framework contains reusable tools for different agent implementations ($2.3); Built-in sandbox & code execution: if it supports sandboxed execution of arbitrary agent- generated code; Built-in web browser:
provides agents access to fully functioning web browser; Human-Al collaboration: if it enables multi-turn human-Al collaboration (e.g., human can interrupt the agent during task execution and/or provide additional feedback and instructions); AgentHub: if it hosts implementations of various agents ($3); Evaluation F Framework: if it offers systematic evaluation implemented agents on challenging benchmarks ($4); Agent QC (Quality Control): if the framework integrates tests ES to
overall framework software quality.
Domole
User Inter face
Graphlc
Standandiond Text Elbeary
Ball- Sandier
Cale Esecttion
Bullih Web Browser
Colidoraiion
Mudli-geee
Hamas-A Cullibrntiin
Agentlia
Fratereark
Eralustion
OC
AusCarT Granta (202))
Cerand
LenChai (Chass 20023
Geeand
MatdPP (loeg sta 3023)
Oceand
Autdka (Wuu ei 2023)
Oceond
Autolges (Clica ci 2004
Oceond
Agebb (Z al, 302300
Occond
Nogess (Tea, 2003)
Oremd
Qpeaggres Oft 2003)
Grend
NasColdkeer (Saag at 21048)
SWE
X
X
SWE-Agos (Vaag 3154)
SWE
X
Openthands
Oceand
support. party commercial options
Browsing Agent. We implemented generalist web agent called Browsing Agent, to serve as a simple yet effective baseline for web agent tasks. The agent is similar to that in WebArena (Zhou et al., 2023a), but with improved observations and actions, with only zero-shot prompting. Full prompts are in $K.
GPTSwarm Agent. GPTSwarm (Zhuge et al., 2024) pioneers the use of optimizable graphs to construct agent systems, unifying language agent frameworks through modularity. Each node represents a distinct operation, while edges define collaboration and communication pathways. This design allows automatic optimization of nodes and edges, driving advancements in creating multi-agent systems.
Micro Agent(s). In addition, OpenHands enables the creation of micro agent, an agent specialized towards a particular task. A micro agent re-uses most implementations from an existing generalist agent (e.g., CodeAct Agent). It is designed to lower the barrier agent development, where community members can share specialized prompts that work well for their particular use cases.
4 EVALUATION
To systematically track progress in building generalist digital agents, listed in Tab. 2, we integrate es-
Table 2: Evaluation benchmarks in OpenHands.
Categary
Benchmark
Required Capability
SWE Bench Diizenez et 2004)
Fixing Gitbub isses Fining SQL
tablished benchmarks into OpenHands.
HanenEvaliss (Micnmigholl sli, 2004)
20233)
These benchmarks cover software en- gineering, web browsing, and miscel- laneous assistance. In this section, we compare OpenHands open-source producible baselines that do not perform manual prompt engineering specifically based on the benchmark content. Please note that we use 'OH' as shorthand for HaenHands for the rest of this section for brevity reasons.
(ling 20246)
Bistnfommatics coding
MI-Bersh
Mushine leamiigg Swttwar AM cailing
(oo la APlilench (Phil
ToolQA (/hong 2013g)
Tooi
Web
MidlWi (Liu al. 2010))) 2014)
WethAreaa (Zhou al. 2023g)
Goal plarning realistic beowong Short injectiey on systletic web
(Mislon cial.,
buooing, multi-m-slallty Graduzee Google proof Q&A
GPDA
Mise.
MINT
Apentlinch 20248)
Operating system groblmm
Deduction Amna (Zhong al, 20134a)
State tracking phining
PoufWwieer (Tifjeed atal. 2011y
Dualcttoon Roowning
4.1 RESULT OVERVIEW
In OpenHands, our goal is to develop general digital agents capable interacting with the world through software interfaces (as exemplified by the code actions described in 12. We recognize that a software agent should excel not only in code editing but also in web browsing and various auxiliary tasks, such answering questions about code repositories conducting online research.

---

Published as a conference paper at ICLR 2025
Table 3: Selected evaluation results for OpenHands agents ($4). See Tab. (software), Tab. 5 (web),
Tab. (miscellaneous assistance) for full results across benchmarks.
Software ($4.2) SWE. Bench Lite
Web (64.3) WehArena
Misc. (84.4)
Agent
Model
GPQA
GAIA
Software Engineeriing Agentts
SWe.Agget (Yang 2024)
gpt-4-11 1166pprv1
AutoCodcRover (Zhang al,, 2024b)
gpt 4 0125 preview
19.0
Aider (Gauthier)
gpt-4o & (aaude-3-opus claude got-4o
26.3
Mootless Tools (Orwall)
26.7
Agentless (Xia et al., 2024)
27.3
Web Browsing Agents
Lemur (Xu al., 2023)
Lemur- Sertzr--hat-790 Trained 72B w/ symbetic data
5.3
Patel et al. (2024)
9.4
AutoWehGI M (i
2024)
Thained 7B w/ human/agent annotation
18.2
Auto Eval & Refine (Pan ot al. 2024)
GPT-4 + Reflexion w/ GPT-4V
20.2
WebArenn Agent (Zhou ol., 2023a)
gpt 4 curbo
14.4
Mia. Assistance Agents
AutoGPT (Gravitas, 2023)
gpt 4 turbo
13.2
Few shot Prompting
Llama-2- 2-hb-hchtt
28.1
Chain-of- Choug-of-Thought (Rein 2023)
gpt-3. -3ttbb3-tbbb--6k
29.6
qpt-4
38.8
OpenHands Agents
gpt 40 mini 2024 07 18
6.3
8.3
CodeActAgent v
20L-44-2024-05-2
14.5
*53.1
claude-3-S-sonree
26.0
15.3
52.0
GPTSwamm v
200--4--024-300
32.1
Numbers reported from CodeActAgent
Tab. showcases a curated set of evaluation results. While OpenHands agents may not achieve top performance in every category, they are designed with generality in mind. Notably, the same CodeAct agent, without any modifications to its system prompt, demonstrates competitive performance across three major task categories: software development, web interaction, and miscellaneous tasks. This is particularly significant when compared to the baseline agents, which are typically designed and optimized for specific task categories.
4.2 SOFTWARE ENGINEERING
Next, we report results specifically for software engineering benchmarks in Tab. 4.
SWE-Bench (Jimenez et al., 2024) is designed assess agents' abilities solving real-world GitHub issues, such as bug reports or feature requests. The agent interacts with the repository and attempts fix the issue provided through file editing and code execution. The agent-modified code repository is tested against test suite incorporating new tests added from human developers' fixes for the same issue. Each test instance accompanies a piece of "hint text" that consists of natural language suggestions for how to solve the problem. Throughout this paper, we report all results without using hint text. canonical subset, SWE-bench Lite, is created to facilitate accessible and efficient testing. We default to this subset for testing for cost-saving consideration.' Result. As shown in Tab. 4, our most recent version of CoteAct v1.8, using claude-3 3. 5-sonnet, achieves a competitive resolve rate of 26% compared to other open-source SWE specialists.
4.2.1 HUMANEVALFIX
HumanEvalFis (Muennighoff et al., 2024) tasks agents fix bug in provided function with the help of provided test cases. The bugs are created to ensure one or more test cases fail. We focus on the Python subset of the benchmark and allow models to solve the bugs by self-debug over multiple turns, incorporating feedback from test execution. We follow the setup from Muennighoff et al. (2024) using @ @k (Chen al., 2021). Results. In Tab. OpenHands CodeActAgent successfully fixes 79.3% of bugs in the Python split. This is significantly better than all non agennic approaches, almost doubling the performance of St Staaroderr222
(Lozhkov al., 2024: Li et al., 2023c). While SWE-Agent achieves 87.7%, Yang et al. (2024) provides the model full demonstration of a successful sample trajectory fixing one of the bugs in the test dataset ("1-shot"", whereas our evaluation of OpenHands is 0-shot. As HumanEvalFix has been created by humans and all bugs
Running the complete of 2294 instances $6.9k, using conservative estimate of $3 per instance.

---

Published as a conference paper at ICLR 2025
Table 4: OpenHands Software Engineering evaluation results ($4.2).
Agent
Model
Success Rate (S)
S.y Cost
SWR-Bench Lite (Fimenez ot al, 2024), 300 instances, wol Hist
SWE-Agent (Yang et ol,, 2024) AutaCodeRover (Zhing 2024b) Aider (Gautheer)
gpc 4-1106 prreview
18.0
1.67
3pt-6-0125-pxee -0-025-pxavioo gpc-co claude-3-apus
gpt
OH CodeActAgent v1.8
gpt-4e- -00---4--22------0
22.0
1.72
clauda-3- 1a-uda-3--55 5-880n002222222622
26.0
1,10
HumanEEalFF ix (Muensighoff
al. 2024), 164 instances
210030--1
16.6
DotooCder-115
Prompting,
DeepSsekCoder: Inacruct
3tazCodez2-15B
48.6
SWE-agent, I-shot (Yang al., 2024)
opt-4-cozbo
87.7
OH CodeAAtAAent Geneealisz, O-shor.
9pt-3.3-turbo- 66x-6613
9pc-to-2026-05-3
20.1
0.11
Promgting, O-shox
CodeOven: 73 73-Chat
Codel-lama Inst
31.3
Off CodeActAgent v1.5
1oe-44-1100 4-11-4--1666-80 --1106-ppreviee
42.7
0.19
2B0--4-22244--133
MI--Bench (Tang 20216), instances
opt-).3-turbo
prompting + RM25,
get 40-2024-05-13
22.1
gpt
26.2
SWF-Agent (Yang et ol,, 2024) Aider (Gautheer)
9-1-4-1106
42.6
1.91
gpt-4o
OH CodeActAgent vi.5
gpt-8-1106-pzene
58.8
1.22
(ptt-3... 3.5-ttrbb--ooo 1e 66-0613
13.2
0.12
BioCuder (Pythom) (Tong ot al 20246)) 137 instances
prompting O-shot
gpt 3.33-.ttbubo
11.0
9pt-4-1106 preview
Olf CodeActAgent
Gorilla gpc-to-2024-05-13
2023),
claude-vi
8.7
Prompting. O-shox
gpt-4-0314
21.2
gp--3.5-turbo-1301
29.7
Gorilla, finetuned API calls, O-shot (Putil 2023; Touvron al., 2023)
11ama-7b
75.0
gpc-3.3-turbo-0225
OH CodeAAtAgent v1.5
9pc-1o-2025-05-13
36.4
0.04
ToolQA (Zhnung al., 2024), instances
Chat CoT
5.1
Prompting. O-shor
ChatGeT
5.5
Chansleon
10.6
ReAct, O-shot (Yio et al., 2023; OpenAl, 2024a)
ap:-3
gpt-b.3-turbo
43.1
OHf CodoActAgent v1.5
gp=-3.3-turbo-D2 op:-1o-2024-05-13
2.3
0103
47.2
0.91
carefully validated, achieving 100% on this benchmark is entirely feasible, which we seek to do in future iterations of OpenHands.
ML.-Bench (Tang et al., 2024b) evaluates agents' ability to solve machine learning tasks across 18 GitHub repositories. The benchmark comprises 9,641 tasks spanning 169 diverse ML problems, requiring agents to generate bash scripts or Python code in response to user instructions. In the sandbox environment, agents iteratively execute commands and receive feedback, allowing them to understand the repository context and fulfill user requirements progressively. Following the setup from the original paper, we perform agent evaluation on the quarter subset of ML-Bench.
Gorilla APIBench (Patil et al., 2023) evaluates agents' abilities to use APIs. it incorporates tasks on TorchHub, TensorHub, and HuggingFace. During the evaluation, models are given a question related to API usage, such as "identify an API capable of converting spoken language in a recording to text." Correctness evaluated based on whether the model's API call is in the correct domain.
ToolQA (Zhuang et al., 2024) evaluates agents' abilities to use external tools. This benchmark includes tasks on various topies like flight status, coffee price, Yelp data, and Airbnb data, requiring the of various tools such text tools, database tools, math tools, graph tools, code tools, and system tools. features two levels: easy and hard. Easy questions focus more on single-tool usage, while hard questions emphasize reasoning. We adopt the easy subset for evaluation.
BioCoder (Tang et al., 2024c) is repository-level code generation benchmark that evaluates agents' performance on bioinformatics-rell.. related tasks, specifically the ability to retrieve and accurately utilize context. The original prompts contain the relevant context of the code; however, in this study, have removed them demonstrate the capability of OpenHands to perform context retrieval,
8

---

Published as a conference paper at ICLR 2025
Table 5: OpenHands Web Browsing Evaluation Results ($4.3).
Agent
Model
Success Rute (8)
$ A.v.g. Cost
WebArena (Zhou al., 2025c). 812 instances
al., 2023)
Lemur-chat
Patel et (2024)
Trained 728 with ceff-imprvement synthetic datn
9.4
AM (Laii et 2024)
Truined 7B with human/ugent hybrid unnotation
18.2
Auto Refine (Pan al., 2024)
GPT-4 Reflexion w/ GPT-4V reward
20.2
WehArena Agent (Zhou al., 2023a)
ept-3.5-turth
6.2
spt-4-Lusbo
4pt-74o-mini-2224-071
8.5
0.01
OH BrowsingAgent v1.0
spt-4c-2024-05-13
14.8
0.15
0.10
spt-44--aa1n1-2224-0--11
8.3
OH CodeActAgent v1.8
cgtt-4g-2024-5--33
delegatiin BrowsingAgem v1.0
15.3
MiniWoB++ (Lia et al., 2018), 125 environments
Warkflow Guided Exploration (Liu 2018)
Truined specialist model with envirommnnt exploration
34.6
CC-NET (Humphreys et 2022)
Trinee specialist model and human annotated
91.1
OHI BrowsingAgent v1.0
gpt-3. 5-turbo-ooo
27.2
0.01
spt 4pt-4e-2021105-13
OH CodeAetAgant v1.8
delegation BeeosingAAent
<pt--te-002105-13
39.8
self-debugging, and reasoning in multi-turn interactions. BioCoder consists of 157 Python and 50 Java functions, each targeting specific in bioinformatics, such proteomics, genomics, and other specialized domains. The benchmark targets real-world code by generating code in existing repositories where the relevant code has been masked out.
BIRD (Li et al., 2023b) is a benchmark for text-to-SQL tasks (i.e., translate natural language into executable SQL) aimed realistic and large-scale database environments. We select 300 samples from the dev set to integrate into OpenHands and evaluate on execution accuracy. Additionally, we extend the setting by allowing the agent to engage in multi-turn interactions arrive the final SQL query, enabling it to correct historical results by observing the results of SQL execution.
4.3
WEB BROWSING
We report evaluation results for web browsing benchmarks in Tab. 5.
WehArena (Zhou et al., 2023a) is self-hostable, ceccution-based web agent benchmark that allows agents to freely choose which path to take in completing their given tasks. WebArena comprises 812 human-curated task instructions across various domains, including shopping, forums, developer platforms, and content management systems. Results. From Tab. 5, we can see that our Browsing achieves competitive performance among agents that use LLMs with fomain-general prompting techniques.
MiniWoB++ (Liu al., 2018) an interactive web benchmark, with built-in reward functions. The tasks are synthetically initialized on 125 different minimalist web interfaces. Unlike WebArena, tasks are easier without page changes, require fewer steps, and provide low-level step-by-step task directions. Note that it contains a portion of environments that require vision capability tackle successfully, and many existing work choose to focus only on subset of the tasks (Kim et al., 2024; Li et al., 2023d; Shaw et al., 2023). Still, report the performance the full set and only include baselines that are evaluated on the full set.
4.4 MISCELLANEOUS ASSISTANCE
Results for miscellaneous assistance benchmarks are reported in Tab.
GATA (Mialon et al., 2023) evaluates agents' general task-solving skills, covering different real-world scenarios. It requires various agent capabilities, including reasoning, multi-modal understanding, web browsing, and coding. GAIA consists of 466 curated tasks across three levels. Setting up GAIA traditionally challenging due to the complexity of integrating various tools with the agent, but OpenHands's infrastructure (e.g., runtime $2.2, tools $2.3) simplifies the integration significantly.
GPQA (Rein et al., 2023) evaluates agents' ability for coordinated tool use when solving challenging graduate-level problems. Tool use (e.g., python) and web search often useful to assist agents in answering these questions since they provide accurate calculations that LLMs are often incapable of and access to information outside of the L.M''s parametric knowledge base.

---

Published as a conference paper at ICLR 2025
Table OpenHands miscellaneous assistance evaluation results ($4.4).
Agent
Model
Success Rate (%)
Avg. Cost
GATA (Mialon et al, 2023), LI validation set, 53 instances
AutoGPT (Gravitas, 2023)
gpt --tubb
GPTSwarm v1.0
gpt---0123-previer
30.2
0.110
gpt 4o 2024
32.1
0.050
GPQA (Rein et al,, 2025), diamond set, 198 instances (refer to $G, Tab. for other subsets)
Expert
Humam (Rein al., 2023)
Non expee human
21.9
Fow-shot Prompting Chain-of-Thought (Rein ot al., 2023)
gpt gpt-3.5 3.5 3.5-Luzbo Luzbo gpt-4
16x
29.0
38.8
OHI Code/AttAgent v188
52.0
0.065
AgentBench (Liu et al, 2023), OS (bash) subset, 144 instances
AgentBench Baseline Agent (Liu al, 2023)
qpt gpt 3.5-3.5.0 turbo t.5-tubb
42.4
32.6
OH Code:ActAgent v1.5
gpt spt-4o-2024-0--13
57.6
0.085
9pt-3.5-turbo-0122
11.8
0.006
(Wang 2024h); mat subset,
MINT Baseline Agent
gpt 4 0613
66.8
OIf CodeAAcAAeent v1.5
gpt spt-4o-2024-0--3
77.3
0.070
1Bt-3.--turoo-16--005
33.8
0.048
(Wang 2024h); code subset, gpt 0613
MINT Baseline Agent
59.6
gpt gpt-4o-2024-05-13 DBt-3.--turbo116-066
50.0
DJ087
OII CodeAAtAgeent v1.5
5.2
0.030
ProofWriter (Tafjori et 2021),
Few-shot Prompting Chain-of- Thought (Pan et al., 2023)
68.1
Logic-LM (Pan et al., 2023)
gpt4 symbolic solver
79.6
CodeActAgent
gpt 2021-022202 2021-05-13
Entity Deduction Arena (Zhang et al., 2024a), 200 instances
Zero-shot Prompting (Zhang al., 2024a)
gpt-4-0314
40.0
spt 3. 305-3.--tuzbo-065
27.0
Code:AtAAennt
gpt
24.0
AgentBench (Liu et al., 2023) evaluates agents" reasoning and decision-making abilities in multi- turn, open-ended generation setting. We selected the code-grounded operating system (OS) subset with 144 tasks. Agents from OpenHands interact directly with the task-specific OS using bash commands in a multi-turn manner, combining interaction and reasoning automate task completion.
MINT (Wang et al., 2024b) is benchmark designed to evaluate agents' ability to solve challenging tasks through multi-turn interactions using tools and natural language feedback simulated by GPT-4. We use coding and math subsets used in Yuan et al. (2024). We follow the original paper and allow the agent to interact with up five iterations with two chances to propose solutions.
ProofWriter
(Tafford et al., 2021) is synthetic dataset created to assess deductive reasoning abilities of L.Ms.. Same as Logic-LM (Pan et al., 2023), we focus on the most challenging subset, which contains 600 instances requiring 5-hop reasoning. To minimize the impact of potential errors in semantic parsing, we use the logical forms provided by Logic-LM.
Entity Deduction Arena (EDA) (Zhang et al., 2024a) evaluates agents' ability deduce unknown entities through strategic questioning, akin the 20 Questions game. This benchmark tests the agent's state tracking, strategic planning, and inductive reasoning capabilities over multi-turn conversations. We evaluate two datasets "Things" and "Celebrities", each comprising 100 instances, and report the average success rate over these two datasets.
CONCLUSION
introduce OpenHands, community-driven platform that enables the development agents that interact with the world through software interfaces. By providing powerful interaction mechanism, a safe sandboxed environment, essential agent skills, multi-agent collaboration capabilities, and a comprehensive evaluation framework, OpenHands accelerates research innovations and real-world applications of agentic Al systems. Despite challenges developing safe and reliable agents (SA), excited about vibrant community and look forward OpenHands's continued evolution.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do can, not i say: Grounding language in robotic affordances. arXiv preprint 1rXiv:2204.01990 2022.
Daniil A Boiko. Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570-573 2023. 1
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi. Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-4,,2022
Harrison Chase. LangChain, October 2022. URL http:////ggiiiib.ooom////aaaaaa/
4, 6,
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation, 2024. 1, 6, 18
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021. 7
Cognition.ai. Introducing devin, the first ai software engineer. URL httpss//www.conniiooonbbbbbb introducing-devin.
CrewAl. CrewAl, October 2024. URL http://ggitubboom///rwwl//ll///t
18
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases, 2023. 18
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024. 3,4
Paul Gauthier. How aider scored sota 26.3% on swe bench lite aider. https///iderrchaa/224455222
swe-bench-lite.hmme htmml Accessed: 2024-06-05, 7,
Significant Gravitas. Auto-gpt: An autonomous gpt-4 experiment, 2023. URL https://githuh.
hmmpiigntiiicat-GGrrvvaaaaatttto 2023. 6, 7, 10, 18
Sirui Hong. Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning
Representations, 2023. 1, 6, 18
Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi-agent- based code generation with iterative testing and optimisation, 2024. 19
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornoon, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pp. 9466-9482. PMLR, 2022. 9
IPython. Jupyter and the future of IPython - IPython. URL htpp://ppyytoonn.rggg. 4
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint ArXivv2201.0000 1.0088, 2024.1
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can Language Models Resolve Real- Real-world Github Issues? In The Twelfth International Conference on Learning Representations, 2024, URL https//opentreiiew.ee
forun 6, 7, 8, 17, 19
11

---

Published as a conference paper at ICLR 2025
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. 2024. 18
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Hanyu Lai, Xiao Liu, Iat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce large language model-based web navigating agent. arXiv preprint arXiv:2404.085 036448 2024. 7,9
Hareton K. N. Leung and Lee J. White. A study of integration testing and software regression at the integration level. In Proceedings of the Conference on Software Maintenance, ICSM 1990, San Diego, CA, USA, 26-29 November, 1990, pp. 290-301, IEEE, 1990. doi: 10.1109/CSM.1990. 131377. URL https:///oi.org// 10.1109/ICMMM.MM 1990.7 19
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303. 17760, 2023a. 18
Jinyang Li, Binyuan Hui, GE QU, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a Blg bench for large-scale database grounded text-to-SQLs. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b. 6, 8, 9,9, 17
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jía Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov. Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Naday Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dalan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023c. URL http:://ariiv.orgaa///23.......
Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A zero-shot language agent for computer control with structured reflection. arXiv preprint arXiv:2310.087400 2023d. 9
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow- exploration. In International Conference on Learning Representationss (ICLR), 2018. URL https://arrxivorr//ab// 80188002.8808 08802. 6, 9, 17
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Evaluating Ilms as agents, arXiv preprint arXiv: 2308. 03688, 2023. 6, 10, 17
Agentbench:
Anton Lozhkov, Raymond Li. Loubna Ben Allal, Federico Cassano, Joel PPirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He. Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze. Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa
12

---

Published as a conference paper at ICLR 2025
Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024. URL https://axiiv.org//b///000..1119
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. CoRR, abs/2311. 122983, 2023, doi: 10.48550/ARXIV.2311.. 129833. URL http://dd...ogr000005555//vxv...
6, 6,9, 10, 17
Mozilla. Accessibility tree MDN Web Docs Glossary: Definitions of Web-related terms MDN. URL http::/deeeloprr.mozlllaa.ooomm http://wwww.aaaoooooooooooooooaa UScdess/lessrry/ccesiiilllyyyy tree. 5
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models, 2024. 8,
Y Nakajima. Babyagi. URL http://ghhia...ooooyhhrriiiiibbb 2023. 18
OpenAl. Chatgpt: May 2024 version. https///www.ooennni.ccmmmaaaggg 2024a. Accessed: 2024-05-
29. 8, 18
OpenAl. Hello gpt-4o. https//lopemiiicommmddddccllleep 2024b. Accessed: 2024-05-15.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga. Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambatista Parascandolo, Joel Parish. Emy Parparita. Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston
13

---

Published as a conference paper at ICLR 2025
Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. 18
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.0647 064444 2024. 7,9, 18
Liangming Pan, Alon Albalak, Xinyi Wang. and William Yang Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint 122. 2023. 10
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. 18
Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison- Burch, and Sepp Hochreiter. Large language models can self-improve at web agent tasks, arXiv preprint arXiv:2405. 20309, 2024. 7.9
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph F. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 2023. 6, 8, 17
Playwright. Fast and reliable end-to-end testing for modern web apps Playwright. URL hitps: //playwright.dww.. 4
Chen Qian, Xin Cong, Wei Liu, Cheng Yang. Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023. 18, 19
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real- world apis. CORR, abs/2307.16789 2023. doi: 10-4855//ARXXV2237...7777 URL https://do..or//I 0. 485500/r2Xi.2307..66666
4
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint arXiv:2311.12022, 2023. 6, 7, 9, 10, 17, 23
Aman Sanger. Near-instant full-file edits. http:://www.ccansccc.oom///nn/na Acces// Accessed: 2024-06-05.
ServiceNow. BrowserGym: a Gym Environment for Web Task Automation, URL https///gituub.com ServiceNow//roosserGyy. 2, 4, 18
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to ui actions: Learning follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36: 34354-34370, 2023. 9
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 18
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 3621-3634, Online, August 2021. Association for Computational Linguistics. doi: 10. 1 1/8853/11/2222 findings-acl.317. URL http::///elaattoooooo///11111nnn
6, 10, 17
14

---

Published as a conference paper at ICLR 2025
Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang. et al. Prioritizing safeguarding over autonomy: Risks of Ilm agents for science. arXiv preprint arXiv:2402.0422 2024a. 1
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang. Tianyu Liu, Zhiwei Jiang, Baobao Chang. Yin Fang. Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. ML-BBenh: Evaluating large language models and agents for machine learning tasks repository-level code, 2024b. URL https://arxii.orr/abs//3111 1.09835. 6, 8. 17
Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. BioCoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40 (Supplement_ (ettrmmeen___ii266666__
06 2024c. ISSN 1367-4811. 6, 8. 17
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning, 2024d. 18
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023. 1
XAgent Team. Xagent: An autonomous agent for complex task solving, 2023. 6, 18
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable Code Actions Elicit Better LLM Agents. In ICML, 2024a. 2, 5,
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. In ICLR, 2024b. 10, 17
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen IIm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08/55. 2023. 1, 6, 18
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint, 2024.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv::3100 2310.10634 2023.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. arXiv preprint 100.v22310006830, 2023. 7.9
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. 2, 5, 6, 7, 8, 19
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL http::/opeanevie..eeffommmm ?id vluYUL-X. 8
Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao. Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, et al. Proagent: From robotic process automation to agentic process automation. arXiv preprint 2rXiv:2311.10755 2023. 18
15

---

Published as a conference paper at ICLR 2025
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. CRAFT: customizing 1ims by creating and retrieving from specialized toolsets. CoRR, 0as/2309.17428.. 2023. doi: 10.48500ARRXI.2300.7722
URL http://bw...........oo////..2/00
4
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024. 10
Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. Probing the multi-turn planning capabilities of Ilms via 20 question games. 2024a. 6, 10, 17
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement, 2024b. 6, 7, 8, 19
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2023a.
1, 7, 7,
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang. Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents, 2023b. 6, 18
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 36, 2024. 6, 17
Mingehen Zhuge, Haozhe Liu, Francesco Faccio, Dylan Ashley, Róbert Csordás, Anand Gopalakr- ishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrman, Kazuki Irie, et al. Mindstorms in natural language-based socicties of mind. arXiv preprint arXiv:2305.17066, 2023. 18
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jurgen Schmidhuber. Language agents as optimizable graphs. arXiv preprint arXiv:2402. /6823, 2024, 6, 17, 18
Albert Örwall. Moatless tools. URL http:://.gttiii.ccoo///o///a//as
16

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
This work was an open-source collaborative effort across multiple institutions. We employed a point-based system determine contributions and award authorships, with technical contributions tracked and measured in units of pull requests (PRs)'. Xingyao Wang led the project, coordinating overall development and paper writing efforts. Detailed contributions were as follows:
Agent Development ($3): Xingyao Wang led the implementation of CodeAct Wang et al. (2024a) and CodeActSWE agents. Frank Xu led the development web browsing agents Zhou et al. (2023a). Mingehen Zhuge orchestrated the integration of the GPTSwarm agent Zhuge al. (2024). Robert Brennan and Boxuan Li lead the development of the Micro Agent.
Architectural Development (Fig. 2): Robert Brennan initiated the architecture design. Boxuan Li, Frank F. Xu, Xingyao Wang. Yufan Song, and Mingzhang Zheng further refined and expanded the architecture. Boxuan Li implemented the initial version of integration tests ($E), maintained the agentskills library ($2.3), managed configurations, and resolved resource leaks in evaluation. Frank F. Xu developed the web browsing environment ($J) for both agent execution and evalua- tion and integrated it with both agent and front-end user interfaces. Xingyao Wang authored the initial code for the agentskills library and the Docker sandbox. Yufan Song implemented cost tracking for evaluation, while Mingzhang Zheng developed an image-agnostic docker sandbox for more stable SWE-Bench evaluation.
Benchmarking, Integration, and Code Review: Boxuan Li and Yufan Song led benchmark integration efforts, including coordination, evaluation, and code review. Yufan Song also helped track PR contributions. Graham Neubig. Xingyao Wang, Mingzhang Zheng, Robert Brennan, Hoang H. Tran, Frank F. Xu, Xiangru Tang, Fuqiang Li, and Yanjun Shao provided additional support in integration and code reviews. Specific benchmark contributions included:
SWE-Bench Jimenez et al. (2024): Bowen Li and Xingyao Wang
WebArena Zhou et al. (2023a) and MiniWob++ Liu et al. (2018): Frank F. Xu
GAIA Mialon et al. (2023): Jiayi Pan (integration) and Mingchen Zhuge (GPTSwarm evaluation)
API-Bench Patil et al. (2023) and ToolQA Zhuang et al. (2024): Yueqi Song HumanEvalFix Muennighoff et al. (2024): Niklas Muennighoff and Xiangru Tang ProofWriter Tafjord et al. (2021): Ren Ma
MINT Wang et al. (2024b): Hoang H. Tran
AgentBench Liu et al. (2023): Fuqiang Li
BIRD Li et al. (2023b): Binyuan Hui
GPQA Rein et al. (2023): Jaskirat Singh
BioCoder Tang et al. (2024c): Xiangru Tang and Bill Qian
ML-Bench Tang et al. (2024b): Xiangru Tang and Yanjun Shao
Souity-DeductionnAreeee Zhang et al. (2024a): Yizhe Zhang
Advising: Graham Neubig advised the project, providing guidance, resources, and substantial paper edits. Heng Ji and Hao Peng offered additional project advice and assisted with paper writing. Junyang Lin contributed advisory support and sponsored resources.
A LIMITATIONS AND FUTURE WORK
We are excited about the foundations our vibrant community has laid in OpenHands and look forward to its continued evolution. We identify several directions for future work:
Enhanced multi-modality support. While our current implementation already supports a wide range of file formats through predefined agent skills, we are interested in enabling mouli-yodaliity in principled way through standard IPython and browser integration, such as viewing images and videos using vasion-language model through browser or processing XLSX files with code.
Stronger agents. Current agents still struggle with complex tasks, and we are interested in building better agents through both training and inference time techniques.
For details, please refer https:/githuh..om/AA/ Hands- Ai0addd-AAOOpppaaad////ll11111
17

---

Published as a conference paper at ICLR 2025
Agent editing improvements. Current agent suffers lot when editing long files, and interested in exploring different approaches to improve the file editing performance of agents.
Web browsing improvements. Due to the extensible nature of OpenHands. orthogonal components that could improve agents can be integrated easily. For example, thanks to OpenHands's 's extensible architecture, Auto Eval & Refine Pan et al. (2024), agent retry-on-errof strategy with Reflex- ion Shinn et al. (2024) prompts and task completion reward models, will be integrated as an optional component attached our browsing agent.
Automatic workflow generation. Currently, OpenHands's workflow still requires a substantial hand- crafted workload. We believe that graph-based frameworks such GPTSwarm Zhuge et al. (2024) and LangGraph Chase (2022) could serve as alternative solutions for building agents. Particularly in GPTSwarm, when agents are constructed using graphs, becomes casier to incorporate various optimization methods (e.g., reinforcement learning, metaa-romppinng. OpenHands considers these methods lay the groundwork for promising solutions in automatic workflow generation future versions.
B ETHICS STATEMENT
Most Al agents today are still research artifacts and lack the ability to perform complex, long-horizon tasks in the real world reliably. However, as their performance continues to improve and they are increasingly deployed in real world, they have the potential to boost productivity while also posing security risks to society significantly. OpenHands helps mitigate risks by:
(1) Enabling systematic evaluation of these agents, which can identify and address risks before they are widely deployed.
(2) Facilitating human-agent -ggent interaction rather than allowing agents to operate autonomously without oversight.
(3) More importantly, we hope OpenHands allows researchers worldwide access the best suites of agents to conduct frontier safety research towards building safe and helpful agents.
C RELATED WORK
The breakthroughs in large language models (LLMs)) like ChatGPT OpenAl (2024a) and GPT- 4 OpenAl al. (2024) have significantly enhanced the capabilities of autonomous agents across various domains Ye et al. (2023); Tang et al. (2024d); Park et al. (2023); Cui et al. (2023). These advances have spurred a multitude of generalist agent proposals Gravitas (2023); Nakajima (2023); Wu et al. (2023) aimed at performing diverse user tasks and have gained attention from both developers and broader audiences. Notable works such as Auto-GPT Gravitas (2023) harness LLMs for task completion by decomposing user goals into executable steps. Multi-agent collaboration systems leverage LLMS for elements like role-playing and task-solving capabilities Zhuge et al. (2023); Li al. (2023a); Zhou et al. (2023b); Team (2023), with MetaGPT Hong et al. (2023) emphasizing standardized operating procedures, and AutoGen Wu et al. (2023) providing a conversation framework for interactive systems. AGENTS Zhou al. (2023b) and AutoAgents Chen et al. (2024) offer new paradigms for customizable agent architecture, while XAgent Team (2023) and GPTSwarm Zhuge (2024) introduce complex management systems and optimizable graphs, respectively, for enhanced agent operations.
This surge agent development has led to specialized frameworks aimed streamlining agent implementation. LangChain and LangGraph Chase (2022) provide foundational building blocks with basic runtime support, while CrewAl CrewAl (2024) focuses on orchestrating multi-agent communications. BrowserGym ServiceNow specifically targets web browsing capabilities, and DSPy Khattab et al. (2024) emphasizes end-to-end prompt optimization. AutoGen Wu et al. (2023) advances beyond basic frameworks by implementing Python and bash execution capabilities, though with stateless command execution, while frameworks like CrewAl offer sandboxed but limited code interpreter features.
Software development, a front-runner in applying LLM-based agents, has seen advancements in frameworks for facilitating the development processes Hong al. (2023); Qian et al. (2023). In-
18

---

Published as a conference paper at ICLR 2025
novations such as ChatDev Qian et al. (2023) automate the software development lifecycle akin to the waterfall model, and AutoCodeRover Zhang et al. (2024b) addresses GitHub issues via code search and abstract syntax tree manipulation. AgentCoder Huang al. (2024) iteratively refines code generation with integrated testing and feedback, while SWE-Agent Yang et al. (2024) integrates LLMMs for automated Github issue fixing, streamlining software engineering.
D GRAPHICAL USER INTERFACE
Besides running from the command line, OpenHands features a rich graphical user interface that visualizes the agent's current actions (e.g., browsing the web, executing base commands or Python code, etc.) and allows for real-time feedback from the user. Screenshots of the UI are shown in Fig. The user may interrupt the agent at any moment to provide additional feedback, comments, or instruction while the agent is working. This user interface directly connects with the event streams ($2.1) to control and visualize the agents and runtime, making it agent and runtime agnostic.
E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS
Integration tests Leung & White (1990) have long been used by software developers to ensure software quality. Unlike large language models with simple input-output schema, agents are typically complex pieces of software where minor errors can be easily introduced during the development process and hurt final task performance. While running a full suite evaluation ($4) is the ultimate measure of performance degradation, running them for every code changes can be prohibitively slow and expensive. 4. In OpenHands, we pioneer an end-to-end agent test framework that tests prompt regression, actions, and sandbox environments. It combines integration testing from software engineering and foundation model mocking for deterministic behavior to prevent the accidental introduction of bugs during agent development.
Defining an integration test. The integration test framework for OpenHands is structured to validate end-to-end functionality by automating task execution and result verification. Developers define tasks and expected results; for instance, a task might involve correcting typos in a document named "bad.txt". Upon task execution through OpenHands, outputs are compared against a predefined "gold file" to ensure accuracy.
Mocking LLM for deterministic behavior. Addressing the challenge of non-determinism in large language models (LLMs) and the associated high costs, the framework intercepts all LLM calls and supplies predefined responses based on exact prompt matches. This method not only ensures consistency in test outcomes but also reduces operational costs by minimizing the reliance on real LLMs.
Regenerate LLM responses breaking changes. Prompt-response pairs are managed through a script that generates and stores these pairs when new tests are introduced or existing prompts are modified. For routine tests, the framework attempts to reuse existing LLM responses by slightly adjusting the prompts. Substantial changes that affect task handling require regeneration these pairs using real LLMs.
Benefits integration tests. The framework offers several advantages, including 1) Prompt regres- sion testing: Stored prompt-response pairs facilitate change tracking and provide reference for new team members to understand LLM interactions, 2) Multi-platform support: Tests are automatically scheduled for every pull request and commit on the main branch, running across multiple platforms, environments, and agents, including Linux and Mac, and in local, SSH, and exec sandboxes, and 3) Comprehensive error detection: It captures errors prompt generation, message passing, and sandbox execution, thereby maintaining high test coverage.
Running a SWF-Bench I. Lite Jimenez al. (2024) evaluation with go around 600 USD
19

---

Published as a conference paper at ICLR 2025
User sarripro Custom Docker Image
Openilands Backend
Spawn
Spown
Agent
Obtain
EventStream
Execute Action
OH Runtime Image
Get Observation Via REST API
Launches
Docker Contalge
Runtime Client
Initializes
Initializes
inttaalizes
Browser
Bash Shell
Plagins
initlelizes
Jupyter Server
Figure 4: OpenHands runtime workflow.
F HOW OPENHANDS RUNTIME WORK
F.1 WORKFLOW
The OpenHands Runtime system uses client-server architecture implemented with Docker contain- ers. See Fig. 4 for an overview of how works.
(1) User Input: The user provides custom base Docker image.
(2) Image Building: OpenHands builds
Docker image (the "OH runtime image") based on the user-provided image. This new image includes OpenHands-specifcc code, primarily the "runtime client" (i.e., runtime API server described in $2.2).
(3) Container Launch: When OpenHands starts, launches a Docker container using the OH runtime image.
(4) Communication:
The OpenHands backend (runtime py) communicates with the runtime client over RESTful API, sending actions and receiving observations
(5) Action Execution:
The runtime client receives actions from the backend, executes them in the sandboxed environment, and sends back observations
Observation Return: The client sends execution results back OpenHands backend event stream as observations.
The role the client:
It acts as an intermediary between the OpenHands backend and the sandboxed environment
It executes various types of actions (shell commands, file operations, Python code, etc.) safely within the container
It manages the state the sandboxed environment, including the current working directory and loaded plugins
20

---

Published as a conference paper at ICLR 2025
It formats and returns observations to the backend, ensuring consistent interface for processing results
F.2
How OPENHANDS BUILDS AND MAINTAINS RUNTIME IMAGES
OpenHands' approach to building and managing runtime images ensures efficiency, consistency, and flexibility in creating and maintaining Docker images for both production and development environments.
F.2.1 IMAGE TAGGING SYSTEM
OpenHands uses a dual-tagging system for its runtime images to balance reproducibility with flexibility:
(1) Hash-based tag:
(target_ iaapeet___maa__sa________ Example:
runtime: :bc123def456
This tag is based on the MD5 hash of the Docker build folder, which includes the source
code (of runtime client and related dependencies) and Dockerfile
Identical hash tags guarantee that the images were built with exactly the same source code and Dockerfile
This ensures reproducibility; the same hash always means the same image contents
(2) Generic tag: (target_image.e repo) : lelarget_image___eeeee__
Example:
runtime voh_v0. 9. 9. s__ubunuu_tag 22.
This tag follows the format: oonntimeoo..ommm [VERSTON) (BASE_ TMAGE. IMAGE_TAG)
(BASF,
It represents the latest build for a particular base image and OpenHands version combination
This tag is updated whenever new image is built from the same base image, even the source code changes
The hash-based tag ensures reproducibility, while the generic tag provides a stable reference to the latest version of particular configuration. This dual-tagging approach allows OpenHands to efficiently manage both development and production environments.
F.2.2 BUILD PROCESS
(1) Image Naming Convention:
oarprrt__ingg_uooootttreeennnnao runtime: abc123def45
Hash-based tag:
Example:
Generic tag: Funtime:oh_ve 9. 9.3_ubuntu_tag__2200
Example:
(2) Build Process:
a. Convert the base image name to an OH runtime image name Example: ubuntu: 20untu:22.00 :22. runtimm :oh_v0,
v.. 9. 2oh_vO...3_uuucc s__ubnntu_ttt ___ta_________ ______2__22
Generate a build context (Dockerfile and OpenHands source code) and calculate its hash
c. Check for an existing image with the calculated hash
d. If not found, check for recent compatible image use as a base
e. If no compatible image exists, build from scratch using the original base image
f. Tag the new image with both hash-based and generic tags
(3) Image Reuse and Rebuilding Logic: The system follows these steps determine whether to build new image or use an existing one from user-provided (base) image (e.g.. ubuntu:22 22.04))22.))
a. If an image exists with the same hash (c.g., runt ime: abc123def456), it will be reused as is
b. If the exact hash is not found, the system will try to rebuild using the latest generic image (e.g., runt ime ao__v__ vO. 9.3 mbuntt__tag _ag_u__t________ 04) as a base. This saves time by leveraging existing dependencies
21

---

Published as a conference paper at ICLR 2025
Start
Convert base
nundimeroon 9.3.bbubu.bgg
Generate build contact and hash
exiting image hush
Not ruund
Check for cankinarolu v..9. Ldbaatit Iay, 22.04
Found mattmmetteeileeeee
frund
imagge
with hesh and
End
Figure 5: OpenHands Runtime Image Build Workflow.
If neither the hash-tagged nor the generic-tagged image is found, the system will build the image completely from scratch
Caching and Efficiency. The system attempts reuse existing images when possible save build time. an exact match (by hash) found, it's used without rebuilding. a compatible image is found, it's used base for rebuilding, saving time dependency installation.
flowchart illustrating the build process shown in Fig.
G ADDITIONAL RESULTS FOR GPQA BENCHMARK
We showcase more detailed results, including performance on other subsets for GPQA benchmark in Tab. 7.
22

---

Published as a conference paper at ICLR 2025
Table Full Evaluation Results the GPQA Benchmark Rein al. (2023) ($4.4).
Accuracy by subset (%)
Evaluation Method and Model
Avg Cost (S)
Diamond Set
Main Set
Extended Set
Expert Human Validators
81.2
72.5
65.4
Non- Expert Human Validators
21.9
30,5
33.9
N/A
Few-Shot CoT Llama-2-70B-chat
28.1
29.1
30.4
N/A
Few-Shot CoT GPT-3.5-turba-16.
29.6
28.0
28.2
N/A
Few-Shor CoT GPT-4
38.8
39.7
38.7
N/A
GPT-4 with search (backoff CoT abstention)
38.8
41.0
39.4
N/A
OpenHands + CodeActAgent v1.5 GPT3.
27.9
23.4
26.1
0.012
OpenHands Code/ActAgent v1.5 GPT4-turbo
51.8
47.4
42.4
0.501
OpenHands + CodeAAtAAgent v1.5 GPT4o
53.1
49.3
52.8
0.054
H IN-CONTEXT DEMONSTRATION FOR CODEACTSWEAGEN
The prompt is re-adopted from the SWE-agent's released trajectory wweeae///eercmmm/ttttrooo/e/rror The prompt can be found https:/gitthu..com All- All-Hands- https//lloggaaaabbbbommoleeeoooo seeesssseee_ee agget/prompt..py.
1
SUPPORTED AGENTSKILLS
As of OpenHands v0.6, we support the following list of skills. Please refer to the source code for the most up-to-date list of skills: https://github..o httpggiihhbb..o/A11/ com/All-Hands- Hall--aand-A-Onnnaaaaaa AlKppplaandkbbbmmiinnetttttaa runtime/plugins/agent hitiie//uggniigggennkkklllleggnn
def open file (path: str, line Optional [int] None)
>>
+ None:
Opens the file at the given path in the editor. If line__umbe number is provided, the window will be moved to include that line.
Args:
path: str: The path to the file to open.
line number: Optional [int]: The line number to move to.
pass
def goto_line line
int) None:
Moves the window to show the specified line number.
Args:
int: The line number move to.
pass
def scroll_down() () None:
"""Moves the window down by 100 lines.
Args: None
pass
23

---

Published as a conference paper at ICLR 2025
def scroll_up up -> None: "'VVe the window up by 100 lines.
Args:
None
pass
def create_file (filename: str) None: """Creates and opens new file with the given name.
Args:
filename:
str: The name of the file to create.
pass
def edit_file (start: int, end: int, content: str) -> None: """Edit file.
replaces lines "start" through 'end' (inclusive) with the given text "content" in the open file. Remember, the file must open before editing.
Args:
start: int: The start line number. Must satisfy start >= w 1.
end: int: The end 1ine number. Must satisfy start <= end
<= number of lines in the file.
content:
str: The content replace the lines with.
pass
def search dir (search_tern) str, dir sir_path: str ='/'") None: """Searches for search_term in all files in dir. If dir is not + provided, searches in the current directory.
dir
Args:
search term: str: The term to search for. term: path: optional[str]: The path to the directory to search.
pass
def search file (search str, file_path: Optional [str] None) -> None:
"""Searches for search term in file. If file is not provided, searches in the current open file.
perm
Args:
term: str: The term to search for. file_path (str)) The path to the file to search.
mmn
pass
def find_file (file_name) name: str, dir_path: str './") -> None: """Finds all files with the given name in the specified directory.
Args:
24

---

Published as a conference paper at ICLR 2025
file name: str: The name of the file to find. Sir_path: [otional[stx]] The path to the directory to search.
pass
def parse_pdf (file_pat str) None: """Parses the content of PDF file and prints it.
Args:
file_path: str: The path to the file to open.
pass
def parse_docx (file_pathh str) -> None:
Parses the content of DOCX file and prints it.
Args:
file path: str: The path the file to open.
pass
def larse_laa str) -> None:
Parses the content of a LaTex file and prints it.
Args:
"** file_path: str: The path to the file open.
pass
def
str, model: str 'whisper-1') None
Parses the content of an audio file and prints it.
Args:
file_paths str: The path to the audio file to transcribe. model: optional(str) The audio model to use for transcription. Defaults to 'whisper-1'.
pass
def parse_image(
file path: str, task: str 'Describe this image as detail as possible."
None:
Parses the content of an image file and prints the description.
Args:
file_path: str: The path to the file to open. task: Optional [str]: The task description for the APT call. Defaults to 'Describe this image as detail as possible.'.
pass
25

---

Published as a conference paper at ICLR 2025
def gesse_video file_path: str, task: str 'Describe this image detail possible.',
frame_interval int = 30,
None:
Parses the content of an image file and prints the description.
Args:
file_path: str: The path to the video file to open. task: Optional(str]: The task description for the API call. Defaults to 'Describe this image as detail as possible.'.
frame_interval: optionallintl: The interval between frames analyze. Defaults to 30.
pass
def (frse__ptx(file__aa str) -> None:
"**
Parses the content of pptx file and prints it.
Args:
File_pathe str: The path to the file to open.
pass
J BROWSERGYM ACTIONS
The following are all the supported actions defined in BrowserGym' as of v0.3.4. The actions can be categorized into several types and can be configured to use only a subset of the functionality. There are agent control actions, navigation actions, page clement-based actions, coordinate-based actions, as well as tab-related actions. We use these actions from the BrowserGym library as our main browsing action primitives.
def send Send_mssg___ wwg_mssg___._... toueerr user (text: str): =**
Sends message to the user.
Examples:
wwg__t__uussseeeeeeeee ("Based on the results of my search, the city was built in 1751.")
pass
def
(reason: str):
Notifies the user that their instructions are infeasible.
Examples:
report_ inforsibiee ("I cannot follow these instructions 4 because there is no email field in this form.")
26

---

Published as a conference paper at ICLR 2025
pass
def noop (wait_ms: float = 1000):
Do nothing, and optionally wait for the given time (in milliseconds).
Examples:
noop ()
noop (500)
pass
def fill (bid: str, value: str): #**
Fill out form field. It focuses the element and triggers an 4 input event with the entered text.
It works for <input>, <textarea> and [contenteditable] elements.
Examples:
fill ('237', 'eexamlle value') fill ( ''4',
"multi-linel \nexample")
fill ('a12', "example with \) ("quotes\\""")
pass
def check (bid: str):
"**
Ensure checkbox or radio element is checked.
Examples:
check ('55')
pass
def uncheck (bid: str):
Ensure a checkbox or radio element is unchecked.
Examples:
uncheck ( ('a5289')
pass
# http:://aaar.aatt.ee////o/pppppp htwp//ocs/ipppteeeeeetttoooooo
def select_option (bid: str, options: str [ist(s
27

---

Published as a conference paper at ICLR 2025
Select multiple options in <select> element. You can specify option value label to select. Maltiple options can be selected.
Examples:
select_option ('a48', "blue")
select_option ('c48', ["red", "green", "blue"])
pass
def
click(
bid: str,
button: Literal Literal("left",, "middle", "right"] = "left", modifiers: list [Literal ("Alt", "Control", "Meta", "Shift"]]
[],
Click an element.
Examples:
click ('a51')
click ('sk('b22 ('b22', button="right")
click ('48', ('' button="middle"" modifiers=
pass
dblclick( def
bid: str, butten: Niteraal""let"" "middle", "right"] = "left", fodifiers: list [Literal ("Allt, "Control", "Meta", "Shift"]] U),
Double click an element.
Examples:
dblelick('12)
Jolclick('caf2 button="right")
dblclick('178. button="middle" Seiifier==("hhfff"
pass
def hover (bid: str):
#**
Hover element.
Examples:
hover (1b81)
28

---

Published as a conference paper at ICLR 2025
pass
# https://playwrigitt httess///paarrttgh..../////////p
def press (bid: str, key_comb: str):
Focus the matching element and press combination of keys. It accepts
the logical key names that are emitted the keyboardEvent. key property
of the keyboard events: Backquote, Minus, Equal, Backslash,
Backspace,
Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight, ArrowUp, F1 F12, Digito - Digit9, Keya KeyZ, etc. You can
alternatively specify single character you'le 'd like to produce such "a" or "g" Following modification shortcuts are also supported: Shift, Control,
Alt, Meta.
Examples:
press ( ('88', 'Backspace')
press ( ('a26', 'Controlta')
press ('a61', 'MetatShift+t")
pass
https://playwrightt def focus (bid: str):
Focus the matching element.
Examples:
focus ('b455')
"**
pass
attte///aauuitttt./////t///////s http::///w........oooooooooooo/a def clear (bid: str): "**
Clear the input field.
Examples:
clear('996)
pass
def arag_and_dde (from_bid: str, bod: str):
"""
Perform drag & drop. Hover the element that will be dragged. Press
29

---

Published as a conference paper at ICLR 2025
left mouse button. Move mouse to the element that will receive the drop. Release left mouse button.
Examples:
drag (156', '498')
"**
pass
http://///ttt/oo//////////////a/ def scroll (delta_x: float, delta_y: float): "*"
Scroll horizontaliy and vertically. Amounts in pixels,
positive for right or down scrolling, negative for left or up scrolling. Dispatches wheel event.
Examples:
scroll (0, 200)
ccroll(-50.2 -100.5)
pass
http://t.w..ee..ooooo/////////// mouse def mouse_move (x: float, y: float):
#
Move the mouse to a location. Uses absolute client coordinates in pixels.
Dispatches mousemove event.
Examples:
mouse_move move (65.2, 158.5)
pass
http:////awiitooe/////////////// mooss-s-
def mouse_up (x: float, y: float, button: Literal ("left", "middle", "right"] "left"): e*"
Move the mouse to a location then release mouse button. Dispatches
mousemove and mouseup events.
Examples:
mouse_up (250, 120)
mouse_up (47, 252, 'right')
pass
mosseessssssesemooooo mousemoosee-omm def mouse_down (x: float, float, button: Literal ( ["left", "middle", "right"] "left"):
Move the mouse to a location then press and hold mouse button. Dispatches mousemove and mousedown events.
30

---

detp///thoo/ooossaa////asss/e/es Published as a conference paper at ICLR 2025
Examples:
mouse_dome down (140.2, 580.1)
mouse_down (458, 254.5, 'middle')
pass
def mouse_click (x: float, y: float, button: Literal [ "left", "middle", "right"] "left"):
Move the mouse to a location and click mouse button. Dispatches mousemove, mousedown and mouseup events.
Examples:
(ous_cclick8887.2 68) mouse_click (56, 712.56, 'right
pass
def mouse
coousedblclickiie (x: float, y: float, button: Literal ("leftt, "middle", "right"] "left"): """
Move the mouse a location and double click mouse button. Dispatches
mousemove, mousedown and mouseup events.
Examples:
mouse_dblelick (5, 236)
mouse_dblclick (87.5, 354, 'right')
pass
def mouse_draq_ame drop (from_x: float, from float, to_x: float, to_y: f ***
Drag and drop from a location to a location. Uses absolute 4 client
coordinates in pixels. Dispatches mousemove, mousedown and mouseup
events.
Examples:
mouse_drag_and_dem (10.7, 325, 235. 6, 24.54)
pass
http://www......../ooo///////aaa def keyboard_press (key: str):
31

---

Published as a conference paper at ICLR 2025
Press combination of keys. Accepts the logical key names that are
emitted in the keyboardEvent key property of the keyboard events:
Backquote, Minus, Equal, Rackslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight,
AzrowÜp, P12, Digito Digit9, KeyA Keyž, etc. can alternatively specify single character you 'd like to produce 4 such as "a" or "#" Following modification shortcuts are also supported:
Shift, Control, Ait, Meta.
Examples:
keyboard_press (BBcckspace')
keyboard_press 'Controlta')
beyboarddpresse (Metattstttt-
page. keye.arybbard..rres ("PageDoww))
pass
httpss//pllayyclghhh..e////seeee def keyboard_upp (key: str):
"**
Release keyboard key. Dispatches keyup event. Accepts the
logical
key names that are emitted the keyboardEvent key property
of the keyboard events: Backquote, Minus, Equal, Backslash, Backspace, Tab, Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown,
PageUp,
ArrowRight, ArrowUp, F1 F12, Digit0 Digit9, KeyA Key?, etc.
You can alternatively specify a single character you'l 'd'dd like to produce such as "a" or "*"
Examples:
keyboard_up up ( ('Shift') kpyboard_up up ('c))
nnn
pass
http:://tttttt.oo/o////////////s def deybcard_down (key: str):
Press and holds a keyboard key, Dispatches a keydown event. Accepts the
logical key names that are emitted in the keyboardEvent. key property
the keyboard events: Backquote, Minus, Equal, Backslash, Backspace, Tab,
32

---

Published as a conference paper at ICLR 2025
Delete, Escape, ArrowDown, End, Enter, Home, Insert, PageDown, PageUp, ArrowRight, ArrowUp, F1 F12, DigitO Digit9, ReyA Key2, etc. You can alternatively specify single character such as "a" " "*"
Examples:
keyboard_up up 'Shift') keyboard_up ('cp ('c')
pass
def keyboard_type (text: str):
Types string of text through the keyboard. Sends keydown, keypress/input,
and keyup event for each character in the text. Modifier keys DO NOT affect keyboard_type. Holding down Shift will not type the text upper case.
Examples:
keyboard_type ( ('Hello world!')
pass
def
(text: str):
Insert a string of text in the currently focused element. Dispatches only input event, does not emit the keydown, keyup or keypress events. Modifier keys DO NOT affect beaboarr_insert_teee the text in upper case.
Holding down Shift will not type
Examples:
keyboard insorrt_eesssrt_teete ('Hello world!')
pass
http:://baauttt...eeeee///////os hogp://pp..cllls-ppppaaaeaaaeeee tageepage--oo def goro(url: str):
Navigate to a url.
Examples: goto
pass
33

---

Published as a conference paper at ICLR 2025
def go _o_back :):
Navigate to the previous page in history.
Examples:
ga_back
pass
def vo_forward():
"**
Navigate to the next page in history.
Examples:
fo_fordard()
pass
def new_tab():
Open a new tab. It will become the active one.
Examples:
new_tab
global page
set the new page as the active page
page page. context...t.....mo. new_pageeeappee trigger the callback that sets this page active in 4 browsergym
pass
def tab_close() (): #**
Close the current tab.
Examples:
tab_closel ()
#*"
pass
http::/wwww...oooooooooccc/ceeee def tab_focus (index: int):
Bring tab to front (activate tab).
Examples:
tab_focus (2)
34

---

Published as a conference paper at ICLR 2025
pass
http:://bbyy.......ee//////////s def upload_file (bid: str, file: str [str]) #**
Click an element and wait for a "filechooser" event, then select one
or multiple input files for upload. Relative file paths are + resolved
relative to the current working directory An empty list clears the selected files.
Examples:
file ("572", "my_receipt.ppo") file ("63",
pass
files
def fiise_uplload_ile (x: float, y: float, file: str | [ist(str)):
"**
Click location and wait for "filechooser" event, then select one
multiple input files for upload. Relative file paths are resolved relative to the current working directory. An empty list clears the selected files.
Examples:
mouse_uplood__ile (132.1, 547, "my_receeit.ppdf") mouse_upl oadd moous__ullodd_file file (328, 812, [" ("/home/ home//oolt /bb./bbbbbb//bbe wttp////ooueett/imaeeee.eeo jpg",
pass
K BROWSING AGENT DETAILS
The following shows example prompt containing all the information required for the current step to make a prediction about the next browsing actions. Note that we also instruct the agent to predict multiple actions one if the agent thinks they are meant to be executed sequentially without any feedback from the page. This could save turns for common workflows that consist of sequence of actions on the same page without any observation change, such as filling the username and password and submit in login page.
Instructions
Review the current state of the page and all other information to - find the best possible next action to accomplish your goal.
Your answer will be interpreted and executed by program, make sure to follow the formatting instructions.
35

---

Published as a conference paper at ICLR 2025
Goal:
Browse 1ocalhost:8000. and tell me the ultimate answer to life. Do not ask for confirmation at any point.
Action Space
16 different types of actions are available.
ncop (wait_ms ms: float 1000)
Examples: noop
noop (500)
(text: str)
Examples:
send_mag_to_user ( ('Based on the results of my search, the city was built in 1751.')
scroll (delta_x; x: float, delta_y: float)
Examples:
scroll (0, 200)
scroll (-50.2, 2, -100.5)
fill (bid: str, value: str)
Examples:
fill ('237', 'example value')
fill ('45', 'multii-iineteexeppa
fill ('a12', 'example with "quotes"")
select_option (bid: str, options: str [ist(str]) Examples:
select_option ('48', 'blue')
select_option ('48', ['red', 'green', 'blue'])
(bick str, button: Literal['lefte 'middle', 'right'] 'left', modifiers: list Cat.ppiin.Litteaal [ ('ll'', 'Control', 'Meta', 'Shift']] [1)
Examples: click('51')
click('b22 ('b22', button='right")
click ('48', 'utton='middle" modifiers= [1 ['Shift'])
dbiclick (bid: str, button: Literal ('left', 'middle', 'right 'left', modifiers:
1ist [typing Literal ['Alt', 'Control', 'Meta', "Shift']] = (])
Examples: dblclick('12")
dbiclick ( 'ca42', button 'right')
dblclick ( (177'' button='middle"" modifiers= [ Shift'))
36

---

Published as a conference paper at ICLR 2025
hover (bid: str) Examples: hover (bbB*)
press (bid: str, key_comb: str)
Examples:
press ('88", 'Backspace')
press ( ' 'Control+a')
press ( ('a61,, 'Meta+Shi ft+t!)
focus (bid: str)
Examples: focus ('b455')
clear (bid: str)
Examples:
clear ('996')
drag_ane _rooppnddopppp_____ee_e fitom_bidi str, bid: str)
Examples:
_rag_andddrop ('56', '498')
uile (bid: str, file: str list (str)) Examples:
upload file ('572', 'my_receipt.pdf")
ggload_file 6'',', ['/home//obb//oocmeetss//// jpg', /hw/e/bbb////cc htt.///ccmeett/fill.....p.
go ba_k ()
Examples:
go_back ()
go forward ()
Examples:
go_Forward()
goto (url: str)
Examples:
goto ('http://www, www.wwwwwwww....mmmm comppleeecmm)) com')
Multible actions can be provided at once. Example: 'example with "quotes"!)
click('511
click ('48', button='middle', modifiers=('Shiee
Multiple actions are meant to be executed sequentially without any feedback from the page.
Don't execute multiple actions at once if you need feedback from the page.
Current Accessibility Tree:
RootWebArea 'The Ultimate Answer', focused
[8] heading 'The Ultimate Answer!
[9] paragraph
StaticText 'Click the button reveal the answer to life, the universe, and everything.'
[10] button 'Click me', clickable
37

---

Published as a conference paper at ICLR 2025
Previous Actions (taatt:://occalhos::000000
Here is example with chain of thought of valid action when clicking on button:
In order to accomplish my goal I need to click on the button with - bid 12 "lclick ("12")*..
And an example response to the above prompt is:
In order to accomplish my goal, I need to click the button with bid 10 to reveal the answer to life, the universe, and everything.
click ("10")**
For the evaluation on WebArena benchmark, since some of the tasks require checking for answer exact match on the agent's message back the user, add the following instruction to the agent reply with only concise answer string when messaging the user to prevent the agent from failing the test due extra text:
Here is another example with chain of thought of a valid action when providing a concise answer to user:
In order to accomplish my goal I need to send the information
asked back to the user. This page list the information of НP Inkjet Fax Machine, which is the product identified in the objective. Its price is $279.49. I will send a message back to user with the answer. tenn__ss_to_uurr22211.
38


Reviews:
[{"review_id": "l16Pc5IIq4", "reviewer": "Reviewer_ujp2", "summary": "This paper introduces OpenHands, is a community-driven platform designed to advance the development of AI agents that interact with the world in terms of code writing, command-line interaction, and web browsing. OpenHands includes an evaluation framework for benchmarking agent performance on tasks like software engineering and web browsing. It supports various LLMs and sandboxed environments for running these AI agents and conducts extensive evaluation of different agents in various benchmarks.", "questions": "This paper introduces an open platform with the event-driven architecture to advance the AI agent development, especially focusing on coding and web browsing. It also conducts extensive evaluations of heterogeneous agents on various benchmarks. This is a solid contribution to the LLM agent community. I have some questions about the implementation details. \n- The authors mention that event streaming enables human intervention in agent tasks, which I believe is a crucial step for calibrating agent outputs. I wonder whether human intervention is enabled during the evaluation of agents and is there a mechanism to determine whether human intervention is required?\n- Regarding the multi-agent delegation feature, I’m unclear on how agents collaborate. Is there an orchestrator agent responsible for task delegation, or is this managed through event streaming? If it’s the latter, how are tasks assigned and dispatched, and where do agents deliver their outputs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation to build an open-sourced agent platform is good. \n- The agent interface, with capabilities for code execution and web browsing, is impressive.\n- Evaluations conducted across various agents and benchmarks are thorough and extensive.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JiWOu2CY4h", "reviewer": "Reviewer_4sob", "summary": "The paper presents OpenHands, an open-source platform designed to facilitate the development of AI agents that interact with the world through software interfaces such as code execution, command-line operations, and web browsing. A key feature of OpenHands is AgentHub, a platform where users can contribute their own agents (architecture) to the community, fostering collaborative development of agent architectures. The platform provides a simple agent abstraction, making it accessible for users to create and extend agents, and provides a modular architecture for agent interactions and task execution, including an event-driven state management system, multi-agent coordination, and an extensible agent skills library. OpenHands aims to enable agents to even create tools by themselves. The authors describe the architecture, implementation details, and evaluate the platform across multiple benchmarks, including software engineering tasks and web browsing scenarios. It's kind of a combination of AutoGen and BrowserGym.", "questions": "Distinction from Existing Platforms: How does OpenHands fundamentally differ from other agent development platforms like LangGraph, CrewAI and DSPy for multi-agent collaboration or BrowserGym for benchmark evaluation? Could the authors elaborate on the unique features or advantages that OpenHands offers over these frameworks? Including these comparisons in Table 1 would provide a more comprehensive overview.\n\nAgent Performance Analysis: Can the authors provide a more detailed explanation about the data collector in the framework? Specifically, what makes the framework the best solution to develop an agent when one approaches such a task?\nAnother question is: how will the platform address this in future iterations?\n\nHandling of Observations: How does the platform treat observations and their full composition? Can the authors clarify how observations are managed and utilized by agents? Providing more details on the configuration and usage of the event stream would enhance understanding, especially for developers less familiar with event-driven systems.\n\nSecurity Considerations: Given that the platform allows the execution of arbitrary code in a sandboxed environment, what security measures are in place to prevent potential exploits or breaches? Has the sandboxing been tested against known vulnerabilities? Discussing security protocols would demonstrate the platform's reliability.\n\nWeb Page Understanding Techniques: Regarding the web experiments, it is not clear which page understanding techniques are used to interpret interactable elements and web pages. Does the framework provide screen/page understanding capabilities, or should a new user implement their own methods? Clarifying this would help users understand the platform's capabilities in web interaction tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality:\nOpenHands uniquely consolidates multiple agent capabilities: coding, command-line interaction, web browsing, and multi-agent collaboration, within a single, open-source platform. The introduction of AgentHub allows users to contribute their own agents, promoting community collaboration and expanding the platform's versatility. This integration distinguishes OpenHands from existing frameworks, although IMO some of the main ones are absent from the comparison (see questions).\n\nQuality: \nThe methodological approach is solid, featuring a well-defined architecture that includes an event-driven state management system and a secure, sandboxed runtime environment. The use of an extensible agent skills library enhances flexibility, allowing agents to perform complex tasks and even create tools themselves. The agent abstraction is designed to be simple, enabling users to easily create and extend agents.\n\nClarity:\nThe paper is generally well-written and logically structured, making it accessible to both practitioners and researchers. Key components of the platform, such as the agent abstraction, runtime environment, and evaluation framework, are explained clearly. Figures and tables effectively illustrate concepts and results.\n\nSignificance:\nBy providing an open-source, MIT-licensed platform with contributions from a large community, OpenHands has the potential to significantly impact the development and evaluation of AI agents. It introduces a varied range of datasets for each general agent domain, although it could be expanded in the browser domain (see weaknesses). Evaluating multiple close-source LLMs on each task in addition to the average costs of evaluating benchmarks is a valuable addition that offers practical insights.", "weaknesses": "Limited Novelty in Certain Aspects: While OpenHands integrates various functionalities, much of the work appears to assemble existing components from the domain into one place. Some features, like code execution in a sandbox and web browsing agents, are present in other platforms. The paper could better articulate how OpenHands distinguishes itself from similar frameworks like LangChain, DSPy, or AutoGen.\n\nBasic Implemented Agents: The currently implemented agents are relatively basic. For example, the Browsing Agent is based on WebArena's agent, which is very simple to implement and not competitive with current SOTA agents. There are more advanced multi-agent architectures available that could be implemented in AgentHub to better demonstrate the system's capabilities in facilitating the general agents' development.\n\nOutdated Datasets: The evaluation includes MiniWoB++, which is an outdated dataset. There are newer datasets, such as WorkArena, WorkArena++, and ST-WebAgentBench, that are more relevant. The last test agents in more complex, restricted environments using policy hierarchies. Incorporating these in the framework would strengthen the evaluation.\n\nEvaluation Depth: The evaluation, while broad, could benefit from deeper analysis. Incorporating an automated data collector to track agent success and failure would enhance insight into agent performance and make the framework more complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hAjilsg8SY", "reviewer": "Reviewer_WpcF", "summary": "OpenHands is an open-source platform for the development of general AI agents. This paper details the motivation behind OpenHands, the architecture of the platform, how a user might implement an agent using OpenHands, description of some existing baseline agents, and comparison of some of the OpenHands agents against other open-source agents on software, web, and other assisting tasks. Selected results show that their agents perform reasonably well across a wide range of these tasks (generally underperform the specialized agents).", "questions": "Can we further motivate why we want a generalist agent? If we have specialized agents for different tasks, can we just have a coordinator deploy specialized agents where they perform best?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "OpenHands is a valuable contribution to the research community. The OpenAgents platform will surely be used to develop many future agents, and be referred to as a baseline for other agent-developing research projects. Tables 3, 4, and 5 present thorough experimentation of their agent(s) compared to many other baselines on various tasks.", "weaknesses": "While the value of OpenHands is apparent, the research question of this paper is unclear. If it is that they can develop a general agent that performs better across a range of tasks than any existing agent does, it is my judgement that the results and analysis do not explore this sufficiently to warrant a research contribution. To do so, discussion and ablations should be included for the design decisions of OH Browsing Agent v*, OH CodeAgent v*, OH GPTSwarm v*, etc. To then isolate the effect of those decisions, the choice of backend model should be held consistent to those of the baselines against which the OH agents are compared. Error analysis of results comparing OH to baselines would be another welcome contribution that has been omitted from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPvRALCQYE", "reviewer": "Reviewer_nJqs", "summary": "This paper describes OpenHands, fka OpenDevin, a platform to develop LLM agents which can take meaningful actions using computers, such as browsing, coding, and interacting with a command line.\n\nSection 2 describes the architecture of the platform, detailing the state and event streams, actions and observations, and an extensible library of tools. Agent delegation is also mentioned.\n\nSection 3 mentions some notable agents implemented using their platform.\n\nSection 4 presents results of many evaluatory benchmarks, showing the OpenHands has a broad and general level of competence across domains including coding and browsing.", "questions": "L305 - I'm uncertain what is meant by \"we compare OpenHands to open-source reproducible baselines that do not perform manual prompt engineering specifically based on the benchmark content\"\n\nIt seems surprising that OH CodeActAgent w. GPT-3.5 would get 2% on ToolQA, while ReAct (perhaps the most minimal form of agent scaffolding) would get 36%. Do you have any idea what's behind this performance difference?\n\nAlso surprising is that CodeActAgent delegating to BrowsingAgent does very slightly worse than straightforwardly using BA directly. Could you discuss this?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "OpenHands is an excellent and significant LLM-agent platform, seeing use e.g. as one of the reference agents in OpenAI's MLE-bench https://arxiv.org/abs/2410.07095 , and there is value in having this scaffold's architecture explained and benchmark performance reported in a peer-reviewed form.\n\n\n- General competence across three different domains (SWE, browsing, and 'misc') is significant\n\n- Solve rate of 26% on SWE-bench is competitive, as are many of the results suggested by the tables (though, see below, I have questions around like-for-like comparison)\n\n- The paper provides a great many benchmark results\n\n- OpenHands has many useful features, such as the API server being inside the docker container, the ability to delegate subtasks to other agents, and an extensible library of tools", "weaknesses": "The paper is an announcement of a software platform, more than it is a description of a contribution to a research area. Normally I'd expect a paper to look like a research question, with some motivation, experimental design, and results. This paper shows little, if any, scientific exploration, instead the paper gives a high-level overview of the architecture with little motivation, and then uncritically showcases performance on a broad range of existing benchmarks. I'm unsure whether this kind of project-announcement is a fit for the conference.\n\nSome smaller points:\n\n- Why do the (e.g.) GPQA results not compare like-for-like LLMs? I would want to see whether a CodeActAgent powered by a particular LLM was better or worse than the bare LLM, but the table compares (Llama 3 70b chat, GPT 3.5 turbo 16k, GPT 4) with (gpt-4o-mini-2024-07-18 , gpt-4o-2024-05-13 , claude-3-5-sonnet), so it's unclear how much of the difference is due to the scaffold and what is due to the choice of LLM. This is also true for many results in Tables 4 onwards, perhaps an unfortunate downside of comparing to results from the literature, but this limitation precluding rigorous comparison should at least be acknowledged in the text.\n\n- A large fraction of the paper comprises short summaries of many different benchmarks used to evaluate the codebase. Since these summaries are not original work, I question whether so much of the paper should be dedicated to them. (As a minor point, I'd expect this exposition to be in a setup / methodology / problem-setting section, rather than in the results.) Instead, the authors could focus more on the creative decisions made during the creation of their platform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)