Programming with Pixels: Can Computer-Use Agents do Software Engineering?
Authors: Pranjal Aggarwal
Sean Welleck
Venue: ICLR 2026 Poster (2026)
Keywords: computer-use agents
evaluation
benchmark
code-generation
multimodal
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce ProgrammingwithPixels (PwP), an environment for evaluating computer use agents (CUAs) for software engineering. Our evaluations on the 15-task PwP-Bench reveals current CUA limitations and potential directions for their improvement.

Abstract: Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. 
It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). 
To investigate this, we introduce Programming with Pixels (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. 
To enable holistic evaluation, we also introduce PwP-Bench, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. 
We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. 
However, when the same CUAs are given direct access to just two APIs—file editing and bash operations—performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. 
Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. 
Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.
PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks.

Published as a conference paper at ICLR 2026
PROGRAMMING WITH PIXELS: CAN COMPUTER-USE AGENTS DO SOFTWARE ENGINEERING?
Pranjal Aggarwal & Sean Welleck Carnegie Mellon University (pranjala, swe 1 leck) @cs cmu edu
ABSTRACT
Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenar- ios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce P Programming with Pixels (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce PwP-Bench, a benchmark of 15 existing and new software-engineente languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely vi- sually, they perform significantly worse than specialized coding agents. How- ever, when the same CUAs are given direct access just two APIs-file editing and bash pperations-performann agents despite having task-agnostic design. Furthermore, when given access additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for fu- ture improvements. PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks.
tasks spanning multiple modalities, programming
jumps, often reaching the levels of specialized
INTRODUCTION
Computer-use agents (CUAS) hold the promise of automating a wide range of economically valu- able tasks by acting through primitive actions such clicking, typing, and observing digital screens, potentially obviating the need for specialized Al agent action interfaces (Anthropic 2024 OpenAl 2025 Yang et al. 2024a). However. current evaluations have primarily focused on simple tasks such as web navigation Koh et al. 2024), basic document editing, or tweaking settings in operating systems (Xie et al., 2024 Bonatti et 2024). Therefore, it remains unclear whether current gen- eralist computer-use agents can automate more sophisticated and specialized tasks such as software engineering. this work, we specifically study how well the current generation of computer-use agents can do software engineering and identify their key limitations.
The choice of using software engineering as the test domain is motivated by two primary reasons. First, software engineering represents economically important and practically challenging task. Second, the field of AI software- -enginrrnngineering agents (SWE agents) has produced numerous spe- cialized agents that use hand-engineered APIs for specific operations (Yang al.) 2024a; [Wang et al. 2024b Xia et al. 2024), providing strong baselines for comparison. These agents use custom functions such as file editing, code search, and repository management, with each tool requiring significant engincering effort and domain expertise. For instance, SWE-agent (Yang et al. 2024a) uses language-specific parsers and editing commands, while Agentless (Xia al. 2022)) relies on Python-specing abstract syntax trees. This specialization has yielded strong performance, but it
Cooe and data released httph:tt/////...... hatp:://roogrrmaaa ngwi

---

Published as a conference paper at ICLR 2026
env htv.steppace (action)
PwP Environment
action
L Languages
observation
JS
Computer-Use Agent
IDE Features
Evaluation: PwP Bench
Code Gan
Pull Requests
Code Editing
DevOps
Data Sdince
Ul Generation
Modalities:
T
docker
Figure 1: Programming with Pixels is an environment for computer- use agents, where they interact with a VSCode IDE through keyboard and mouse actions while observing the screen. The framework supports multiple programming languages, tests interactions with multiple IDE features, modalities (eg: text, images, data files). PwP-Bees -Bechh evaluates agents across 15 diverse software engineering tasks such code generation, UI generation, Data Science.
raises fundamental question: can general-purper computer-use agents match specialized agents in complex domains like software engineering?
To investigate this question, we introduce Programming with Pixels (PwP), the first envi- ronment for systematically evaluating computer-use agents software enginecring tasks. The PwP environment provides a VSCode-based IDE where agents perceive the screen and use primitive ac- tions such typing and clicking to perform variety of SWE tasks. This design enables two critical properties. First, the environment is expressive, allowing agents to complete any software engineer- ing task achievable in an IDE without language- domain-specific modifications. Second, agents can access all IDE tools tools- debuggess,,, linters, code suggestions throough the same visual interface available to human developers specialized SWE agents. Hence, PwP provides general-purpose, purpose, realistic software engineering environment for testing computer-use agents.
To evaluate computer-use agents, we construct PwP-Bench, benchmark of tasks spanning different tasks such as code generation, pull request resolution, UI development, and data science across multiple programming languages and modalities. The benchmark represents unification of 13 existing SWE tasks ported for evaluating computer-use agents, and 2 additional tasks developed by us. Our evaluation of wate-of-the-ar computer-use agents reveals that when restricted to pure visual interaction, these agents achieve only 22.9% average accuracy, significantly underperforming specialized coding agents. However, when augmented with just two basic text APIs file edit- ing and bash operations-the same agents achieve 50.7% accuracy, often approaching specialized agent performance despite their task-agnostic design. Furthermore, our analysis reveals substantial opportunities for future work. First, even state-of-the-ar computer-use agents suffer from visual grounding issues. Second, we show that current computer-use agents lack the ability to use many of the tools available in the IDE, including ones that could make their tasks trivial. This suggests that training computer-use agents to explore and leverage the functionality present in their computer environment fruitful future direction. Overall, our results highlight software engineering realistic and challenging benchmark for evaluating and improving computer-use agents.
In summary, our contributions are as follows. First, we introduce Programmi ng wi with Pixels (PwP), the first software ingineering-focuseee environment for evaluating computer-use agents. Sec- ond, we introduce PwP PwP-Benn -Benhh, a benchmark spanning 15 diverse SWE domains, allowing for systematic comparison of computer-use agents. Third, through extensive evaluation, we highlight the limitations of current computer- use agents, identifying the need for models that have better visual grounding and that better take ad advantage of their environment as key future directions. Finally, open-source our environment and benchmark, allowing it to serve as an open platform for evaluating and improving agents software engineering tasks.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Multimodal and Computer-Use Agents. Recent works have explored using multimodal LLM
agents to operate user interfaces such as web browsers (Koh et al.) 2024, Deng et al.) 2023 Zheng et al. 2024) and operating systems (Xie et al., 2024 Bonatti al. 2024)). Agent designs in these settings fall into two categories: (a) agents with predefined action sets (e.g. new tab, go_back, cl ick [element id]) that receive auxiliary information such as HTML accessibil- ity trees Yangg et al. 2023a) for visual grounding; (b) pure computer-use agents operating with prim- itive keyboard and mouse actions, relying solely on screenshots (Anthropic) 2024; OpenAl [025] 2025). PwP supports evaluating both agent designs. Further, existing benchmarks such as World Xie et al., 2024). AndroidWorld (Rawles et al. 2025), and WindowsAgenttArena (Bon-) atti et al. 2024) evaluate agents on simple tasks like document editing and calendar management, leaving unclear whether performance on these tasks translates to complex, specialized domains like software engineering. PwP-Bench fills this gap by providing the first benchmark specifically de- signed to test whether computer-use agents can handle software engineering tasks. While some prior works explores specialized domains such as game playing (Tan et al. 2024) and concurrent work explores scientific software (Sun et al.) 2025), PwP is the first environment and PwP-Bench the first benchmark systematically evaluating computer-use agents for software engineering, a domain that is particularly noteworthy due to the presence of strong specialized agent baselines.
Qin et al.
Software Engineering Agents. Software engineering agents have primarily relied on specialized scaffolding tailored to specific tools, languages, or tasks (Jin et al.) [2024] Yang et (024b)). For in- stance, Agentless (Xia al. 2024) uses Python-specific parsers, SWE-agent employs task-specific modifications (Abramovich et al., 2024 Yang et al. 2024b), and others depend on hand-engineeee components like IPython kernels (Wang et al. 2024a) or custom browser views (Yang et al. 2024b). Our work takes fundamentally different approach by evaluating whether computer-use agents which interact through the same visual interface as human developers- can match these specialized agents. This also tests whether visual interaction with standard developer tools is sufficient for soft- ware engineering or if specialized APIS remain necessary. As PwP supports evaluating both designs, it enables direct comparison between computer-use and specialized agents across the diverse tasks in "wwP-Bench, establishing a unified platform for understanding the capabilities and limitations of different agent designs. We refer readers to Appendix C for more detailed related work.
3 PROGRAMMING WITH PIXELS (PwP)
Testing computer-use agents (CUAs) on software engineering (SWE) requires an environment that captures the full complexity of modern software engineering, which involves multiple programming languages, tools, and modalities. Furthermore, a fair evaluation must provide access to the wealth of tools that human developers use and specialized AI SWE agents have access to, such as lin- ters, visual debuggers, and even project management tools. To enable such evaluation, we create P Programming with Pixels an IDE environment that satisfies these two requirements. First, it is expressive, meaning that an agent can perform any task that is achievable through a sequence of primitive operations (e.g., typing or clicking) within an IDE, which includes a wide range of soft- ware engineering activities. Second, an agent has access to any functionality implemented within the IDE, since using IDE functionality amounts to performing a sequence of primitive actions.
PwP environment. We represent the PwP environment as a partially observable Markov decision process (POMDP). We define the PwP POMDP (S, A, O,T, T, R) as follows. S is the set of states describing the IDE and the operating system (OS) context, including open files, active editor panels, and cursor positions. A is the action space, encompassing all possible keyboard and mouse events. The atomic actions in PwP are provided by the xdotool library (Sissel), which allows specifying all possible keyboard and mouse events in a simple syntax. The specific action space varies based on the agent setting, described in ($5). O is the observation space. The observation space varies based on the agent setting, described in (5)) T is the transition function. Actions like inserting character typically lead to deterministic changes in the IDE state, whereas background processes can introduce stochasticity in timing and responses. R is the reward function that measures performance on a given task. For instance, after the agent finishes editing code to fix a bug, the environment can run test suite the updated files to compute a reward. Trajectories in PwP thus resemble real-
3

---

Published as a conference paper at ICLR 2026
world development work: agent can fix a bug in repository, use suggestion tool to help with writing code, or create documentation. The IDE and OS environment track changes, run tests and return reward signals. In addition, we discuss five key features of PwP.
1. Expressive observation and action space. PwP provides computer-use agents with an unre- stricted environment where they can attempt any software engineering task achievable through an IDE's visual interface, as humans do. Unlike environments with predefined action sets, agents can navigate IDE menus visually, move cursors, and press keys perform more complex actions.
2. Full Spectrum of Developer Tools. When evaluating computer-use agents on SWE tasks, it is imperative that they have similar level of access tools specialized SWE agents, such those with custom APIs for debuggers, linters, refactoring utilities, and more (Xia et al. 2024 Yang et al., 2024b). PwP provides all these tools through IDE's visual interface. creating a comprehensive test of whether CUAs can leverage the same rich functionality that specialized agents access through APIs.
Multimodality and language agnosticism. CUAs promise generality across tasks and domains. Software engineering spans many languages such as Python, Java, JavaScript, Lean, and more, with tasks involving multiple modalities, such as text, images, data files, and PDFs, providing a rigor- ous test of this generality. In PwP, the same CUA must handle code generation, UI development, data science, and theorem proving without task-speeific modifications. For agents requiring visual grounding support, modified VSCode's source code to provide rich DOM trees and Sct-of-Marks annotations, ensuring fair evaluation across different CUA architectures.
4. Ease of verification. PwP provides direct access to the IDE's internal state, file system, and OS processes for verification. When an agent modifies code, we can run test suites, check com- pilation, and verify correctness. This separation between agent interaction (visual) and evaluation (programmatic) makes it easier to verify task completion and provide other sources of feedback.
5. Future adaptability. Computer-use agents are improving rapidly, and ing agents. PwP is designed for future adaptability. First, adding new benchmarks as simple as modifying configuration files. Second, PwP's checkpointing is useful for search and RL training methods. Third, PwP's gymnasium interface (Towers et al., 2024) provides a standard interface for evaluation and development. Finally, as agents improve and become capable of using more complex tools, the environment (IDE) would automatically incorporate these without architectural changes. This makes PwP an extensible platform for evaluating and developing computer-use agents.
software engineer-
Infrastructure and Implementation PwP is deployed in secure sandboxed docker environment, running open-source VSCode and a minimal operating system. Each container is isolated, prevent- ing interference between experiments, ensuring parallel evaluation and facilitating reproducibility. We implement checkpointing for the environment state, which is especially useful for backtracking in search algorithms or training RL agents. The environment interfaces to VSCode using four chan- nels for real-time screen capture, DoM information, and customizable configuration such display, CPU/memory limits, etc. However, the complex interaction is abstracted away from the user, as they simply interact with the environment through gymnasium python API (See (Sguue 9) and install the environment using a simple pip command. We refer to subsection A.3 for more details.
4 PwP-BENCE
We introduce TwW-Bench, a benchmark containing 15 diverse software engineering tasks that span 14 programming languages and multiple modalities. Each task provides agents access to the IDE via the Pw? environment. The goal of PwP-Bench test whether computer-use agents (CUAs) can handle the depth and breadth of software engineering activities.
Tasks. PwP-Bench contains 5400 instances sourced from 13 existing code-generatie datasets and 2 newly created by us. These tasks are designed to be representative of software engineering activities that take place within an IDE. Since the IDE is simply computer program, in principle, these activities should be achievable by a general-purpose purpose computer- use agent. We selected the tasks in PwP-Bench according three key principles: (1) tasks must require substantial interaction with software engineering tooling, (2) each task should require multiple steps, and (3) the benchmark must cover multiple languages and modalities. Accordingly, tasks are grouped into four categories:
4

---

Published as a conference paper at ICLR 2026
Code Generation and Editing (n = 6): These tasks evaluate the ability to generate and edit code. This c includes datasets such as HumanEval for code completion, SWE-Bench (Jimenez (Yang al.] 2025) for resolving pull requests, DS- Bench for data science tasks (Jing et al.) 2024), and Res-Q LaBash et al. 2024) or CanITE- dit (Cassano et al. 2024) for code editing. Fach dataset benefits from different IDE functionality.
ct al. 2023) and SWE-Bench-Multlingg
For example, SWE-Bench can take advantage of debuggers and linters, while DSBench may lever- age an IPython kernel and extensions for analyzing large data files. Code editing tasks can leverage refactoring utilities and repository searches, covering varied input-ouiput formats and end goals.
Multimodal Code Synthesis (n = 4): These tasks involve creating code based on input im- ages or other visual data. Examples include Design2Code (Si et al., 2024b) for UI development, Chart2Mimic (Shi et al. 2024) for generating Python code from chart images, SWE-Bench- MM (Yang et 2024b) for multimodal code editing, and DSBench tasks that rely on images, data files, or PDF documents for data analysis.
Sppein-Specific Programming (n = 3): These tasks focus on specialized fields such as ethical hacking (CTF) (Yang et al., 2023b) and interactive theorem proving (miniCTX) (Hu al. 2024). which demand significant use and interaction with the IDE's functionality, For example, theo- rem proving requires continuously inspecting goal states via the IDE, while CTF tasks involve analyzing images, running executables, or installing VSCode extensions (e.g., hexcode readers).
IDE-Specific and General SWE Tasks (n = 2): Since code generation is only one aspect of software engineering, we introduce two novel task sets that evaluate broader SWE skills. The first, IDE Configuration, evaluates an agent's ability to modify IDE settings such as themes, extensions, and preferences. These skills involve substantial interaction with the IDE, and are often a precondition for using IDE functionality such as new extensions. The second, which we term General-SWE, targets five different non-code activities: performance profiling, code refactoring, debugging bugs in standard libraries, UI mockup design, and code restoration. These tasks target practical software engincering tasks typically absent in conventional benchmarks. Full details are in Appendix B.2
The distribution of tasks across categories and modalities is shown in Figure I0 the Appendix. Computer-use agents that perform well across these tasks would demonstrate strong potential for automating diverse SWE activities across multiple languages, and working with varied input/output modalities such as text, images, data files, and other data types. Furthermore, taking ad advantage of the functionality provided by the agent's environment essential.
Benchmarking Design and Task Setup. All tasks evaluated within the PwP environment. Unlike traditional benchmarks, PwP-Bench presents agents with a realistic IDE environment: each agent receives an initial IDE state S, and instruction I, with the goal to achieve final state Sy evaluated via execution- whether CUAs can find relevant information from files, directories, and other resources, which is important for complex software development. Furthermore, a task is defined by a simple setup script that defines the initial IDE state, the instructions, and the evaluation logic. This makes easy add new tasks, allowing PwP-Bench to evolve as new benchmarks or better agents are developed.
criteria (e.g., unit tests). Among other capabilities, this setup tests
-wP-Bench-Lite. Because PwP-Bench contains more than 5400 instances in total, running a full evaluation can be computationally expensive. To address this, we also provide PwP-Ben.hh Lite: a smaller subset of 300 instances. Specifically, we randomly sample tasks from each of the 15 benchmarks. This subset preserves the overall difficulty and distribution while ensuring equal representation for each task, thereby making rapid experimentation more accessible.
EVALUATING AGENTS IN PROGRAMMING WITH PIXELS
We evaluate three distinct agent designs in the PwP environment to understand the capabilities and limitations of computer-use agents for software engineering tasks.
Computer-use agents. Computer-use use agents interact with the IDE through primitive actions, i..., keyboard and mouse inputs, while observing the interface visually through screenshots. Each agent operates in a curn-aased manner, receiving a screenshot each turn and returning an action to progress toward the goal. Since most vasion-language models without GUI-specific training struggle with raw
5

---

Published as a conference paper at ICLR 2026
Figure 2: Example of Successful Use Live Preview Tool in the UI Replication Task The agent successfully uses the live preview tool in the VSCode browser to compare the UI design made versus the reference design.
Figure 3: Example of Successful Use of Tool in the Chart Generation Task The agent can com- pare the generated chart with the reference chart side by side and refine its code accordingly.
pixel coordinates, we incorporate Sel-of-Marks (SoM) (Ynng et al. 2023a). With Set-of-Marks, an agent receives both the image and parsed representation of available interface elements (e.g.. buttons, text fields), allowing them to interact via element IDs rather than pixel coordinates. This design follows previous works (Xie et al.
2024 Koh al. 2024).
Computer-use agents with File/Bash APIs. Computer-use agents are augmented with direct access file-editing and bash commands through text APIs. The file-editing APIs include operations such as 'read file' and 'string replace', while bash operations allow command execution in the terminal. Agents receive screenshots only when requested via a screenshot action, rather than automatically each turn. This design strictly follows Anthropic's computer-use implementation (Anthropic 2024)
Specialized software engineering agents. comparing how well current computer-use agents perform relative to specialized agents, evaluate mini-swe-agent (SWE-agent) 2024),, agent scaffold specifically designed for software engineering. Unlike computer-use agents that interact visually with the IDE, mini-swe-agent operates entirely through text APIs. For multimodal tasks, receives required images directly as input in its prompt. We chose mini-swe-agent due to its near state-of-the-a performance for adapting to different programming tasks. See Appendix D.I for implementation details and Appendix
the widely-used benchmark SWE-Bench, as well its flexibility
for for the prompts used for each agent.
Experimental setup. We test multiple models as the parametrization for the two computer-use agent designs. Specifically, we test four vision-language models: Gemini-Flash- 1.5 Gemini- Pro-1.5, GPT-4o, GPT-4o-mini, and we test six models with Ul-specific training: closed-source Claude-3.5 Sonnet, Claude-3.7 Sonnet, Claude-4.0 Sonnet, and oeen-weights Qwen-2.5-VL, Qwen- GUI-OwI-32B, and Dwen33-VL-30B3
For mini-swe-agent (with multimodal support), we test Claude-4.0 Sonnet. We also evaluate text-only version of mini-swe-agent withholding image inputs to assess the importance of visual modality (see Appendix F). We keep the experimental setup consistent across all tasks and models: for each task instance, the maximum number of iterations is capped at 20 steps; if the agent either exhausts these steps or issues a stop command, the environment's final state evaluated using task-specific metrics (see Appendix B.2 for full details). For BWE-BBncch related tasks, we further evaluate with a maximum of 250 steps in Appendix F Due computational and budget constraints, we evaluate on PwP-Rench- -ennch--iitee which has 300 task instances.
multimodal tasks by
5.1 RESULTS AND ANALYSIS
Table I summarizes performance across different agent architectures and base models over the four categories of PwP Benhh (task-wise results are in Table 8). As seen in the top half of the table, computer-use use agents using only primitive keyboard and mouse actions achieve poor performance, with maximum overall average of 22.9%. This is significantly lower than the software-enginerrine specific agent mini-swe-agent, agent, which achieves 48.8% accuracy. We attribute this poor performance primarily to limited visual grounding and an inability interact effectively with the IDE, particularly

---

Published as a conference paper at ICLR 2026
Table 1: Performance Evaluation of Different Agents PwP WwP-Bert -eee-Bennh-iii by Task Categories. Best numbers are in bold, and best numbers for computer-use agents are underlined.
Model
Code Generation
Multimodal Code Gen, (m=4)
Domain-Specific Code Gen. (m=3)
General SWE Tasks (m=2)
Overall
& Editing (n=6)
Computer- Use Agents
Gemini- Flash
0.0%
4.3%
0.0%
0.0%
1.1%
GPT-4o-mini
0.8%
3.7%
0.0%
2.5%
1.7%
Qwen2.5-VI.-722
0.0%
4,3%
0.0%
5.0%
1.8%
GUI-Owl-32B
0.0%
0.0%
22.5%
3.0%
Qwen3- VL-30B-A3B
9.0%
0.0%
37.5%
7.7%
Gemini-Pro
2.5%
5.7%
0.0%
7.5%
3.5%
GPT-4o
0.8%
12.4%
1.7%
10.0%
5.3%
PlaudeSSonet-333
10.7%
8.3%
5.0%
22.5%
10.5%
Saude-Sonnet-333
11.8%
28.5%
8.3%
27.5%
17.7%
Claude-Sounel Sonnet-4.0
1.2%
37 37.310.6%
6.7
10.0
20.3
Computer- Use Agents with File/Bash APIs
Gemini-Flash
9.5%
11.7%
8.3%
2.5%
8.9%
GPT-4o-mini
23.6%
17.6%
15.0%
5.0%
17.8%
Qwen2.5-VI.-725
13.7%
11.8%
6.7%
7.5%
11.0%
Gemini-Pro
30.0%
16.7%
3.3%
12.5%
18.8%
GPT-4o
36.2%
41.9%
28.3%
10.0%
32.6%
Claude-Sonnet-3.5
47.9%
55.1%
43.3%
22.5%
45.5%
Claude-Sonnet-3.
51.9%
58.7%
46.7%
27.5%
49.4%
33.520.2 %
37.8
43.9_2.0
38.3
50.7 2
Software Engineering Agents
mini-swe-agent OpenHands
49.4%
60.3 %
40.0%
37.5%
48.8%
50.4%
50.8%
43.3%
25.0%
45.7%
for file editing and tool usage; see Section 5.1 for further analysis. Among all evaluated models, the Claude computer-use agent performs best, likely because it specifically trained for UI interactions. We found that it can leverage basic IDE tools such HTML live preview, chart visualization, and file navigation, boosting performance on tasks that require visual understanding and IDE navigation.
Nonetheless, when the same computer-use agents are granted access to just two text APIs (file editing and bash operations) we observe consistent improvements across all categories, with the maximum average accuracy reaching 50.7
However, models still struggle to fully leverage the tooling available the IDE. This is evidenced by poor performance the "General SWE' category, where tasks often require fewer than ten steps when using appropriate IDE tools. We analyze the poor performance on General SWE tasks further in the following sections, confirming that these tasks would become simpler if models could use IDE tooling more effectively. Overall, our results show that computer-use agents to have some facility for software engineering, but currently require better visual grounding, tool usage, and planning. In the following paragraphs, we analyze these strengths and deficiencies in more detail.
Qualitatively, we found that Claude Computer-Use agent can use basic IDE functionalities, including file explorer naviga- tion, file editing, search, browser-based live preview, and image generation and visualization ca- pabilities. Figure demonstrates the agent's s effective use of browser tools in UI replication tasks. Similarly, Figure illustrates the agent's ability to utilize multiple tools while editing specific lines in repository. relying solely screenshot observations and primitive keyboard/mouse actions.
Claude Computer- Agent Demonstrates Basic IDE Tool Proficiency.
Furthermore, we hypothesize that agents have additional latent abilities to use tools that can be activated through prompting or fine-tuning. To investigate this, we examined the project refactoring task (such as symbol renaming) our 'General-SWE' benchmark, where Claude initially achieves 25% accuracy when attempting the task. However, when explicitly instructed to use precise tools (such as rename or move to file), its accuracy improves to 75% (see Appendix E
Computer-Use Agents Demonstrate Poor Visual Grounding Capabilities. While, Claude Computer-Use agent is able use basic IDE tools, found that in general all current CUAs have

---

Published as a conference paper at ICLR 2026
User Mekuper
Model Inputs Value at Wirong Location
Hgp
Model Response
The search results for "pythonniininng pylintArgs" pyng displayed. The input pythonnlinting pylintArgs' now contains
(
Correct Location to add setting value
mMUM
Figure 5: Agent Misidentifying UI Elements tents The agent correctly mentions, search re- The agent fails to identify the correct input field, sults are displayed (green), it hallucinates an in- put field containing "disable import error" (red). the word wrap column setting field (red arrow).
Figure 4: Agent Hallucinating Screen Con-
typing '50' into the settings search bar instead of
significant limitations in visual grounding. i.e., the ability to understand the visual input and take actions on the visual IDE interface. We identify three primary failure modes. First, the agents can the often fail to use the correct UI elements. For example, in Figure the agent types in the search bar rather than the settings field, while in Figure [55 the model clicks the wrong location. Surprisingly, Sct-of-Marks did not resolve these issues; agents would instead select incorrect elements.
Second, the agents often struggle to comprehend the current UI state, such as linter errors indicated by wavy underlines (Figure 13) hallucinate screen contents (Figure 4. Finally, even when the agent can identify a simple error, such as incorrect indentation, it is often not able to fix the error due to struggling with clicking and typing in the proper locations. Furthermore, in Appendix [F] models frequently completely ignore the visual state information and instead rely on completely memorized action sequences. Quantitatively, we found that 20% and 95% of trajectories have at least one visual grounding error in GPT-4o and Claude Sonnet-4.0, respectively (see Appendix
While grounding has been highlighted mains (Koh et al., 2024] Xie et al. (2024), the limitations were primarily observed in models without Ul-specific training. However, our work shows that even models explicitly trained for UI interac- tion, such as Claude Computer Use (Anthropic 2024), exhibit these issues in PwP. We hypothesize that the deficiencies come from the IDE being particularly information-dent as well as potentially not being covered by computer-use training datasets.
weak point of computer-use agents in web and OS do-
Agents Struggle to Use Advanced IDE Functionality. Although the best computer-use agent we tested could use basic IDE functionality, all agents lack the ability to leverage more sophisticated IDE tools. Specifically, we can see this through the low performance on the 'General-SWE' dataset, which focuses on software engineering activities (c.g., profiling, refactoring, debugging) that can be often completed without direct code edits. Although these tasks sometimes require only 4-5 steps when using appropriate IDE tools, agents achieve minimal performance, highlighting substantial room for improvement. Furthermore, we observed no successful uses of profilers, debuggers (even when explicitly instructed to) when performing the other tasks in our benchmark (see Appendix E.. We further quantify the IDE features used across all trajectories of Claude-Sonnet-4 CUA in Ap- pendix
Distribution of Functionality Used by Computer-Use Agents with File/Bash Operations. As we observed in Table computer-use agents perform much better when they have access to file and bash API calls, which are based on text inputs and text outputs. A natural question is to what extent these are using the visual interface versus relying on text-only APIs. We study this Figure(s) which shows the distribution of file/bash API calls versus computer-use interactions on four representative datasets. The figure shows few interesting patterns. First, for HumanEval, agents rely entirely on file APIs. This is because HumanEval tasks involve simple function completions that are achiev- able without IDE interaction. The lower performance of pure CUAS on this task (25% compared to 100%) demonstrates their inability perform basic file editing visually. Second, for MultiModal, surprisingly there are minimal computer-uu use interactions, primarily using screenshots to understand the open repository or occasionally attempting open the built-in browser.
8

---

Published as conference paper at ICLR 2026
HUMANEVAL
VSCODE
Percentage (%)
90
66
95.4%
87. 77.5%
100.0%
40
12.5%
Canpurer
#lxchokk
Riohad
Figure 6: Distribution of file/bash calls vs computer-use interaction for computer-use agents.
In contrast, the distribution shifts dramatically for Design2Code, where agents frequently open live preview tools compare generated designs with reference images, and continuous refining the output (see Figure 2 In a similar vein, for VSCode tasks, the agents rely entirely on visual IDE functionality to update settings, install extensions. and edit themes. These patterns demonstrate that computer-use agents with file/bash APIs have some ability to choose between visual and API based interactions based on the task requirements. datasets such as HumanEval, their performance improvements stem from bypassing their inability to visually perform edits, instead using text APIs
Computer-Use Agents Are Rapidly Improving. Tigure/compiin the performance of Claude-Sonne 3.5, 3.7, and 4.0 released over period of 7 months. The line shows steady improvement in pure CUAs, with performance nearly doubling from 10.5% to 22.9%. Furthermore, from Table we see that the
Sonnet
Computer Jua Agart
22.95
2.2 2 im orover ent
17.7%
gap between pure CUAs and CUAS with file/bash op-
Sounet
7 months
erations has steadily decreased from 35.0% 27.8%.
These results highlight that while substantial gap remains, rapid progress is being made and continued improvements may eventually close this gap.
Ralease
2013
Figure 7: Performance of Claude-Sonne Computer- Use Agents over time
Leveraging the IDE functionality better would improve performance. While single computer-use agent design can perform non-trivially across a wide variety of tasks, our analysis indicates that these models do not fully exploit domain-specific tools. To quantify the potential performance gains if agents could effectively perform an "assisted" experiment. this experiment, we manually engineered a set of IDE-based tool calls representing commonly used IDE functionalities (e.g., .., live HTMI previews, repository structure, symbol outlines). Importantly, each API call is achievable using basic operations in the IDE, meaning that in principle, agent could learn perform it. See Appendix E for full details.
the IDE,
Table summarizes the performance im- provements of assisted agents, highlighting an average gain of up to 13.3%. These re- sults demonstrate that current CUAs have poor interaction capabilities with complex interfaces, yet there is significant scope for improvement. The results also suggest that in the near term, performance gains can be achieved by introducing specialized hand-engincered tools into computer-use agents and incorporating existing agent designs in our PwP environment.
Figure 8: Assisted versus Computer-Use Agents
SWE-Bench
DesignOCade
Chartminic
BIRD (T2 SQL.
Camputer Use Agents
0%
23 55
2.7%
0%
CUA File/Bash Assisted
15%
18.1%
25.3%
7%
79.5%
17%
6 CONCLUSION
We introduce Programming with Pixels, an environment designed to evaluate computer- use agents on software engineering tasks. We also introduce PwP-Bench, a diverse benchmark of 15 tasks spanning the breadth of software engineering across multiple languages and modalities. Our extensive evaluations of nine models reveal that pure computer-use agents relying solely on visual interaction perform poorly, while augmenting these agents with simple file and bash text

---

Published as a conference paper at ICLR 2026
APIs dramatically improves performance. Our analysis pinpoints poor visual grounding and an inability to leverage the rich set of functionality in the Pw2 environment as primary weaknesses. Despite these limitations, findings show that CUAs improving rapidly, signaling significant potential. PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents reach specialist-level performance sophisticated tasks.
7
ACKNOWLEDGEMENTS
We thank Daniel Fried, Graham Neubig, Zora Wang, Saujas Vaduguru, Atharva Naik, Riyaz Ahuja, and Weihua Du for helpful feedback. We thank Convergent Research the OpenAl Researcher Access program. This work was supported in part by the National Science Foundation under Grant Nos. DMS-2434614 and DMS- DMS-2502281. Pranjal is supported by SoftBank Group-Are Arm Fellowship.
REFERENCES
Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang. Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Enigma: Enhanced interactive generative model agent for etf challenges, 2024. URL
org/abs/ 2409. 16165
Aider. ol tops aider's new polyglot leaderboard. polyglot. html 2024. Accessed: 2025-02-12.
Anthropic. Developing computer use model, October 2024. URU/ttpp::/www.ttbboooo com/
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: petttps:/aaxxvv.ott www.iiivorg..... org.abs/////oooooooooooo.....e.a 2108.100100 07732
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL
abs 2409. 08264
Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili,, Anton Lozhkov, Carolyn Jane Anderson, and Arjun Guha. Can it edit? evaluating the ability of large language models follow code editing instructions, 2024.
URL
abs 2312. 12450 2450
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. URL https:
org abs / /217 03374
Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Oni Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The browsergym ecosystem for web agent research, 2024. URL http:///araav..orrr//////22222/ 05467
10

---

Published as a conference paper at ICLR 2026
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL https://arxiv:
abs/2306. 06070
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2024.
org. abs 2410. 05243
Jiewen Hu, Thomas Zhu, and Sean Welleck. minictx: Neural theorem proving with (long)ccontexss 2024. URL
03350
Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, and Omer Tripp. Qualityflow: An agentic workflow for program synthesis controlled by Ilm quality checks, 2025. JtR.hhtts:///axxit http://///awwv...//aaaaaa/e//e/e /2501 17167
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL httpps:/arxiv..rrr wwwparwwwww.rggb....oo//e
24033 07974]
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github is- sues? ArXiv, abs/2310.06770, 2023. URL https: DLLLLDoo////DD...cct...cooooar org/
semanticscholar
Corpus ID:
Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. From Ilms to lim-based agents for software engineering: A survey of current, challenges and fu- ture. ArXiv, 2a//248.002799 2024.
/taps: stw../cco...mmttccccccc.........
CorpusID:2717000
Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents to becoming data science experts? arXiv preprint arXiv:2409007700 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov. and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL
2401. 13649
Beck LaBash, August Rosedale, Alex Reents, Lucas Negritto. and Colin Wiel. Res-q: Evaluat- ing code-editing large language model systems at the repository scale, 2024. URL https:
//arx/v.ore/aas/2200..
Bingxuan Li, Yiwei Wang. Jiuxiang Gu, Kai-Wei Chang, and Nanyun Peng. Metal: multi-agent framework for chart generation with test-time scaling, 2025. URL https://arxiv org/
Abss2552.1765
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants, 2023. URL https::/aaxxxi.......m org/
abs abs/2311 12983
Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. URL
2406. 12552]
OpenAl.
Introducing operator.
OpenAI, 2025.
http:://ppeni..com
Introoduiing-operater
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jia- hao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma. Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang. Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL https: ///xxii/ wttpa//wwwwogg////////////////// 12326
11

---

Published as a conference paper at ICLR 2026
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 59708-59728. Curran Associates, Inc., 2023. URL. htttps::/proceeeinngss neur cc wwtppc/ppppppcc.ccccc wttp://aasssaaa/e22///////
(bbba008000eee000033322933322220 Paper wwwprrpataaeeeeeeeeaaa Datasets and Benchmarks.
pdf
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Camphell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmark- ing environment for autonomous agents, 2025. URL / absgg//bb/4455
arxiv.
14573
Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li. Yuxiang Zhang, Gongye Liu, Xiaomei Nie. Deng Cai, and Yujiu Yang.
Chartmimic: Evaluating Imm's 's cross-modal reasoning capability via chart-to-code genera-
tion. ArXiv, his/2400.0996611 2024. URL. https://api /tttss:/ap..semmnticceee..
Corpus D: 200221101100 907
Vladislav Shkapenyuk, Divesh Srivastava, Theodore Johnson, and Parisa Ghane. Automatic meta- data extraction for text-to-sql, 2025.
.ttp://2255111988888 19988
Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang. Ruibo Liu, and Diyi Yang. Design2code: Benchmarking multimodal code generation for automated front-end engineering. 2024a. URL.
https: //atxii wwwp//gg////o/..oo../tt../t..e.e /at.s/243.000 03163
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang. Ruibo Liu, and Diyi Yang. Design2code: How far from automating front-end engineering? ArXiv, abs/2403.03163, 2024b. URL https:
7/api. wwmpntsemmiiiioomm cacho ar. Craccholar..rr//oopus...... TD: ID: 26822D::668800
Jordan Sissel. xdotool: Fake keyboard/mouse input, window management, and more. Accessed: 2025-02-12.
https:
//github.
Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. Coding agents with multimodal browsing are generalist problem solvers, 2025. lattts://aaaaiiii
abs/ 03011
Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang. Tianbao Xie, Xiachong Feng. Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi. Lingpeng Kong, and Zhiyong Wu. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows,
2025. URL (tttttss:: ptnttts://arxi... http:///ww....///////b......
SWE-agent. mini-swe-agent. https://gittie http:/aweeeeaateoo./......oo/ana 2024. GitHub repository. Accessed 2025-09-22.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Jun- peng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yu- jie Wu, Xiaoqiang Chai, Yifei Bi, Tianbao Xie, Pengjie Gu, Xiyun Li, Ceyao Zhang, Long Tian, Chaojie Wang, Xinrun Wang, Börje F. Karlsson, Bo An, Shuicheng Yan, and Zongging Lu. Cradle: Empowering foundation agents towards general computer control, 2024. URL https::/arrxt wrg. 20..:///2233.00 03186
Trae Research Team, Pengfei Gao, Zhao Tian, Xiangxin Meng, Xinchen Wang, Ruida Hu, Yuanan Xiao, Yizhou Liu, Zhao Zhang, Junjie Chen, Cuiyun Gao, Yun Lin, Yingfei Xiong, Chao Peng. and Xia Liu. Trac agent: An llm-based agent for software engineering with test-time scaling, 2025. URL URL /tttps:///aaaeee
org/ abs /aabe//255 /2007 23370
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforcement learning environments, 2024. URL https: lorg/abs 2407. 17032
12

---

Published as a conference paper at ICLR 2026
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe- cutable code actions elicit better Ilm agents, 2024a. BtLltttppa:/aaxiv.cro abspg//bbs/4222
01030
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for software developers generalist agents, 2024b. URL
org/ abs /0t..//////20222
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying 11m- based software engineering agents, 2024. URL attps://arxiv> httpr///aaa///00/ 01489
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Sil- vio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https:
org/ 2055/bb//50 2404. 07972
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking Ilm agents consequential real world tasks, 2024.
URL httpttts///aai....ooop org/ abs/ 221.2
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023a. Uttpsttss///axxv
orq/abs/2310. 11441
John Yang. Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing
and benchmarking interactive coding with execution feedback, 2023b. URL. https://axxxii..
org/abs/ 2300 14898
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. 2024a. Uttpsttss://arxi hwt.:/org/abb.... 2405. 15793
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang. Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do systems generalize to visual software domains?, 2024b.
(ttLhtts::/ /ttp:i/aaaivvvv/aa o / 03859
John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig. Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025.
wwwp:///wo/wa///oo/ooooo/oosss/a 22504.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents, 2023. URL https://////
abs/2207.01200 01206
Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng. Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, and Qianxiang Wang. Swe-bench-java: A github issue resolving benchmark for java, 2024. URL htt.://pps///aaa....rr//////2222 2045 14354
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury, Autocoderoveer Autonomous
program improvement, 2024. URL httpt::///// /tttps://aaxiii wtwpviiiiii..//.. org/ wttpg//////.......e/e 05427
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint Xiv: 01611, 2024.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: Á realistic web environment for building autonomous agents, 2024. httpthttp////atttee org/ abs/
2307. 3854
13

---

Published as a conference paper at ICLR 2026
Table Comparison of Hand-engineered Tools across Methods versus PwP. PwP natively supports all tools.
Method
Hand-engineered Tools
Supported in PWP
Agentless (Xia 2024
Edit, Repository Structure, File Structure
CodeAct (Wang 2024aa
File Edit, IPython, Bash
SWB-agenr Yang AU244
Search File, Search Text, File Edit
EnIGMA Abramovich
2024
SWE-agent Tools Debugger, Terminal, Connection Tool SWE-agent Tools View Webpage, Screenshot, Open Image
swebench-mm Yang et al.
20246
Table 3: Comparison of different environments across multiple dimensions
Computer-Use Execution-Bar Specialized Domain
SWE Specific
Environment
Environment?
Reward
GAIA (Mialon 2023
WEBSHOP Yao el al. 2023
X
x
WEBARENA 7houu cf al.
2024
X
x
VWEBARENA Koh et al.
2024
X
x
BrowserGym CCezzelles
2024
X
X
OSWORLD (Xe et 2024
X
X
Android World (Rawles el al..
2025
X
X
X
WindowsAgent AgentArena (Bonatu al.
2024
X
ScienceBoard (Sun et 2025
Cradic (Tan al., 2024
X
PwP (Ours)
V
A PROGRAMMING WITH PIXELS (PwP) ENVIRONMENT
A. TOOLS
Previous methods have proposed use of various hand-engineered tools. For a fair comparison, all tools should be accessible the PwP environment. Aas shown in Table 2 these tools.
PwP natively supports all
A.2 COMPARISON WITH OTHER ENVIRONMENTS
In Table 3 we compare PwP with existing environments across multiple dimensions. We evaluate environments along the following dimensions:
Computer-use environment: Whether the environment is designed for computer-use agents, and thereof whether it supports multimodal interaction.
Execution- bbased evaluation: Use of runtime execution to verify the correctness of agent actions
Whether the environment is designed for general and basic tasks, such as web navigation, is it designed for more sophisticated, specialized and potentially econom- ically important tasks. Only Cradle (Tan et al. 2024) and ScienceBoard (Sun et al. 2025) are specialized for Game Playing and using Scientific softwares respectively.
Specialized:
SWE-specific: Whether the environment is purposefully designed for software engineering tasks
Further, ours support other engineering features that others do not. For instance, PwP also support streaming video and audio, something other environments do not support out of the box. Further, unlike environments such OS-World, which require manual creation of environment image, PwP natively docker based, and based on simple scripts, that can be easily used to modify startup scripts and other configurations for future adaptations. Finally, also specifically suppport state checkpointing which supports storing file system and complete process state, and is especially useful for search-based methods.
14

---

Published as a conference paper at ICLR 2026
bench PwPBench (dataset=' swebench')
# Replace with any dataset from PwP-Bench
dataset bench.get_dataseet
# set up environment and get initial observation
env = bench. seatcch.gee_ettege (dataset [0])
observation: Image - www.get_obbeervtiennn ('screenshot'
# Generate and execute action
10 action = aetttteet_acttoooobereeeeee
print (action)
il Output: xdotool mousemove 1000 1200
13 it click & xdotcol type 'hello world'
14 observation, info anv.step (action)
15
env. render)
18 Environment control aav.pause ()
19
env. resume ()
Get reward and reset
21 is_success env. env.get_reward( env.reset ()
24
Figure 9: Example demonstrating interaction with PwP environment, including keyboard/mouse actions, checkpointing, and state management. The code shows basic initialization, action execution, environment control, and reward handling.
A.3 INFRASTRUCTURE AND IMPLEMENTATION
PwP is deployed in secure sandboxed environment. particular, we run a modified version of Vi- sual Studio Code (VSCode) and minimal operating system inside Docker container, ensuring secure and isolated environment. We chose VSCode for extensive language support, rich ecosys- tem of extensions, widespread adoption in the developer community, and open-source nature that enables customization and modification of its core functionality, Each container instance maintains its file system and processes, preventing interference between experiments, facilitates repro- ducibility, and ensuring parallelization of evaluation. We further provide the ability to checkpoint the environment state, which is especially useful for backtracking search algorithms while training RL agents.
The environment interfaces with VSCode through multiple channels: 1.) A controller that manages Docker container lifecycle and configuration, 2.) A port-forwarding system for real-time screen and video capture. 3.) A modified VSCode codebase that exposes DOM state information, and 4) The VSCode Extension API for accessing fine-grained IDE state. This multi-channel approach enables both high-level environment control and detailed state observation.
Screen capture is handled via ImageMagick for static screenshots and f Empeg for streaming video output. These tools were selected for their low latency and ability to handle various screen resolutions and color depths. For actions, a lightweight controller executes xdotool commands within the container, which turn simulates keyboard and mouse events on the IDE. Agents can thus insert code, open new files, navigate menus using the same actions that human developer would.
As shown in Figure 9 a Python API is provided for interaction, following a style similar to common reinforcement learning libraries such as gymnasium (Towers et al.) 2024). The API abstracts away the complexity of container management, benchmark management, and handling observations and actions, allowing researchers to focus on agent development. Users can query the environment for the latest screenshot, issue xdotool command, and receive updated states rewards. Examples
15

---

Published as a conference paper at ICLR 2026
SWE Bench
Res-Q
SWE Bench Java
Canit- Edit
SWT Bench
Code Gen & Editing
DSBench
Human Eval
Chart Mimic
Multimodal Code Gen
CTF
Domain Specific
Design
2Code
SWE Tasks
SWE
MiniCTX
Bench-MM
BIRD
VSCode
General
SWE
Figure 10: Distribution of tasks in PWP-Bench across four main categories: Code Generation and Editing, Multimodal Code Synthesis, Domain-Specift. Programming, and General inner ring shows the main categories while the outer ring shows specific datasets and tasks within each category. Note that the figure is not drawn based on relative size of tasks.
SWE Tasks. The
of xdotool commands include 'xdotool mousemove 1000 1200' and 'xdotool type 'hello world" and are shown in Fngiug 9. The environment's 's container configuration is flexible, allowing for software installations, customizable CPU/memory limits, and display settings (e.g., resolution). This versatil- ity is crucial for large-scale evaluation, especially when tasks vary in complexity and resource needs. Finally, the environment has been tested on three different operating systems: Ubuntu, MacOS, and Windows.
B PEP-BENCE
B. TASKS
Figure shows the of tasks across all categories. Further, Table 4shows the number of instances for each task in the full benchmark, along with the languages used each of the tasks. PwP-Bewn -Bench- Lite contains 300 instances, which is random sample of 20 instances from each task.
16

---

Published as a conference paper at ICLR 2026
Table 4: Number of instances for each task in PwP-Bench
Task
Number of Instances
Languages
HumanEval
165
Python
Design2Code
485
HTML/CSS/I
ChartMimic
600
Python
InterCode
100
Python, Bash
RES-Q
100
Python
CanItEdit
105
Python
VSCode
20
Bird
500
SQL
DSBench
112
Python
SWE-bench
2000
Python
Me--Bench-Muurilingeee
91
C++, Typescript, Javascript, Rust, Go, C, Ruby, PhP, Java
Swehench-MM
510
Javascript
SWT-Bench
276
Python
Minictx
381
Lean
General SWE
20
B.2 EVALUATION
All tasks are evaluated using programmatic verifiers. These verifiers are typically run separate environment, not accessible to the agent. This typically works based on fetching relevant files and information from the agent environment, and then running through task-specific evaluation scripts on a separate environment. However, to the user, this is abstracted away, and they simply have to call 'env.get reward()' to get the exact score correctness signal based on task.
Metrics We use individual metrics mentioned in the original datasets. When reporting results on -encBench, we report marco average of all these metries. In particular, 11/15 used Accuracy as their metric. However, due to complexity of dataset, these often goes beyond simple accuracy metries and in some cases, the dataset is evaluated on multiple orthogonal metrics, instead of one. We detail, these metrics for each of the datasets.
SWT-Bench evaluates generated tests by the agent, and reports different metries: Ap- plicability, Success Rate, F- X, F- P, P- and Coverage. We report the average of all 6 metrics.
ChartMimic evaluates generated code on various metries such as accuracy of text, colors used, legend etc. We average all metrics similar to the original dataset.
Design2Code evaluates generated code on various metrics such as accuracy of text, posi- tion, clip score, etc. We average all metrics similar to the original dataset.
DSBench has two categories, one containing MCQ questions, while the other containing generating code for Kaggle Competitions. We use 10/10 instances from each category in PwP-Bench-LLis While MCQ questions are evaluated using Accuracy, the code genera- tion part is evaluated using linear normalization between the baseline score (of the compe- tition) and the score of the winner of competition.
VSCode and General SWE Tasks In this section, we detail the VSCode and General SWE tasks in BBeche created by us. The VSCode tasks are mostly designed to evaluate the ability of agents to use basic VSCode features, such renaming all instance of symbol in file, installing extensions, changing themes. modifying specific settings. All these tasks are evaluated based on final IDE state, either by invoking the "code" cli tool, configuration files stored in environment filesystem, or through direct access to VSCode state provided by PwP (see [subsection A.3). General-SWF tasks, involves categories of tasks: 1.) QA based on code profiling (evaluated based on final answer by model which requires using appropriate profiling tools), 2.) code refactoring (assessed through automated tests on the final repository state). 3.) debugging bugs in standard libraries (evaluated based the correctness of final code state), 4.) UI mockup design (assessed using CLIP scores), and 5.) code restoration, where the agent leverages VSCode's timeline feature to recover corrupted codebases, evaluated by the correctness of the restored state.
17

---

Published as a conference paper at ICLR 2026
Full List of Tasks SWE below.
We provide the full list of tasks for IDE Configuration (VSCode) and General-
Table 5: List of VSCode Configuration Tasks
ID
Task Description
1
Install the pylance extension in VS Code.
2
Please help me change the background of VS Code to the photo background.jpg. Change VS Code's ' color theme to Solarized Dark.
3
Please modify VS Code's settings disable error reporting for Python missing im- ports.
5
Please help me open the autosave feature of VS Code and delay AutoSave operations for 500 milliseconds in the VS Code setting. Please help me modify VS Code setting to hide all 'pycache' folders the explorer view.
7
Can you delay VS Code autoSave for 1000 milliseconds? Please help me configure VS Code settings so that "Format On Save" is enabled specifically for Python files, but disabled for all other file types.
9
Please modify the "Files: Exclude" setting to hide all files ending in ".log' and '.tmp' from the Explorer view, ensuring they don't I clutter the workspace.
10 Please help me create standard 'launch.json configuration file for a Flask application within the current workspace, setting the port to 5000.
11
Please help me select the Python interpreter located at '/venv/bin/python' for the cur- rent workspace, rather than using the system default.
12 Please help me search for and install specific color theme extension called "Dracula Official", then immediately activate after installation.
13 Please help me set a conditional breakpoint on line 45 of 'data cattapprocesor.py that only pauses execution when the variable 'retry_com .count ? 3'. Please help me configure VS Code so that on startup reopens the last used workspace.
15 Please help me change all the places in this document that say "text" to "test" and "test" to "text".
16 Please help me remove the shortcut "Ctr++F" for Tree view Find the Explorer, and then assign "Cul+Allt+"" the new shortcut for Tree view Find to avoid conflict with editor search.
17
Please modify VS Code's settings to disable error reporting for Python missing im- ports.
18
Please configure the suggestion list that Code Snippets always appear at the very top of the suggestion list, above standard variable names or keywords. Please help me install the Black Formatter extension and configure it as the default formatter for all Python files in this workspace.
19
20
Please install flamegraph extension in VS Code.
Comparison with Other Benchmarks In Table further compare PwP-Berd with other existing benchmarks.
C RELATED WORK
C.1 COMPARISON TO SOFTWARE ENGINEERING AGENTS
Tpsk-specific SWE benchmarks Early neural code generation approaches were typically evalu- ated on fixed or from general textual descriptions (Austin et al. 2021). Subsequent benchmarks extended these
pairs- for example, generating code from docstrings (Chen et al. 2021)
evaluations to interactive settings, such as resolving GitHub pull requests or writing unit tests for
real-world code repositories (Jimenez et al. 2023 Zan et al. 2024 Mündler et al. 2025 More
18

---

Published as a conference paper at ICLR 2026
Table 6: List of General SWE Tasks
Category
Task Description
Timeline
Use the Timeline view to find the local history version of main.py where import re- quests was deleted ( mins ago) and restore
Identify the Timeline entry immediately before the formatting action that changed spaces labs, and revert the file that stale.
Locate the "Git: Staged Changes' entry in the Timeline and copy the validate _user function from that version into the current file. Use the Timeline identify when timeout changed from 5000 to 10000 in utils.js and revert to the version immediately prior.
Profiling
Analyze the generated flame graph to identify the innermost C-level function consum- ing the most CPU time during the NumPy random generation phase. Use the flame graph to determine the execution time ratio between the compute heavy function and the io_save function. Locate the widest bar the top of the stack trace (the 'tip') and identify which specific Python library call it corresponds to. Identify the deepest stack level in the graph where the application spends at least 50% of its total execution time.
Refactoring
Rename the DataProcessor class to LegacyData DataaPooessso globally, ensuring that currences within comments and string literals are excluded from the update. Swap all assignments and references of variables width and height. Move the AuthHandler class from main.py a new file named auth utils.py, making sure all import references across the workspace are updated.
Rename the random class to pseudo random across the whole workspace.
Mockups
Replicate the image exactly into the currently open Drawiio canvas using standard flowchart shapes. Replicate the image exactly into the currently open Drawioo canvas using standard flowchart shapes. Replicate the image exactly into the currently open Drawio canvas using standard flowchart shapes. Replicate the image exactly into the currently open Drawio canvas using standard flowchart shapes.
Debugging
Trace the execution flow through the Middleware class to find the exact method call that is silently modifying the hequees dictionary before it reaches the endpoint. The script fails during the np.dot operation; use the debugger to inspect array shapes inside the function call and identify the dimension mismatch. We have some numpy code which isn't working correctly due to issue inside library function. Debug the code fie-by--line inside the library function's source to identify the problematic area.
internal
The code failing specifically the np.dot operation. Use the debugger inspect the runtime values of the matrices involved to figure out the exact dimension mismatch issue.
recently, efforts have broadened the scope of code generation include multimodal tasks, where vision models must interpret images to generate correct code or edits (Si et al. [2024b] (Sh et al.) 2024 Jing et al. 2024; Yang al. 2024b). However, each of these benchmarks confined to specific languages, modalities, or task types. In contrast, our proposed PwP-Rench unifies these diverse evaluations into single framework, encompassing multimodal and multilingual challenges that require interaction with broad suite of IDE tools. Using this unified approach reproduce the performance of established benchmarks and encourage the development of general-purpose agents capable of handling variety of new software engineering tasks. We further compare work with previous efforts Tables andd] 7
Software Engineering (SWE) Agents Recent work has explored "code agents" that move beyond single-step neural code generation toward interactive methods, where intermediate feedback from tools informs subsequent actions. However, many of these approaches specialize in particular tools
19

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of existing software engineering benchmarks. PwP-Bench provides the largest dataset (5400 instances) and uniquely covers all aspects: multiple languages and modalities, real IDE interaction, interactive coding, and both code generation and general software engineering tasks.
#Instances
Multiple Languages
Muliiple Modalities
Real IDE Env
Interactive Coding
Non Code
Code Generation
Benchmark
SWE Tasks
SWE Tasks
SWE-Bench Uimencz et al. 2025
2K
X
X
X
SWT.Dench MM
2024b
1K
X
/
LiveCodeBench Jai et al 2024
1K
x
x
x
Aider Polyglot Aider 2024
1K
/
x
x
/
TheAgenCCompury
VisunlWehArrnaa (Koh
A124
1R
X
/
x
X
OSWORLD (Xe al
1R
X
/
WindowsAgellAreess Bonatt ctall 2024
X
FwP-3ench (Ours)
54KK
J
V
or programming languages (Jin al. 2024 Yang et al. 2024b), limiting their broader applicability. For example, Agentless (Xia et al. 2024) relies on tool that parses files into Python-specific class and function structures. This fails to perform well in other languages or settings (Yang et al. 2024b) without manual modifications. Similarly, the SWE-agent requires modifications adapt different tasks (Abramovich et al. 2024 Yang et al. 2024b). contrast, agents designed for PwP are inher- ently task and language-agnen agnostic due to the expressive action and observation spaces mandated by our environment. Moreover, the diverse tasks in PwP-Bench require agents to generalize across a wide range of SWE challenges rather than excel in narrowly defined area such resolving pull requests.
Many existing agents also depend hand-engineered tools that require human effort implement and are susceptible to bugs. For instance, Agentless (Xia et al.) 2024) leverages tools for pars- ing files into Python-specific structures; CodeAct relies
IPython kernel (Wang al. 2024a); SWE-Agent uses dedicated search and file editing tools (Yang et al. 2024a); AutoCodeRove
re-
quires a linter (Zhang al. 2024); SWE-Agent EnIGMA develops specialized tools for CTF-style competitions (Abramovich et al. 2024); and SWE- SWE-Bench-Ne Bench-MM (Yang et al. (024b)) implements browser view. In PwP, these tools are inherently available within the IDE (as detailed Table 9.
and the agent's 's task is to effectively use them rather than being explicitly guided on which tool to use for each specific task.
Finally, current approaches often blur the line between the agent and the environment, each agent designed with its own specified action and observation spaces within a self-created environment. Progr amming with Pixels addresses this issue by unifying existing environments into sin- gle, general-purper platform on which agents operate. This clear separation of environment design from agent design standardizes evaluation and also allows any existing agent to be modeled within our framework, making it an important testbed for both current and future SWE agents.
C.2 COMPARISON TO GENERAL VISUAL AND COMPUTER- USE AGENTS
Visual Agents and Computer-Uss Use Agents A family of recent multimodal agent benchmarks re- quire agents to operate user interfaces using predefined, limited set of actions (e.g., new_tal abb go back, cl click [element id]) (Koh et al. 2024 Deng et al. 2023 Zheng et al. 2024) These visual agents typically rely on additional prompting -uch set-of marks techniques that supply an HTML accessibility tree containing textual and positional information overcome their inherent poor visual grounding capabilities (Yang al. 2023a). Despite such aids, these agents of- ten fail when faced with the complex and dense IDE interfaces found in environment.
A separate family of computer-use agents (Anthropic) 2024; OpenAl 2025 Gou et al. 2024) are trained to operate with expressive action and observation space using primitive operations like clicks and keystrokes, without the need for external accessibility elements. However, there is no SWE-specific environment for evaluating and further training these agents. PwP fills this gap by providing unified, expressive IDE platform that challenges computer-use agents with realistic and diverse SWE tasks.
20

---

Published as a conference paper at ICLR 2026
Expressive Agent Environments Prior work expressive agent environments has predomi- nantly targeted the web domain (Koh et al. 2024 Deng et al. 2023). entire operating systems (Xie
et al. 2024; Bonatti al. 2024; Rawles et al. (003)) or other general scenarios (Xu et al. (024). Some of these environments, such as OSWorld (Xie et al. 2024), feature general action and ob- servation spaces similar ours. However, although these benchmarks are capable of expressing wide range of tasks, they do not focus on the unique challenges inherent to software engineering within an IDE. For example, while OSWorld offers a broad set of tasks, it is not specifically de- signed for SWE, resulting in increased computational overhead. Software engineering is diverse and important domain that merits its own dedicated environment.
Additionally, we design PwP so that existing tool-based software engineering agents can be readily incorporated into our framework. Specifically, we modify the source code of the IDE to open up API calls that let us test current tool-based agents. Furthermore, PwP-Bench is tailored specifically for multimodal SWE tasks within an IDE, encompassing activities such as pull-request handling, debugging, and image-based code generation across multiple programming languages. We also observe that existing agents built for generic UI control often struggle in the PwP environment, as they must interact with richer set of tools and achieve precise visual grounding within complex interface containing a large number interactive elements. We further distinguish PwP from other environments in Table 3
D EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
D.1 AGENT DESIGN
In addition to the details mentioned in Section S we provide more implementation details in this section. First, the exact version numbers used for different API models are: ept-4o-2024-11-20 9p--40-mini-2022---111,, 2024-07-188 pro-preview-000 claude-377-oomne-2203333 19, souude-sonne---2025566
gemini-1.5- For the three Claude models, we use the computer-use variants by passing the "computer-use" beta flag in API calls. For open-weights models, we run inference on L.40s using vLL.M. We use temperature=0.3 consis- tently across models. For our main experiments, the number of iterations is set to 20 because: a.) for most tasks, 20 iterations is enough to complete the task, b.) increasing the number to more models didn't support caching
than 20 would increase the computational cost, and since the time of running the experiments, the cost grows quadratically, c.) we ran experiments with 250 steps -laude-Sonnet-4.0 on SWE-bench related datasets (see Appendix F); however, we found no difference in trends.
D.2 -INENSTE-AGENT
For mini-swe-agent, use Claude-4.0 Sonnet. We the same code the official source code (SWE-agent 2024), except that we modify it for multimodal tasks so that the agent receives required images input in its prompt.
D.3 OPENHANDS
We evaluate OpenHands, a strong baseline that uses SWE-specific prompts and has access to variety of tools. Specifically, lowing tools enabled: command line execution (CmdRunAction) IPython interactive shell (TPythenRunCel Cww.RunneelAliiiiooo i on), and file operations (Fi "PileReacActiin and browser tool (BrowserAct ion). The agent also uses Agent ThinkAct ion for reasoning and AgentFind shAct ion to conclude tasks. The agent uses Claude-Sonnet-4.0 with tempera- ture of 0.3.
use the CodeActAgent configuration with the fol-
FileMriteActis
RESULTS
Table presents comprehensive results for all agent designs across datasets in PwP -BeBBench. Note, that non-trivial performance of mini-swe-agent--extt on design2code and chartmimic is because
21

---

Published as a conference paper at ICLR 2026
of noisy metrics, since if there is even some similarity between the actual image and the generated one (eg, chart type is accidentally the same), the metric returns a non-trivial score.
E. COMPARISON WITH BEST REPORTED SPECIALIZED SWE AGENTS
In this section, compare computer-use agents with the best reported specialized SWE agents scores on individual datasets. particular, for each dataset, we use different strategies to identify the best reported scores:
Citations: For each dataset, we manually go through the citations and find the most relevant works and look for reported scores.
Official Leaderboard: For some datasets, such as SWE-Bench, we use the official leader- board find the best reported scores.
Web-Search Agents: We further prompt ChatGPT-5 thinking to find the latest and highest reported scores on each of the datasets. We then manually verify the results based on the links provided.
For each dataset, we follow all three strategies and take the highest reported score. Typically these results are achieved using specialized approaches including finetuned models, custom tool inter- faces, specific pipelines, prompts, inference scaling, and verifiers. Therefore, is important note that direct comparisons on individual datasets may not provide complete picture. Further, since our evaluations are done on 20 examples from the whole dataset, the results may not be directly comparable. Further, while we make our best effort to include the latest publicly available results, there may still be discrepancies. Finally, the search was conducted on 22nd September 2025, and future numbers may change.
We now list the best reported scores for each dataset:
Hvaa QualityFlow (Hu al. 2025) achieves 98.8% performance using Claude- 3.5-Sonnet.
SWE-Bench:
Highest scores (75.2%) are achieved by method named TRAE agent Team et 2025), with best reported performance with Claude-4-Sonnee as base model as 74.6%.
WWWE-Bech-Muutillnguaa Highest score publicly reported is 43% (Yang et al., 2025)
using Claude-3.7-Sommt and Swe-agent framework (Yang et al. 2024a).
ResQ: Highest score publicly reported is 58% (LaBash al.
2024) using Claude-3.5-
Sonnet in the official dataset report.
SWT-Bench: Highest score publicly reported is 63.3% (Cassano et al.
2024 using GPT-
4o in the official dataset report.
Design2Code: Highest score publicly reported is 90.2% (Si et al. 3.5-Sonnet in the official dataset report.
2024a using Claude-
Chartmimic: Highest score publicly reported is 86.46% using GPT-4o and METAL method (Li al. 2025). Further they use inference scaling with n=5.
Intercode-CTF The publicly reported state of the number is 72% using SWE-Agent- Enigma (Abramovich et al., 2024). This is much smaller than the numbers reported by computer-use agent evaluation, which reaches 100% with the same 3.aude-3.5-Somet model. This is surprising, since the method employed numerous specialized tools for static analysis, dynamic analysis, and networking, and confirmed that the improvement is statistically significant (p-value- = 0.014, McNemar's test).
The best reported score is 76.14% (Shkapenyuk al. 2025 as per the numbers reported in offciail leaderboard.
BIRD:
SWE-Bench- Beeneee-MMeiiddaaaa The best reported score is 35.98% using scaffolding over O3, and 34.33% when using OpenHands-Verss (Soni et al. 2025 with Claude-4-Sonnet.
Overall, the results are often much higher than the numbers achieved by computer-use agents, even with access file and bash APIs Overall, the discussion points out that present specialized coftware-engineene agents still perform better, and built scaffolding around computer-use agents might also be helpful.
22

---

Published as a conference paper at ICLR 2026
Table 8: Performance Evaluation of Different Models Across Task Categories, Leged: HE: Hu- manEval, SB: SWEBench, SJ: Twebench-Multilinge RQ: ResQ. CI: CaniteEdit, ST: SWTBench, DC: Design2Code, CM: ChartMimie, DS: DSBench, SM: Swebench-MN. IC: Intercode- CTF, BD: Bird SQL, MC: Minictx, VS: VSCode, GS: General-SWFF Tasks.
Cal Genrrados
Coda Coneralie
SWE Tacks
Overall
60
Nig
Coepare Apri
0006
0.65
0.76
0659
0.65
0U55
15.25
3,06
0,06
6.5%
0.66
E,06
6.8%
8226
1,85
1204
10/5
006
105
01/5
1075
72.15
00/5
0,75
1.5%
0376
0.05
1.83
1015
0.00
180
(096
000
1076
100
146
10180
1.8%
3100
15.90
Clack
32.9
1375
1399
23/76
4.7
3519
DUPS
E.8%
135
1007
1035
Clack Smael
167
213
66
1
DE 20
00
547
333
55 123
M5o/71100000 4P%)
5%
15%
1216
13%
256
56
EB6
1676
1.4%
12.99
5,99
0075
15.91
20/95
1619
13.9
015
15%
19.9
139
133m
6.9
1156
(P74)
$3G
25%
130
390
596
17.90
$1.0
15.99
11.00
385
105
100
1008
31.4
Cluck 5
195
25%
15%
45%
75%
41.2%
1428
61.5%
55.4%
15%
1975
12%
25%
40%
12.04
-0.4%
Cludl Seacc44
265 S 25
48.8 20
53
133
367
383
58. 23
3395
1599
8.10
$196
000
100
3006
30AS
3.55
Table Tools available in different environments. The table shows the various tools provided by different environments for assisted analysis. Common tools like file manipulation and bash opera- tions are shared across environments, while specialized tools cater to specific tasks like web design and chart replication.
Category
Tool
Description
bash file_edit
Perform bash operations
Common Tools
Perform file manipulation operations
search repossiroy
Scarch the repository for string the entire repository
SWEBench
fle_name_scarcl view structure
Search for file by its name
View the structure of the current directory
view.hi wht.mlllppppp.. previeww view .original image
Get a preview of the index. html page as rendered in the browser
Get a screenshot of the html image for replication
Design2Code
zoom in zoom out
Zoom in on the current rendered html page
Zoom out on the current rendered html page
view www.python.pgg oython_preview preview
Get preview of the graph generated by python file
ChartMimic
niww.original.iii
Get screenshot of the graph for replication
test sql
Test a SQL query against the database
BIRD
get relevant schemas
Get relevant descriptions of relevant databuse tables
F ADDITIONAL RESULTS
Visual Grounding Errors. In Section 5.1 show that current agents struggle in visual ground- ing, despite some of these models being specifically trained for visual interfaces. To quantify the extent, we manually analyzed 20 random trajectories of two best performing agents: GPT-4o and Claude-3.5-Sonnet In particular, we quantify the number of trajectories where the model had at least one visual grounding error, where a visual grounding error is defined as any of the following: (1) incorrect click, (2) incorrect interpretation of the current state, or (3) interacting with the wrong element. Surprisingly, we find that 20% of the trajectories of Claude-Sonnet-4 3.5- 3.5-Sonnet, and 95% of the trajectories of GPT-4o contained at least one visual grounding error, indicating significant scope for improving these models for complex visual interfaces such as those demanded by PwP.
35% for Claude-
Training models to use IDE tools better would improve performance. In Section 5.1 we demonstrate that models can achieve superior performance when effectively utilizing IDE tools. In particular, Table shows the performance of assisted agents (averaged across models: GPT-4o, Gemini-1.5-Por and Vlaude-3.5-Sonnet) highlighting average gain of up to 13.3%.
23

---

Published as a conference paper at ICLR 2026
Step The agent needs to bug
sgent use tool. that performs name match across repository.
Step agent searches
"pracess" and reads "def process function".
Stop 6:
sgent
Step S. The agent reads the surnunding
Stop agent identifies edit,
with correct content, solving the task!
code the Line
and jumps to it. using VSCode tool.
Figure 11: Example the Claude Computer-Uss agent successfully using multiple IDE tools to complete repository level code-editing task.
However, analysis reveals two primary limitations in current models' tool usage: (1) poor visual grounding and inability to handle complex tool interfaces, and (2) failure to prioritize IDE tool-based solutions manual approaches.
To evaluate the second limitation specifically, we developed refactoring tasks within our 'General- SWE' dataset. These tasks require agents rename symbols across a project repository opera- tion that cannot be reliably accomplished through simple search-and-replace due to potential naming confliets and contextual variations. The IDE provides robust solution through its rename feature, which leverages the complete AST to ensure accurate symbol renaming across the codebase. This operation requires only pressing F2 on a symbol and entering the new name. In our evaluation, the Claude agent initially achieved 25% accuracy across four tasks when given no tool guidance. How- ever, when explicitly prompted with "You can utilize the rename feature VSCode perform this task," its accuracy improved to 75%.
We observed similar patterns across other tasks designed to evaluate tool usage. For instance, tasks that could be efficiently solved using the debugger showed limited success. While agents could sometimes set breakpoints, their poor visual grounding prevented them from effectively interpreting the debugging cntorface-paaticullee location. These findings suggest significant potential for improving agent performance through better training on IDE tool utilization.
in understanding the current execution state and paused line
Successful Use of Tools We further show a couple of examples of successful tool use in Figure 2 However, we do note that while the agent is able to use the IDE tool through UT interaction, still may not be able to make optimal use of it as shown in Figure 17
Agents Fail to Edit Files. File editing is a basic capability required in most SWE tasks. However, we find that the deficiencies in visual grounding significantly impact the file editing capabilities of current agents that use basic actions (clicking and typing). For example, even when provided with cursor location information in textual form, these models struggle to interpret such data amid complex UI elements. Models funetuned for UI interactions still commit basic editing errors-suck as incorrect indentation and text misplacement-anne are unable to recover from these errors (see Appendix for examples). We speculate these limitations could stem from two factors: (i) model overfitting to user interfaces in their training domains, or (ii) the increased complexity of the PwP IDE interface, which contains substantially more interactable elements than typical web or OS envi- ronments. Addressing these limitations represents an important direction for future work. Although direct file access via tool operations is available, UI-based editing confers unique advantages for
24

---

Published as a conference paper at ICLR 2026
Table 10: Performance Evaluation of Different Agents on 250 steps on SWE-Bench related tasks.
Model
SWE-Bench
SWE-Bench-Multimoda
SWE-Bench-Multligun
Average
Computer- Use Agent CUA w/ File/Bash Tools
10.0%
30.0%
15.0%
18.3%
60.0%
30.0 %
40.0% %
43.3%
mini-swe-agent
60.0%
30.0% %
35.0%
41.7%
tasks such as editing Jupyter notebooks, comparing changes, or modifying specific sections of large files. These results underscore two limitations: (i) current VLMS are challenged by complex UI in- teractions beyond simple web/OS interfaces (Xie et al. [2024] Koh et al.) 2024), and (ii) the inability effectively perform Ul-based editing prevents agents from leveraging valuable IDE features that could have improved their performance.
Agents Are Incapable of Recovering from Errors. Next, we find that current agents show lim- ited error recovery capabilities. When action fails to execute correctly, models tend to persis- tently repeat the same failed action without exploring alternatives. Similarly, if an agent selects an incorrect action, it continues along erroneous solution path without recognizing or correcting the mistake. In an experiment designed to probe this behavior, we deliberately suppressed one of the model's (Gemini 1.5-Pro) actions. Despite the environment's screenshot clearly showing un- changed state, the models proceeded with their planned action sequence as though the suppressed action had succeeded. This behavior suggests heavy reliance memorized action sequences rather than dynamic responses to visual feedback, resulting in exponentially increasing errors and poor performance. However, when we repeated the experiment with Claude-Sonnet-4 we tested such scenarios, and found only in one case, the agent ignored the screenshot, potentially highlighting that computer-use agents are improving over time.
Performance on Long Horizon Tasks. In our main experiments, we had capped the maximum number of agent steps to 20, owing to high cost associated with each of the models. However, certain datasets, such SWE-Bench, typically require much larger number of steps for agent to complete the task. In this section, we therefore evaluate 3 agents based on Claude-Sonnel 4.0, with 250 steps 3 relevant datasets: SWE-Bench, SWE--ench-MMutimen
SWEE-Bench. Mheecc--ultilinguall The results are shown in Table We note, that almost all agents show consistent improvement in performance with higher number of steps. However, overall tends remain consistent with 20 steps: Computer-Use Agents with File/Bash APIs show 43.3% performance, and mini-swe-agent shows 41.7% performance, and pure computer-use agents show 18.3% performance.
Robustness and A Ablations To address potential robustness concerns, we re-ran the experiments for the best-performing computer-use agent (with and without bash/file APIs) over multiple trials. We find the variance in performance any of the conclusions the work.
PWP-Bench-LLin is very low (Ixx)), and does not affect
We also investigated the importance of visual modality by running mini-swe-agent without pro- viding any images on the multimodal task category. Overall, the performance drops sigmificantly, demonstrating the importance of the visual modality. However, the effect varies by task: perfor- mance on SWE-Bench-MN. sees a non-significant difference, whereas performance on tasks like ChartMimic drops to near zero.
IDE Feature Usage Analysis We analyzed the distribution of IDE feature usage across all trajec- tories of the Blaude-Sonnet-4.0 Computer-Use Agent. Specifically, we prompted Claude-Sonnet-4.0 with complete trajectory (including textual steps and each step's 's screenshot), to generate what fea- ture the model used at each step. We then aggregate this information across all tasks, and report the results in [2] Importantly, the results just show when the agent attempted to use tool, and not whether succeeded or not. As shown in Figure [2 the agent primarily relies on basic editing and navigation features. Specifically, Text/Code Editing accounts for 26.2% of interactions, followed by File Explorer usage (16.8%), Terminal interaction (12.5%), and Global Search (10.6%). These four categories alone comprise over 65% of all IDE interactions. In contrast, advanced features significantly lower usage: Debug/Run capabilities are used in only 1.8% of interactions, and Code Intelligence features (such go-to-definition) account for just 2.8% Further, manual inspection
25

---

Published as a conference paper at ICLR 2026
shows that while the agent is attempting these advanced features, fails very commonly. This analysis reinforces our finding that while current agents can perform basic IDE operations, they struggle to leverage the full depth of specialized software engineering tools available in the environment.
Text/Code
25.8%
File Explorer
16.5%
Teminal
12.2%
10.4%
File
Others
5.7%
Raplae
3.0%
Code
Notebooks
2.7%
Image Viewer
2.3%
Dialogs
2.2%
Tabe
Gmd Palette
2.1%
Settings
2.0%
Editor Nav
Debug/Run
1.8%
Relative Frequency (%)
Distribution of VS Code Feature Usage by Claude-Sonnet-4.0 Agent. The agent pre- dominantly uses basic editing and navigation tools, with limited usage of advanced features like debugging or code intelligence.
Figure 12:
G QUALITATIVE ANALYSIS
In this section, we consider both positive and negative examples of agent grounding and ability to interact with the complete IDE interface in PwP.
H DISCUSSION
Computational Overhead of Running PwP While PwP provides a much more general interface for software engineering agents, natural question what computational overhead it introduces. The added computational requirements primarily come from: (1) capturing screenshots using the xdotool library, (2) running the IDE, (3) maintaining a VNC server, and (4) processing video and audio streams via ffmpeg. Importantly, only components (1) and (2) essential for all agents, as video and audio processing are only necessary when agents must interpret visual or auditory cues-a universal requirement for any environment supporting these modalities. The VNC server used solely for debugging or pair programming scenarios and can disabled when not needed. The xdotool commands consume negligible CPU resources (ii 1%) and minimal memory. While VSCode does increase memory and CPU utilization, the latency overhead remains limited, and the computational cost is substantially lower than running the large-scale computer-use models that power the agents. In summary, despite its comprehensive feature set, the computational overhead of PwP is minimal, with the primary computational demand stemming from the computer-use models themselves rather than the environment.
Why use IDE over simple Bash Agent? While computer-use agents perform worse than even simple API based SWE agents, intuitively there still remains lot of value in utilizing a general in- terface such as IDE, for software engineering. The reason being modern IDEs, have been developed multiple years of effort, and provide several advantages that are not possible with say bash in- terface. While, theoretically may still be possible to create equivalent tools, would take similar tremendous effort, to develop them again for agents, with less reliability.
26

---

Published as a conference paper at ICLR 2026
Figure 13: Example of Agent Missing Visual Figure 14: Example of Agent's Inability to Per- Error Indicators The agent fails to recognize form File Editing The agent incorrectly posi- linter error indicators (wavy underlines).
tions new content in the file editor.
Settings Icon
Model types search of file editor
Model Clicks-at
Wrong Kocation
Figure 15: Example of wrong mouse click by couude-Compuerr Use Agent The agent at- tempted to click Settings icon but clicked at the wrong location.
Figure 16: Example of Agent Misidentifying Active Panel The agent fails to recognize the active editor panel, incorrectly typing into the search bar (red arrow) instead of the file editor.
To give few examples of myriads advantages of IDEs:
Interactive Debugging Capabilities
IDEs provide rich, stateful debugging interfaces that allow Al agents to set break- points, inspect variables, and evaluate expressions dynamically
Unlike CLI debuggers (GDB, LLDB, pdb), IDE debuggers maintain visual context and state, making it casier for AI agents to track program flow and debug complex scenarios
The visual representation of stack traces and variable states is more structured and machine- machine-parseable compared to text-based CLI output
Intelligent Code Refactoring
IDEs maintain complete Abstract Syntax Tree (AST) of the project, enabling aceu- rate symbol renaming and code restructuring across multiple files
Al agents can leverage IDE's semantic understanding to perform complex refactoring operations with higher confidence
Unlike text- ext-based search-ande search-and-replace in Bash, IDE refactoring tools understand code context and prevent accidental modifications to unrelated symbols
Test Management and Coverage Analysis
IDEs provide structured APIs test discovery, execution, and result analysis
-
AI agents can efficiently track test coverage through visual indicators and program- matic interfaces
Real-time test feedback and coverage data is more readily accessible compared to parsing CL.I test runner output
-
27

---

Published as a conference paper at ICLR 2026
c.)
Fuitineuring
a.)
A11
OLINA
Fut Fineurinn
b.)
OLORA POT
M42
Usaed
Opiiion
ISS Netoros DENY
Optinee
Figure 17: Performance comparison of GPT-4 agent in Computer- Use and Assisted settings the ChartMimic dataset. a) Image as seen by the Computer-Use agent. b) Replication in Computer-US setting. Replication in Assisted setting. The Assisted agent demonstrates superior performance despite seeing the same image but in different context and state.
Performance Profiling and Analysis
IDE profilers offer structured data about CPU usage, memory allocation, and runtime behavior
Visual representations of performance metrics (flame graphs, memory usage) are eas- ier for Al agents analyze systematically
Profiling data available through APIs rather than requiring parsing of complex text- based output
Code Indexing and Semantic Search
IDEs maintain comprehensive code indexes that enable fast, context-aware code search and navigation
AI agents can leverage these indexes for more accurate code understanding and mod- ification
Unlike grep or find, IDE search capabilities understand code structure and can filter based on semantic properties
Extension Integration and Automation
IDE extensions can be programmatically controlled through APIs, allowing Al agents leverage additional tools seamlessly
Extensions can provide structured data and interfaces that are more reliable for au- tomation compared parsing CLI tool output
Configuration and coordination of multiple tools be managed through unified IDE interfaces rather than managing separate CLI tools
I PROMPTS
In this section, we provide the prompts used for the agents our evaluation.
L.1 COMPUTER-USE AGENT PROMPTS
mt__eeaaeee vneyou auconamous intell ligens Lasked with h Int with code 1D2 (e.g., VSCoder. given tasks accomplieh within the use epecific wos lons you dan isxue.
acoomplishes through
28

---

Published as a conference paper at ICLR 2026
Scre's a the information yourll have:
user's eb fefectiver::
the you're crying
++the current scrsenshot This scresashot
each clickable eiement assigned
unique aumerical ID. Each nounding box and its respective ID chares the sand color.
The observationrs, which lista the Ils of all clickable elamenta the cuzzent IDE screan with their text fornat '[1d] [elenent tyse]
any, link, Lexibos, wic. Por exaaple, *[123] [buitun] ["Rua"] neans .barefa button wich 123 Lexs content "Ras" on the current IDE screon, weleit Inagere; The differsace betxeen the current inage and the previcus inaca, hichiighting the changes information
tyre
the accion executed effect er not, adEicionally, this serves purpose clearly showing content that you may want recus oa.
have
the intended
The curaos positienve Information about the cueren: curaoz position, provided DOM element in both image fornats.
11
The actions you can parfern fail into several categories:
12
can the computes_contee soal issue these secions. example, you can and pass arqunenca as "xdotool type "hello world"! to type "hello world" as the carzent cursor position.
computer_ponteed
15
.*Keyboard Actionssex
18
'adaLoul Lype (eomteni)""s Type the specified consen- as the carzent turser pus-tion, "xdotool key Lonyonebbiattio "s Simulate pressing key or combination of keys (6.9., "xdotool key Messles file).
19
21
*ooose Actions:re
22
element with Lse specified id and elick oxdatonl Housenove [x] [y] olick [el pode)) Move the mouse to the nooedinates (9 3) on the screen and click.
"xdoLoul on 1t. [id] glick pod_eode]
24
--repeat 1h bouble-click 'adotool nousenove [id] click 5% Seroll down on the elament with the apeci fied sd.
the specifled
27
28
Navigat Actioosses
31
*=Inceract ing with ThE Boal mest You dan use any bools inside the ThE, such as fille expil orer aearch, ge to definit ion, by ger peefooning she agpropeiate keyboard or mouse wo.lons.
34
35
iCCorgietion Actionise
'execution ion Taxue this conmand when you belleve the Lask is complete. Do not genezabe anything aftez this action.
38
41
To be succesefol, it is very inportant to follow chese rules:
42
+sStart
achieve objectivenss
Regin the task by creating a plan how you achieve the object vie. Think about istepa you need take, the tools you can use, and the actions you need to pertorn. This will help you stay organized. and focused throughout the task.
+*Start every step with on Image Descriptions::
47
Regin every step by desor bing the provided ded IDE screenshot.
Enclose your description within teimage doscription>*
<image descriptions
51
<!-- Your descristion here m->
52
</inage descriptions
(1
55
2. *Tinnk Betore Actinges:
screanshot and
action carefully.
IISou Only Velid Actionssr:
59
Only perdorn actions that are valid given the current absorvation.
62
Complation:::
63
Use 'oxesution_dons" when you think you have sealeved the objective.
64
**Corso: Pociticnings::
67
Befora editing file or field, nake sure whare the cursor la. Clear thines 1f you have alzeady weitzon You siso sending key etritg, typing rumber, press Retuen, then
right iscation right using acrow keys. 1f unsure, use keybeard shortcure oz nouse actiors to place the curser approoria:ely before typlac. Despita being sure, you night still make nistake. Reviow serecachots and texz informacion efter cact.
sonething,
utility
69
action
71
*PPrecision in Actionses:
72
Be precise when pezforning nouse actions.
kayboard actions
actions
29

---

Published as a conference paper at ICLR 2026
+00:1liee Available Toolave;
76
Leverage any fanctionalities available vithia the IDe to accomplish the task.
77
+Cther ripgee:
**llo Uliee Use ur whtn porsible instead of editing riles.
S1
reevies the text on Scraeneri Previous exparimenta with you show, that you aften confuse what la shown text infermation provided to varidy you secing
in the
inage.
82
.=Loazn Erom Mistakeser: If an action or szep of actions didn't get the interded result, think of difforent stratcey in order to achieve the gea.. *-Heybuard Shorteutars:
keyboard
possible
eff.cieney,
order to opan sottings, use "xdoteol kay stal-," instead clicking on the settinga icon. +*ihat is en Screenes: If you do not see somethira in nent/sctting that you ware planning to use, look For appcopriaca search bas, and
84
relevant queries to find the op.ion are looking for. you plarning
**llear the Bditer/ IItte Fialdre;
sonathing in editor input field, suze to clear the existing ing bortent before tyging the new pontent. For instance, you tan use "xbooul key cizlra BackSpace" to cleat the concent.
36
Locationi Do not autonatically assune you are at the right location batoze typing. For instance, it you
sonething, make despite the comand being cosrect, you are ausposed find the tigh: input field click it and then type aqair.
the
irgut field.
gets typed,
87
VSDCod musto Vicode shorccuts oot recesserily ly sene sdotool somerdaa For example in two commanda: "xdatool ctrlek' folloved
order
strlsk etrite, you will xdotool key sirl-e'.
88
89
should nechodinnlly guide towards accerpl ishing regui task, using precise and asomic comands. Psioricias keyloard Intecactions over mous actions to echanow wfficiency.
91
92
CIMPORTANT PTLE RDTTING>
Keep mind theas Lipe while wdiling filea: To jurp particular lina nunber, you can "otritg" followed 'line number" (and optionally colurn number, egs 11:12) and then preas "Return",
95
If you execube Lype command, howwves, file dows aot change, it can likely sean, the focus is not Lbe alick it, eneure
nove your While typling make sure that cocrecs indentat being used.
foruzed.
97
1.2 MINI-SWE-AGEN PROMPTS
systen pertenttenplate:
You are a helpful assistan: that can interact kith computer.
Your response mast contain exactly
with command conmanda connected sa
[II.
Include a TROUGHT section before your command shere you explain your reasoning procoss. response shown in cxumpet_
<tornst exemple>>
<tommat_exempie>
Your reasoning and analysia heze. Explain why you want to serforn the action.
12
yau
13
</Earnal_esample>
16
Failure to follow these ruics will cause your response to be rejected.
17
instancs_ienge iemplate:
Flease selve this
[[task]]
20
You can execate bash coomands and edit tiles to inglement the mecessacy changea.
21
becomennded Workflow
24
This warkfieus should be dona step-by-stop so that you can iterate on your changes and any possible problens.
25
Analyzo codobase by finding roading relavant files
27
2. Creale a suript ipl Le repcutuce the issue
23
3. Edit the sourcs coda to reseive the issue
5. Twal edge cassa Le easure your your scrigt again
Verify your
by
zobuat
31
6. Bubnit your changes and finish your work by issuing the follewing comaand: CONETRTR_TACE AND SUWNTY, [ir__PP FINAT_
'echo
csubine it with any otleez task.</imoztant>
ciaporaan this command, you working
33
34
14 Inportant Rulea
1. Every reagonse must contain exaco ly one action
37
2. The action must be snclused in Lriple backsieks
38
3. tirectory envi ronment veriable changes are persistent. sction executed new subshell.
Hoveves, you can prafix any action
wo__ay_vaewr_vae VALIE /pact//o/wwackin//dii ... weite/
lead enviromment varishles from files
40
Formatting your resporse
30

---

Published as a conference paper at ICLR 2026
44
Bare is an exumoic of carract xaspense:
45
coxample_regpons>>
IHONGRI: uadorstand structure roposátory rlest. chock curzent direszory to get better understanding of the codabase.
48
bosch
51
52
</ </esampie_respcce>>>
examples
56
244 Create new films
"beah
59
cac cetBop' newtile.py
60
Import numpy 25
bella *aurld"
peint (hello)
63
ECF
64
67
08
"bash
Replace hww.uu/iiiiggg/euuuiinogg filenaneepy
71
72
Replace only first oncurrence
i 's/uldatrrnn aow//////aaaaaaoo Cilemame.py
ace first reourrence on Line
76
sed 1 "Ia/ wnw.///..anaaaeaaoooo....o cung// filmname.py
Regi ane commr renows lines 1-10
79
sed 1 11 10//e/l wtw.aa/ttee///enaaleeeoooo
Cilerams,
50
contents
$1
54
specitic with cumbers
r1 lenenee-py sed 2002
S7
83
you
91
1xtpash
92
anything
31


Reviews:
[{"review_id": "SOmIVrYRM8", "reviewer": "Reviewer_f9o7", "summary": "This paper presents Programming with Pixels (PwP), the first IDE-based environment and benchmark for evaluating computer-use agents (CUAs) on software-engineering tasks. PwP enables agents to control a VSCode IDE visually through keyboard and mouse, while PwP-Bench unifies 15 multi-language, multimodal SWE tasks. The authors systematically compare pure visual CUAs, CUAs with minimal text APIs, and specialized SWE agents. Results show that visual CUAs perform poorly (22.9%) but reach near-specialist performance (50.7%) when given file-edit and bash APIs. PwP thus establishes software engineering as a fair, comprehensive testbed for generalist computer-use agents.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Addresses an important intersection between two major agent paradigms — IDE-based (CUA) and API-based (SWE) — by providing a unified testbed for fair comparison.\n\nNovel environment design: the first fully interactive VSCode-based setup enabling realistic software-engineering evaluation.\n\nComprehensive benchmark: integrates 15 diverse SWE tasks across modalities and languages.\n\nInsightful analysis: identifies visual grounding and tool-usage limitations as key failure modes, offering concrete directions for improvement.", "weaknesses": "Only one specialized SWE baseline (Mini-SWEAgent) is included; broader baselines would strengthen comparisons.\n\nConceptual clarity: definitions of computer-use agent vs SWE agent and the rationale for “fair evaluation” (why expressiveness and access matter) could be better explained.\n\nPwP-Bench-Lite covers many task types but has very few instances (20 per task), and the paper reports no repeated trials or variance, raising robustness concerns.\n\nTechnically, the benchmark adapts existing datasets rather than introducing new ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTpMKFU4xZ", "reviewer": "Reviewer_xKNb", "summary": "The paper introduces an environment to approach software engineering using a visual interface. The visual interface consists of screenshots, sometimes enhanced by the set-of-mark technique, which provides a list of elements in addition to the screenshot. They provide an evaluation benchmark consisting of 15 tasks with an OS and a VSCode-like IDE. The VSCode IDE is changed so that it supports the set-of-mark technique and gives the DOM tree.\n\nThey then conducted a comprehensive evaluation of different computer-use agents and found that they perform poorly on this benchmark using only visual inputs and actions like 'type' and 'click'. Then they added two functionalities that boost the performance of agents. They gave them text-based terminal access and text-based file editing tools. These two additional text-based abilities boosted the performance of agents. They also investigated and saw that on some tasks the agents mostly use the text-based interface, while on others they only use the visual interface.\n\nThe paper also investigates why pure visual agents fail, identifying two main issues. The first and primary weakness is poor visual grounding. The second issue is the agent's inability to leverage sophisticated IDE features, such as debuggers or profilers, even when these tools could simplify the task.", "questions": "1. The testing environment needs more clarification on its safeguards, as it has been seen that LLMs might simply remove tests if they have access to those files. Do they have access to them through bash, file editing, or the visual interface, in theory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The provided benchmark is very valuable, and its container-based approach makes evaluations consistent.\n2. The provided VSCode IDE that returns screenshots with set-of-marks and DOM is valuable and can be used in future research.\n3. The evaluation of various computer-use agents is comprehensive.\n4. The ablation studies are also comprehensive, and the results on how much the text-based interface vs. the visual interface is used are very interesting.", "weaknesses": "1. To measure the importance of the visual interface, it would be interesting to measure the performance when agents only have access to text-based tools (file edit, terminal, and task-specific tools). It would show the computer-use agents' ability in text-only settings.\n2. Some tasks aren’t relevant to agentic software development. For example, changing the theme of the IDE is not that important, while similar IDE setting changes, like turning on auto-suggestion, could be helpful. \n3. Some deeper insights into which IDE features agents are using would be very beneficial to add to the paper. For example, can the paper provide a number on whether the agent uses debugging tools, or if it ever did? The framework would benefit if these insights are generated after running tests, which would help those developing agents using this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WyiUBLRRFM", "reviewer": "Reviewer_pzk4", "summary": "Following are the key components of the research article, although there are issues with some of the sections that will be listed later.\n\nProblem Statement:\n- The problem statement is clearly articulated in the Introduction (Page 1-2):\n- Primary question: \"can general-purpose computer-use agents match specialized agents in complex domains like software engineering?\"\n- Motivation: Current evaluations of computer-use agents focus on simple tasks, leaving unclear whether they can handle sophisticated tasks like software engineering. \n\nState of the Art:\n- Related work is covered in Section 2 (Page 2-3):\n- Comprehensive coverage of multimodal and computer-use agents\n- Software engineering agents literature\n- Comparison with existing environments and benchmarks\n\nMethod Explanation\n- The methods are explained in Section 3 (Pages 3-5):\n- Programming with Pixels (PwP) environment as a POMDP\n- PwP-Bench benchmark with 15 tasks across multiple categories\n- Evaluation methodology clearly described\n\nResults Presentation\n- Results are presented in Section 5 (Pages 5-11):\n- Table 1 shows comprehensive performance evaluation\n- Clear comparison between different agent types\n- Detailed analysis of limitations and capabilities\n\nClaims vs Results\n- The abstract makes specific claims that are supported by results:\n- Claim: \"pure visual interaction performs significantly worse\" -> Supported by 22.9% vs 48.8% performance\n- Claim: \"file editing and bash operations dramatically improve performance\" -> Supported by 50.7% accuracy with APIs\n- Claim: \"clear room for future improvements\" -> Supported by visual grounding analysis and improvement potential demonstrated\n\n\nThe authors systematically address whether computer-use agents can handle software engineering by creating a comprehensive IDE environment that captures real software engineering complexity. Then they evaluate across diverse tasks spanning 15 different scenarios, which provides fair comparison with specialized agents through the same environment. Authors state that no prior comprehensive environment existed for this domain. While individual components exist (computer-use agents, SWE benchmarks), their combination is novel. Table-3 mentions which areas were covered by other works as compared to this work.", "questions": "The following recommendations address critical gaps in statistical rigor and model diversity that would significantly strengthen the study's scientific validity:\n\n- The current study's minimal statistical testing (only one p-value reported) represents a fundamental weakness for a comprehensive benchmarking effort. How the p-value is derived is also missing. Implementing proper statistical methodology with confidence intervals, effect sizes, and multiple comparison corrections would transform the findings from descriptive observations to statistically significant conclusions. Power analysis would ensure the 300-instance sample size is adequate for detecting meaningful differences, while systematic statistical testing across all 15 datasets would provide robust evidence for claimed performance gaps.\n\n- The absence of mixture-of-experts models represents a significant oversight, as these architectures are increasingly important in the LLM landscape and may exhibit different performance characteristics on computer-use tasks. Expanding beyond Claude-centric evaluation to include these diverse model architectures would provide crucial insights into whether performance differences stem from fundamental model design choices rather than implementation details. This broader coverage would make the benchmark more representative of the current AI landscape and more valuable for future research direction.\n\n\nThe following are a few minor issues identified in the article\n- Page 5: \"evluate\" should be \"evaluate\"\n- Page 11: \"intutively\" should be \"intuitively\" \n- Page 13: \"potetntially\" should be \"potentially\"\n- Page 21 (and some other places): \"os\" should be \"OS\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The PwP environment addresses a critical gap in evaluating computer-use agents for software engineering tasks. By providing a realistic IDE interface through VSCode, the environment enables comprehensive testing of agent capabilities while maintaining the expressiveness needed for diverse software engineering activities. The extensible design, with features like checkpointing, multimodal support, and easy benchmark addition, positions it as a lasting contribution to the research community.\n\nThe study maintains fidelity to established benchmarks by using programmatic verifiers and metric calculations that match the original dataset specifications. This approach ensures meaningful comparisons with prior work and establishes a reliable baseline for future research, while the 15-task PwP-Bench provides comprehensive coverage of software engineering activities.\n\nThe combination of detailed performance tables with concrete examples of agent behavior (such as visual grounding errors, successful tool usage, and failure modes) provides a balanced perspective. This multi-faceted analysis helps explain why certain performance patterns emerge, making the results actionable for both understanding current limitations and guiding future improvements in computer-use agent design.", "weaknesses": "The following are key issues identified in the work\n\n- The reliance on PwP-Bench-Lite (300 instances) due to computational constraints creates potential sampling bias, potentially missing important edge cases that would emerge in full-scale evaluation. The 20-step limitation, while computationally necessary, may artificially constrain performance on genuinely complex software engineering tasks, though the supplementary 250-step experiments provide partial validation.\n\n- The heavy focus on Claude-based agents (3.5, 3.7, 4.0) introduces anthropocentric bias, as these models may share architectural similarities that don't represent the broader landscape of computer-use agents. Additionally, evaluating only VSCode limits generalizability across the diverse IDE ecosystem used by developers worldwide.\n\n- The inconsistent file editing capabilities across models and the limited investigation into API-driven improvements suggest insufficient mechanistic understanding of why certain interventions succeed. The study documents performance differences but lacks deeper analysis of the underlying factors driving these disparities, limiting its prescriptive value for future agent development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpwvgqAsVV", "reviewer": "Reviewer_5ZwW", "summary": "This paper introduces PwP, an environment for evaluating the software engineering (SWE) capabilities of computer-use agents (CUAs). It also presents PwP-Bench, a benchmark covering 15 task types, including newly proposed datasets for VSCode-use and general SWE tasks. Experimental results show that current CUAs perform poorly on SWE-related tasks. Only when a bash tool is added do they achieve performance comparable to a purely text-based mini-swe-agent that relies solely on bash operations.", "questions": "- What is the fundamental difference between using IDE images as input in PwP-Bench and using webpages as input in SWE-Bench-MM?\n- Is a CUA with bash tools still a general-purpose CUA? Does it become a SWE-specialized CUA or even a vision-enabled SWE agent?What exactly are we evaluating here?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Evaluating CUAs on more complex tasks to comprehensively assess their capabilities is an important and valuable direction.\n\n- The paper aggregates a wide range of existing datasets and adds two new SWE-related task sets, resulting in a large and diverse benchmark.", "weaknesses": "- The motivation is somewhat confusing. General-purpose CUAs are primarily designed for everyday computer-use tasks, such as web navigation or basic UI operations. While CUAs **can** perform SWE tasks, that doesn’t necessarily mean they **should** be expected to do well on SWE. This feels similar to asking *whether an image generation model can produce good videos*—theoretically possible, but not aligned with its intended design or primary evaluation purpose.\n\n- I also find it difficult to identify the essential difference between PwP-Bench and SWE-Bench (-MM). If the goal is to test SWE performance, then a simple bash environment with read, write, and execute (-rwx) capabilities is sufficient (which is also the design strategy of Mini-SWE-Agent). Notably, code is inherently *textual*. Even web front-end layouts are generated from *textual style configs*. If one wants to expose the model to rendered visual outputs, webpage-based inputs (as used in SWE-Bench-MM) already achieve this. Could the authors clarify what is the core different between using webpages as vision input a  using IDE screenshots? Furthermore, everything visible in an IDE can already be accessed in text form, without introducing noise or visual ambiguity. Thus, if the goal is to evaluate SWE ability, feeding noisy visual input to the agent seems counterproductive. PwP-Bench assumes that the agent must read from IDE images,  which introduces an artificial constraint that inherently weakens any agent’s performance on SWE tasks.\n\n- Moreover, the authors seem aware of the limitation of solely using visual input for SWE: They add a bash tool to CUAs, and indeed, Claude-Sonnet-4.0 with bash achieves the best performance—nearly identical to Claude-Sonnet-4.0 Mini-SWE-Agent. Again, it is a very, very small agents with only **100 lines** of code and **only one simple bash tool**. This strongly suggests that SWE tasks are simply not well-suited for general-purpose CUAs. Even a minimal agent—an LLM with just a bash tool—can match or outperform them.\nThis further reinforces the view that CUAs are not suited for solving SWE tasks, thereby undermining the claimed contribution of this work.\n\nI agree that CUAs should be tested on more complex tasks, but SWE may not be the right choice. Since CUAs’ key advantage lies in their visual interaction capabilities, it might make more sense to evaluate them on tasks that genuinely depend on visual understanding, rather than on code-centric problems that are mostly text-based.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eadHMXttFq", "reviewer": "Reviewer_Syoz", "summary": "This paper introduces a new environment and benchmark to test whether general-purpose computer-use agents (CUAs) can perform complex software engineering (SWE) tasks. The authors develop Programming with Pixels (PwP), a VSCode-based environment where agents interact visually via clicks, typing, and screen observation to complete tasks such as code editing, debugging, and UI design. They also propose PwP-Bench, a 15-task benchmark covering multiple languages and modalities, to systematically assess agent performance. Through extensive evaluation, the study finds that purely visual CUAs perform far below specialized coding agents, but when granted limited text-based APIs (file editing and bash), their performance nearly doubles, approaching specialized agent levels. The analysis highlights poor visual grounding and limited tool usage as key weaknesses, while emphasizing the rapid progress and potential of CUAs for future generalist software engineering applications.", "questions": "Since file and bash APIs drastically boost performance, do the authors believe that future progress should prioritize improving visual grounding or integrating more structured textual interfaces? and how might that affect the generalist nature of computer-use agents?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper introduces Programming with Pixels (PwP), the first comprehensive IDE-based environment that allows computer-use agents to perform real software engineering tasks through visual interaction, offering a realistic and general testbed. They proposes PwP-Bench, a well-designed and diverse benchmark spanning 15 tasks across multiple programming languages, modalities, and software engineering domains—covering code, UI, data science, and IDE configuration.\n\n- It provides clear empirical insights showing how adding minimal text-based APIs dramatically boosts agent performance, helping isolate the impact of visual grounding and tool-use limitations.\n\n- It establishes software engineering as a meaningful and economically significant domain for evaluating generalist computer-use agents, setting the stage for future research toward specialist-level generalist systems.", "weaknesses": "- There is limited discussion on the cost, efficiency, or scalability of running PwP which are important factors in order to serve as a community standard benchmark.\n\n- Although PwP mimics a realistic IDE, the experiments do not explore longer, multi-step development workflows or collaborative settings common in real-world software engineering.\n\n- The evaluation metrics are primarily accuracy-based and may not fully capture partial progress, reasoning quality, or the richness of multi-step interaction trajectories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VAh9wgEn7i", "reviewer": "Reviewer_KWZS", "summary": "The paper benchmarks multimodal LMs on solving software engineering tasks using an IDE.\nThis PwP benchmark is built from many existing benchmarks, plus 20 tasks that have been added.\nOverall, the paper presents results based on a random sample of 300 task instances (though the full benchmark contains more than 5000).\nIt clearly demonstrates that when only given computer use controls, all models significantly underperform on almost all areas of tasks when compared to models that are given simple APIs.  The paper also makes some observations about how performance could be improved on the benchmark.", "questions": "* It would be interesting to have a baseline completely without multimodal tools, for example by running an unmodified mini-swe-agent. This would be an obvious comparison for code generation & editing, but some multimodal problems can also be solved without access to screenshotting etc. (and if not this would show us the extent to which the benchmarks _require_ multimodality to be solved). For example many instances SWE-bench multimodal can definitely be solved without having the agent use screenshots for verification etc. (though including the image in the initial task instance might be required for some instances)\n* Since the pwp benchmark mixes so many very different benchmarks, understanding how the categories are made up and how the lite split is built might be main-text-worthy. Maybe just mentioning on p. 4+5 what the percentage of each of the benchmarks is in the corresponding category? Or even just some `n=` number of instances in tab 1 to understand how many instances are evaluated. \n* If I understand the appendix correctly, and the paper samples 20 instances for the lite split from each task source, then this also leads to very different numbers of tasks for the different categories (e.g., General SWE tasks only has 20 instances in total), so these instances might have a disproportionate sway over the overall avg if you take the unweighted average. It will also significantly increase the statistical uncertainty on the scores. \n* Could you show the prompts that you use for the agent in the appdx? One through-line of the paper is the comparison of task-specific software engineering agents (like mini-swe-agent) and general purpose agents. However, mini-swe-agent seems to be pretty general, too, since it doesn't have any specific tools for software engineering (other than that the prompt mentions a few strategies). So unless your prompt is much more general, I'm not sure if this generalized vs specialized narrative really is supported.\n* How can mini-swe-agent (even with screenshot support) score in some tasks of the \"General SWE task\" category? The appdendix for example gives \"installing of vscode extensions\" as an example. Can this be done from the bash?\n* Since there are only 20 tasks that are added and because they have a strong influence on the overall score that is reported, it might be worth putting a summary of each of them in the appendix.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Significance: Computer use is a very relevant skill and the constructed benchmark is very challenging (with a current best score of 23% for computer-use agents) while staying close to real-world use cases. Especially if LMs were to be evaluated on the full set of > 5000 task instances, this would show a very fine-grained picture of LM skills. Because so many different benchmarks are combined, the data obtained is rich and could be analyzed even further. Overall, I expect this benchmark to become relevant as a computer-use agent benchmark especially as easier benchmarks get saturated. \n\nOriginality: The benchmark is mostly constructed from existing benchmarks, but using them to evaluate computer use is new.\n\nQuality: The paper is solidly executed and evaluating on tasks from so many different existing benchmarks, as well as running computer use agents to scale is not easy. The quantitative and qualitative observations on computer use agent behavior are interesting.\n\nClarity: Well written and easy to understand.", "weaknesses": "I mostly had some questions about the composition of the benchmark as well as the score, I've put them all in the questions section.\n\nMinor presentation details:\n\n* p.6 official name seems to be mini-swe-agent (with dashes) (also appears completely without dashes in tab1). It might also be good to call your version mini-swe-agent multimodal or similar or clarify in tab 1 (when I first looked at tab 1, I thought this was the off-the-shelves mini-swe-agent, i.e., without any multimodal support built in)\n* Tab 1: There are two \"best numbers\" in bold in column 3 \n* Tab 1 caption should point out that these are the results on the lite split of the dataset. It would also be helpful to put `n=` for the different evaluated categories", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)