SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
Authors: John Yang
Carlos E Jimenez
Alex L Zhang
Kilian Lieret
Joyce Yang
Xindi Wu
Ori Press
Niklas Muennighoff
Gabriel Synnaeve
Karthik R Narasimhan
Diyi Yang
Sida Wang
Ofir Press
Venue: ICLR 2025 Poster (2025)
Keywords: Language models
Natural language processing
Software engineering
Primary area: datasets and benchmarks
TL;DR: Our new multimodal software engineering benchmark reveals that current LM agent systems do not generalize to other programming languages.

Abstract: Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains
(e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent’s flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task
instances compared to 6% for the next best system.

Published as a conference paper at ICLR 2025
SWE-BENCH MULTIMODAL: Do AI SYSTEMS GENERALIZE TO VISUAL SOFTWARE DOMAINS?
John Yang"
Carlos E. Jimenez
Alex L. Zhang²
Kilian Lieret
Joyce Yangå
Xindi Wu?
Ori Press
Niklas Muennighoff Gabriel Synnaeve
Karthik R. Narasimhan
Diyi Yang!
Sida I. Wang
Ofir Press
Stannoord University
Princceton Language & Intelligence, Princeton University
*Cornell University
*Tübingen Al Center, University of Tübingen
*Meta Al
ABSTRACT
Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE- bench (Jimenez et al. 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual el- ements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g.. front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE- bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE--bench 17 JavaScript libraries used for web interface design, diagramming, data visual- ization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top- por-perfrnnng SWE-bench systems struggle with SWE-bench M, revealing limitations in visual coablemmsolvingg and languaee generaliza- tion. Lastly, we show that SWE-agent's 's flexible auguuage-gnostte features enable it substantially outperform alternatives SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.
features 617 task instances collected from
INTRODUCTION
Language models (LMs) are being increasingly deployed to assist software engineers (Bagalkote 2024 Yepis & StackOverflo 2024). As LMs gain in prominence, the research community has been expanding from building LM-based assistants that work at the code line or function level (Chen) et al. 2021 Hendrycks et al. 2021) to building autonomous systems that can maintain and improve large codebases with hundreds of files and thousands of lines (Wang et al., 2024b) Xia et al. [024] Yang et al.) 2024; Zhang et al.) 2024b). These systems provide LMs with tools and environments that let them engage in multi-step interactions to solve complex software development tasks.
SWE-bench (Jmenez al. 2024a) is the most popular benchmark for evaluating the performance of these systems. Drawn from GitHub issues and pull requests, SWE-bench task instances capture a range of software bugs and verify solution behavior by executing unit tests. Since the introduction of SWE-bench in October 2023, state-of-the-art performance on SWE-bench Lite, the most commonly used subset of SWE-bench, has soared from 3% 43% (Jimenez al. 2024b).
However, with respect to the broader landscape of software engineering, SWE-hhnch reflects only a fraction of real-world applications. Its 17 repositories are predominantly written in Python. Task stance codebases tend to be structured similarly because every repository is a PyPI package. Though
Equal contribution. Correspondence to jownby@stan edu carl ose j@pr www.on.eee.. .dd Data, code, and leaderboard hww.wennch.om/mmlllmmo....

---

Published as a conference paper at ICLR 2025
Diagramming
Interactive Mapping
Syntax Highlighting
Web Frameworks CasoodrrSricejjmaaiiiiaaa
Show message element name
KML Symbol Align/Pooemnn//Ste
Brooket highlighted with different color class inheritance contect.
Ourrendy namos message
There brgg
paint some syrrisols
# t
diements mussuge Ilovis nel rendend
I've attached a screen citpping from Google Earth
Reproduced J5Ficdle hitposs plidile not hwwhgn/eeehh//heee
Companent CascaderGnleet
Givan this sccemple cluggamn Impga]
alww now it supposed to look.
### Stupe to reproduce
[Lefi Imagel Light Imrupe)
Dmoge)
maapp
class
WV
clace TABLE: TAg !
tr(inie: Unie)
class TR: Tag
fun td(init: 10.0 <> Unic)
Tag
bpmn-jj
openlayers
Ms highlight.js
next
Four task instances from SWE-bench Multimodal. The 17 JavaScript repositories featured in SWE-bench M introduce new categories of software development challenges, such as the use cases shown above, along with performance profiling, art, and more. In addition to multi- modal comprehension, SWE-bench M contains more multi-language issues for both programming (JavaScript, TypeScript, HTML, CSS), and natural (English, Chinese) languages.
Figure 1:
SWE-bench features several commonly used backend and data science libraries, many other use cases are not represented. In addition, many domains of software development rely visual assets, such as user interface design, gaming, virtual reality, and data visualizations (Foley & Van Dam 1982 Shneiderman 1998), but only 5.6% of SWE-bench tasks contain an image.
In light of these limitations, this paper poses the research question:
"Do AI Systems Generalize to Visual Software Domains?"
A popular sub-field capturing several of these unexplored dimensions is front-end web develop- ment, which focuses on building user-facing applications and websites with visual and interactive elements Si et (024). Here, coding problems frequently contain both textual and visual com- ponents (Cherubini et al.] [007]] Moody 2209). As of 2023, JavaScript, the primary vehicle for front-end development, was the most popular programming language for the last decade (Daigle) & GutH 2023). The JavaScript ecosystem encompasses a wide range of frameworks, such as Node.js, React, and Angular, each embodying unique architectural principles (Wittern et al.) 2016).
In this work, we introduce SWE-bench Multimodal (SWE-bench M), dataset of 619 task instances focused on visual JavaScript problems. Like SWE-bench, we derive instances from real-world issues on GitHub. Unlike SWE-bench, our collection targets JavaScript-based, user-facing applications, such as UI design systems, web app development, interactive mapping, and syntax highlighters, filter exclusively for task instances
with four example instances shown in Figure Furthermore, that contain images or videos in their problem descriptions or testing scenarios. Finally, through verification by human experts, find that for 83.5% of task instances, images are necessary for solving the corresponding issue.
We discover that existing systems perform signiticantly worse on SWE-bench M than they do on SWE-bench, due in large part to the challenges of visual problem solving and JavaScript's diverse development practices. The performance results on SWE-bench M appear to vary depending on the diverse types of images and challenges presented. Different visual elements, such as code snippets, website screenshots, and diagrams, seem to require distinct comprehension abilities. Furthermore, JavaScript's support for object oriented, functional, and procedural programming introduces sub- stantial variance how codebases are structured, which standardized solutions struggle with.
Our efforts to adapt existing systems highlight gemeralizability as a desirable but overlooked con- sideration for building LM systems. Current systems evaluated on SWE-bench rely heavily on Python-only parsers to perform fault localization steps independent of an LM [Örwall 2024 Xia et al., 2024 Zhang et al., 2024b). These approaches either force system builders to re-engineer tools for other programming languages or fail entirely when similar tools are not available. Among our takeaways, we offer suggestions for building agent systems that operate efficiently in numerous programming languages and repositories with visual content.
2

---

Published as a conference paper at ICLR 2025
Table I: Comparison of repository-level coding benchmarks. We characterize benchmarks by their number of repositories (# Repos), programming languages (Lang.), whether they employ execution- based evaluation (Exec.), and include tasks with image content (Images). Programming languages for front-end apps and visual assets are novel attributes introduced by SWE-bench M.
Dataset
# Repos
Lang.
Exec.
Images
RepoEval (hhagg et al. 2023b)
14
Python
X
RepoBench (Li et al. 2023b)
> 10,000
Python, Java
X
SWE-bench Uimenez al.
2024a)
17
Python
SWE-bench Multimodal
17
JavaScript
2 SWE-BENG MULTIMODAL
We first review SWE-bench's 's task formulation and limitations (Section 2.1 and describe how SWE- bench M builds upon this work. We then discuss SWE-bench M's data collection heuristics (Sec- tion 2.2 Finally, thoroughly characterize SWE-bench M (Section 2.3 highlighting the novel
challen engge it poses to LM agent-based software development.
2.1 PRELIMINARIES
Formulations. SWE-bench (Jimenez et al. 2024a) has emerged as a popular LM agent benchmark. By drawing from real GitHub workflows, task instances reflect diverse, practical software challenges and require systems to ingest meaningful, long-fore form inputs. Repository-lever coding also involves meticulous refactoring of interwoven modules, each with dependency trees spanning multiple source files. SWE-bench's 's collection strategy yields Haman--rittee unit tests that ensure robust evaluation.
SWE-bench consists of 2,294 task instances derived from pull requests (PRs) collected across 12 open source Python repositories. Each task instance corresponds to a PR and one or more resolved issues; issue(s) describe bug or feature request, and the PR contains the corresponding solution code along with unit tests validating its correctness. The unit tests fail before the solution code is applicd to the codebase but pass afterwards, also referred to as fail-to-pas (F2P) tests. Additional auxiliary pass-to-pass (P2P) tests verify that the codebase's 's existing functionality is maintained.
A task worker is shown the codebase and the issue, also called the problem statement. The worker must then modify the codebase solve the problem. The proposed modification is run against both F2P and P2P tests to check (1) the issue is fixed and (2) prior working behavior is not broken. all unit tests pass, the task instance is considered resolved.
Limitations. Within the SWE-bench task formulation, several facets of software development re- main unexplored. This work primarily investigates two such facets. First, SWE-bench task instances are predominantly text only, and there are no discussions of the implications of the interplay between images and videos with software development. For the 5.6% of SWE-bench task instances with an image, it is unclear what these images portray and whether they are necessary to solving the task. To fill this gap, we focus on task instances with visual elements and demonstrate the significance of multi-modal reasoning in software engineering evaluations. Second, SWE-bench is comprised exclusively of Python repoxitories. We show that benchmark containing "WF--hench-style task in- stances from an alternative programming language, i.c., JavaScript, highlights previously unconsid- cred complexities (e.g., web development, asynchronous programming, DOM/state manipulation).
2.2 COLLECTION
JavaScript repositorics have a high concentration of visual assets due to its popularity for full stack web development and browser manipulation. We summarize modifications to SWE-bench's 's task collection pipeline to identify task instances with visual components. See Appendix B for details.
1. Find user-facing JavaSeript repositories. Using GitHub's 's search feature, we look for JavaScript repositories with 5,000 or more stars and 500 or more pull requests. We then manually pick 17 repositories, filtering for user-facing libraries that have visual aspect, such as mapping, plotting.
3

---

Published as a conference paper at ICLR 2025
Table 2: Median values different attributes of SWE-bench M task instance.
grommet (21)
carbon (134)
next (39)
scratch-gui (11)
Median
prism (38)
Issue Text
Length (Words)
105
prettier (13)
bpmn-js (54)
# Lines (non-test)
549K
highlight.js (39)
Codebase
quarto-cli (24)
# Files (non-test)
1799
cslint (11)
lighthouse (54)
openlayers (79)
# Lines edited
27
Gold Patch
# Files edited
2
Web Frameworks
Interactive Mapping
# Funcs edited
3
Syntax Highlighting
Publishing System
# Fail to Pass
1
Tests
Performance Profiling
Diagramming
# Pass to Pass
5
Aspect Ratio File Size (KB)
5:3
Figure 2: SWE-bench M Test Split Distribu- tion of 517 task instances from 12 open source GitHub repositories written mainly in JavaScript.
Images
42.95
Resolution (Pixels)
262K
or syntax highlighting. We scrape 135k PRs from these selected repositories. Every repository's source code is at least 70% JavaScript TypeScript code, with the remainder usually being HTML or Cascading Style Sheets (CSS). None the selected repositories have Python code.
2. Filter for issues that have visuals in their description or unit tests. We first look for PRs with one or more issues. We then filter these [issue(s), PR] pairs down to ones with visual assets in either the issue testing code. Specifically, we inspect the issue text and test patch for working hyperlinks that point to an image (e.g., jpg, png) or video (e.g., gif, mov). From 135k PRs, this new filtering criteria yields 1,478 candidate task instances.
Set up the environment for each repository. SWE-bench's Docker containers do not support JavaSeript out of the box. Therefore, first add foundational infrastructure, such Node.js and Chrome, to support JavaScript execution, visual testing, and webpage rendering in-browser. Next, for each repository, testing scripts. This process requires several rounds of manual trial and error not only to validate the setup procedure, but also overcome undocumented obstacles. Designing environments suited to run the majority of task instances took an average of ten hours of manual labor per repository. We are able install the repository and execute tests for 679 of the 1478 task instances.
read its public contribution guidelines and then write tailored installation and
4. Remove inconsistent tests. During manual inspection, we found that a small percentage of tests exhibited inconsistency, i.c., given the same patch, test's output (pass or fail) was inconsistent across multiple evaluation runs. This phenomenon was also reported in SWE-hench (Brown et al., 2024]. To eliminate such cases, we run validation per task instance 10 times and remove tests with inconsistent results, reducing 679 candidates to 643 viable instances.
Human validation. To remove impossible task instances and perform dataset characterization that is difficult to conduct automatically, the authors manually inspect each task instance. We iden- tify the kinds image(s) provided and assess how necessary image(s) are to solving task. We include the annotation procedure and queries in Appendix D Annotators removed 24 impossible tasks for a final total of 619 task instances. We discuss takeaways from annotations Section 2.3
2.3 FEATURES
SWE-bench is a dataset of 619 multimodal, JavaScript task instances collected from open- source GitHub repositories. It contains diverse set of libraries, including ones for data visual- ization, building diagrams, website UI components, displaying maps, and syntax highlighting. As shown in Figure the test split consists of 517 task instances from 12 repositories. The development split contains 102 task instances from repositories.
Diversity of images. Across all SWE-bench M task instances, there 862 "problem statement" images, meaning the image hyperlink is in the issue text. These images feature a multitude of visual processing challenges. Our annotation procedure grouped these images into seven distinct
4

---

Published as a conference paper at ICLR 2025
categories. Interpreting UI issues in website screenshots (401 images), such malformed front- end layouts or accessibility issues, involves spotting design flaws and assessing spatial relationships between elements. Grounding visual cues from code screenshots (194) and error messages (54) to specific codebase entities can provide hints for bug localization. Images of diagrams (107), art (38), maps (35), and data visualizations (28) depict repository-specifi challenges that address properly generating complex statistical, geographic, and creative content.
Videos, actual/expected reference images. SWE-bench M task instances often have multiple pic- tures (221 instances) or videos (70) to succinetly communicate helpful context and reproduction steps. For instance, gi fs illustrate issues for 43 instances from bpmn-js, diagramming toolkit. Unexpected behaviors are easily depicted by recording diagram interactions, drag-and-drop actions, and zooming in/out. instances from ca carbon, a UI design system, contain "actual" and "ex- pected" image pairs that highlight visual discrepancies and incorrect implementations.
Visual testing. A subset of 69 SWE-bench M task instances test submissions' functional correctness with pixel-level visual testing. This approach renders web pages and then compares screenshots pixel-by-pixel, identifying visual differences that functional unit tests cannot detect.
Necessity of images. To gauge the importance of problem statements' images, we pose two ques- tions to human annotators. First, does an image corvey more information than the text in the image? For instance, syntax-highe highhlhigg bug screenshot would show several lines of code, but the text col- oring is an important visual element. Annotators answer "Yes" to this question for 80% of problem statement images. Second, is it feasible to solve task instance without its associated image(s)? Of 557 instances with image(s), 83.5% were considered crucial for resolving the corresponding task. When provided, images are essential for diagnosing software bugs, and the information they convey is necessarily visual. See Appendices] D.2 and D.3 for details.
Difficulty curve. Human annotators estimated the time it would take for developer solve the task instances in the benchmark. This analysis shows that SWE-bench M features a range of difficulty levels, with tasks classified as follows: 13% take less than 15 minutes, 43% take 15 minutes to an hour, 38% take 1-4 hours, and 6% take more than 4 hours. Comparing these ratios to a similar study performed on SWE-bench (Chowdhury al. 2024) shows that SWE-bench tasks are longer on average overall. Reference solutions in SWE-bench M also edit more files, functions, and lines than in SWE-bench. See Appendices A.3 and D.4 for details.
3 EVALUATING ON SWE-BENCH M
We now diseuss the challenge of generalization for existing open-source solutions for SWE-beech and describe how we have adapted these methods for evaluation on SWE-bench M (Section 3.1 Additionally, we provide details about our the setup of our experiments (Section
3.1 Do EXISTING SYSTEMS GENERALIZE?
We attempted to run existing open-source systems that perform well on SWE-bench (Jimenez al., 2024b) on SWE-bench M. However, when adapting these systems to SWE-bench M. we found that several solutions were so heavily tailored to Python and SWE-bench to the point that they were unusable for JavaScript repositories and SWE-bench M evaluation.
Our observations from exploring a number of existing solutions led us to identify generalizabillty as desirable but overlooked property of automated software engineering systems. Specifically, do existing agent systems perform well on bugs that are not similar to the ones found in SWE--benh (i.c.. Pothon or issues that include images)?
Though current LMs perform well in multiple programming languages (Cassano et al., 2022). a rigidly defined system can force LMs to follow a particular problem solving pipeline that restricts its capabilities. Although such workflow-oriente repository or even programming language, they can fail when confronted with minor distribution shifts outside their original design parameters. Simply put, the shift from Python to JavaScript or the addition of the image modality exceeds the abilities of many existing approaches.
systems may work well for a specific type of
5

---

Published as a conference paper at ICLR 2025
We attempt to adapt the top four open- systems the SWE-bench leaderboard (Jmenez et al. 2024b) to work on SWE-bench M tasks. For each system, we discuss below whether adaptations were feasible based on the preceding questions and the changes we made, if applicable.
SWE-agent YYng et al., 2024) is a lightweight framework connecting an LM to an operating sys- tem and shell process. It integrates text-based agent-computer interface (ACT) for LMs to edit files in addition to the ability to execute shell commands. The primary modifications necessary for SWE-bench was adapting the message processing and history functionality to work with images for multimodal LMs. To explore the effects of further adaptation, we evaluate three ACI config- urations for SWE-agent: (1) SWE-agent Base, the original SWE-agent ACI, initially designed for evaluation on Python and SWE-bench tasks; (2) SWE-agent JS. which converts SWE-ageet Base's 's editing interface to detect JavaScript edit errors, mirroring the linting feature used by SWE-agent for Python files; and (3) SWE-agent M, which extends the SWE-agent JS ACI to provide simple web browser, screenshot, and image viewing capabilities, allowing models to visually reproduce image-based issues and verify changes generated images.
Agentless (Xia et al. 2024) proposes a two stage localize-then-repair pipeline. A repository is first pre-processed with the Python ast module into a simplified overview of the repository's ss structure. Based on this syntax tree, the LM can then view files' content and select which files to fix. In the repair stage, based the identified files, the LM then generates multiple candidate patches. The most commonly repeated solution among the candidates is its final submission.
The main adaptation needed is swapping out the Python ast module with custom JavaScript parser written from scratch. Based on feedback from one of the authors, we also update Python- centric prompts with more JavaScript-oriented instructions. Without these changes, Agentless gets 0% resolved on SWE-bench M's development split. We denote our adapted version Agentless JS.
AutoCodeRover (Zhang al. 2024b) employs two-phase approach similar Agentless. The first phase involves a customized retrieval process where an LM performs multiple iterative searches. This is followed by a patch generation and testing phase, where an LM, provided with the search history and results, generates and refines a code patch using results from executing tests.
We found that most tools AutoCodeRover provides to the agent rely heavily on Python- specific program analysis features and even prior knowledge of the particular repositories included in SWE- bench. We ultimately chose not benchmark AutoCodeRover since its specialized tools would require extensive redesign for SWE-bench M, likely resulting in fundamentally different system.
Moatless (Örwall, 2024) subscribes to the pecalize-then-repale workflow, but does not use an LM for localization. Instead, code files are first converted in abstract syntax trees (AST). These ASTs are then aggregated into a single, searchable code graph represented as a Faiss index (Johnson et al., 2019). The LM then queries this index to generate a repair.
We worked off the author's in-progress implementation for generating ASTs from JavaScript and TypeScript files"] Although the code graph representation is language agnostic, the input AST repre- sentation nceded to generate the Faiss index is heavily based on Python's 's object oriented design. As reflected by the magnitude of the pull request's changes, writing equivalent parser for JavaScript that (1) reflects the code and (2) subscribes to the index's 's input format is non-trivial due to usage of functional and declarative programs being more commonplace. We do not benchmark Moatless.
RAG. Jimenez et al. (2024a) proposed a retrieval augmented baseline using BM25 (Robertson & Zaragoza 2009) from which we inherit the document formatting, retrieval method, and prompt structure. We adapt the prompt template used in Jimenez et al.) (2024a) to use a JavaScript example for patch generation instead of Python. We also include new section in the prompt for inserting reproduction code collected from links in the problem statement, as described in B.3
3.2 EXPERIMENT SETUP
Models. Resolving SWE-bench M issues with existing systems requires handling very long con- texts, processing images and text simultancously, and producing sophisticated structured outputs. Given these constraints, we focus all of our evaluations on GPT-4o cgpt-4c-2024-08 006-6) (Ope-)
nAl 2024) and Claude 3.5 Sonnet (claude 3 wcnnenttt------enn--nn 202400200 (Anthropic) 2024). the two

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison of various baselines on SWE-bench M. The table shows re- sults for different software development agent frameworks, including SWE-agent (with multimodal and JavaScript-specife variations) and retrieval augmented generation (RAG) approach. Each sys- tem's s success rate (% Resolved) and average cost ($ Avg. Cost) per task are reported.
System
Model
% Resolved
$ Avg. Cost
SWE-agent M
GPT-4o
12.2
2.94
Claude 3.5 Sonnet
11.4
3.11
SWE-agent JS
GPT-4o
9.2
0.99
Claude 3.5 Sonnet
12.0
3.11
SWE-agent Base
GPT-4o
12.0
2.07
Claude 3.5 Sonnet
12,2
1.52
Agentless JS
GPT-4o
3.1
0.38
Claude 3.5 Sonnet
6.2
0.42
RAG
GPT-4o
6.0
0.17
Claude 3.5 Sonnet
5.0
0.15
most well-supported mulitmodal LMs for long-context RAG and agent systems. Though alternative models continue to improve, they currently lack the combination of long- long-context handling, mul- timodal processing, and structured prediction abilities required to work with existing autonomous software engineering systems.
Baselines. Based on Section 3.1 findings, we benchmark the performance of five systems: Retrieval Augmented Generation (RAG), SWE-agent (Base, JS, M), and Agentless JS. Each of our adapted and developed using the development split of SWE-hench M. We describe more about how the final test configurations, including hyperparameters, were selected in Appendices C.I and C.2
baseline systems
Metries. We report two evaluation metries: (1) % Resolved, the main performance metric, which represents the proportion of successfully resolved task instances and (2) Avg. $ Cost, the mean per-instance inference cost incurred from running the baseline, as used in Yang et al. 2024).
4 RESULTS
We compare the performance of each baseline system in Table While overall performance on SWE-bench M is relatively low, observe substantial performance gap between the interactive SWF-agent systems (11.5 % resolved on average) and the Agentless and RAG baselines, which achieve 3.9 and 5.5 % resolved average respectively.
Across all SWE-agent configurations, we observe similar absolute performance, suggesting that the JavaScript-speccfic customizations in SWE-agent JS and SWE-agent M had minimal impact on performance. Additionally, we also observe minimal performance differences when swapping the underlying LM (GPT-4o or Claude 3.5 Sonnet) within a system. However, study of SWE- agent ablations on the development set reveals that the added multimodal tooling can improve agent performance in some cases, though the overall picture remains ambiguous.
An analysis of solution rates segmented by the date each task instance was originally solved, is performed in Appendix C.3 It finds no evidence for a test set advantage from solutions leaking into the LM training sets: we show for example, that SWE-agent M using GPT-4o performs better on task instances based on issues that were resolved after knowledge cutoff date.
4.1 ANALYSIS
We present insights into how images influence system performance, evaluate the impact of multi- modal actions in SWE-agent M, and explore the generalization of different approaches to automated software engineering. All analyses are conducted the development set of SWE-bench M.
Resolving SWE-bench M issues requires improved visual understanding. Table 4 compares the performance of the RAG and SWE-agent JS systems, with and without images. Thee results
7

---

Published as a conference paper at ICLR 2025
reveal significant performance decrease when models lack access visual information. Table analyzes performance based on the two key questions from our human validation: "Do the images contain more information than just text?" and "Are the images necessary to solve this task?" (For detail, see Appendices [.2 and D33 respectively.) As expected, system performance generally improves when images provided with the problem statement. In addition, for instances where annotators deemed images necessary to solve the task, performance plummets without visual input. Performance changes vary based image content. When images contain primarily text, SWE- agent JS maintains 23.1% % performance with or without images. Larger performance drops occur when images contain non-textual elements. For non-textual visuals, performance drops from 13.0% to 8.7% without images for SWE-agent JS. Visual elements, especially those that are non-textual, appear to provide additional context that substantially improves system performance.
Table 4: Performance of RAG and SWE-agent with (W/) and without (Wo) Images. The consistent improvement across both models and systems emphasizes the critical role of multimodal inputs, particularly for complex tasks requiring visual reasoning.
System
Model
W/ Images
Wo Images
GPT-40
11.0
8.0
SWE-agent JS
Claude Sonnet 3.5
16.0
13.0
GPT-4o
10.0
8.0
RAG
Claude Sonnet 3.5
14.1
11.2
Table 5: System performance segmented by annotator response. We show performance for SWE-agent IS and RAG using Claude 3.5 Sonnet broken down by response to human validation questions detailed in Appendix [.2] and D.3 Each performance value corresponds to the perfor- mance of the configuration (e.g. SWE-agent JS with images) on the subset of instances correspond- ing to the annotation class for each question. The questions in the tables have been simplified from the original for readability.
Does the image contain more information than just text?
Are the images necessary to solve this task?
System
With Images?
Yes
No
System
With Images?
Yes
No
13.0
23.1
17.6
11.1
SWE-agent JS
SWE-agent JS
X
8.7
23.1
X
8.8
22.2
11.6
13.8
12.4
11.9
RAG
RAG
X
6.1
16.2
X
9.1
8.1
3.1 except for SWE-agent, the systems that study (Agentless, Moatless, and AutoCodeRover) impose fixed, procedural problem-solving workflows. Every system starts with bug localization step that relies on abstract syntax tree (AST) parsing libraries to identify programmatic symbols.
Localization modules are overly engineered for Python. As discussed in Section
These Python-specific modules do not work for other languages. Overcoming this design is contin- gent upon either finding or creating a similarly reliable tool. For Moatless and AutoCodeRover, no such replacement tool exists; reimplementation was prohibitively laborious and would have resulted in a system with little resemblance to the original. For Agentless, replacing invocations of the Python ast package with the JavaScript t ree-ss s tter library still yielded 0% resolve rate on the de- velopment split; this is due Python-centric assumptions the localization module makes about the AST representations. Therefore, wrote custom JavaScript parser from scratch compatible with the Agentless logic, which took 15 hours of labor.
However, because the original design of Agentless is influenced by Python, fundamental differences between Python and JavaScript's design patterns (e.g., multiple programming styles, object proto- types, arrow functions) lead to many failure modes during localization, resulting in Agentless JS's low performance (4.6% Table For instance, for grommet grommet 67 49, the localization module, designed based on Python's imperative and object-oriented style, does not recognize Tab,
8

---

though some of its successes from Published as a conference paper at ICLR 2025
Successful submission
Terminated due cost/crror
Successful despite early teminaation duc to cost
False submission
(GPT-40)
10%
3
46%
41 %
(GPT-40)
11%
15%
74%
JS- no imgs (GPT-40)
9%
15%
76%
M (Claude)
8%
7%
32%
54%
IS (Claude)
16%
1
10%
73%
JS imgs (Claude)
12%
14%
71%
Figure 3: SWE-agent performance across configurations. The frequency outcomes by success and exit status of SWE-agent under different configurations on the development set. Configurations include multimodal (M), JavaSeript-specifie (JS), and no images, for both GPT-4o and Claude 3.5
Sonnet models. A similar analysis by repository is shown in Figure
the buggy component, because it was defined declaratively as an arrow function. In the subsequent repair phase, Agentless JS writes a completely new Tab imperatively, while the reference solution makes a minor change to its existing declarative implementation. We also found that given the raw code, GPT-4o correctly identified Tab as the buggy component.
We calculate the F1 Score for file localization on the development split with Claude Sonnet 3.5 as the base LM. Agentless JS achieves a score of 0.142, while SWE-agent gets 0.367. Flaws with language-specific design are only amplified in SWE-bench M. Unlike SWE-bench solutions, which edit only Python files, 28% of reference solutions SWE-bench M edit multiple files types (e.g., TypeScript, HTML, CSS, Lua). Adapting current solutions requires writing unique AST generator and navigator functions to pre-process code files for many languages.
From these observations, hypothesize that generalizable LM-based systems for software engi- neering will emphasize interaction, nol problem solving. Instead of inserting LMs at specific points of a workflow, our results show the superiority of an LM-first approach that focuses on building tools that enhance an LM's ability to navigate and manipulate an environment. SWE-agent's 's supe- rior performance and Agentless's localization struggles suggest that the burden problem solving should remain on the LM instead of being offloaded to manually engineered pipelines.
Multimodal tools increase task complexity but improve agent performance. To evaluate the impact of the presence of images on agent behavior, we conduct an analysis of the SWE-agent trajectories generated while resolving the development
We run SWE-agent in three different configurations: SWE-agent M. SWE-agent JS, and SWE-agent JS without image inputs. The web- specific tools introduced in SWE-agent M lead to approximately 20% of actions being dedicated to building and taking screenshots of websites, though this proportion varies significantly across repositories (see aigures|oland7 7 SWE-agent M with GPT-4o builds websites and takes screenshots for 38.3% of the instances. For these instances, average of 7.5 screenshots are taken, showing that the multimodal tooling is used as part of an iterative problem-solving process for some instances.
However, reproducing issues and verifying solutions by building and taking screenshots of websites makes the task for SWE-agent M more complex than directly performing and submitting edits as in SWE-agent JS. This reflected in almost threefold increase of attempts that terminated because they exceed their cost limit (see Figure 3). The web-specific tools lead to degraded perfor- mance with Claude 3.5 Sonnet, while GPT-4o improves, runs that are terminated prematurely due to costs.
Another key figure for agent performance is the fraction of correct submissions, as incorrect submis- sions increase manual reviewing workload. To increase this fraction, submissions from SWE-agent runs that were terminated prematurely due cost be excluded, they have lower probability

---

Published as a conference paper at ICLR 2025
to be correct. In this setup, adding multimodal tools almost doubles the fraction of correct submis- sions with GPT-4o to 19.6% compared to 10.4% without image inputs. This result is not observed in the Claude 3.5 Sonnet-based SWE-agent.
Together, these findings show that LM agents can use web-based multimodal tools to reproduce issues and verify solutions, in some cases increasing success rates solution correctness. However, using these tools to their full potential requires complex workflows that are still challenging for agents and result in an ambiguous record overall.
RELATED WORK
Multimodal Code Benchmarks. Code generation, the task of synthesizing code from natural lan-
guage, has served as long-standing measure of LM performance (Austin et al. 2021 Chen et al.)
2021 Hendrycks al. 2021). As performance on these benchmarks has plateaued Claude Sonnet 3.5 achieves 92% on
Subuesuuuuuuuuuunee works have extended this task along different di-
mensions to increase difficulty, such as more robust evaluation (Liu et al. 2023a Dain et al. 2024),
multilinguality (Cassano et al.] [0022 Wang et al.] [2233 Zheng et al. 2024), open-domain gener-
ation (Wang et al., 2023b] Zhuo et al., 2024). data science (Lai al. 2022, Yin et al. 2022 Cao
et al.
2024), understanding execution traces (Gu et al., 2024: Muennighoff et al. (0224) repository
comprehension (Liu et al. 2023b Zhang et al.] 2023b). cybersecurity (Yang et al. 2023b Zhang
et al.
2024a; Shao et al. 2024|] Abramovich et al. 2024). and efficiency (Huang al. [0224] Liu
ct al.
2024] Waghjale et al., [2024]. SWE-bench M represents a merger of two promising directions:
software engineering (Jimenez et al. 2024a) and multimodal code generation (Li et al. 2024 Si
jet al. 2024 Wu et al. 2024] Nishina & Matsui 2024). By drawing on real-world problems from GitHub, SWE-bench M overcomes the synthetic and short-form limitations inherent in code gener- ation task formulation. In addition, SWE-bench M represents a significant multimodal upgrade to SWE-bench, which has already become a popular benchmark for LM evaluation.
LM Agents for Web and Code. Prior to the proliferation of LMs, earlier works explored translating between user interfaces and front-end code (HIML/CSS//avaSripp (Beltramelli, 2018 Robinson 2019). More recently, LM agents (Yao et al.] 2023b; Sumers et al., (024) have been increasingly
employed and evaluated on various tasks (Wang et al.
2024a] Xie al. 2024). Two popular use
cases, web or application navigation (Hong et al. 2023 (Soselia et al.
2023; Yan et al.
2023 Yao
(ct al.
2023a| Zhang et al. 2023aa Koh et al. 2024 Putta et al. 2024 Rawles et al. 2024 Press
jet al.
2024| Yoran et al. 20224) and software engineering (Yang al.
2023a 2024 Zhang et al.
2024b Xia et al.) |llaal222222 2024), have typically been studied separately. To the best of our knowledge, SWE-
bench M is the first benchmark that meaningfully couples these two tasks. Though prior approaches have provided web browsing tools to programming agents (Wang et al.) 2024b), such tools usually represent webpages in text form and were not designed with a clear downstream objective. SWE- bench M demonstrates how agents can iterate meaningfully between code updates and their effects rendered as images or browser.
CONCLUSION
This work introduces SWE-bench Multimodal (SWE-bench M), the first benchmark evaluate coding agents on real-world software engineering tasks involving visual elements. SWE-bench M contains 619 task instances from 17 user-facing JavaScript repositories, including ones for web user interface design, data visualization, art and mapping. Our analysis reveals that SWE-bench M contains diverse visual challenges and increases task complexity compared to SWE-bench. Fur- thermore, existing systems perform poorly on it, with the top resolve rate reaching only 12.2%; SWE-bench M presents repository-level programming tasks that are challenging even for state-of- the-art systems built top of the strongest LMs. Incorporating multimodality in SWE-bench M not only expands the coverage of exciting, practical challenges in software engineering, but also encourages practitioners to develop more general- purpose, aunguage-agnostic solutions that do not overfit to SWE-bend' or Python repositories.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank the authors of the Agentless and Moatless systems, Steven Xia and Albert Órwall, for responding to our questions and verifying implementation details during our attempts modify these approaches to work on SWE-bench M.
We thank Open Philanthropy, Oracle and the National Science Foundation (Grant No. 22393633 for providing funding for this work. Any opinions, findings, conclusions, or recommendations ex- pressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Ori Press thanks the International Max Planck Research School for Intelligent Systems (IMPRS- IS) for its support.
All experiments were conducted by the authors at Stanford University and Princeton University on Princeton University's servers. Meta affiliated authors acted an advisory role.
REFERENCES
Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Enigma: Enhanced interactive generative model agent for etf challenges, 2024. URL https:t/axx::ii
org/abs/24D0 2409. 16165
Anthropic. Introducing claude 3.5 sonnet, 2024. URL http:://ww..aaatrpp...c... /cl /cllaude3355 --o-n-----d-
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: www.aaavvvv.oo org/ abs /22///1183 2108. 07732
Kaustubh Bagalkote. Amazon ceo andy jassy says company's ai assistant has saved $260m and 4.5k developer-yeaar of work, 2024. URL
http://iiianneeaaoooooooooo wtw.:yawhoo...../.oooooooo ama zon says- 213018283 htmmm2833mmll
20
Tony Beltramelli. pix2code: Generating code from graphical user interface screenshot. In Pro- ceedings of the ACM SIGCHI symposium on engineering interactive computing systems, pp. 1-6, 2018.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le. Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. httpttps: www.aaxiivvv....... org / abw/abaaaa abs/2407. 21787
Ruisheng Cao. Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang. Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Vic- tor Zhong, Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? CoRR, abs/2407.10056, 2024. URL httts:://arxat http///wwwr./////......22 /ttp:/240..11...
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e A scalable and extensible approach to bench- marking neural code generation, 2022. URL
https: httpp/////aaxxtvoo/////////// /200 08227
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis. Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec
11

---

Published as a conference paper at ICLR 2025
Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. httph:httt/a/ataaaaa http://wwwwwgga///a//aaaaaaaa/ aaaaaabss21100 03374
Mauro Cherubini, Gina Venolia, Rob DeLine, and Amy Ko. Let's go the whiteboard: how and why software developers use drawings. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 557-566, 2007.
Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing swe-bench verified, 2024. URL htwp::ttp///tt /otttps://opeai... com/1
Kyle Daigle and GitHub. Octoverse: The state of open source and rise of ai in 2023, 2023. URL httppp//gg.hhho...booeeeesssssss
rce ice-and-aii wtw.//thee-tte http://fhwhhetooooooooopoooaaaaa
Langguuugnnnng--ggggggeeeeeeg languages
James D Foley and Andries Van Dam. Fundamentals of interactive computer graphics. Addison-
Wesley Longman Publishing Co., Inc., 1982.
Paul Gauthier. Aider, 2024. URL httpa///a//////taaa.a.ooaaaaaaaa
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024. URL
https://arxiv http:///axxvv.r/////// 201p:///.wrg//a///.11...111 03065
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021. URL http:: /ttp://aaxvv.v//.m///i//.......a org/ abs / 2105 09938
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023. URL https:: /arxivv.vv.v.////aa/ wttp:bs////........//// 089
Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie Zhang. Effibench: Benchmarking the efficiency of automatically generated code, 2024. URL https::/axxiv.rr/////// 02. 02037
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL
.07974
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024a. URL
lattps://arxiv. httpaaiiiv..og/..oaa 2230. 2310. 06770
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R.
Narasimhan. Swe-bench leaderboard, 2024b. URL www. swebench. com/
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions on Big Data, 7(3):535 547, 2019.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. ACL, 2024.
Yuriy Kotsarenko. Measuring perceived color difference using yiq ntsc transmission color space in mobile applications. Programacion Matematica y Software, 2018.
Yuhang Lai. Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 10-1000: A natural and reliable benchmark for data science code generation, 2022.
221111110111111111111111 11501
12

---

Published as a conference paper at ICLR 2025
Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. Mmcode: Evaluating multi-moda code large language models with visually rich programming problems, 2024. URL https:
//
09486
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by
chatgpt really conrect? rigorous evaluation of large language models for code generation, 2023a.
URL htttps:://axxi..rrg//p. ab33005. 01210
Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluat- ing language models for efficient code generation, 2024. URL htt.tttss///axiiii wwwpiivvvrrg/b/sooo//oooo 2408. 06450
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023b. URL htt.::://arcwi..rr///aaas
03091
Daniel Moody. The "physics" of notations: toward scientific basis for constructing visual notations in software engineering. IEEE Transactions on software engineering, 35(6):756-779, 2009.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models, 2024. URL ULL/sttpss//arrat...rrr/////////
Kunato Nishina and Yusuke Matsui. Svgeditbench: benchmark for outere/eeeeeeeeeeomnneaooooononn of I1l's 's svg editing capabilities, 2024. URL htt.L//ttgg////rrr...//////000.0
OpenAl. Hello gpt-4o, 2024. URL http:::tpppppooiieee /ttp/pppp////paaaaaacc//////////
Albert Örwall. Moatless tools, 2024. URL
ools
Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. Citeme: Can language models accurately cite scientific claims?, 2024. URL https:
7/ arxiv. wtt.:o//bbbb/....22..2
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024.
URL httph:ttts/s/tt http://wwwww......aaa..ooooooo //t.////.... 07199
Christopher Rawles, Sarah Clinckemailiie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Camphell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworid: A dynamic benchmark- ing environment for autonomous agents, 2024.
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and be- yond. Found. Trends Inf. Retr. 3:333-389, 2009. URL httpps://ppp, semanticschon r. F.ott//orrussD:22011777
Alex Robinson. Sketch2code: Generating a website from a paper mockup, 2019. URL https: //azcii.orr//ab//19966..33
Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan DavintGavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin. Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu etf dataset: A scalable open-source benchmark dataset for evaluating Ilms in offensive security, 2024. URL https: /ttl:://axxvva wttp://.rggbbbbbb/bb/eeee hbtp://b.s/2422... 05590
Ben Shneiderman. Designing the User Interface: Strategies for Effective Huwan- Inter- action. Pearson, 1998. URL https:://////. /ttp::///pppiieee www.tttccchoolo.ooooooooo.
54156244
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang. Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering?, 2024, URL http://:://aaxxx.////a//aa
03. 631
13

---

Published as a conference paper at ICLR 2025
Davit Soselia, Khalid Saifullah, and Tianyi Zhou. Leamingg code reverss re critic without rendering, 2023. URL /tt.://e//aattoooor////////////0
generator using visual
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive archi- 02427
tectures for language agents, 2024. URL httppp:://axxxt..orrraaa////////
Siddhant Waghjale, Vishruth Veerendranath, Zora Zhiruo Wang. and Daniel Fried. Ecco: Can we improve goderegeeertted code efficiency without sacrificing functional correctness?, 2024. URL
https:://rxiiii wttpaaawwvv...ggaaoo... wwg./bss///oooooooooo/ooooooooo/ 2407. 14044
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang. Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024a. ISSN 2095-2236. doi: 10.1007/s1 170-0/11770---44442231111-- 07/s11704-024-40222
URL httpht doi. org/10.10
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Opendevin: An open platform for software developers generalist agents, 2024b. URL wwwp::ttp///ccm /arxiv org/ abs 2407 6741
Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank Xu, and Graham Neubig. Mconala: bench- mark for code generation from multiple natural languages, 2023a. URL https::/axxiv.o. g aaabb/2003 08388
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation, 2023b. URL http:/ttt///rrstt. http://aaaw...ooo/////////////// 10481]
Erik Wittern, Philippe Suter, and Shriram Rajagopalan. look the dynamics of the javascript package ecosystem. In Proceedings of the 13th International Conference on Mining Software Repositories, MSR 11, pp. 351-361, New York, NY, USA, 2016. Association for Computing Machinery, ISBN 978145034 868. 1868. doi: 10. 10.114572901739929017777 2901773399999..... URL http:::/ddoo
g/10. 1/10.1145/2901/2 2901743
Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots, 2024. URL http::://///// /ttes////xxiiii wttpaaawww...aaa///////aaa 05. 07990
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying Ilm- based software engineering agents, 2024. httpst /ttp///xxvvv///// http/////b///4.cccc 01489
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL hattps:/aaxxv.co
ab
8//244.07972
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023. URL https::/
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing
and benchmarking interactive coding with execution feedback, 2023a. URL httph:tt///aaaa
v. w.orggabs/23056 220.//13..6.88888.) 14898
John Yang. Akshara Prabhakar, Shunyu Yao, Kexin Pei, and Karthik R Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop@ NeWr 2023b.
14

---

Published as a conference paper at ICLR 2025
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL htt.t/////aw.a..o..o.////////...
org/
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents, 2023a. URL https://axxxx..orrr
abs/2207. 01206
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https://arxiv:
org/abs/2210.00 03629
Erin Yepis and StackOverflow. Developers get by with little help from ai: Stack overflow knows code assistant pulse survey results, 2024. URL htthhttpss//staceeeeellle bltp:///w/wo///2///////// 05 05/29/developers get by by--by-www. liitleeeee frrp-irro ai sat-sstackkee Sttack-ore overf
low kww.knowwwccoooooo code -es---ssistatn suressssttttttssss-------------e survey- results/
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sut- ton. Natural language to code generation in interactive data science notebooks, 2022. URL
https::/arrxt org/abs /ttpa////21..222 09248
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL
https://arxiv. wttpaaawwwwoooaaaaaaaaaaaaaaaaaa 2407.15711
Andy K. Zhang, Neil Perry, Riya Dulepet, Eliot Jones, Justin W. Lin, Joey Ji, Celeste Menders, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjís, Kenny Osele, Gautham Raghu- pathi, Dan Boneh, Daniel E. Ho, and Percy Liang. Cybench: framework for evaluating cyber- security capabilities and risk of language models, 2024a. URL htl.:ttss:// //ttt::////xiii org./bs/ 2408 140880083 08926
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users, 2023a. URL http:/tttt/:xxi..ttt.. s 1/2312.1377
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation, 2023b. /ttp:/ttas///axxi...o////bbbbb//
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury AutoooooddnuuyyAAuuouuueoeeooe Autonomous program improvement, 2024b.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang. Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x, 2024. URL https://aaxi
w.org/abs//300 17568
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. StR.:ttts:://// wwwpaawwwww.ooo....ooaaaaaaoaaa
15877
15

---

Published as a conference paper at ICLR 2025
APPENDIX
In the appendix, we provide additional details about the collection process for curating SWE-bench M, more characterizations of the SWE-bench M dataset, supplementary experiments and ablations of LM agent performance on SWE-bench M, limitations and more.
A DATASET
A.1 DEVELOPMENT SPLIT
As shown in Figure the SWE-bench M development split consists of 100 tasks instances from open source repositories largely written the JavaScript family of programming languages (e.g- js, jsx, ss, sx). Repositories were chosen for the development split based on two criteria. First, the number of task instances about one-fifth the size of the test split. Second, each repository has "equivalent" test repository, meaning for each repository in the development set, there is repository with similar purpose in the test set. This parallelism is meant to make solutions crafted on the development more transferable the test set.
We have observed that, in practice, practitioners developing solutions to the SWE-bench task have often iterated task instances from the test split of the dataset. For those interested in working on SWE-bench M, we encourage practitioners to follow the preferable practice of iterating on the development split, and then only evaluate on the test split once the approach is finalized.
Table 6: Median values different attributes of SWE-bench M task instance.
Chart.js (24)
marked (14)
Median
Web Frameworks
p5.js (16)
Syntax Highlighting
Issue Text
Length (Words)
145
Creative Coding
Lines (non-test)
228K
Codebase
Files (non-test)
1324
react-pdf (11)
wp-calypso (37)
Lines edited
35
Gold Patch
Files edited
Distribution of SWE-bench M devel- opment set tasks (in parenthesis) across 5 open source GitHub repositories.
Figure 4:
Funcs edited
3
Fail Pass
2
Tests
Pass to Pass
7
As reflected in Table compared to Table a task instance from the development split has slightly longer issue text, but also a smaller codebase in terms of number of lines and files. The median number of files (2) and functions (3) edited identical. Development split task instances change slightly more lines, and have generally have somewhat more Fail-to-Pass and Pass-to-Pan tests.
A.2 ADDITIONAL CHARACTERIZATIONS
We provide further characterizations about the SWE-bench M dataset. In this section, we include both extended discussions of filtering criteria and features discussed in Sections 2.2 and 2.3 as well additional, noteworthy details not mentioned in the main paper.
Visual Testing, A total of 69 task instances from Chart. js, a data visualization framework, and openlayers, interactive mapping library, verify functional correctness with pixel-level visual testing. Across this subset of instances, there are 273 reference screenshots for visual testing.
Within the JavaScript development ecosystem, visual testing like this is commonplace, with Pup- peteer and Pixelmatch being two popular libraries. Such libraries are necessary because they provide several sensible advantages simple binary comparison of two images to check if
16

---

Published as a conference paper at ICLR 2025
they'r identical. Generally, the overarching reason is that direct diff is too strict and doesn't t account for the discrepancies of how different browser render user interfaces and webpages.
There several additional benefits. First, such libraries provide the ability to specify tolerances for minor pixel level differences that might arise from anti-aliasing or sub- uub-pixel rendering (Kotsarenko) 2018). Second, visual testing libraries allow for targeting certain components of webpage. Specif- ically, Puppeteer has tools for taking screenshots of specified sub-parts of a page, and Pixelmatch allows for region-specific comparisons of images. This use is particularly important for selec- tively ignoring dynamic website content, such as timestamps or ads, that irrelevant to the code.
Task instances by year. In the following Table 7 we show the number of task instances for each repository across different years. Task instances are distributed fairly uniformly, with the majority of tasks being from 2019 to 2022. Some repositories have higher distribution of older task instances, such as lighiight ghthouse and kp-calypas
Table Number task instances per year for each repository represented SWE-bench M.
Repository 2018
2019
2020
2021
2022
2023
2024
bpmn-js
4
carbon
36
30
30
24
12
2
eslint
0
3
0
0
grommet
1
0
highlight.js
14
21
4
0
0
lighthouse
25
12
3
0
next
11
10
4
0
2
openlayers
12
16
14
8
prettier
3
1
1
prism
14
quarto-cli
0
0
8
16
0
O
scratch-gui
0
Chart.js
17
3
0
marked
1
p5.js
1
1
6
3
0
react-pdf
8
wp-calypso
26
11
0
0
0
0
0
Total
83
114
100
130
112
67
13
Asset collection. For each task instance, we download all image, video, and reproduction code assets referenced by hyperlinks in the issue. This step, not done in SWE-bench, ensures that problem statements are reproducible in the future even if these hyperlinks expire.
Multilingual problem statements. 55 task instances have languages other than English in the prob- lem statement text or images, with Mandarin Chinese being the second most frequently occurring language (38 task instances). The next repository, design component library based React, contributes the most of these instances, with 26 task instances. This is in large part due to the fact that the repository's owner is the Alibaba Design team. Although not explicitly emphasized in SWE- bench M's collection process, we find that collecting from repositories owned by maintainers whose primary language is not English is a fruitful source of multilingual SWE-bench style task instances.
References solutions edit multiple file types. In SWE-bench, all reference solutions edit exclu- sively Python files. In SWE-bench M. 174 task instances have references solutions that modify two or more file types. We did not consider non-code files such as text, Markdown, or images files when calculating the counts. We list the frequencies for the different sets of file types edited by reference solutions in Table g
A.3 ADDITIONAL ANALYSES
We provide further details and breakdowns the SWE-bench M dataset, such as repository specific statistics and additional information from seraping task instance information GitHub.
17

---

Count Published as a conference paper at ICLR 2025
Table 8: Counts for the number of task instances with reference solutions that edit a specific set of file types. 28% of SWE-bench M task instances modify two or more files types.
File Types Edited
Count
File Types Edited
js
400
js, scss, jsx lua
8
js, scss
8
jsx
19
js, hbs, scss html, ja, css
6
ts, js
4
js, jsx
16
scss, jax
4
html, is
14
js, lock
3
10
js, css
3
Repository Statistics. Following Tables 2 and 6 show the same median statistics for each repository in Table 9 The discrepancies in the issue text length, codebase size, and gold patch changes can be directly attributed to the codebases themselves relative to the original online sources for the code and issue text, the SWE-bench M collection process does not affect these values at the number of tests per repository, particularly Pass-
all. On the other hand, the differences to-Pass tests, is outcome of how the testing specifications and test log parsing manually specified. Two factors lead to these differences. First, for some repositories, the entire testing suite run (e.g., highlight specific test files, usually derived from file paths modified by the test patch, are run (e.g., carbon, open openlayerre ayers, wp-cal ypso). Consequently, for the latter of repositories, the number of Pass- to-Pass tests is usually few to none. Second, the granularity of test logging is also different across codebases. The primary distinction that while some logs show Pass/Fail on per-file basis, other test logs will show Pass/Fail on per-test case basis where one files has multiple test cases.
js. p5. is, quarto-cli) while for the majority of repositories, only
Table 9: Median values for metries about each task instance, mirroring the statistics shown in Tables and 6 Repositories from the test set are listed above the separator, while development set repositories are listed below. "Fail to Pass" abbreviated as "F2P." "Pass to Pass" is "P2P."
Issue Text
Codebase
Gold Patch
Tests
Repository
Length
Lines
Files
Lines
Files
Funcs
F2P
P2P
GoogleChrome//ighhhouue
94
1076K
834
29
2
3
1
1
PrismJS/prism
79
184K
2,721
12
2
2
1
9
Sababb-fusion/nng
14
200K
1,731
57
5
5
3
25
bpmn-io/bpmn,
99K
593
15
1
2
1
0
aubhn-ddsign-sytemmrrrom
116
822K
4,250
45
3
4
1
33
eslint/eslint
324
398K
1,567
23
2
2
0
grommet/grommet
105
612K
1,186
8
1
1
1
77
highliggtjs/hiihhlghtijj
139
105K
1,476
16
2
2
2
1,641
openlayers/openlayers
128
544K
1,761
32
2
3
1
0
prettier/prettier
105
348K
6,855
48
2
5
1
0
quarto-dev/quarto-do
202
1399K
3,639
12
2
2
1
434
148
100K
514
562
12
9
1
0
Automattic/wppcalyyss
121
562K
7,990
76
3
5
2
11
chartjs/Chart.js
148
154K
1,376
2
0
diegomura/react-pdf
88
188K
806
39
2
2
1
210
markedjs/marked
38K
283
11
0
processing/p5.j
254
436K
1,009
30
2
4
12
2,373
As shown in Figure we include plots of the cumulative distribution functions for the statistics presented in Table across all task instances from both the development and test splits. We also do the same calculations for the SWE-bench dataset (2519 task instances total) and overlay the CDFs for each statistic. The frequencies normalized to range of zero one.
18

---

Published as a conference paper at ICLR 2025
Problem Statement Words
Files Edited (Gold)
Functions Edited (Gold)
Lines Edited (Gold)
0.29
SWE beacs NM
2
16
20
100
200
300
Fail to Pass Tests
Pass to Pass Tests
Num. Files (Codebase)
Num. Lines (Codebase)
110
0.50
0.25
0
2
4
6
8
10
12
0
500
1000
1500
2000
2500
2k
16
66
8k:
10k
500k
Wilk
Figure 5: Normalized cumulative distribution functions for different task instance statistics. We compare these statistics between SWE-bench M (blue) and SWE-bench (orange).
While the median number of files and functions low, with a majority of changes being local and limited to few lines, there is long tail of tasks that edit multiple files and functions. The distribution edits is unlike SWE-bench's s. In bench, 83% of task instances edit one file, and 65% edit one function. In SWE-bench M, just 40% of task instances change one file, and 32.5% change one function. On average, the changes by reference solutions in SWE-bench are larger than those in SWE-bench, with multi-file edits being more commonplace.
The average SWE-bench M codebase is somewhat smaller than SWE-bench repository. The me- dian SWE-bench M codebase has 200k fewer lines of code than its SWE-bench counterpart (535k compared to 734k). However, in the long tail, SWE-bench M has codebases that are larger in terms of both number of files and lines.
Testing compared to SWE-bench is quite similar. For SWE-bench, 80% of task instances have two or fewer fail pass tests compared 75% for SWE-bench M. Finally, another noticeable trend is that SWE-bench M issues tend to be shorter than SWE-bench problem statements. This trend can be attributed the inclusion of images as complementary form of communication.
GitHub Issue Tags. For every issue collected in the SWE-bench M dataset, we check what labels, any, were assigned to each issue. These human-annotated labels provide further insight into the kinds of fixes and contributions that a SWE-bench M task instance makes to the codebase. Table [T] summarizes the categories tags along with the counts individual tags.
Table 10: Categories of 1018 tags associated with 535 issues associated with one of the 517 task instances from the test split of SWE-bench M (499 task instances have issue, and 18 have 2).
Category
Count
Examples
Bug
267
"bug" (207); "type: bug" (56)
Feature
79
"type: enhancement" (44); "enhancement" (19); "feature request" (10);
"feature" (6);
Other
672
"ally" (40); "language" (37); "help welcome" (35);
"language-deffnition" (32): "role: dev" (32); "good first issue" (28);
"package: react" (21); "package: @ @carbon/react" (14); modeling (13);
While bug fixes are the most popular type of tag in terms of raw frequency, they only constitute quarter of all tags. In addition to features, there are variety of more repository specific bug fixes that highlight the diversity of the types fixes corresponding to SWE-bench M task instances. For example, the "al ly" tag, shorthand for "accessibility", is used in both the carbon and bpmn- js repositories. Issues with this tag usually describe how the accessibility features of a UI component from the JavaScript library is either missing or defective. The "language" and "language-deffittoon tags are used by syntax highlighting libraries including hi highlight ighht is and prism. Issues with this tag refer to bugs where language-specific symbols not being highlighted correctly in code
19

---

Published as a conference paper at ICLR 2025
editor. The "modeling" tag, used exclusively in the bpmn-js repository. refers to unexpected behavior the user encounters when manipulating bpmn-js diagrams. Every SWE-bench M task instance associated with issue that has this tag contains either video annotated picture that captures the phenomenon via screen recording and/or describes what the expected action should be. Such problem statement require not only visual reasoning acumen, but also the ability to ground accompanying issue description text in the visual elements.
Categories 186 tags associated with 109 issues associated with one of the 100 task instances from the development split of SWE-bench M (91 task instances have 1 issue, have
Table 11:
Category
Count
Examples
Bug
52
"type: bug" (21); "[Type] Bug" (17); "bug" (14);
Feature
45
"[Type] Enhancement" (9); "[Feature] Signup & Account Creation" (6);
"[Feature] Plans & Upgrades" (6);
Other
89
"Area:WebGIL (13); "L1 broken" (7); "released" (6); "L2 annoying" (5);
"Store" (5); "FixTheFlows" (5);
Mirroring Table TO, redo the analysis of the distribution of tags for issues associated with task instances from the development set in Table The findings for the development set are similar to the conclusions drawn from the of tags, categorized under "Other", are specifically to subsets of 1+ repositories and reflect more fine- grained, interesting purposes. For example, the "Area: "ebG." tag. which is specific p5. js, refers to problems that arise when using WebGL, a popular API for rendering 2/3-D graphics, to render js components. Such problems require visual reasoning to fully comprehend; usually, the problem is related to how the component is created properly when using the default P2D renderer, but not when WebGL used instead.
Bug fixes constitute quarter of all found tags. The majority
A.4 LICENSING
We fully list the license associated with each repository included in the SWE-bench M dataset in T2 These licenses all allow for non-commercial use of a repository's 's code and content, which the collection and evaluation processes of SWE-bench respects.
Table 12:
Summary and licenses for all GitHub repositories represented in SWE-bench M. Test split repositories are listed above the delimiter, while development split repositories are listed below.
Repository
Summary
License
carbon-design- carbon-deiign-systeecaaaaat system/carbon
A design system built by IBM
Apache-2.0
GoogleChrome/lightoonn
Automated auditing, perf. metries for web React-based framework for web app dev.
Apache-2.0
grommet/grommet
Apache-2.00
openlayers/openlayers
ceature-packed library for all your mapping needs
BSD-2-Clause
highlightts/highlighttis
JS syntax highlighter with lang. auto-detection
BSD-3-Clause
scrachhoounndaiio//cccacccggggl
GUI to create + run Scratch 3.0 projects
DSD-3-Clause
bpmn-io/bpmmnjjs
BPMN 2.0 rendering toolkit + web modeler
Custom
quarto-dev/quarto-cli
Open-source scientific and technical publishing system built on Pandoc.
Custom
Configurable component library for web Find and fix problems in JS code.
MIT
eslint/eslint
MIT
PrismIS/prism
Lightweight, robust syntax highlighter
MIT
prettier/prettier
Prettier is an opinionated code formatter.
MIT
markedjs/marked
Markdown parser and compiler. JavaScript and API powered WordPress.com
Custom
Automaticc/wp-calye
GPL-2.0
processing/p5.js
JS library for learning to code, create art Simple HTMLL Charts w/ <canvas>> tag
LGPL-2.1
chartjs/Chart.js
MIT
diegomura/react-pdff
Create PDF files using React
MIT
20

---

Published as a conference paper at ICLR 2025
B COLLECTION
Additional information about new dataset curation processes introduced for SWE-bench are in- cluded here, such as the reasons for and number of task instances removed due to inconsistency testing, challenges of setting up automatic evaluations for JavaScript repositories, and the procedure with which digital assets were automatically downloaded.
B. STATISTICS
provide summary of the number of pull requests that were eventually successfully converted into viable SWE- -rEchb task instances along with how many pull requests were removed per step of the filtering process.
In Table B
Table 13: Per repository, the number of pull requests eventually converted into viable SWE- bench M task instances and task instances kept per stage of the filtering process. The red subscript denotes the number of task instances that were filtered out at this stage with respect the total from the previous stage.
Inconsistent Test
Manual Filter
PRs Crawled
Conversion
Validation
GoogleChrome/lighhhooss
6,022
116
55.61
55.0
54
PrismJS/prism
1,942
52 5890
42 110
38 14
38.0
fiibaba-usioonnexxt
2,133
85
50 50g35
50.0
39 ,9
bpmn-io/bpmn-jj
820
86
54
54.0
54.0
8,096
196,7900
143 j53
134
134.0
eslint/eslint
7,454
42 1412
11
11.0
11.0
grommet//grommet
3,848
68 (,780
22 (46
22.0
21
highlighrisshighlighhe
1,955
60 1895
39 121
39.0
39.0
openlayens/openllyers
9,466
143 19323
88 L55
80
79,1
prettier/prettier
9,424
34 34 13930
13 13
13.0
13.0
quarto-dev/quartoccl
1,620
60 1560
255
25.0
24
weauchoounndaioo//srrtcchhhg
6,347
38 16309
11 127
11.0
11.0
Automattic/wppcalyyee
61,165
214 L 60951
37 1177
37.0
37-0
chartjs/Chart.js
3,792
200 13592
33 L167
2419
24.0
diegooura/reect-pddr
784
22 762
11 11:11
11.0
11.0
markedjs/maaredd
1,816
16 16 1800
14
14.0
14.0
processing/p5.j
3,186
42 42,3144
27 27
16411
16.0
Total
134,866
1,478
679 L799
643
619
B.2 INCONSISTENCY TESTING
All task instances in SWE-bench M should yield consistent test case results, scheme to filter out candidate task instances that are inconsistent. For each task instance, we run the test cases five consecutive times and check whether 1) all runs contain the same test cases, and 2) all runs contain the same passing test cases and failing test cases. either condition is false, we filter the task instance.
use simple
B.3 RESOURCE COLLECTION
Besides text, issue descriptions can contain
1. Images, often in the form of web browser screenshots detailing bugs or specifications of features. For issues describing unexpected user interface behaviors that involve com- plex mouse actions (dragging. etc.) or a longer sequence of reproduction steps, animated images (GIFs) are often used.
21

---

Published as a conference paper at ICLR 2025
2. Links to online integrated development environments (IDEs) for testing and showcasing HTML, CSS, and JavaScript code snippets. Around 17% of the instances contain at least one such link, and 15% contain more than one.
While images can be directly downloaded. copying the resources from the online IDEs is more involved. We handle four different such IDEs, codesandbox.io (63), jsfiddle. (25), codepen. (21), stackblitz.com (5), and edioorp5js.org where numbers indicate the number of occurrences in the final dataset.
The IDEs typically consist of three panes for the code in the three respective languages and an additional preview pane. Furthermore, menus provide options to link external resources, such as JavaSeript and CSS libraries, from content delivery networks. Depending the IDE, we have written extraction scripts (downloading and parsing source code or using web browser automation tools) or manually downloaded content obtain all resources. Most snippets are a combination of HTML, CSS, and JavaScript, while others take the form of React apps. For the former category, the online IDEs apply a variety of heuristics to interpret the HTML (linking external resources, adding missing tags, etc.), so we similarly apply a series of postprocessing steps to ensure that the websites can be immediately served without additional steps.
the occurrence of
C EXPERIMENTS
In this section, we include additional details concerning the experiments and ablations we ran on the SWE-bench dataset. We provide information such as how the optimal context window for retrieval augmented generation development agent scaffolds compatible with SWE-bench M task instances. We also show more detailed analyses of experimental results, such as solve rates by repository and year.
selected and the refactoring efforts required for making software
C.I BASELINE ADAPTATIONS
Here we provide further technical details on how adapted each of the software development scaffolds for evaluation on SWE-bench MM.
Retrieval Augmented Generation. We largely inherit the retrieval pipeline, prompts, and format- ting for RAG from those used in Jimenez et al. (2024a). This includes building a standard BM25 index using pyserini Adaptations include the inclusion of "reproduction code" where present ref- erenced in links as described in Appendix B.3 and adapting the patch generation example in the prompt from Python example to an equivalent JavaSeript example.
Lastly, we perform some minor reformatting of issue text. We convert all images to Markdown-style links with the following format:
[alt text or image url if not provided] (image url)
Agentless (Xia et al. 2024) is split into main phases, localization and repair, the former of which is heavily dependent on Python-specific constructs. The original localization phase relies on the Python ast module collect the line numbers of all function and class declarations. Unfor- tunately, ast is specific to the Python abstract syntax grammar, and therefore cannot be applied other languages such as Typescript and Javascript. We also found that the Esprime library for parsing Javascript code had issues with Typescript code, so we used the Tree-sitter parsing library, which supports a wide variety of languages. Due differences in function variants in Python vs. Javascript (e.g. arrow functions in Javascript), we also added extra parsing steps in the localization step. Furthermore, Agentless uses several in-context Python code snippets for both the localization and repair phases
replace all prompts with Javascript equivalents. Finally, when provide the repository structure as input to the model, we replace their tab-indented nested repository struc- ture with full file paths- hallucinate false file paths.
found this to be extremely important in ensuring the models do not
The original Agentless work generates a list of candidate solutions and manually checks whether they 1) pass syntax checks and 2) pass all pass-to- to-sssssss P2P tests. We do not include these filters for our implementation of Agentless because 1) we found syntax parsing to be inconsistent between
22

---

Published as a conference paper at ICLR 2025
different versions of Javascript, leading to many valid candidate solutions being filtered unless particular parser was used per instance and 2) we were interested in a fully automated framework. Finally, the original Agentless implementation did not implement support for Claude models, in- cluding how to handle extraneous outputs when querying for structured outputs. We added extra functionality to support Claude outputs for our experiments while remaining faithful the original implementation.
Aider (Gauthier 2024) similarly parses an abstract syntax tree for Python code to help the language model localize code with a limited context window - however, they use Tree-sitter instead of the ast module, enabling their framework to generalize easier to other languages. Other than adding the Javascript/ / Typescript versions of Tree-sitter, we do not modify Aider. Unlike Agentless, which simultaneously generates multiple candidates and does not require a syntax-checking step, Aider uses syntax-checking and linting in an iterative fashion to produce plausible generations. We found that due to syntax differences in Javascript versions, the parser would often invalidate correct or syntactically valid solutions generated by the language model, leading us not to use the baseline in its current form.
AutoCodeRover (Zhang et al. 2024b) also uses the ast module to generate an abstract syntax tree (AST) for Python code in its context-retrievat phase. Furthermore, it defines a set of APIs that the agent can use to traverse the AST. In our attempted implementation, we replace the ast module with the Tree-sitter library and some extra custom functions, and re-define the APIs to interface with the Javascript AST while retaining the same stratified search functionality the original method. We found the traversal APIS to be too specific to the ast module, as the structure of the Tree-sitter Javascript AST was entirely different, leading faulty localization.
To re-purpose AutoCodeRover for SWE-Bench M, a faithful implementation would require entirely swapping out the localization/ sode-searching APIs.
Moatless Orwall 2024) employs a multi stage pipeline for solving task instances. Similar to Agent- less, Moatless also generates an AST of the desired language, and converts this AST into custom code graph object that they use for their search and localization APIs. Because this code graph object is, in theory, language agnostic, it enables flexibility in applying Moatless to different programming languages. However, while the authors wrote code for converting a wethon-generated AST to their code graph, we noticed that performing an analogous conversion for Javascript/Typessee capture all of the relevant design patterns (e.g. arrow functions). Thus, we chose not to use Moatless as a baseline.
does not
SWE-agent Base. SWE-agent involves relatively few Python specific tools and components. The most notable exception to this is that for tasks in the original SWE-bench dataset, it sets up the OS environment with the necessary dependencies and installs the packages locally to provide the agent with a local testbed for experimenting with changes. To enable similar behavior in Javascript, we also install package dependencies and build the project locally when applicable. We also globally link the package name to the local repository when the instance contains an npm package.
New for SWE-bench M, we supply agents with "reproduction code" from links available in the problem statement. As described in (ppendix[B.3) these links and assets can contain useful code for reproducing issues locally. For each link containing reproduction code, we create a directory in the repository directory where the agent starts, and paste the code into the corresponding files (usually 'index.html" html', 'script.j'', and 'style.css'). Finally, we initialize those repositories as an npm package and link the package from main repository directory so the agent may use the package as installed locally.
SWE-agent JS. Extending SWE-agent Base, SWE-agent JS adapts the base edit command to work more effectively with JavaScript and related programming languages. In particular, the original SWE-agent edit command includes a component that performs an linting operation after every edit an agent applies to a Python file. This linting operation checks whether the resulting file would be syntactically correct and can detect certain other mistakes such as a reference to undnefined variable. During editing, SWE-agent will "roll-back" an edit if it is found to be invalid according to the linting operation. It then reports this error to the agent and requests that the agent to correct and try its intended edit again.
23

---

Published as a conference paper at ICLR 2025
Table 14: Specialized tools provided the agent for web interaction and image handling. Required arguments are enclosed in <> and optional arguments are in [].
Command
Documentation
open_ webnae <website.dir>
Opens the webpage in new display window us- ing Google Chrome. website..e ieeedieedir is the directory where an index.html file exists and will be served.
restart.webpage
Restarts the webpage with the given directory. website_dir is the directory where an index.html file exists and will be served.
<website.dir>
close_webpage
Closes the open webpage.
screenshot <num_images>] [<interval>]
Takes screenshots of the display. num-images (op- tional, default 1) is the number of screenshots to take. interval (optional, default 1.0) the interval be- tween screenshots in seconds.
open image <image .aiaagepath>>> [< [<image path> ...]
Opens the image(s) in base64 format to be fed to the LM. Multiple image paths can be provided.
seurch
read
modiff
website
other
17%
2.1
31%
30%
21%
28%
33%
18%
(GPT-o)
(Clande)
18
28%
27%
39%
0%
34%
20%
(GPT-o)
(Clande)
28%
24%
46%
33%
31%
30%
5%
(GPT4o)
(Clande)
Figure 6: Action space of SWE-agent with different configurations for Claude 3.5 Sonnet and GPT- 4o using the three configurations mentioned in Section on the development set. For version of this figure stratified by repositories, see Figure
To adapt this for SWE-bench M, we use ESLint with the local repositories configuration files, to primarily detect fundamental syntax errors. While typically ESLint is capable of enforcing wide variety of rules, we found that relying too heavily on these rules from the projects' configuration substantially degraded performance.
SWE-agent M. Further extending SWE-agent JS, SWE-agent M introduces a variety of new com- mands and features ot the ACI, which show in Table [4] These new commands require some environmental support for simulating displays. We simulate a display for the X window system using Xvfb, and take sereenshots with xwd.
C.2 BASELINE CONFIGURATIONS
SWE-agent. Due the expense and flexibilty in developing new SWE-agent interfaces, de- veloped new features and for SWE-agent IS and SWE-agent M primarily iterating while validating performance on small subset of the development set. By default, for long trajectories, SWE-agent "collapses" past environment observations by replacing the content of old observations with single line: "n lines omitted". SWE-agent will usually show the model the last five observations, and col- lapse all observations prior to that except the initial problem statement and demonstration. For the final configuration evaluated for each system, we perform very small grid search over two different options for the number of past observations to show in {5. 9}. We perform this search set of 50 instances from the development set and show the results of this search Table 15]
Retrieval Augmented Generation. For RAG systems, the amount of context provide, either in terms the number of documents or simply the absolute length of the context retrieved, is an important hyperparameter that may affect model's performance. As in Jimenez et al. (2024a), we determine the final RAG system to evaluate by performing a grid search over three possible context lengths in (32K, 64K, 100K) and the inclusion not of images input with the problem
24

---

Published as a conference paper at ICLR 2025
SWE-agent M (GPT-4o)
SWE WWgea (GPT-4o)
up-calypoo
30%
28%
36%
up-calypo
31%
24%
30%
15%
Chartjs
2.1
Chantjs
conct-pdf
7%
200%
23%
50%
reac-pdrr
25%
52%
marked
8%
13%
39%
40%
markod
29%
35%
24%
12%
p5.s
3.5%
3.5
14%
18%
14%
12%
73%
SWE-agent IS imgs (GPT-40)
SWE-agont M (Claude)
up-calypso
3.4%
20%
43%
wp-calypso
29%
35%
34%
Chartje
25%
18%
39%
Char
24%
2.5%
31%
19%
rao
26%
47%
27%
neach pdf
2.5%
2.%
32%
16%
marked
marked
35%
p5
IWE-agent (Claude)
SWE-agent imgs (Cloude)
op-calypso
wp-calypso
Chartjs
40%
37%
23%
34%%
31%
29%
react
31%
54%
27%
8%
pdt
40%
36%
22%
marked
19%
31%
37%
13%
narked
21%
30%
38%
11%
p5 js
34%
33%
15%
p5
32%
20%
30%
search
read
medify
website
other
Figure 7: Action space of SWE-agent with different configurations for Claude 3.5 Sonnet and GPT- 4o using the three configurations mentioned in Section 4 on the development set broken down by repository.
automatticcpp-cattto instanced
groonesinggg Instanome)
(0P744)
M
M
195
11%
11%
14%
(3P74))
11%
33%
(Caud)
675
X05
(Cuad))
11%
(Casée)
116
charts-Uhart.js (23 Instances)
diegomune/rrot pdf Instanoes)
107766
274
65%
33%
(C7546
27%
J5. 10
1:5
12%
36%
12%
(Casde)
18%
43%
715
79%
41%
10 mo0gr
markeq@ilsoreed isstances)
(0P740)
131540
116
1F%
XF4
(2F740)
Successful submission
Suocesstul desplte early pertnaatinn coal
(Caado
Teemitated due to costeerre Pelos submiedio
(Caado)
39%
45%
Figure 8: Frequency of outcomes by success and exit status of SWE-agent under different configu- rations the development set.
25

---

Published as a conference paper at ICLR 2025
Table 15: Hyperparameter search for SWE-agent system for non-collapsed history length.
Model
System
History Length
% Resolved
18%
SWE-agent Base
9
14%
Claude 3.5 Sonnet
28%
SWE-agent JS
18%
24%
SWE-agent M
16%
18%
SWE-agent Base
9
12%
GPT-4o
24%
SWE-agent JS
10%
26%
SWE-agent M
16%
Table 16: Hyperparameter search for RAG models with differing lengths of context and image input.
Model
Retrieval Context
With Images
% Resolved
9.4
32K
7.8
Claude 3.5 Sonnet
X
11.2
64K
V
14.1
X
7.642.7
100K
V
13.7
X
6.7
32K
6.9
GPT-40
X
6.7
64K
7.6
X
8.041.5
100K
10.0
statement. We run each configuration times on the development set and select the configuration with the best mean performance. We report the results of this grid search Table [6] highlighting the performance of our selected configuration.
C.3 FURTHER ANALYSES
Here, we include additional analyses of the experimental results and performance by different base- lines on SWE-bench M.
Resolved by Repository. We show the resolved rates per repository accomplished by different baselines in T7
26

---

Published as a conference paper at ICLR 2025
Table 17: % Resolved Rates by repository for different baselines. All performance numbers listed here use GPT-4o as the base model. Performance for the test split repositories (above de- limiter) are pass@1, 1, while performance for the development split repositories (below delimiter) represents the best result achieved from hyperparameter search.
Repository
Count
SWE-agent
Agentless JS
RAG
GoogleChrome/lghhtooss
54
5.6
7.4
1.9
PrismJS/prism
38
0.0
0.0
0.0
fusiona-fusionnnextt
39
0.0
2.6
0.0
bpmn-io/bpmn-js
54
27.8
11.1
13.0
carbon-design-syteem/rrooo
134
1.5
0.7
0.0
eslint/eslint
11
0.0
0.0
0.0
grommet//rrommet
21
0.0
0.0
0.0
highlightj/hhighhihttis
39
2.6
2.6
0.0
openlayers/openlayers
79
51.9
3.8
30.4
prettier/prettier
13
7.7
0.0
0.0
quarto-dev/quarto-cll
24
0.0
0.0
0.0
scrathhfoundaationssrcthhggggg
11
0.0
0.0
Automatiic/wp-calysso
37
0.0
0.0
2.2
chartjs/Chart.js
24
20.8
0.0
11.7
diegomura/react-pff
11
0.0
0.0
0.0
markedjs/marked
14
7.1
0.0
20.0
processing/p51 js
16
25.0
6.2
23.8
27

---

Published as a conference paper at ICLR 2025
Temporal analysis does not reveal any indication of solution leakage. Table 18 shows resolution rates for different baselines by year. While agentless with GPT-4o shows somewhat increased per- formance for older task instances, all other system shows significantly higher performance newer instances with a pronounced peak in 2024. However, the instances of the different repositories have uneven temporal distributions and average resolution rates (see Tables 7 17). making it difficult to interpret the yearly solution rates. For example, out of the 13 instances from 2024 are from openlayers/ openlayers, which has by far the highest solution rates with SWE-agent M and agentless across all years.
To account for this, we calculate average resolution rates before and after the GPT-4o training cutoff (October 2023). We then compare the post resolution rates to the pre-resolution rates reweighted to the post-cutoff repository distribution" The results are shown Table 19 and show that the post- cutoff performance of all tested systems exceeds that of the pre-cutoff even when accounting for the shift in repository distribution.
Since Claude 3.5 Sonnet's training cutoff of April 2024 would leave too few instances, similar analysis is not feasible. However, SWE-agent with Claude 3.5 Sonnet follows a very similar trend SWE-agent M with GPT-4o and shows even more pronounced performance jump in 2024.
Table 18: % Resolved performance for SWE-bench M test split task instances from different years. Each row corresponds to a year, while each column is the agent's performance for task instances from that year.
SWE-agent M (Claude)
SWE-agent M (GPT-4o)
Agentless JS (GPT-40)
RAG (GPT-40)
Year
Count
2017
22
4.5
4.5
9.1
4.5
2018
31
0.0
0.0
3.2
0.0
2019
96
9.4
7.3
3.1
1.0
2020
91
8.8
4.4
5.5
2.2
2021
103
13.6
11.7
2.9
3.9
2022
101
12.9
19.8
1.0
13.9
2023
60
13.3
20.0
0.0
6.7
2024
13
46.2
53.8
7.7
46.2
Table 19: % Resolved Rates with GPT-4o before and after training cutoff. This table shows the performance for the SWE-bench M test split task instances before and after the GPT-4o training cutoff date (Octoher 2023). Pre-cutoff (reweighted) is the pre-cutoff performance reweighted to the post-cutoff repository distribution (see text).
Count
SWE-agent M
Agentless JS
RAG
Pre-cutoff
500
11.0
3.0
5.0
Pre-cutoff (reweighted)
500
27.6
3.1
13,6
Post-cutoff
17
47.1
5.9
41.2
*i.c., we calculate Enpo propo re3o where fpon the fraction of instances a specific repository among the post-cutoff instances and 1/pce is the corresponding pre-cutoff solution rate.
28

---

Published as a conference paper at ICLR 2025
D HUMAN VALIDATION
We provide full details about the human validation process for SWE-bench M task instances. We include each question with corresponding examples that were provided to the papers' authors. We also show additional analyses about the authors' answers and agreement.
D.1 IMAGE CATEGORIZATION
This question aims to categorize the types of visual content commonly associated with open-source software issues. By classifying images across a diverse range repositories, we aim to analyze the visual ways in which developers convey information about software development. To accomplish this, we randomly sampled 50 images from the dataset and manually derived 8 categories from these samples. Human participants then manually inspected and labeled all images with of the categories.
Classify the image as one these categories: (1) Code Snippet Screenshot (2) Web Inter- face (UI/UX Element) (3) Map/Geospatial Visualization (4) Diagram (5) Data Visualization (Plots) (6) Artwork / Photography (7) Error Message (8) Miscellaneous.
Prompt:
Here examples of each category of images.
(nunc) quory)
WHUes "sslec "hollo world" Free 615
INSERT INT0 RESSANESE4RCPPATTTER (nomo, query) "hallo world" froe w!)
vaues ('v') velect
(a) Code Snippet Screenshot
(b) Web Interface (UI/UX Element) (c) Map/Geospatial Visualization
Thac (Eo Th, dac (or/ 21 tratalia, tanernea,reaan
20
2.1
1.
Pull Sea
(d) Diagram
(c) Data Visualization
(f) Artwork Photography
Neva
Warearteeiindngg beffero cePEESSEs ou (ertl, namee ant anceents saenecurres)
(g) Error Message
(h) Miscellaneous
The images are from the following task instances:
1. Code: PriemJSS_prssm-22 1500
5. Data Viz.: -uw.atooooo--55
2. Webpage: carbon-desiga--satem..
6. Art: guarto-dev.qurrtooc111
3. Geospatial: openlayers, wel.oppennalle appmlayyrrs..opnllaerertte lay4r----19
7. Error Trace: diegomura.reacctpptt.. react-pdt--188)
4. Diagram: bpmn-i o..bpmm-js-1548
8. Other: openlaayere_ogeenlaaer--
29

---

Published as a conference paper at ICLR 2025
Table 20: Response to Image Categorization indicates the majority of images are code and website screenshots. Other categories tend to be more repository -peecfffi............
Code
Website
Geospatial
Diagram
Data Viz.
Art
Errors
Other
Total
bpmn-js
carbon
21
136
0
36
0
0
30
1
224
Chart.js
5
2
0
18
0
26
eslint
10
0
0
0
2
0
13
grommet
3
25
0
2
0
0
0
32
highlight.js
4
0
lighthouse
5
55
0
3
0
68
marked
22
0
2
0
0
25
next
36
openlayers
6
6
35
2
0
49
p5.js
5
0
0
22
2
32
prettier
9
3
0
prism
41
3
0
0
0
0
0
47
quarto-cli
7
22
10
43
react-pdf
6
7
0
0
0
20
scratch-2e gui
0
12
0
1
0
0
13
wp-calypso
55
0
3
0
59
Total
194
401
35
107
28
38
54
2
859
Results: We applied this labeling procedure the 862 problem statement images across all SWE- bench M task instances. The counts for each of the categories are presented below in Table 20
The majority of SWE-bench M images are screenshots of webpages or code. These categories dominate because they are most often used to point out problems with linting (e.g. eslint int, highlight. js, pr sm, prominently feature code-related images due to their focus on code anal- ysis and syntax highlighting) and incorrectly rendered web elements (e.g. carbon, hight next, wp-calypso, as these tools are used to address issues in web design, layout and perfor- mance). The representation for other categories correlates heavily with specific repositories. In other words, images falling under a certain category are usually distributed across at most two to three repositories. For example, diagrams tend be found in bpmn-js, a tool for creating BPMN (Business Process Model and Notation) diagrams, and ca rbon, which often uses design component specifications to communicate details about web elements, such spacing and size.
Geospatial images, including maps or location- based data, are exclusive found in openlayers. Data visualization, such charts graphs, are predominantly found in the Chart. js repository, which specializes in creating data-driven visual content. Additionally, creative coding repositories like p5. js often include images related artistic outputs, those images typically include visual glitches or unexpected rendering behaviors. Lastly, categories such as errors are found sporadically across repositories and represent instances where tools fail to operate correctly, such as compilation failures. Based on these findings, we conclude that SWE-bench M features a wide range of images that capture various aspects of software development, with code and webpage scrcenshots domi- nating the dataset. At the same time, certain types of images tightly coupled with the specific functionalities of individual repositories, reflecting the unique focus and purpose of each project.
D.2 IS AN IMAGE REPRESENTABLE AS TEXT?
How important it for SWE-bench image to be an image? To answer this, we attempt to quantify the proportion of images that solely text. For these images, it may not be necessary for the image's information to be communicated visually. For instance, it's possible that a large proportion of images screenshots of code error messages, some of which could be represented
30

---

Published as a conference paper at ICLR 2025
perfectly text. If this is the case, the fact that the information is presented image becomes somewhat trivial.
To clarify, we not asking whether the image can be processed with Optical Character Recognition (OCR) or verbalized, as such approaches would still likely result in loss of information and are hard to answer without considering the problem context. For instance, for syntax highlighting libraries, although many images are code screenshots, there could be color highlighting of different characters that is crucial to understanding the problem correctly. Therefore, while traditional OCR could turn the code in the image into text, it would fail to capture additional visual details. Asking whether the same image can be verbalized is also not straightforward. as the answer could likely depend on the type of problem being solved and what information about the image is actually necessary. In a small labeling trial, we also found that answers are sensitive to the annotator's subjectivity, reflected by low inter-annotator agreement.
Similar to Ql: Image Categorization, we ask human participant to look at the image and respond with either "Yes" if the image can be represented faithfully as text, and "No" otherwise. A more detailed discussion of how to decide is included in the following prompt.
Prompt: Can you assess if this image can be faithfully represented with text? A faithful represen- tation means that the contents of the image can be represented as text with no loss in information. An image fits this criteria if (1) The image only contains text (2) The text in the image is mostly or entirely monochrome, and (3) The image does not contain any meaningful visuals patterns inde- pendent of text that could be important to know for solving a problem. Please only respond with [1/0], 1 meaning "Yes" and 0 meaning "No". Do not include any additional justification or text.
Here are examples images that can and cannot be conveyed effectively with OCR.
Show
(a) Label: No
(b) Label: No
(c) Label: No
atte "qaae pereet Amultt
jnceeedt deprecaciasar buttert) diprecaced Leadttry
the
Rorfon-dien Sy*erulbbuitutc teffor.fves) initme..
pos 0g
(d) Label: No
(e) Label: No
(0 Label: Yes
Figure 10: Examples of images that cannot (top row) and can (bottom row) processed into text by OCR software. Typically, code and webpages are OCR-able while diagrams, plots, and art are not.
Top Left ott.://oounnnatioeeeeeeeerrriaaa
The eye icons cannot be represented as text. While there text, it does not take up much of the image.
Top Middle (chartjs- -as--5667777
The bar plot cannot be converted into text. Top Right (processing.p5.. js-3799...3399) This is an graphic with no text in it.
649): Although significant portion of this image is made up of text, the code syntax coloring is an aspect of the image that cannot be conveyed effectively in text representation. In addition, the black highlighting and striped background cannot be portrayed as text.
Bottom Left (open layers
Bottom Middle (diegomura react seact-pdf-2ee -pdf-12f111 Similar the bottom left image, this is also "No" because of the text coloring.
Bottom Right (PrismJS pri smiimm27) The contents of this image can be faithfully represented as text, as the majority of it is colored red. The main purpose of this screenshot is to communicate an error message. There are no meaningful visual elements.
Results: We again answer these questions for problem statement images in SWE-bench M. Out of 862 problem statement images, 691 (80%) were labeled as "No", while 171 (20%) were labeled as
31

---

Published as a conference paper at ICLR 2025
Q2: Yes/No Counts per Repository
200
150
No
60
bpmn-jj
carbon
Chartj
eslint
grammet
highightje
lighthouse
marked
opentayers
ls
preitier
primm
quarto-cli
raact-piff
scratch-gul
wp up-calypsa
Figure 11: Responses to whether image can be represented with text per repository,
Q2: Yes/No Counts per Category
300
Yes
200
100
Web
Data Viz
Diagram
Code
Error
Geospatial
Figure 12: Responses to whether an image can be represented with text per image category.
"Yes". Below, present two figures. Figure I shows the labeling results per repository. Figure 12 shows the labeling results with respect to image category.
With the exception of prettier. across repositories, the majority of images were labeled as not representable with text, with several repositories having no images labeled "Yes" (e.g. bpmn- js, highlight js, quarto-cli), The principal component that corresponds whether image can be labeled the image category. As shown in Table different image categories have varying rates of whether not the image can be represented with text. Error messages are often laheled "Yes" than not. A smaller proportion of web and code screenshots were labeled "Yes". Although first glance it may seem like this proportion should be higher, through the annotation procedure, we found out that for code, text color and highlighting are often very important signals for solving task instances, particularly linting repositories such as pri sm. and eslint. For web screenshots, while there is usually text on webpage, the spatial arrangement of web components is critical aspect of understanding problems design framework libraries, such as carbon and next. For the remaining categories, low to no images are labeled as being representable in text.
Based on our findings, we conclude that information from problem statement images in SWE-bench M frequently must be represented as images. The rate which images not just screenshots of text varies by repository and image category. Images that are errors are more often representable as text. Other categories are less often due either complete absence of text or the text coloring and organization being an important part understanding the problem being communicated.
D.3
IMAGE NECESSITY.
We clicit human feedback on how necessary images are to solving the task at hand. This question attempts directly determine how the visual asset(s) provided with a problem statement actually necessary to solving a GitHub issue when provided. We label the 557 SWE-bench task instances that have one more images in its problem statement.
For the annotation procedure, human participant asked to look at each task instance's problem statement, codebase, gold patch, and test patch, and judged whether it could be solved. They should
32

---

Published as a conference paper at ICLR 2025
then look the associated problem statement image(s) and assign label according to the procedure described below. Task instances with no images in the problem statement are not considered for this question. If task instance is deemed generally unsolvable, it is removed from the dataset.
Prompt: For each task instance, please determine whether its associated images necessary to solving the problem described by this task instance. The following workflow is recommended:
Read the problem statement. you't d like, check the codebase, gold patch, and test patch needed. Make mental note of whether the task instance can be solved.
2. Then, look the image(s) associated with the task instance. Now, re-answer whether the task instance can be solved.
3. Provide answer according to the following logic:
"Yes": If the answer changed from "No" to "Yes", then the image is necessary.
"No": If the answer remained "Yes", the image is not necessary solve the task.
Remove: the answer before and after seeing the image was both "No", please note this and will remove the task instance.
Below, provide examples of task instances where the image is necessary to solving the task (PrismJs_.ppe sm-3442), and where is not
PrismJS prism-3442. A sereenshot of code is provided for this task instance. It is shown together with the corresponding problem statement in the text box below.
The main contribution of this image is the syntax highlighting. Specifically, it shows the undesired effect of Prism highlighting the quotes with the incorrect color. The double quotes around "Month", "Days", and "Jan" should be highlighted gray.
Without the screenshot, it not exactly clear how the quotes are being highlighted incorrectly. For instance, it's unclear what color corresponds to "highlighted punctuation". In the absence of the image, while it is still possible that an experienced maintainer of this repository could make educated guesses about the exact nature of the bug or reproduce exactly, the image provides concrete evidence of what the user is experiencing.
PriamJ8 3442: [language-markupl Quotes in HTML attribute values are highlighted
**Information **
Prism version: Latest (reproducible on test.html)
Plugins: none
EEnvirommen: Browser
Quotes are highlighted as punctuation inside HTML attributes. E.g. this:
""hml
<google-chart dato="(""Mont",, "Days"I, ["Jan", </loolc<goollllhhhhat
is highlighted as:
agoogle-chart data= "M"nth"", "Days" ["Jan", 31]] >> goo.go
might decide that this a feature, not a bug, but figured I'd flag this case, as looked wrong to me.
On the other hand, the image quarto-dev quarto-cli- 6659's problem statement is not necessary.
33

---

Published as a conference paper at ICLR 2025
The provided image is an arbitrary link that is used in the reproduction code shown in the issue description. While the image is helpful that it is readily usable by a developer for recreating the problem, there is no additional information being conveyed by the image's content that helps with understanding the task at hand.
quarto-dev.quartoo-e i- 6559: Figures from R code blocks don't render "fig-cap: lexpr evaluates to 'character(O)'
With Quarto 1.4.330, quarto-r 1.2, and knitr 1,43 on R 4.2.1, 1, the command 'quarto render exam- ple.qmd -to html renders the following Quarto document expected:
"I
#| fig-cap: texpr caption
caption "hello world"
knitr::include graphics(" http:///atccentttnilly/aaaaaaaaa
But "caption = "hello world" is changed to 'caption character(O)", the figure disappears from the output.
I originally hit this issuc with mistaken 'sprintf" call, which can produce zero-length character vector.
Results: Out of 557 task instances, 465 were labeled as the image being necessary to solving the problem. The remaining 93 were labeled as "No", meaning the image is not necessary. The follow- ing Figure shows the labeling splits for this question by repository.
Figure 13: Responses to whether an image is necessary to solve task instance per repository.
Q3: Yes/No Counts per Repository
No
Spmojjo
Clarjj
eniny
prommn
lphiloune
mered
next
pottee
Hesdeed
wor pralchgu
apecalyyo
Annotations suggest the significant majority (83.5%) of problem statement images are necessary to solving the problem at hand. Several repositories have zero issues with unessential images.
Annotators attributed several factors that affected their decision making. First, if the task instance included reproduction code, annotators were more likely to rate the task instance as requiring the image. For these cases, the image was redundant because running the reproduction code would produce the exact effect that the image was capturing. Second, repository's contributing guidelines influences the role of the image. For instance, or screenshot that details the error message, which is why for repositories such as bpmn- 1ight house, there zero occurrences of unessential images. On the other hand, images sometimes provided under an "Additional Information" section, in which case the image helpful, but crucial. These observations also hold for reproduction code. Task instances from carbon, p5. js, and next will often have links to online editors reproducing the described bug.
repositories will explicitly ask for an image
and
34

---

Published as a conference paper at ICLR 2025
D.4 TASK DIFFICULTY
Following the instructions used in Chowdhury et al. (2024), we label the difficulty of a task instance. We define difficulty as the estimated amount of time would take for experienced developer to accomplish this task. There are four possible labels for difficulty:
15 min fix (e.g., a trivial change adding some assertions to function)
15 min-1 hour (e.g., a small change that requires bit of thought)
1-4 hours (c.g., substantially rewriting a function or editing multiple files)
-4 hours (e.g., a very esoteric issue that clearly requires a substantial amount research to fix, changing > 100 lines of code)
For every estimate, we assume the developer is approaching the problem from the following context:
The developer has neither worked on nor used the codebase before.
The developer familiar with JavaScript code and has experience working on JS related frameworks (e.g.. React).
The developer has had a couple hours to become familiar with the codebase prior to starting work on the task instance.
To label, each annotator looked at the full details of the task instance (problem statement, images, gold patch, codebase) and made best-effort judgment of how intensive the fix would be. Annotators were asked to not consider the time spent on writing or modifying tests.
Because of the potential variance in annotators' answers to this question along with the amount of time it takes to comprehend a task instance before making a judgment, we do not label the entire SWE-bench M dataset. Instead, we take 100 task instances from all of SWE-bench M, uniformly sampling with respect to the ratios of task instances in each repository. Three annotators then label each of the tasks with one of the four labels. For each task instance, the final difficulty level is assigned to be whichever label has majority vote. If there is no majority vote, the "middle" of the three annotations (either 15 min-1 hour 1-4 hours used the label.
Results: From this labeling procedure, 100 SWE-bench M task instances are labeled with differing levels of difficulty, shown in Table 21] The Fleiss' kappa score is calculated to be 0.78, reflecting substantial amount of agreement between the three annotators who carried out labeling.
Table 21: Estimated level of difficulty for different splits of SWE-bench in addition to SWE-bench M. Each number is the percentage of sampled task instances classified at that level of difficulty.
Dataset
# Samples
< <55 min
15 min-1 hour
1-4 hours
> hours
SWE-bench
1,699
24.5
53.3
19.4
2.8
Lite
231
37.7
56.3
6.1
0.0
Verified
500
38.8
52.2
8.4
0.6
SWE-bench M
100
13.0
43.0
38.0
6.0
After the labeling procedure was completed, annotators agreed that the strongest signals for estimat- ing the level difficulty came most frequently from three indicators.
First, smaller gold patches tend be labeled as requiring less time and visa versa. For instance, for the 13 task instances labeled to take < 15 minutes, the gold patch change sizes (calculated as Lines Added+ Lines Removed) are 3, 4, 3, 163. 5, 5, 10, 7, 6J4 The one aberration of 163 lines due to updates to an auto-generated package-lock is on. On the other hand, for task instances requiring 4 hours to fix, the gold patch sizes are [350, 186, 557, 367, 2682, 937]..
Task instances with more descriptive problem statements tend to get rated as casier. The annota- tors' noted that more detailed descriptions and the presence of reproduction code made them more
35

---

Published as a conference paper at ICLR 2025
inclined to reduce the estimated amount of time required. The images in the problem statement also had an effect on the labeling. Annotators mentioned that screenshots of web elements and source code helps developers localize errant code in codebase faster. For instance in carbon and next, many screenshots would point out spacing or coloring issues with a specific web component. A screenshot is often informative for not only describing the problem, but also directly point which component folder the task worker should look into.
Finally, annotators noticed that sometimes, a small change can still take some time, particularly if the edit is precise and requires a good amount of context to find. In addition to the size of a gold patch edit, the quality of an edit, specifically the types of entities and symbols that were used in the fix, influences the amount of time a problem takes. For instance, a larger edit that mostly consists of JavaSeript primitives is usually regarded as simpler than shorter edit mostly made up of entities from other parts of the codebase.
In conclusion, SWE-bench M presents task instances at range of difficulties, from short quick fixes to problems that require large scale refactoring of multiple modules in codebase. The difficulty distribution suggests that SWE-bench M can be effective for tracking improvements model and agent capabilities.
E LIMITATIONS
Broader scope To comprehensively evaluate multimodal Al systems, we curate 617 task instances from 17 JavaScript libraries for SWE-Bench M. However, the scope could be broadened across three key dimensions: (1) More programming languages: Besides JavaScript, multimodal content also appears issues related to code in other programming languages, such as Python, C++, or Rust. (2) More modalities: While images and videos likely the most commonly used media besides text in GitHub issues, users can also upload audio files or other media and file types.
(3) More tasks: Our 617 tasks across 17 libraries cover various domains including web frameworks, data visualization and syntax highlighting. however, there are many more JavaScript libraries in other domains that could be added to the benchmark. We are excited about research broadening the scope across these and other axes. However, here we focus on quality over quantity. Scope extensions likely warrant separate dedicated future work, as we found that adding task instances to SWE--benh M was labor-intensive task.
Improved models and environments While we primarily focus on developing the benchmark and evaluating current models, also extend SWE-Agent to handle multimodal issues. There remains substantial opportunity for future enhancements, both by advancing the underlying models (e.g., leveraging future versions of GPT and Claude) and by enriching the agent's environment with, for example, better browsing capabilities or more tools. We look forward to seeing future work along these lines improve performance on SWE-Bench Multimodal beyond the systems that present.
36


Reviews:
[{"review_id": "imyj4vPgLh", "reviewer": "Reviewer_wvpw", "summary": "This work introduces SWE-Bench MultiModal, an extended version of SWE-Bench, which addresses two limitations of the original version: (1) its focus solely on the Python programming language, and (2) its reliance on a single textual modality. The proposed benchmark overcomes these shortcomings by incorporating JavaScript as the primary programming language and adding images that reflect the issues being addressed, in order to assess the capability of language models (LMs) in generating multimodal patches. Additionally, the authors test the performance of existing LMs after adapting this benchmark to three systems: SWE-agent, Agentless, and RAG. Experimental results demonstrate that this benchmark remains a challenging one.", "questions": "1. I performed a preliminary review of the benchmark provided by the authors and noticed that a significant number of patches and test patches are empty. Does this suggest potential concerns regarding the quality of the benchmark?\n\n2. I have questions about how the benchmark is evaluated. For example, how is the effectiveness of the generated patches in fixing new front-end pages assessed? The authors mentioned pixel-level visual testing, is this pixel-level comparison reasonable?\n\n3. The authors should consider providing a complete repair process and examples to demonstrate how the system performs on the benchmark. This would offer a clearer understanding of how the system operates in practice.\n\n4. The authors should consider showcasing several successful and failed cases to demonstrate that the successful cases across all models are not merely fixes for very simple problems. Additionally, it would be valuable to provide insights into the reasons behind the repair failures. This would help clarify the challenges and limitations of the current approach, offering a more comprehensive understanding of the model's capabilities and shortcomings.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, this work is commendable, and I believe the following points are particularly praiseworthy:\n\n1. The introduction of multimodal autonomous systems for software engineering is a forward-thinking direction. By introducing this benchmark, the authors contribute to the community's understanding and research in this emerging field.\n2. The experimental comparison of existing models is thorough, yet the primary aim is to highlight that this is a challenging and unsolved problem.\n3. Significant contributions have been made in data collection and system adaptation, providing a solid foundation for guiding future research in this field.", "weaknesses": "However, from my perspective, I believe the paper could be improved in the following areas:\n\n1. The authors claim that SWE-Bench only uses Python, limiting the ability to assess the generalization performance of language models, and use this as the motivation for proposing SWE-Bench MultiModal. However, this argument seems unconvincing. Although SWE-Bench M introduces JavaScript (and potentially other languages like HTML in the repository), a truly \"generalizable\" benchmark should encompass multiple programming languages, not just extend the original benchmark with a new one.\n\n2. Despite the extensive engineering adaptations made in this work, the benchmark seems somewhat lacking in innovation compared to SWE-Bench. SWE-Bench was created from scratch (0 -> 1), whereas SWE-Bench M is an extension (1 -> 1.5, maybe).\n\n3. Multimodal software engineering is a highly attractive topic, but it seems unlikely that, in the foreseeable future, we will be able to solve it comprehensively in both practical applications and research. As a benchmark, simply increasing the difficulty of the tasks does not seem like a sustainable or effective direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjYSNF0CgI", "reviewer": "Reviewer_RMn2", "summary": "This paper introduces SWE-bench Multimodal (SWE-bench M), a novel benchmark designed to evaluate coding agents' ability to handle real-world software engineering tasks involving visual elements. The benchmark comprises 619 task instances collected from 17 JavaScript repositories, focusing on user-facing applications such as UI design, data visualization, and interactive mapping. Through rigorous analysis, the authors demonstrate that 83.5% of these tasks require images for proper resolution, highlighting the crucial role of visual elements in software development. The study reveals significant challenges in generalizing existing software engineering systems to handle both JavaScript and multimodal contexts, with even the best-performing systems achieving only a 12.2% success rate. The paper makes several key contributions: it provides comprehensive analysis of visual content in software development issues, introduces new evaluation methodologies for assessing visual reasoning capabilities, and reveals important insights about the limitations of current systems designed primarily for Python codebases. The authors also demonstrate that more flexible, interaction-based approaches perform better than rigid, pipeline-based systems when handling multimodal tasks, suggesting important directions for future development of software engineering systems.", "questions": "1) Could you provide more details about the engineering efforts made to adapt existing systems (Agentless, Moatless) for multimodal tasks?\n2) Have you considered expanding to other visually-rich domains beyond web development (e.g., game development, mobile apps)?\n3) Do you plan to expand the benchmark to include other front-end frameworks or visual-heavy domains?\n4) How do you envision SWE-bench M evolving as models become more capable at handling multimodal inputs?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work exhibits rigorous data collection and validation processes, including comprehensive human validation of task instances, careful consideration of image necessity, and thorough experimental design with detailed ablation studies. The work addresses a significant gap in existing benchmarks by incorporating multimodality and tackling real-world challenges in software development. The dataset shows a diverse range of visual content types, and well-documented collection processes with careful consideration of licensing and reproducibility.", "weaknesses": "The paper has several areas requiring improvement, primarily in its framing and presentation. A significant issue is that the paper sometimes misframes its contribution by emphasizing Python-centric limitations of existing benchmarks, when its true novelty lies in targeting front-end development's inherent multimodality. The motivation for choosing JavaScript repositories should be more prominently centered on JavaScript's dominant role in front-end development and its natural incorporation of visual elements, rather than being positioned as addressing a gap in language coverage. \n\nIn addition, the evaluation methodology raises significant concerns about fairness and thoroughness in baseline adaptations. A major weakness is the oversimplified approach of merely swapping Python AST parsers with JavaScript parsers when adapting existing systems like Agentless and Moatless Tools. This superficial adaptation strategy leads to potentially premature conclusions about these systems' inability to generalize to multimodal tasks. The paper lacks a convincing demonstration that sufficient engineering effort was invested in properly adapting these baselines for multimodal scenarios.\n\nThe authors should have:\n- Made a more rigorous attempt to adapt these systems for multimodal input handling, beyond just language parsing\n- Documented failed adaptation attempts in detail to substantiate claims about generalization difficulties\n- Explored architectural modifications that could enable these systems to better handle visual elements\n- Referenced and learned from existing frameworks like HyperAgent[1] that successfully handle multiple programming languages and SE tasks\n\nWithout demonstrating that serious engineering efforts were made to strengthen these baselines, the paper's conclusions about the systems' generalization capabilities are not well-supported. The low performance of adapted baselines (3.9% for Agentless vs 11.5% for SWE-agent) might reflect inadequate adaptation efforts rather than inherent limitations of these approaches. A more compelling evaluation would require investing significant engineering effort to properly adapt these systems for multimodal tasks, even if this requires substantial work. This investment is necessary to make meaningful claims about generalization capabilities and to truly understand the challenges of adapting existing systems for multimodal software engineering tasks.\n\nFinally, Table 1's comparison between SWE-bench M and other repository-level coding benchmarks is incomplete, notably missing comparisons with important related works such as DevEval[2], CrossCodeEval[3], and RepoExec[4]\n\n[1] HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale, https://arxiv.org/abs/2409.16299\n\n[2] DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories, \nhttps://arxiv.org/abs/2405.19856\n\n[3] CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion, https://arxiv.org/abs/2310.11248\n\n[4] On the Impacts of Contexts on Repository-Level Code Generation, https://arxiv.org/abs/2406.11927v3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QQ3rriS41", "reviewer": "Reviewer_yGjW", "summary": "This work creates a new benchmark SWE-Bench Multimodal which address some limitations of prior benchmarks in this field that mainly focus on evaluation on Python only repositories. SWE-bench M provides a benchmark suite in JavaScript with the input  being to include also visual elements. SWE-bench M contains more than 600 different tasks complete with test cases to measure functional correctness. The authors further modify some existing state of the art open-source techniques and found that many of them require python-specific designs that limit their ability on the new benchmark", "questions": "- Given the low amount of changes required to go from SWE-agent to SWE-agent JS and SWE-agent M, I am curious if the authors have tried applying SWE-agent JS or SWE-agent M on the original SWE-bench tasks? if so what would be the performance?\n- How are the human annotators chosen? the paper only mentions the number of annotators (3) in the Appendix, but no further details are provided", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Creates an important benchmark for an important area of measuring repository level code generation especially when the input is multi-modal\n- Extensive manual effort used to ensure the benchmark is useable as well as adopt and modify baseline techniques", "weaknesses": "Overall, this is an important and usable benchmark, below are some weakness that I would love to see the authors address\n\n**More analysis on the difficulty of the dataset**\n- As the current demonstrate that SWE-bench M seems to be more diffcult than SWE-bench and SWE-bench lite, I would suggest the authors take a deeper look into the types of problems in SWE-bench and the reason why it is more difficult\n- While the authors mentioned the mulit-modality aspect and the requirement of images, is that the only factor?\n- It seems from looking at table 5 (the tasks without requiring images according to human annotators), the problems still appears to be more difficult that original SWE-bench lite problems\n- More analysis into what makes these problem difficult would be beneficial to the research community \n\n**Improvements to Testing**\n- Given the state of software development is test driven, I would like to see the authors make some effort in improving the test quality.\n- As mentioned in text, the authors already filter out some \"flaky\" tests which are a good sign, however looking at the number of tests in Figure 5 and Table 9 in Appendix it seems the number of Fail to Pass tests are still limited.\n- It would be great if the authors can apply some test augmentation or manually apply (while this might require tons of manual effort, it woud definitely improve the robustness of the benchmark) \n\n**More metrics than just pass or fail**\n- One issue with repository level benchmarks (including SWE-bench) is that the metric used seems to be always whether the issue was resolved or not.\n- For real-world software development, there are many steps to solving the problem from being able to identify the root cause, localize the code elements, writing tests that can reproduce the errors, and then finally applying the patch\n- I believe that more intermediate metrics that can be used to gain more information from the results should be designed and applied. \n- This would aid in the design of new approaches as well as provide more analysis options espeically given the large cost of evaluation on repository level benchmarks like SWE-bench M", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lKPBc9NF56", "reviewer": "Reviewer_v5Tx", "summary": "The paper introduces the SWE-bench Multimodal (SWE-bench M), an extension of the SWE-bench benchmark for evaluating the performance of autonomous software engineering systems on their ability to fix bugs in visual, user-facing JavaScript projects. The dataset includes 619 task instances from 17 repositories, incorporating visual elements like screenshots, diagrams, and other assets to reflect real-world programming challenges more accurately. The authors also assess existing models developed for SWE-Bench on SWE-bench M, highlighting the limitations of current state-of-the-art approaches in handling visual information and cross-language generalization.", "questions": "1. This work extended SWE-Bench, while major challenges encountered during data collection may be similar to those in SWE-Bench. Could authors clarify what unique challenges were faced in data collection for this work compared to SWE-Bench?\n\n2. There is an established research body on reproducing/fixing multimodal bugs in mobile apps, while this work focuses on JavaScript bugs. What is the motivation behind focusing specifically on JavaScript-based multimodal bugs, given there already exist mobile app bug datasets?\n\n3. The data collection process inlcudes five steps, but the criteria for each step are unclear. For example, in step 1, the authors selected 17 repositories. What was the project sample size after applying the filters (e.g., 5,000 or more stars and 500 or more pull requests)? Step 5 includes human validation, how many authors involved? How to solve the conflict during the process?\n\n4. \"From 135k PRs, this new filtering criteria yields 1,478 candidate task instances.\" Does this indicate that the identified bugs are relatively uncommon or corner cases? How important is the importance of these collected multimodal bugs compared to text-only bugs?\n\n5. Do the baseline models perform significantly differently on multimodal issues than text-only issues within these JavaScript projects?\n\n6. The impact of visual data (images/videos) on the bug-fixing process is unclear; the performance decline without images/videos is around 3%. Does this imply that images/videos are unimportant for these tasks?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ a new dataset for evaluating LLM-based issues resolving models\n+ limitations of existing approaches on the new data were discussed", "weaknesses": "- criteria for project and issue selections are unclear\n- generalizability of the JS-based multimodel issues is limited\n- there already exists a rich research body on multimodal bugs/issues---mobile bugs/issues\n- the adaptions for existing models are mainly on the language level, it's unclear how to handle images/videos\n- the comparison between these approaches on multimodal issues and text issues is missing\n- how important the visual data (images/videos) in the process of bug fixing is unknown", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)