Automated Benchmark Generation for Repository-Level Coding Tasks
Authors: Konstantinos Vergopoulos
Mark Niklas Mueller
Martin Vechev
Venue: ICML 2025 poster (2025)
Keywords: Code Agents
Benchmark Generation
Coding Benchmark
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: We automatically create large and diverse repository-level code benchmarks and show that performance on SWE-Bench is not representative of these more realistic settings.

Abstract: Code Agent development is an extremely active research area, where a reliable performance metric is critical for tracking progress and guiding new developments. This demand is underscored by the meteoric rise in popularity of SWE-Bench -- a benchmark that challenges code agents to generate patches addressing GitHub issues given the full repository as context. The correctness of generated patches is then evaluated by executing a human-written test suite extracted from the repository after the issue's resolution.

However, constructing benchmarks like SWE-Bench requires substantial manual effort to set up historically accurate execution environments for testing. Crucially, this severely limits the number of considered repositories, e.g., just 12 for SWE-Bench. Considering so few repositories, selected for their popularity runs the risk of leading to a distributional mismatch, i.e., the measured performance may not be representative of real-world scenarios running the riks of misguiding development efforts. 

In this work, we address this challenge and introduce SetUpAgent, a fully automated system capable of historically accurate dependency setup, test execution, and result parsing. Using SetUpAgent, we generate two new datasets: (i) SWEE-Bench an extended version of SWE-Bench encompassing hundreds of repositories, and (ii) SWA-Bench a benchmark focusing on applications rather than libraries. Comparing these datasets to SWE-Bench with respect to their characteristics and code agent performance, we find significant distributional differences, including lower issue description quality and detail level, higher fix complexity, and most importantly up to 60% lower agent success rates.

Automated Benchmark Generation for Repository-Level Coding Tasks
Konstantinos Vergopoulos'
Mark Niklas Müller
Martin Vechev
Abstract
1. Introduction
Code Agent development is an extremely active research area, where a reliable performance met- ric critical for tracking progress and guiding new developments. This demand is underscored by the meteoric rise in popularity of benchmark that challenges code agents gen- erate patches addressing GitHub issues given the full repository as context. The correctness of gen- erated patches is then evaluated by executing a human-written test suite extracted from the repos- itory after the issue's resolution.
Code Agents are quickly becoming one the most promis- ing and actively researched applications of Large Language Models (LLMs);; partly due their potential to revolution- ize the 700 billion dollar software industry (Statista). To measure progress and more importantly steer further de- velopments in this field, high-quality datasets and bench- marks are crucial. In particular, it essential that they are representative of real-world use cases, sufficiently large to allow meaningful statistical analysis, and diverse and recent enough to avoid unintentional overfitting and contamination.
However, constructing benchmarks like SWE- Bench requires substantial manual effort to set up historically accurate execution environments for testing. Crucially, this severely limits the num- ber of considered repositories, e.g., just 12 for SWE.-Bench. Considering so few repositories, se- lected for their popularity runs the risk of leading to a distributional mismatch, i.e., the measured performance may not be representative of real- world scenarios running the riks of misguiding development efforts.
Existing Benchmarks However, function-level bench- marks like HumanEval (Chen al., 2021), popular for evaluating LLM's coding performance, are unrepresentative of real-world use, lack diversity, and are becoming satu- rated. To address these limitations, WWE-Bench (Jimenez et al., 2024) was proposed as the first repository-level cod- ing benchmark based on real-world tasks, i.c., resolving GitHub issues. Yet, it still suffers from several limitations. (i) It is limited to few repositories, potentially leading to overfitting to these specific codebases. (ii) Its sole focus on libraries in contrast to applications raises generalizability questions. (iii) Its focus on popular repositories not only makes it less representative but also increases the chances of contamination with general codebase knowledge. (iv) static nature leads to most or even instances being created before recent models' knowledge cutoff, allowing even the exact instances to be present in the training data.
In this work, we address this challenge and in- troduce SETUPAGENT, a fully automated sys- tem capable of historically accurate dependency setup, test execution, and result parsing. Using SETUPAGENT, we generate two new datasets: (i) SWEE-Bench extended version of SWE- Bench encompassing hundreds of repositories, and (ii) SWA-Bench benchmark focusing ap- plications rather than libraries. Comparing these datasets to WWE-Bench with respect to their char- acteristics and code agent performance, we find significant distributional differences, including lower issue description quality and detail level, higher fix complexity, and most importantly up 60% lower agent success rates.
Creating Repesitory-Level Benchmarks To address these challenges, we would like to create benchmarks and update them frequently with new tasks, However, while the GitHub Issues and Pull Requests (PRs), serving as task descriptions and reference solutions, respec- tively, for SWE-Bench--lie benchmarks can be scraped automatically, evaluating the correctness of solution, re- quires the repository's 's test suite be executed. This, in turn, requires setting up historically accurate execution environ- ments, identifying the correct test commands, and parsing the results. Prior work addressed this problem either manu- ally (Jimenez et al., 2024) or by aggressively filtering out instances where default commands were unsuccessful (Jain et al., 2024c). However, both approaches yield limited di- versity and don't 't lend themselves to frequent updates.
diverse
Equal contribution 'LogieStar AI Department of Computer Science, ETH Zurich. Correspondence Mark Niklas Müller <mark logicttaaai.............
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Automated Benchmark Generation for Cspository-Level Coding Tasks
Cmd. Extraction
Outputs
GitHub Repository
Context
Iterative Improvements
Validation
Install Cmds
CV/CD Files
Install Cmds
Install Exec
Parse
Initial Commands
Test
Text Files
Cmds
Check Results
Test Cmds
Test Exec
Online Does
Links
Result
Parser
Rejected
Reference Commands
Figure 1. Overview of SETUPAGENT where -iiio represents an LLM driven step and -icon represents execution feedback.
This Work: SETUPAGENT To address this challenge, we propose SETUPAGENT, the first method to automate this setup process, enabling us to create repository-level code benchmarks fully automatically from list of GitHub repositories. SETUPAGENT works in three key phases (il- lustrated in Figure I): (i) Command Extraction (green in Figure 1), (ii) Iterative Testing and Improvement (blue m), and (iii) Validation (purple 1). In the extraction phase, SE- TUPAGENT analyzes relevant context, such as README. md files, CI/CD configurations, and referenced web pages, to propose installation and testing commands. During the itera- tive improvement phase, SETUPAGENT then executes these commands in clean environment and leverages LLM to systematically diagnose and resolve issues. Finally, in the validation phase, SETUPAGENT ensures that the generated commands are reliable by verifying the correctness of the setup based on test results, only accepting configurations that meet a predefined success threshold.
Key Contributions of this work are:
We propose SETUPAGENT, the first method for au- tonomously creating historically accurate execution environments.
We leverage SETUPAGENT to create two datasets for repository-level code generation SWA- and SWEE- Bench, focusing on applications and diverse projects, respectively.
We extensively analyze SWA- and SWEE-Bench in terms of their characteristics and corresponding code agent performance.
2. Related Work
Code Agents To fully leverage the potential of LLMs for code generation, they have been equipped with tools to interact with their environment without additional user input, e.g., by searching, viewing, and editing code, (Wang et al., 2024a). These so-called code agents have shown great promise on complex tasks (Bouzenia et al., 2024a; OpenDevin, 2024; Zhang al., 2024; Yang et al., 2024b; Xia et al., 2024; Aider, 2024; Ridnik et al., 2024; Wang et al., 2024b). In this work, we evaluate some of the best- performing open-source agents.
This Work: Generated Benchmarks We demonstrate SETUPAGENT's capability to generate coding benchmarks from list of repositories by creating SWA- and SWEE- SWE-
Bench, each addressing specific shortcomings Bench. Both are designed to be representative of real-world use cases, consider many repositories leading to diverse benchmarks, and can be frequently updated without manual effort to avoid contamination and overfitting, SWA-Bench focuses on software applications. containing 44 projects while SWEE-B -EE-Bench focuses on diversity and less popular projects containing 366 Python repositories. Comparing SWA- and SWEE-Bench to SWE-Bench, we find signifi- cant distributional differences, including lower repository age and popularity at issue creation, larger focus on recent issues, and significantly more complex reference code fixes (2-4x more modified files and lines). Evaluating popular code agents on these datasets, we find significant perfor- mance differences for some models and statistically signifi- cant signs of contamination, highlighting the importance of evaluating on representative benchmarks.
Code Generation Benchmarks With the success of LLMs in the domain of code generation, an increasing va- riety of function-leved code generation benchmarks were proposed to assess their capabilities (Chen et al., 2021; Hendrycks et al., 2021; Austin et al., 2021; Jain et al., 2024a: Huang al., 2024). However, not only were these increas- ingly saturated by state-of-the-ar models but their focus interview-style function-lewe coding challenges makes them also unrepresentative of the complexities of real-world codebases and software engineering tasks.
To address these limitations, a range of repository- code-gen sede-generation benchmarks have been proposed recently (Liiu al., 2023; Jain al., 2024b; Jimenez al., 2024).
2

---

Automated Benchmark Generation for Repository-Level Coding Tasks
However, repository-level context not only makes code generation but also dataset generation more challenging it requires a historically accurate execution environment to be set up, the project's s test suite to be run, and detailed results to be extracted. The required manual effort led to existing datasets focusing on a relatively small number of popular repositories. As a result, they are prone to overfitting, often lack diversity, and can easily contaminate the training data.
3.1. Notation and Definitions
We first introduce notation to describe repository-level cod- ing tasks, adapted from Mündler al. (2024). Given codebase R. we obtain Ro X by applying the code patch X. We similarly denote the test suite T with T o S after apply- ing the test patch S. A single test / E T can either pass (P) or fail (F) when executed against the codebase R in an ex- ecution environment E. We write: exec (t, R) E { F)
Automatic Dataset Generation These challenges could be addressed via automatic dataset generation, which has been successfully applied to function-level benchmarks by seraping tasks from coding challenge websites and doing varying levels of manual post-processing (Hendrycks et al., 2021: Jain et al., 2024a; Huang et al., 2024).
and let the order P > F hold.
A repository-level coding task can be written the tu- I, ple (R, TI, E, 8 x**,, where R and T are the original codebase and test suite, respectively, I is the issue de- scription, F the execution environment, and 5 and X* the reference test and code patch, respectively. By ex- ecuting all tests 4, E T o S* in the execution environ- ment E, first against the original (R) and then the patched codebase (R o X*), we obtain the reference test behavior bj = (exec (ts, Rt + execg(ti, Ro X*)) We call 4; with, e.g., 69 = F + P a fail-to-pass test as it fails before the reference fix is applied but passes afterward. We let the partial order F F > F F and P-P> P-F hold.
Jimenez et al. (2024) transfer these ideas to repository-level benchmarks, automatically scraping GitHub repositories, is- sues, and pull requests resolving these issues to create SWE- Bench consisting of 12 repositories and 2294 instances. However, they still created the required execution environ- ments and test commands manually. Further, the resulting issues were shown to suffer from underspecified descrip- tions and overly specific tests (Chowdhury et al., 2024).
The task is now to generate a patch X', given only (R, such that the test behavior b9 execg(t; R) + exec R o X') matches or improves on the reference result, i.e., b, 2 bg for all tests I; E S
Jain et al. (2024b) create R2F, function-level synthe- sis benchmark with repository context by scraping GitHub repositories and masking out the function to be generated. They automated the setup by applying a default approach for projects with a aetupppp or pyproject, tomm file, au- tomatically generating equivalence tests, and filtering out all instances where this approach fails. However, this approach aggressively filters projects with more complex installation procedures, not only introducing selection bias but also yielding only 246 instances.
3.2. Setup Agent Requirements
A generic setup agent targeting individual, up-to-date reposi- tories only has to satisfy one main requirement: Correctness - It must extract and run the installation and testing com- mands before parsing the test results. However, benchmark generation, i.e., generating the execution environment E given the remaining components of a coding task, imposes additional requirements: Historical Accuracy Benchmark instances are based specific, often outdated versions of codebase R. The execution environment E must thus use historically accurate dependency versions to reproduce the original issue faithfully and avoid version incompatibilities. Efficiency - To generate a dataset of many hundreds of in- stances, the setup agent must be efficient enough to keep total runtime reasonable (hours or at most few days). Gran- ularity - Evaluating agent success requires test- results to be parsed from the test suite output.
In this work, we combine the more interesting repository- level tasks with a fully automated benchmark generation process, by introducing and leveraging SETUPAGENT to automatically extract the installation and testing procedures for every task instance, allowing us to create larger and more diverse benchmarks efficiently.
Bouzenia & Pradel (2024), concurrently proposed EXECU- TIONAGENT, a tool to automatically set up and test reposi- tories. However, it is 60 times slower than SETUPAGENT, does not support historical states, and does not extract results at test-level granularity. Even if the latter two shortcomings were addressed, it would remain infeasibly slow taking, e.g., over 4 months to generate SWEE-Bench
3.3. SETUPAGENT
Overview SETUPAGENT works in three phases illustrated
3. Autonomous Environment Setup
in Figure 1: (1) Extraction (m in Figure 1), (2) Iterative testing and imporvement the first phase, SETUPAGENT extracts first version of the installation and testing commands from all relevant files, ref- erenced webpages, and, if available, successful commands
and (3) Validation (m). In
In this Section, we first outline the requirements for a setup and testing agent to be used for benchmark generation and then describe the agent we develop for this purpose.
Extrappllted from ~ 155 repositories.
3

---

Automated Benchmark Generation for Repository-Level Coding Tasks
from similar versions of this repository. In the second phase, SETUPAGENT iteratively executes first the installation and then testing commands, analyses the results and updates the commands. Finally, in the third phase, SETUPAGENT vali- dates the resulting commands by executing them, extracting the test results, and rejecting the proposed commands, if too few tests pass. Validated commands are then returned the user and saved in a reference database facilitate installations of different versions of the same repository.
Input:
You trying install and test <project_neme> using the following commands. " rrbaa
<commands>
...
Modify the commanda to reselve the following error: ...
Modul FootFoundrrrrr No module named 'rustworkx'
..,
Reaponse:
We Lave to install rustwerks first: rrbah
Input:
pip inatall zustworkx
Please extract all commands required install sproject__rane
clean envi ronment and its test suite from the context below.
Updated Installation Commands:
apt gez instell y graphviz
...
uv pip install -I req.txt exclude-rewer 2022-04-11 uv pip install rustworkx exclide-newer 2022-04-11
** READM2.nd
<file_content>
## CONTRTRUTTNG.
Figure 3. Illustration the iterative improvement phase of SETU- PAGENT, where the message was obtained by executing the commands from the previous iteration.
<clle_content> ..,
LIM Response:
"baahh
apt-get install -y graphviz pip install -r req.txt
instellation installation testing
commands fail with an error, SETUPAGENT selects the rel- evant sections of the error messages and then prompts an LLM identify the issue and propose a modification of the installation commands to fix it. We illustrate this in Figure 3, where SETUPAGENT adds a missing dependency. Now SE- TUPAGENT tries to resolve issues apparent in the test results, analyzing the logs to determine whether a failure due to incorrect test commands, an incorrect installation, or a bug in the codebase. Depending on the result, SETUPAGENT prompts LLM to modify the installation testing mands to fix the issue passes the repo to the validation phase. This iterative improvement is repeated until all er- rors are resolved or an iteration limit is reached. Using a moderate iteration limit of 4 steps, we achieve significantly improved Correctness without sacrificing Efficiency.
nox -e test
Installation Commands:
opt-get install -y graphviz
uv pip install -r txq.txt -exclude-newes 2022-04-11
Testing Commands:
nox -e test
Figure Illustration of the extraction phase SETUPAGENT. Please see App. B for the full-length prompts.
Extraction Phase The goal of the Extraction Phase is to extract an initial version of the installation and test- ing commands. We illustrate it in Figure 2, using a sig- nificantly simplified prompt and shortened response for illustration purposes. In the first step, SETUPAGENT views all relevant files to identify the correct Python version. If available, it is directly parsed from the setup. py or pyproject. 1omp toml file, otherwise version constraints are extracted from different sources using an L.M and then resolved while taking issue creation time into account to en- sure Historical Accuracy. Next, SETCPAGENT uses heuris- tie to identify possibly relevant context like README. md and CONTRIBUTING.. mdd files, CI/CD configurations, and referenced webpages. It then reviews these sources and directly rejects those not containing information relevant to installation or testing, thus keeping its context size small and improving Efficiency. Finally, SETUPAGENT prompts an LLM to extract the installation and testing commands from the chosen sources. To ensure Historical Accuracy, SETUPAGENT enforces the use of the environment ager for Python dependencies, configuring it to exclude dependency versions released after the issue creation.
Input:
asseas whether sproject_nane inatalles and its test suite executed correctly given the resulting printout.
Answer YES or NO.
2597 passed, 3 failed in 10.
Response:
YES
Figure 4. Illustration of the first step the Validation phase.
Validation Phase the validation phase, SETUPAGENT first queries an LLM to assess whether the installation and testing were successful, illustrated in Figure If the LLM judges the installation be successful, SETUPAGENT up- dates the test framework configuration return test- results, e.g., by adding - to pytest command, thus ensuring Granularity. It then selects the correct parser from a pre-defined set to extract test-level results and checks the number of passing and failing tests. We consider the installation to be successful if at least 95% of tests pass.
Iterative Improvement Phase SETUPAGENT now tries to install and test the project in a clean Docker container using the extracted commands. If any of the installation steps fail, i.e., exit with a non-zero exit code, or the test

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
SWE
SWE
1.0
SWA
SWA
SWE
SWEE
SWEE
0.5
SWA
SWEE
0.0
2013-00
2019-00 01
2025-01
0
4000
8000
0
40000
80000
Iastte Creation Date
Repo Age [Days]
# Gitllub Stars
Figure 5. PDFs (left and middle) and CDF (right) of PR creation dates (left), repository age PR creation time (middle), and number of GitHub stars (right) for SWA, SWEE, and SWE-Bench.
4. Code Generation Benchmarks
Table 1. SWEE pipeline from projects to tasks. A PR is valid if it resolves an issue, modifies a test file, and merged. An instance valid, if it has additionally at least one F test.
In this Section, we describe how we leverage SETUPAGENT to create SWA- and WWEE-Bench, two new benchmarks addressing specific limitations of SWEE WWE-Bench. We compare these datasets with SWE-Bench and provide insights into distributional differences.
Step
Repos
PRs
Initial Projects
8000
GH Repo Found
7057
+ Preprocessing
5097
Automatically Generated Benchmarks By creating ex- ecution environments automatically, we address two core limitations of manually generated repository-level bench- marks: (i) we can consider many more repositories without requiring infeasible manual labor, thus improving diversity and reducing the risk of overfitting and (ii) we can easily update benchmarks by creating new tasks from recent PRs and issues, thus ensuring that models are not contaminated with benchmark instances (see Figures 5 and 6).
Permissive License
3800
+ Has valid PR
2377
+ SETUPAGENT succeeds
514
+ Get Reper_repo valid PRs
2115
+ SETUPAGENT succeeds
1513
+ valid instance
885
downloaded PyPi projects at the time (van Kemenade et al., 2024) with between 100k and 1.5B monthly downloads and O to 25k stars, leading to good diversity while focusing on relevant projects.
SWA-Bench Many practitioners using Code Agents de- velop software applications that suffer from different types of bugs compared to libraries due to architectural and struc- tural differences. As SWE-Bench only considers libraries, we design SWA-Bench to focus only on applications.
Dataset Creation with SETUPAGENT We combine the original PR filtering process from Jimenez et al. (2024) with our SETUPAGENT follows: For every project, first locate the corresponding repository, deduplicate the results, and filter out repositories that are not published under a permissive license. We then scrape issues and pull requests for each repository until we find the most recent PR that is merged, resolved an issue, and modified a test file. We call this valid PR. We then use SEtUPAGENT to set up an execution environment F for the corresponding codebase R (see Section 3). For repositories where this succeeds, we scrape additional PRs until we have R repo valid ones or. for SWEE, reach a maximum of 500 PRs. We then use SEtUPAGENT to create the execution environment for each corresponding codebase R in reverse chronological order per repository, populating SETUPAGENT'S reference commands database to speed up the setup process. Finally, split every PR into reference code patch X* and test patch S* We execute the full test suite T S* before and after the code patch is applied, i.c., on R and R o X*, respectively, to obtain the reference test behaviors by We then filter out PRs, where test execution fails in one of these settings or which have no F - P test, i.e. At
SWEE-Bench We observe that more popular repositories tend to have higher-quality codebases and issue descrip- tions. This includes, e.g., more consistent (file) structure and naming conventions, better documentation including de- tailed docstrings for most functions, and issue descriptions following a precise template (see Figures and 8). As SWE- Bench focuses on particularly popular Python repositories, the resulting tasks can be unrepresentative of real-world design SWEE-Bent with focus
use. Therefore, diverse and less popular (median of 365 vs 16k stars) Python repositories (see Figure 5).
4.1. Dataset Creation
Source Repositories For SWA-Bench, we combine list of 468 popular Python applications (Hashemi, 2024) with a list of 50 Python projects from Bouzenia et al. (2024b), leading to a total of 475 candidate repositories after dedu- plication. For SWEE- EVE.-Bench, we consider the 8000 most
5

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
To S" ex R) execg(l, Ro X*) P.P. The remaining PRs form the valid instances of the generated benchmark. We choose Plperrrepe preoo 50 for SWA-Bench and Repe_repo = 10 for SWEE-Bence to obtain the desired number of tasks and show the number of repositories and PRs this leads to in Tables and 10, respectively.
Table 2. Comparison of mean dataset characteristics.
SWA
SWEE
SWE
# Files
899
77
1491
Codebase
Lines
112k
14.8k
32lk
# Words
240.2
125.1
181.3
Issuc Descriptions
# Error Messages
0.20
0.13
0.19
# Code Blocks
1.53
1.19
1.06
Ease of Use To make benchmark generation and use as easy as possible, SETUPAGENT only requires a list of repositories to generate dataset in a format compat- ible with SWE-Bench along with docker images with all dependencies installed. We publish SWA- on HuggingFace and the corresponding docker containers at logicstarai/swa-eet bewch. A suitable evaluation harness is avail- able at www./oooimstarr-iliiiiooi
# PPP+P # F P # F-F
564.2
226.6
120.1
38.8
38.1
13.5
Tests
3.7
1.4
3.4
P
0.11
0.03
0.04
# Edited Files
1.89
2.05
1.52
# Edited Lines
74.8
91.5
39.2
Test Patches
# Added Tests
9.10
23.78
6.37
Removed Tests
16.77
2.49
0.54
# Edited Files
3.26
3.26
1.66
Fix Patches
# Edited Lines
104.3
169.9
41.0
SWFF
SWW
1.0
0.5
SWE
DLS
SWE
SWT
SWEE
SWEE
Reposberr Proportins
Figure 6. Comparison of the repository distribtuion of SWEE- SWA-, and SWE-Bench across instances.
Words lane Desc.
ClddbMooks
Urro Nissages
1.00
SWF
SWF
SWA
SWA
0.85
4.2. Benchmark Characteristics
SWEE
SWEE
SIVE
0.50
Diversity We compare the distribution of instances over repositories in Figure and observe that while instances in SWE are heavily concentrated in only a few repositories, with over 50% of instances belonging to only two out of 12 total repositories, SWA- and SWEE-Bench show much more diversity with 535 instances from 44 repositories and 885 from 366 repositories. See App. C for full list.
Choraizad
Lemest Ovedaapeu Striex
Figure 7. CDFs over issue description characteristics. Number of words (top left), number code blocks (top middle). number error messages (top right), number of filenames contained in the issue description and modified in the reference solution (bottom left), the overlap hetween the issue description and the reference solution terms of longest string match (bottom middle) and complete lines (bottom right). A CDF further down and to the right indicates a higher value.
Codebase Characteristics We compare benchmarks with respect to codebase characteristics in Table 2 and Figure 5 and observe that SWE-Bench, compared to, SWA- and especially SWEE-Bench contains significantly older and more popular (# GitHub stars) repositories and larger, more complex codebases (# files and # lines of code).
observe longer issue descriptions and slightly more overlap with the reference solution in SWE-Bench but otherwise similar characteristics.
Fix Complexity To assess the complexity of required fixes, we measure the number of lines and files modified in the reference solution and the number of tests that flip from passing to failing (and vice versa). We show CDFs in Fig- ure and observe that while SWEE- and SWA-Bench have similar distributions across all these metries, SWE-Bench fixes are significantly less complex by all metrics.
Issue Description Quality To assess the issue description quality, we measure the number of words, еттor messages, and code blocks they contain as well as the overlap between the files mentioned there and modified in the reference fix and the overlap between the issue description and the ref- erence solution itself. We show cumulative distribution functions (CDFs) of the aforementioned characteristics in Figure 7. We observe that while WWA-Bench has more detailed issue descriptions (longer, more code blocks, and more error messages), they do not seem to be of higher qual- ity (less overlap with the reference solution and equal file mentions). Comparing SWE-Bench and SW W.EEE--Beee we
4.3. Manual Review
While SETUPAGENT ensures that at least 95% of tests pass, some quality issues may still remain. To assess their fre- quency and impact, we conduct a manual review focusing
6

---

Automated Benchmark Generation for Repository-Level Coding Tasks
1.0
1.0
both 0 and I are considered acceptable.
65
SWE
0.5
BWB
We manually review the same 30 instances above and observe the following: All instances run the correct tests with 23 (77%) using exactly the same test commands as provided in the reference. 22 (73%) of these instances additionally have a fully correct installation/setup. These results indicate that SETUPAGENT has no trouble extracting the correct testing steps, with the setup proving slightly more challenging but also forgiving considering that 95% of tests passed despite minor setup errors.
SWA
SWTR
SWEE
SW022
000
DCC
003 Fioe Fix
Linas
Plopell Tinte
Figure& 8. CDFs over fix-compexiity characteristics. Number of edited lines (left), number of edited files (top middle), number affected tests, i.c., P+ P- (right). A CDF further down and to the right indicates higher characteristic values.
on two types of issues: (i) dataset quality along the lines of SWE-Bench Verified (Chowdhury et al., 2024), i..., whether the issue is sufficiently well-specified to be resolved with- out further information and whether the tests are suitable to check its resolution and (ii) setup success, i.e., whether SETUPAGENT was able to set up the environment correctly and run the tests expected.
5. Experimental Evaluation
In this Section, we first evaluate the effectiveness of SETU- PAGENT for dataset creation and then analyze Code Agent performance across datasets.
5.1. Experimental Setup
Dataset Quality In line with prior work (Chowdhury et al., 2024), we focus on two main aspects: (i) task specificity, i.e., whether the issue is sufficiently well-specified to resolve it without further information and (ii) test quality, i... whether the added unit tests check for the described behaviour or are overly specific to concrete implementations such as exactly matching error messages. We assess both of these criteria on a scale from 0 to 3, where is the best score and the worst, using the same annotation guide as Chowdhury et al. (2024). That is, we scored issue specificity and clarity on a scale of O (well-specified issue with clear success criteria) to 3 (almost impossible to solve correctly without further instructions) and test quality from 0 (test perfectly covers valid solutions) to 3 (tests are too narrow or broad or requiring information not provided in the issue description). On both scales, 0 and 1 are considered acceptable, while 2 and 3 are insufficient.
Models We consider a range of models across sizes, cost points, and model providers. For exact versions, Table 9 in App. A. Unless otherwise specified, we use GPT-4o- MINI as the underlying model for all agents. For decod- ing, we use the default parameters for all Code Agents and greedy decoding for SETUPAGENT.
Code Agents We evaluate three state-of-the-art Code Agents from the top of the SWE-Bench leaderboard which most likely have been optimized for SWE-Bench (Open- Hands (Wang et al., 2024b), AutooCodeRover-v2.0 (Zhang et al., 2024)), and SWE-Agent v1 (Yang et al., 2024a) and ZeroShot (Jimenez et al., 2024) with oracle context (files modified in the ground truth fix) and BM25 retrieval which prompts LLMs directly without any optimization for SWE- Bench. We report the portion of resolved instances as accu- racy (Acc.) for all Code Agents.
We conducted manual review of 30 randomly chosen SWA instances and observed the following: 21 (70%) instances have a meaningful and sufficiently complete issue descrip- tion, and 20 (67%) of these additionally have suitable tests (27 or 90% across all instances) to check whether the issue was fixed. This in line with the results of Chowdhury et al. (2024) and shows that the majority of issues are solve- able, with current performance levels still leaving significant room for improvement, making SWA a suitable benchmark for current and future code generation systems.
Code Execution We run all code execution (both for SE- TUPAGENT and all Code Agents) in separate Docker con- tainers to improve reproducibility and security. For SETU- PAGENT, we use an Ubuntu 22.04 container as the base image and pre-install a range of common build dependen- cies but do not provide any Python dependencies.
5.2. Effectiveness of SETUPAGENT
Setup Success To validate the automated assessment used by SETUPAGENT to determine whether an instance was set- up correctly, we assess both the extracted setup and testing steps as follows. (i) We score the setup on binary scale of 0 (correct setup that is functionally equivalent with the described setup and 1 (incorrect setup). (ii) We score testing on a scale from 0 (functionally equivalent to described test- ing) to (tests only partially or not at all executed), where
We evaluate the effectiveness of SETUPAGENT in creating S SWA- and SWEE-Bench by analyzing the frequency of fully successful environment and testing setups in Table 3. We observe SETUPAGENT is able to extract historically correct execution environments for 20-30% of repositories without reference commands and for 55-75% of instances
swebench.com accessed in November 2024
7

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Table SETUPAGENT success rates extracting installation and test commands as well as parsing the resulting test output.
Table 5. Issue resolution rates (accuracy) of various agents on SWA- SWEE-, and SWE-Bench, all with GPT-40-MINI
Success
SWA
SWEE
SWE
Repos Instances
28.6%
Openhands
3.9%
4.4%
4.6%
SWA
AutoCodeRover v2
8.4%
9.0%
8.2%
58.5%
SWE-Agent 2.6%
7.5%
7.1%
Repos 21.6%
7eroShot(Oracle)
0.9%
2.2%
2.8%
SWEE
Instances
71.5%
ZeroShot(BMM5)
1.3%
2.8%
1.5%
surprisingly small differences in performance between all three datasets when using GPT-40-MINI for most agents, with SWE-Agent performing significantly worse on SWA.
Table 4. Ablation study on SETUPAGENT, reporting the number of successfully extracted execution environments for WWA-Bench.
# Repositories
To assess the interaction of agent performance and model selection, we evaluate AutoCodeRover v2 (Zhang et al., 2024) across a range of LLMs, showing results in Table 6. Interestingly, we observe large variance in the accuracy difference between SWE and SWA across models. While GPT--0-MINI performs similarly well on all benchmarks, all other models perform much better SWEE- and even better on to lower performances for instances created after the models knowledge cutoff.
SETUPAGENT
44
only CI/CD Files
33
only Text Files Iterative Improvement
15
11
for these repositories. Without reference commands, SE- TUPAGENT takes 76 minutes to attempt install all 154 repositories considered for SWA after deduplication and license checks and thus takes only about 30s on average per repository. When creating SWEE-Bench, we deactivate the web browsing ability of SETUPAGENT.
We show later that this may be due
Table 6. Performance of Rutocode v2 (Zhang al., 2024) using different underlying LLMs.
Ablation We evaluate the impact of SETUPAGENT's com- ponents in an ablation study on SWA-Bench, reporting results in Table 4. We observe that especially the use of CI/CD config files and the iterative improvement are crucial for SETUPAGENT's success.
SWA
SWEE
SWE
GPT-40-MINI
8.4%
9.0%
8.2%
GPT-40
10.2%
15.1%
16.6%
HAIKU-3.5
10.8%
12.9%
13.0%
LLAMA 3.3 70B
8.8%
10.8%
12.5%
QWEN2.5
356
256
43
DEEPSBEK V3
858
13%
20%
Failure Analaysis To understand SETUPAGENT's failure cases, we conduct small case study, manually inspecting five failed instances from SWA-Bench, and observe the fol- lowing: In all instances, errors in the build process cause the failure. For all but one instance, finding the installation instructions requires following two or more links on web pages. In all but two instances, the only described way to test the application requires running docker containers, which SEtUPAGENT does not support. In two instances, installa- tion and/or testing requires the use of makefiles, referencing multiple substeps. Finally, in one instance SETUPAGENT chooses the wrong requirement file and then begins to in- stall missing testing dependencies. We believe this points to exciting future work improving SETUPAGENT'S web- browsing capabilities and docker support.
Evallated on random instances.
5.4. Benchmark Analaysis
In Section 4, we observed interesting distributional dif- ferences between the instance characteristics of SWA- SWEE- and SWE-Bench. Now, we explore how these characteristics correlate with agent performance, reporting Spearman's 's rank correlation coefficients p and p-values for Rover v2 and GPT-40 in Table 7. We observe that only characteristics computed with knowledge of the solution have statistically significant correlation with per- formance. In particular, the overlap of the issue with the reference code patch in terms of file names, and number of lines has a strong positive correlation with performance, while all fix complexity metries have a strong negative cor- relation with performance.
5.3. Agent Performance Across Datasets
We conduct all below experiments on the full SWA and uniformly subsampled versions of SWEE and SWE-Full of identical size (535 instances) due to cost constraints.
Data Contamination We analyze the accuracy (Acc) of AutoCodeRover v2 on SWA- and SWEE--Bnnche depend- ing on whether a PR was created before or after a model's s knowledge cutoff (KC), showing results in Table 8. We re-
We report Code Agent performance in Table 5 and observe
8

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
We extensively analyzed SWA- and SWEE- observ- ing significant distributional differences compared to SWE- Bench in fix-complexity characteristics that are strongly correlated with agent success. We further found statisti- cally significant performance degradation for SWA-Bench instances created after the knowledge cutoff for one model. Together, these findings highlight the importance of eval- uating on diverse, representative, and frequently updated benchmarks and thus the value of our automated bench- mark generation approach. We believe SETUPAGENT can facilitate this by enabling practitioners to quickly turn their specific target domain into a high-quality representative benchmark.
Table Spearman's 's rank correlation coefficients p and p-value be- tween accuracy and instance characteristics, separated by whether statistic can be computed without axes to ground truth. Statistically significant (p < 1%) correlations are highlighted in bold. Positive p indicate that a larger characteristic value is associated with better performance.
SWA
SWAE
SWR
Charaterattic
p-value
p-ralue
p-valoe
Repo
2.0
5.5
1.0%
10"1
I1-3
1.3% 10-1
Words in Issue
188% 10-1
9.6 1011
1 10-1
Cisie Blacks
0.7 10--
3.1 10-1
-0.55
1.2% 10-1
Emmor Messages loue
0.00
x6x:10 10
0.08
3.7 x 10
00.0
x 10
Fix File Names Issse
0.12
x 10m!
0.19
12
0.18
2 20-x
Longest Fix Salstrin in Ioouz
-0.04
8.7 10
-0.11
1. 10
0.04
8.1x 10
the Lines in lone
0.00
37 10m2
0.06
1.7 10"3
I.17
1:1 10-4
Lines in Fix
-0.25
3,0 x 10-11
-0.40
1,3 10
40.28
5.2
13-11
Filzs in Fix
-0.12
6.2 10-1
-0.26
1.6
10-1
-0.16
14 10-4
Affected Teuts
33xx
0.0
7.2
Impact Statement
Table 8. Accuracy of AutocodeRover v2 (Zhang et al., 2024) on SWA-Benc instances split between those created before and after the model's 's knowledge cutoff (KC) and the p-value of the underlying resolution rate being the same or higher after the KC
This paper presents work advancing Code Agent evaluation and may thus amplify all positive and negative societal im- pacts of improved Code Agents. Our work shows that eval- uating Code Agents on diverse and up-to-date benchmarks is critical to obtain representative results, with SWE-Bench (and even more SWE -VERIFIE) consisting of unusually easy problems. These findings may relativize some recent predictions of code agents soon replacing human software developers and show that the field is still far from achieving this goal. Beyond this, our work on automated execution environment setup has the goal of advancing the field of Machine Learning for code more generally. There are many potential additional societal consequences of our work, none of which we feel must be specifically highlighted here.
Dotaset
Model
aateer KC
Acc before KC
Acc after KC
p-valuc
GPT-40-MINI
249
9.4%
7.2%
17.90%
SWA
GPT-40
249
12.2%
7.2%
2.65%
HAIKU-3.5
44
11.0%
9.1%
34.83%
GPT-40-MINT
230
8.2%
10.0%
76.50%
SWEE
GPT-40
230
18.4%
15.2%
47.56%
HAIKU-3.5
102
13.6%
9.8%
15.01%
port the (one-sided) p-value of observing these results under the null hypothesis that the success rate is not lower after the KC (computed using a t-test and normal approximation of the binomial distribution). We observe that on all considered models have a lower success rate after the KC with the difference being statistically significant only for GPT-40. Interestingly, SWEE-Bench which contains much less popular projects and is thus less prone to contamination. While all SWE instances are too old to conduct a similar amalysis, we ob- serve that the performance delta between SWE and S SWA is correlated with the drop accuracy over the KC on SWA.
Acknowledgements
observe such signs
We would like to thank the anonymous reviewers for their valuable feedback and suggestions, which helped improve the quality of this paper.
6. Conclusion
We introduced SETUPAGENT, the first method for auto- mated and historically accurate execution environment setup for Python codebases. SETUPAGENT enables us to cre- ate repository-level code benchmarks fully automatically from a list of GitHub repositories. We demonstrated its effectiveness by creating two new benchmarks, SWA- and SWEE-Bench, focusing on applications and diversity of codebases, respectively, and addressing several limitations of existing repository-level code benchmarks. In particular, their automated generation allows us to consider many more repositories, increasing diversity and reducing the risk of overfitting, and update the benchmarks over time, minimiz- ing the risk of data contamination.
9

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
References
Hashemi, tions.
M.
Awesome python applica- https://githhiie com homm/mahoon mmahmudd/
Aider. Aider is SOTA for both SWE Bench and SWE Bench Lite, 2024.
awesome python- aatitoonapplicaiionnn.. 2024.
Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Mcasuring coding challenge competence with APPS. In Proceedings of the NeurlPS Track on Dalasets and Benchmarks, 2021.
Anthropic. 3.5 haiku net. cd cotp:///wccoo000000336666////110
Model
card
addendum:
Claude 3.5 son-
and upgraded
claude
Www--MMoed--dd----------- Adderduuu. pdf, 2024.
Claude-3
Huang, Y., Lin, Z., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., and Chen, W. Competition-level problems are effective LLM evalua- tors. In Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, 2024. doi: 10.18653/V1/ 2024. FIA.IININNNS-.......
Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021.
Bouzenia, I. and Pradel, M. You name it. run it: An
LLM agent execute tests of arbitrary projects. CoRR, abs/2412.10133,, 2024. doi: 10448550/AXXI.2412. 10133.
Jain, N. Han, K., Gu, A., Li, W., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Live- codebench: Holistic and contamination free evaluation of large language models for code. CoRR, 0122244030777444 2024a. doi: 10.48550/ARXI.2403.000
Bouzenia, I., Devanbu, P. T., and Pradel, M. Repaira- gent: An autonomous, Ilm-based agent for program repair. CoRR, 2024a.
Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2E: turning any github repository into programming agent environment. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024b.
Bouzenia, I., Krishan, B. P.. and Pradel, M. Dypybench: A benchmark of executable python software. Proc. ACM Softw. Eng.. 1(FSE), 2024b. doi: 10. 10.1145/3643742.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M.. Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L.., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J.. Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M.. Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code.
Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, I. R2e: Turning any github repository into a programming agent test environment. In ICLR 2024, 2024c.
Jimenez, C. Yang, J.. Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In Proc. of ICI.R, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., al. Deepseek-v3 technical report. ArXiv preprint, abs/2412.19437, 2024.
Liu, T., Xu, C., and McAuley, J. J. Repobench: Bench- marking repository-level code auto-completion systems. CoRR, Abs/2306.03091, 2023. doi: 10.48550/ARXIVV 2306.03091.
ArXiv preprint, abs/2 03374, 2021.
Chowdhury, N., Aung, J., Shem, C. Jaffe, O., Sherburn, D., Starace, G., Mays, E., Dias, R., Aljubeh, M., Glaese, M., Jimenez, C. E., Yang, J., Ho, L., Patwardhan, T., Liu, K., and Madry, A. Introducing SWE-bench verified, 2024.
Mündler, N.. Mueller, M. N., He, J., and Vechev, M. Swt- bench: Testing and validating real-world bug-fixes with code agents. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama herd models. ArXiv preprint, abs/2407.2178., 2024.
OpenAI. Openai model docs. https:/ httlss:/plaffootto openai coamai..cm/daee saen/docs/moddele gpt- 4o, 2025.
OpenDevin. Opendevin: Code less, make more, 2024.
10

---

Automated Benchmark Generation for Repository- Coding Tasks
Qwen Team. Qwen32.5 technical report. ArXiv preprint, abs/2412.15115, 2024.
Ridnik, T., Kredo, D., and Friedman, Code genera- tion with alphacodium: From prompt engineering to
flow engineering. CoRR, abs/2401.08500 2024. doi:
10.48550ARRIVV.200000555
Statista. Statista market insights.
https:
/wwww, statista................. wwwm///tlooo/ttttooooooooooooooe software/worldwiee
van Kemenade, H., Paterson, C., Thoma, M., Si, R., and Dol- lenstein, Z. hugovk/top-pypiiaeee papi-aaca Release 2024.12,
2024.
Wang, L.,, Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J.. Chen, X., Lin, Y., Zhao, W. X., Wei, Z., and Wen, J. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 2024a.
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li. B., Singh, J.. Tran, H. H., Li, F,, Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig. G. OpenHands: An Open Platform for AI Software Developers as Generalist Agents, 2024b.
Xia, C. S., Deng, Y., Dunn, S., and Zhang, L.. Agentless: Demystifying based software engineering agents. CoRR, 0bss22407001883 2024. doi: 10.48550/ARXIV. 2407.01489
Yang. J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing. CoRR, abs/2405.15793, 2024a. doi: 10.48550/ ARXIV.2405.15793.
Yang. J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. SWE-agent: Agent Com- puter Interfaces Enable Software Engineering Language Models, 2024b.
Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. Au- tocoderover: Autonomous program improvement. CoRR, abs/2404.05427,, 2024. doi: 10.4850/AAXII:24000 05427.

---

Automated Benchmark Generation for Repository-Level Coding Tasks
A. Appendix: Experiments
Below, we provide the exact model versions we used Table
Table 9. LLM Details inlcuding Knowledge Cutoff (KC)
Model Name
Model
API Provider
KC
Reference
GPT-40
upt-4o-2024-00-12
OpenAl
Oct 2023
OpenAl (2025)
GPT-4O-MINI
9P2-1B--aaaa-2224-07--8
OpenAl
OpenAI (2025)
HAIKU-3.5 3.5
clauss------hakkk--220-100
Anthropic
Jul 2024
Anthropic (2024)
DREPSERK
neepSeek-V3
TogetherAl
Liu (2024)
LLAMA 3,3 70B
TogetherAl
Dec 2023
Grattafiori al. (2024)
QWEN2.5
TogetherAl
Qwen Team (2024)
A.1. Ablations
Repeatability While benchmarks need only be constructed once
Table 11. Repeatability of SETUPAGENT
and thus the repeatability of SETUPAGENT has no impact on the value and repeatability of benchmark's it creates, repeatability is an important aspect for the usability of such a system. Therefore, we conducted an experiment running SETUPAGENT three times on 100 random candidate repositories from SWEE We show results in Table 11, where report the number of successfully installed repositories and the symmetric difference in installed repositories compared to the first run. We find that SETUPAGENT is highly repeatable, with only one repository being installed in the first and second run that was not installed in the third run.
Run
Successes
Symmetric Difference Run
27
3
26
setup like for Table 5.
B. Appendix: Prompts
In this Section, we provide the full-length prompts used by SETUPAGENT.
12

---

Automated Benchmark Generation for Repository-Level Coding Tasks
Table 10. SWA pipeline from projects tasks. A PR is valid if resolves issue, modifies file, and is merged. An instance valid,
it has additionally at least one / - P' test.
Step
Repos
PRs
Initial Projects
475
+ GH Repo Found
440
+ Preprocessing
427
+ Permissive License
227
+ Has valid PR
154
+ SETUPAGENT succeeds
44
Get up 50 valid PRs
1527
+ SEtUPAGENT succeeds
893
+ valid instance
535
Prompt suggest relevant files
You are senior developer contributing to the www.githu. ctwpl.cco//eeeeeeee com/<repo id> project by solving issues. You have created a Docker environment with Ubuntu, and you want to install the repository in development mode (meant active development and testing) and run the tests. The first step is to locate the installation instructions and the test commands. I will provide you a list of filenames or filc paths (c.g., README.md, contribut.ing..d md), which typically include instructions for installation file and testing. The files can be either filenames (e.g., README. From the paths (e.g., provided list of filenames or file paths your task is: 1. Identity those likely related to installaticn those that your response.
testing pased en their names. 2. Exclude clearly irrelevant. 3. If unsure, include the file/path in
Return only the files/paths from the given list, exactly as they appear, without modifying their names or structure 5. If full path is given, return the full path, not just the filename. 6. Use the following format for your respense <ANSWER>: file 1, file n, filepath 1, filepath
<REASONING>: <YOUR REASONING>
Example input:
readme md, contributing md, contributors.mde docs hn -or.inttaalllfoo--ooorre..e rst, Docs///oree/iii/nstallaaaaaaaa rst, htt.o/eeeeeerrr-eeeegggnoooooooe
reasonable output is:
<ANSWER>; mdd contributing.me.
<REASONING>; The files readme .dd and contributing.mn
commonly contain installation and testing instructions, while docs/ coce://miimaaiinnnn//ssaaaaaiiim
cooom-suureerss is likely related to installation as the name suggests
Here the file names ...
<file 1>, <file 2>,
<file k>
Flease read the names carefully, ask yourself the purpose of each file based the before inclucing it in your response. Use the given format for your answer and please do not add any extra comment or text.
Figure 9, Prompt for choosing relevant files to installation and testing
13

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Prompt suggest external sources of information
You are senior developer contributing to the GitHub project at www. www.gitub. comp/reeo_ide by solving lesucs. Your goal is to install the repository in development mode and run its tests.
You have created Docker environment with Ubuntu, and now you searching for the installation instructions and test commands.
will provide you with the content of common repository files (e.g., INTTRBBUTIN (WTRIBBUTING.m)) RRADME. md,
Your task is to analyze the provi ded text and icentify all external links that contain relevant information to Installation instructions for this project.
2. Test commands or instructions for running the tests for this project.
3. Contribution guidelines.
Flease provide the links you found following the criteria below. a. Exclude links to generalpurpose documentation for external tools (e.g.,
Tex, Pytest, or other frameworss/Libraries
b. If you are unsure about the relevance of a link, better include it. Order the links from most to least relevant. not add any comment text.
e. Use the following format:
LINK: <LINK 1>
LINK: <L.NK 2>
...LINK: <LINK
Here the text:
'1/
[eext_content>>
Figure 10. Prompt to suggest potentially relevant external sources
Prompt to determine importance of url content
You are senior developer working on GitHub project www. git.ubttubb... crom/rrepoicc id>. You have set up Docker environment with Ubuntu, and now your goal is to install the repository in development mode and run its tests.
Your task is to carefully review the content of the following link: scurrent_link: and determine test commands for the <repo_id> project. Please follon these steps:
includes installation instructions or
Look carefully in the provided content for any potential instal lation commands or test commands related the <repo_id> project.
Ask yourself if the located instructions are reasonable, legitimate and can be practically executed to install or to test the <repo_ id> project only.
Flease provide your answer using the following fermat: INSTALLATION/TES, COMMANDS; <TRUE FFALSE> REASONING: <REASONING> * Important Notes**
Answer with TRUE only if the content explicitly includes valid and usable installation test commands. If you do not find any relevant commands, or the instructions are vague, ambiguous, impractical, or unrelated answer FALSE. When in doubt, answer FALSE. Content of the link kcurrent_link>: ''
<clean_content>
Figure Prompt for determining if a link relevant installation and testing the extraction phase the SETUPAGENT
14

---

Automated Benchmark Generation for Repository- Coding Tasks
Extract Install Command Prompt
You are senior developer working on the project located at www. www.github. github. com/ <repo_id__ You have created a Docker environment with Ubuntu, cloned the repository, and navigated to the directory <repo_dir>> Your next step is to install the project in development mode, which is intended for active development and testing. I'11 provide you with important text files (e.g., README. md) and important continuous integration (C1) confi guration files, which typically contain instructions for developers on installation and testing. The format provided will be the file name followed by its content.
Your task is to identify and return the bash commands recessary for the correct installation of the repository. This includes system dependencies, project installation in developnent mode, and any prerequisites or configuration commands.
** IMPORTANT NOTES **
1. Include system dependencies installation commands required for the project (e.g., via apt, yum, curl, etc.).
2. Include installation commands necessary for setting up the project in development mode.
3. Include prerequisites installation and contiguration commands, such as those for при or any other required setup.
3. If comprehensive installation instructions provided, return them with any modifications.
4. Only exclude commands related to creating or activating virtual environments.
The returned commands should meet the following criteria: Enclosed in quotes.
2. Focused strictly on commands necessary for both system dependency installation and development -mode installation of the project.
3. Free from any comments text.
Accurate and executable without errors.
If no installation commands are present, return NONE.
Here is the text:
<context>
Take your time to carefully analyze the content. Make sure that your response includes only the necessary installation bash commands. Ask yourself is the provided content is sufficient for installation. And for each command, ask yourself what's purpose the command and if it is necessary.
An example of the expected response is: 1bash
install [
instal 1_mmmman___mm___
Please provide the installation commands in the above specified format.
Figure 12. Prompt used for extraction of installation commands in extraction phase of SETUPAGENT
15

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Extract Test Command Prompt
You are senior developer working on the www, www. github, con/<repo_itt idd project. You have created a Docker environment with Ubuntu, cloned the repository, and installed it in development mode (meant for active development and testing).
<repo_dir> rectory and your the unit tests. will provide you with some important text files (e.g., RRADME, md) and important continucus integration (CI) congiguration files, which typically include instructions for running tests. The format provided will be the file Your task is to identify and return the exact bash commands zequired to run the tests.
You
inside
goal is to
followed by its content.
The returned commands should meet the following criteria:
1. Enclosed in quotes.
Free from any comments text.
3. Accurate and executable without errors.
If no test commands are present, return NONE. Here is the text:
<context>
Take your time to azalyze the content carefully. Ensure that only the recessary bash commands for running the tests are included. Ask yourself the purpose each command before including it in your response. example the expected response is: 'bashh
test cest__oomand_] 1
test_command_2 ...
Flease provide the test commands in the above specified format.
Figure 13. Prompt used for extraction test commands the extraction phase SETUPAGENT
16

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Prompt for determining causes
You are developer working on the project at www. www.github.co con/<repo idd. You created an environment with python version <pythen_versions Your goal is to install the repository in development mode (meant for active development and testing) and run the unit tests. The installatin commands "baah
www.githib.com//eepo....
kinstall_commawe
[enttall_commaty 2>
dinstall_comnane
The testing commands are:
'bash
<test_command_.3
<test__comaans
<test_command
Yeu received the following error message after executing the command cerror_command>
<error_message.
'''
Your task is to analyze the error nessage and determine its causes. You can return one of the following answers:
1. <PYTHON>, if the error is caused by incompatibilities between the python version and any used package. <INSTALLATION,,
the errer is caused by an installation command or related to any missing package, regardless if it a testing related framework or not. All the required packages must be installed in the installation phase.
<TESTING>, if the invalid 2lag the test command)
caused by any testing command (e.g.,
4. <UNDECIDABLE>, if you cannot determine what causes the error. Please read the error message carefully and try to spot the commands that are responsible for the error. Always provide the reasoning for your answer.
Use the following format: RESULT: <PYTHON, INSTALLATION, TESTING, UNDECIDABLE> REASONING: <YOUR REASONING>
Figure 14, Prompt for determining the error cause in the iterative improvement phase of the SETUPAGENT
17

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Prompt for fixing python version
You are senior developer working on the project at
www. www. github, ctm//oo///////ee conp/rrpp_idd Your goal is to install the repository in development mode (meant for active development and testing) and run the unit tests.
You created envi ronment with python version poython_erriicn on>, but you are unsure if the python version is correct.
You received the following error message while testing the repository:
<error _eroo_ ...
A senior software developer colleaque has provided an explanation of why things are not working as expected with the current commands:
<Reasoning from the answer to the prompt for determining the error cause>. Use his reasoning to resolve the current error we are facing. Your task is to determine a compatible Python version for the current state of the repository. Carefully read the error message and identify the most suitable Python version.
Please follow this answer format:
1. Return <NONE> if the error unrelated to the Python version or you cannot determine a compatible version.
2. If a specifig Python versien is compatible, return only the version
number (e.g., 2.7).
include any aéditional comments text your response.
Figure 15. Prompt for fixing python version used in the iterative improvement phase of SETUPAGENT
Prompt for fixing installation commands
You are senior developer working the project at
github. com/<repo_id>. You are working in enviroment with python version <python_version> You have attempted install repository in development mode (meant for active development and testing) using the follcwing bash commands:
"bash
<install_commande 1>
cinstall_commadd__
cinstall.comand.
However, the command <error_command> failed and we received the following error message:
<error satror_esssaee
Your task is fix the above and try spot the commands that are responsible for it. Please provide the updated installation steps in a bash code block, following these rules: 1. You have to use always uv pip instead of regular pip.
Think carefully what
the
2. Return <NONE> if you can not fix the command. add any comments text. For example: "bahh
apt-get install <package nane> pip install requirements. txt ...
Figure 16. Prompt for fixing the installation commands used in the iterative improvement phase of SETUPAGENT when the error occurs in the building process containers
18

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Prompt for fixing installation commands 2
You are senior developer working on the project at You tried to install the repoeitozy in development mode, which is intended for active development and testing, however the installation failed.
www. www. github. com/
working in envir with python version <python> and you tried to use the Fol lowing bash commands for the installation: "bash
<install_com comnand 1>
2>
patsstall_comarde n>
During the execution of these commands, you received the following error message: ..
<error_message
A senior software developer colleague has provided an explanation of why things are not working expected with the current commands: <Reasoning from the Use his reasoning to resolve the current error we are facing,
to the prompt for determining the error cause>.
Your task is to carefully read the error message and determine which commands are causing ng the error. Reason about every command if it is causing the error. If you sonclude that the problem is related to any of the commands, update the installation bash script that you can also add new commands to fix the problen. If you decide to update the installation bash script you have to follow these rules:
sclve the problem. Note
1. Provide the updated installation steps in bash code block.
2. Use pip instead of regular pip.
errer is not related to the installation steps or you are not able to fix it.
Return NONE
3. Do not add any comments or text.
For example:
The initial installation command is: 'bash
uv pip install
However, the error message states that the <package_name> package is not installed. Then you would update the installation command to:
bbash
uv pip install
pip install spackage_name name>
Figure 17. Prompt for fixing the installation commands used in the iterative improvement phase of SETUPAGENT
19

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
Prompt for fixing testing commands
You are senior developer working on the project at www.github. github. comt///co//////oo/ <repo_iddd You installed the repository in an enviroment with python version <python_version> and now you are trying to run the unit tests.
www.
You the tests using the followi ng bash commands:
"bash
<test_conmand_''
#test__oomand. 2>
However, at the moment we receive the following errer message: <error_mest cerror_message> ''.
senior software developer colleague has provided an explanation of why things are not working as expected with the current commands:
<Reasoning from the anawer to the prompt for determining the error cause>. Use his reasoning to resolve the current error we are facing.
Your task is to read the produced error message carefully, determine what the problem and try to fix it. Ask yourself which test command could cause this problem. It you conclude that the problen. is related to the test commands, update the test commands to solve the problem.
Please provide the updated test commnds in bash code block, following these rules:
1. You have to always use uv pip instead regular pip.
2. Return NONE if the error is not related to the test command or you
cannot fix it.
3. not add any comments text. Add command only you are sure that it is correct. For example: The initial testing command was: "bash
pytest test_file.pp flag '-v' and the maximal number of failing tests be we would have correct the command to:
all However, if in this case would need the
"baah
pytest test fist_file.py maxfail=1
Figure 18. Prompt for fixing the installation commands used in the iterative improvement phase of SETUPAGENT
20

---

Automated Benchmark Generation for Repository-Level Coding Tasks
C. Appendix Dataset Details
Below, we list all repositories along with the number corresponding tasks
SWA-Bench- Repositories
1. iterative/dve 42
23. cookieauster/coolleuttee:
2. streaml/nk/streat ink 35
24. pypa/ptperv
3. spack/spack
graphql pyyth//gaaaaoa
Prefectlll/pzeeect
pypa/banderrenttch
xonsh/xansh
6. aalmproxy//iitmrrowe
28. hynek/doc2dasht
7. python pillow/Fillow 29
29. PyLhonChaermess//yyhho Future 3
6. =kdeas/zkdocs - 23
30. simhtbio/aim
9. bynek/struetiog 22
31. dbcli/pgel1
pallets/click
geapython/pycaw
locustib/locust 20
dbader/schedule
12. jpacills/pyjHt
34. kibstar/<ibit::
13. alertie/lastticeacch-l---/
35. getnikela/nikole
14. pallets-eco/wtffomae 17
36. gespy/gecpy
15. 1python/1python 16
37. Maratysrcaa/Peschhy
postr/pyces.y//oeetty
gawel/pyquery
conan-la/coman
Suar/funcy
cabnzbe/sabmzbd
simorw/satassste
19. Bulko/moviegy 14
41. cowrie/cowrie
20. nvbn/thsfuck 12
42. pypa/pip -1
21. arre=-py/arroи 11
43. Stavenslack/hon /oosts 1
22. benoite/qunicozn
jupyter/bbgrader
Below, we list all repositories along with the number of corresponding tasks SWEF-Bench.
21

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
SWEE-Bench- Repositories Part
1. python attrs/attes
40. Lovasoo//marremmelooo..oo
2. dgsemitth//otL_eennsmm
aia-liba/yarl
3. jazzband/tabl.ib
PyCos/yyllakea
Ma:LiuThema// simplity
43.
44. konrsdhales/dacita
6. siegpeeeilllleeeaaarraaaeapppppe
45. 2leokkwwwkps//yppinngnnnpe
3. accuuuoorct//ppttooo---icrrrrree
46. Junyter//usytes__orre
6. davichalterpparso
getsentryyrregonse:
9. ma ma:shmallow-oode
beartype/plum
10. addmchaiang//laccenn-osss 7
opan?c/bioframs
11. Marreesspuurr//rooottreeeoo
50. deva dew.aloommttttaoc/erraacccc
Lundberg/respx
51. nats io/nats.py
seperman/deepoi
52. nipy/nipype 1
braagget667/yypcchiiii--rgggg
53. prthorrrpppttttt..rrrrr...raaete
cantools/cantools
stac-mtill/pyttte--llnn.
didix21/mdut!
55. wuolingehun/ -pank-opeappp
17. pa:stmalloo-ccoc//ppieee..
56.
18. coftllaper/aatttllyyr-eyyttto
57. jpadills/pyjw:
19. gorakhhergoshh/wetchhhogg
58. hewaar/iiidem/ppoaattttoo
20 pygments/pygmenta
john-kurkowsk rexttxtact
21 hank--contttb/bbaa---hitttageer
gespandaae/gooeendas
22. endialbreechteellparss
aaddovants//dkk-yyyhhr
23. rumee//rriddo
52. jupyter/nbformat
Lableau/Labenn
63. matthew brett/delocate
64. iterative/shtab
26. 8:0wphow/snowplo> pytp:hphwwooppphhoooooooo python-ttacceeeeeeeeeee tracker
65. jeappickle/jaonpiiiiee
27. joeriokkpy1nsszrmmett
etheremm/eth-ttile
26. scikiit-rf/scckitt-r
67. mhe/pynrrd
29. matthewwaard:oop/ttrruuaaa -
66. adamjstswsrt/fisccayyaas
30. laspy/laspy
69. pytest--aw/gr.est-xddtt
70. face facssesssuserr//eaattt
moouts/itables
sellklt-hhep/awwwrrd
templue/kulerretts...ecciii
34. Citebase/ri:ebase admin python
73. ipython/szaillets
35. sthere::/etb-accommt -
74. haw.dwwwbbrrtb//glll....oor
36. davidhalter/jed
75. omzy/omegaconf 4
agrenholim/sypeeuardd
76. python lap/pyshen 1sp server
Delgan/loquzu
cogeotiffrrio=ttiler
irtransitions/taaniiiiiii
wjshnson/ppappcchesslaa
22

---

Automated Benchmark Generation for Repository-Lewe Coding Tasks
SWEE-Bench Repositories Part II
edamchainz/daanpo htmx
118. scrapy/w31ib
80. swclient/awclient
119. htg.
executabbeebooks/phhinn bock theme
120. agrorho m/ m/obor2
82. ocikitshhep/vacto
121. weiwei/ junitarses
pitriick-kt.ggpr/oggitxxx
conar: is/conan
123. prthon//mmorr.llt__rcorree
regebro/pyrona
124. tnwehh//ildccrrr----uooorrpppppp
rephiile/gi.urrlpssee
125. egg.otholmmeeeceelliono....
cooiiecuute::/cooheeeettt:::
126. magmax/python- inquiser inquuiiiiee--ieieer
127. PrafectHg/prefect
jsvane/pofpllumbo:=
128. relp/deetettseereets
91. scrapy/protego
129. Cnllllppaaatotoocaum
92. OmileyChris/djangg countries
130. jaraco/keyring
93. (ttorlee//iiallhheaatee
131. Pylons/waitzess
94. pythologgical/kennnn
132. pypa/setuptcols
95. pypa/virtualenv
133. barr:att/pypaallohccee.
96. fastavro/fastavro
134. bluesky/ophyd
97. ma:shmallow wwww///ooe//d/eeeaeeeeeae wwwee//aaaaaaallwawsssaasoooo sqlalchean
135. OpenMathh/y openmath 2
136. readtheddocs/spphinee not found- page
gezpachoking/jii jsonref
137. canonicalloparrtor -
lepture/mistune
136. ekzhs/dstaske:ch
1.00 / htr.rrrrrrooooooooooooooottt
101. simenw/scllie-utti.
140. Shaobx/xmldiff
102. executablebooks/mdili pyyuiittt----uuingg
141. ewtls/rich-click
103. cmutsul//inkttf--ittrry
142. jaraco/path
104. hastto//utopepé
143. http:://dbb-arrttaaaaaatt-ae
105. cubevise-code/mdang
144. symerio/pgocoode
106. joblib/joblib
daggau/
python-trilo/rrio-tyynng
146. Jazzbend/d] database url
nallepee/panndralllell
147. nipun:advvilkarrpppsss
146. catmbbans/jjgg--llee--iiissssaee -
=lchardjon3e/parsee
149. jwouts/jupytext
ipythoa/ipythen.
150. XrFi:999/djano-vvite
112. pypa/reaame_reeddere:
a152:15/aungy-grropiie
113. jaraco/zipp
152. regebro/svg.path
114. docker/dboker-py
153. gmr/flatdict
115. joshy/stripsté
154.
116. goggleapis/yython-pubeee
117. tylerte//torchinso
155. behave/behar
23

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
SWEE-Bench Repositories Part III
156. Lhesim
196. vinitkumat/jsnn2mn
157. codingjoe/djange select2
197. tri.ppta//eprrnn..cereee..
158.
allieson/pythen: simple rest client
198. pentsaakk-/yyyaeeaagttttttrrrr
159.
cartectiahhllleauttttfaaaas
199. woottweeneervt//aavccooo
1.6.
erphomee/alocspphcmppp.
200. slackkepl/gyyhhomsslakk---tt
161. oauthlib/ocuthhibb
201. nginxiine/cressplaane
rastedpy/zesult
202. betzznerr.oos/hhsood----rr..
163. graphęl WyLthon/gzapheng
203. dbader/schooule
164. benmozan56/esper
204. amp ht
165. eerimog/bineapy
205. jazzbend//conteetllib2
166. kelesher/schems
206. theskumaz/python dotenv
167. PyCoA/flake$
203. raimon44/ppp-llieoneee
168. kjd/idaa 2
208. lomustio/locust
169. jupyter/nbconverl
ssta:in/python Labillll Labulate
170. scikit-hep/hist
210. eleethoenss/Voluutuuus
171. apulec/freezagun
211. abano-rraaa--tomm///ccaaoobbeeee
172. jupyter/nbelient
212. geoppaca-ccodd/yympll -
173. PythonChharmerss/yythn Tuture
213. tedderr/noqpe::s-aaaabhhhh
174. Dottoiae/pypllke-ootliiss 2
214. pyviss//pyvtsapyy
175. sthalley/dhspyyhon
nithimmaeelkppyysheet.
176. akeranasee/pyaml_eny
216. mlenten/collections extended
175. terraforrm-coplliaeessiii
213. amcconville/wan.
176. poopleapss/yyttaa--firsssss -
218. rsolmes//llla--proree.
179. googleegpis/yythoonai---oooo
219. jycusB/feeeeeeeeees__as...eeeee.
180. scrapy/casselect
181. hychoorruumanneeemmnnezzee
221. geagle/ncbly
182. :alposx/pottbbbttnnrrr....g
222. serapy/1:enadapter
183. dedupeio/dedupe
223. necllent/roolient
sitaariokeedataar:.kkellll
224. geogle/duet
bluesky//venttmmodell
workca//worko--python
225. di/celver
kyman/sbstrrpout
226. besncount/mmorttiimpon
asserqpy/assertty
227. bridaeperrosiooppthhon--hhli
dbt-labsshologram
228. construes/ccoattru::
199
229. dewrimmaw.seoll//yybbuees
191. keta/baseb8
230. richaropentan//hoia
192.
sttwad/python--scc
231. evxpy/cvxpy
193. wizeservice/csvkiie
elaatic/ecs-Loggl python
194.
seterhainz//immmmmete
233. gyhhonarcaade/pppttlleLLesssssss
195. XagicStack/ inmutables
234. hetzeopy/ettessiio--eeeeeee
24

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
SWEE-Bench Repositories Part IV
235. SAP SAP/pythen pyodat
274. Pylons/hupper
236. Acure/auure functions durable python
275. abbdeceeleerc/ccccccooeeeleeeeao
233 teenthtppphhnm//hpppssmll2
276. alaseendonaagiii//rrronnngn
238. Jwodder/check
273. imageic/immgiiooffmaee
Zulko/moviepy
278. gcsttmo/ggthhon-nocd--ommm
240. <itml2paffxhttm2pph
279. netbox- petbox--commnityypppetto
241. ckmd//ttack:intte:
kumar303/mchawl:
242. guillp/ Jaskate
281. SpamScope/mail parser
243. jmaarp/fleesk-apispe
282. peztygeo/pythen rastersiats
244. Limotucrer/colorful
283. pahaz/schtunnel
245. zisc-belica/sumy
284. python-tsper/hl1
246. kvesteri/intervale
raxerpay/raaarrpyyyyyhh..
247. marcocer/lime
286. zercsteciner//uleeennniia
246. wkentaro/gdown
237. mercbeta/janome
249. resipptthen/rrndeiiinngg
238. glut23/webvtt-py
jaraco/tempora
benoitc/gunicozn
jendrikeeippg/uutuue
290. nomt:offfessppybee. decutile
Pysen:ribs/zuyaml
291. alaxmojskd/oxocmutii
253. ionsttteam/ bumentationn
292. cigmavirus24/ithhene
254. nose nose-devs dev-dddeeeeeeeeeee
293. ecpen/mrefile
255. jongracecox/sny/leeee anybaagg
esinva/imodele
256. patrys/httmock
295. cicce-oarsi//iiic--bb---....
257 maxfische::27///eyyeeae
296. scossross/zpafile
258. pgzip/pgrip
297. bgrecco/pint
259. arvkevi/knede /kneed
djanço ses/django
260. tasterio/affine
299. gmr/pamap
261. civqua-tont/oirous
300. spatify/anncy
262. schearaaa/aamunngee-w----
301. PysuA//ycodes:yle
Jazaco/porLend
302. regebre/szlocal
264. fiasoasccam./pprrtan-tennedt
265. rump//ammy-financia
303. sapadb/haversine
266. peaw dev/prawcore
304. scieentiiie-gyhhho/lee----be
265. wccpy//lldest--pppprre---ng
305. grappe-py/grapo
268. logtall//ogtallloyytt
306. Blexmock/Elexmann
269. saikaacca//y-acalle codec
307. 1g rp/liquid
270. Kaic/pynmea2 -
308. pomp--oookkss/phhhorrrppprttttt
271. aartheed/jaannn--ontiigrttttt
309. saraco/jaraco.cortt
272. allenai/cachee_pathh
310. htb-libs/multidice
273. sllck-cotrrib/lliekaalll
311.
25

---

Automated Benchmark Generation for Repository-Leve Coding Tasks
SWEE-Bench Repositories Part
312. akdeca/nkdocs
340. pyca/service identity
313.
341. di affrmaccrrporcrrrythooo........ oim.
1
DaLaDog/daLadog lambda python
342. xlwings/ jaendiif
315. storative/dvalive
343. maphox/alig]
316. cogeotiff/rio cogeo
344. cthoyt/pystow
315. arikrons/parriimniicc
345. kagptz/dizzord.py
318. Iaceleessuerr/pyndbonneettt oon
346. gahjells/pyplugs
319. pypa/build 1
347. colan-bbpytess._httpx
320. akdoca/ kktoos/nkdooe-redd..eek
348. LLNL/cersipy
321. dlint-py/dlint
349. spec-first/comnexion
klen/peowee_migratee
350. Yelp/bravado
323. a2g384/python-cwddtee
351. skorpela/pabo:
324. kinverarity1/laato
scrapy/parsel
325. Turbo87/utm
353. alexmo jaki jaxi/pure_evall
326. ajengo/daphne
329. hwwpaaabwlbookkk//phii-dao
354. graptgl python/ccphhhhhh////ppppp -core 1
328. Laterpretal/slicer
355. jckežk/faker
329. google/yapf
356. awasbis//weerbis--yyhhh--sss
sensein/etelemel.yy cliest
357. juppter//uuyterr_client
331. Kcuranowkk//sioorv
358. jaraco/inflec:
332. bpeeutablebookesphhinx Labs
359. GreyZmeem/ python logging -lkiinngg-okkii loki
333. pexpect/pexxect
360. suminb/base62
334. I
youks.owoee/vreeroee
335. frankiee57/Att:px-oaae
362. xnuinside/simpll ddl parser
336. Barugaku/rreoolvel11b
363. exeettableboosssppptoo.leee
337. Python213/:elegrapp
364. Pylors/webob
365. SelhMoor:on//asttuniii
339. rostat pdbx
356. pyhoo--emmeer/yyttoo--eaer
26


Reviews:
[{"review_id": "qGPtvOR0g0", "reviewer": "Reviewer_eETZ", "summary": "To achieve the automatic generation of challenging and realistic repository-level coding benchmarks, this work proposes an LLM-driven method, SETUPAGENT, to automate the extraction of valid information from complex real-world repositories, ensuring the correct setup of the environment for perfectly reproducing issues encountered in practice. Using SETUPAGENT, this work constructs two coding benchmarks with different characteristics: SWA-Bench and SWEE-Bench, to evaluate the properties of the generated benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and positive feedback, recognizing the relevance and quality of the benchmarks we created. Below, we address their remaining questions:\n\n**Can you assess the repeatability of SetupAgent and what it implies for the generated benchmarks?**  \nFirst, we want to emphasize that benchmarks need only be constructed (and then updated) once to evaluate a large number of code generation methods. Therefore, a lack of repeatability would not have an adverse effect on the benchmark’s value and reliability. Nonetheless, we conducted an additional experiment corresponding to the repository setting in Table 3, rerunning SetupAgent 3 times for 100 random candidate Repositories from SWEE. This led to 26, 27, and 27 successes, where the last two runs install exactly the same instances and the first just drops one. This demonstrates that SetupAgent is highly repeatable. We thank the reviewer for suggesting this experiment and will include it in the revised version of our paper.\n\n**How was the sample size for the generated benchmarks chosen ?**  \nWe combined our own experience developing code agents and the community's preference for first the Lite and later the Verfied Version of SWE-Bench to choose a sample size of ~500 instances. As the reviewer noted, larger versions can be created easily, however, making evaluation more costly.\n\n**Can you add an experiment analysing the effect of the underlying model on SetupAgent?**  \nWe first want to highlight that our work focuses on the framework we propose, which we analyse in the ablation presented in Table 4. Further, we specifically chose GPT4o-mini for its great affordability, as it allows us to demonstrate that benchmark generation is now accessible to the wider community. Finally, we note that few cheaper models are available, e.g., even an 8B model hosted by together.ai having the same inference costs, and more capable models being significantly more expensive, e.g., GPT4o having ~15x higher inference costs. We conduct an additional experiment with GPT4o corresponding to 100 SWEE samples in the repository setting in Table 3. We observe GPT4o succeeds for 24 repositories compared to 26 with GPT4o-mini, demonstrating the robustness of our framework to the underlying model. Interestingly GPT4o uses fewer iterations to improve its output (2.1 instead of 3.4 on average), explaining the slightly lower success rate.", "claims_and_evidence": "I think the main claims have been supported by enough evidence, especially the characters of the generated benchmarks: real-world, diverse, efficient, generalizability.", "methods_and_evaluation_criteria": "Although I am not an expert in this field (LLM-driven coding analysis), I have carefully studied the method proposed by the authors, which involves detailed and well-reasoned sub-processes. Overall, the method appears convincing to me.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "In the experiments, the authors tested different code agents and benchmark generators. However, I believe the number of experiments conducted is somewhat limited, especially for benchmark generators. It would be more beneficial to include generators with different capabilities and varying degrees of relatedness, as this would help us better understand the impact of choosing generators with different attributes.", "relation_to_broader_scientific_literature": "The automated construction of benchmarks is of great significance to the community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "I think it can be better to further validate the effectiveness of the benchmark, which may include:  \n\n1. Whether running SETUPAGENT multiple times on the same set of repositories produces stable benchmarks and whether these benchmarks lead to consistent evaluation results. If not, it implies that the evaluation results of model capabilities may fluctuate due to randomness in the benchmark construction process, making the evaluation unreliable.  \n\n2. The automated construction of benchmarks enables good scalability in sample size. Therefore, for this task, it is recommended to determine the optimal number of samples (benchmark size) that balances evaluation efficiency and stability.", "other_comments_or_suggestions": "The first line of chapter 5.3 misses a 'W'", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zj4zLhRIcj", "reviewer": "Reviewer_FowC", "summary": "This paper introduces SETUPAGENT, a system for automatically generating repository-level benchmarks for code agents by setting up historically accurate execution environments. It extracts installation and testing commands from GitHub repositories using LLMs, iteratively refines them based on execution feedback, and validates correctness through test suite execution. Using SETUPAGENT, the authors create two new large-scale benchmarks: SWA-Bench, focused on software applications; SWEE-Bench, focused on diverse and less-popular Python repositories.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback, highlighting the depth of our analysis, the scalability of our approach, and the value of our work for the community. Below, we address their remaining questions and concerns:\n\n**Can you conduct a manual review of generated tasks to assess their quality?**  \nWe have conducted a manual review of 30 randomly chosen SWA instances for both task and setup quality.   \nTo assess task quality, we follow the protocol used to create SWE-Verified. That is, we scored issue description quality and clarity on a scale of 0 (well-specified issue with clear success criteria) to 3 (almost impossible to solve correctly without further instructions) and test quality from 0 (test perfectly covers valid solutions) to 3 (tests are too narrow or broad or requiring information not provided in the issue description). For both questions, 0 and 1 are considered passing scores.\nWe observe the following: \n23 (77%) instances have a meaningful and sufficiently complete issue description, and 22 (73%) of these additionally have suitable tests (27 or 90% across all instances) to check whether the issue was fixed.  \nTo assess setup quality, we score the extract setup as either functionally equivalent to the described setup or incorrect and the test setup from 0 (functionally equivalent to described setup) to 2 (correct tests only partially or not at all executed). \nWe observe the following: \nAll instances run the correct tests with 77% using exactly the test commands provided in the reference. 22 (73%) instances additionally have a fully correct installation/setup.   \nWe will include an extended version of these results in a revised version of our paper. \n\n**How do you ensure the generated benchmarks are meaningful beyond passing (trivial) generated tests?**  \nWe first want to highlight that all tests are taken from the original repositories and were thus written by human contributors and considered to be sufficiently valuable to be merged. We further filter out all instances where not at least one such test that failed before the fix and succeeds after. Finally, our manual evaluation above demonstrates that cases where no meaningful tests are present are rare (only 10%).\n\n**Can you extend the evaluation of the created benchmarks?**  \nYes, please see the response to Reviewer XqFN.\n\n**How can your benchmarks/approach be integrated into existing agent evaluation pipelines?**  \nTo allow for a seamless integration of our benchmarks, we have made sure to create our benchmark instances such that they are compatible with the popular SWE-Bench evaluation harness (only requiring minor changes to load installation and testing commands from the dataset rather than being hardcoded) and will release both our dataset and the modified version of the harness. \n\n**How do you plan to motivate the community to use your tool and benchmarks?**  \nWe believe that the research community can benefit strongly from our benchmarks as the increased sample diversity will reduce the effect of overfitting and thus level the playing field for new methods that were not tuned to SWE-Bench at a significant cost. Further, we believe the ability to efficiently create new domain-specific benchmarks will be especially important for practitioners working on domains not well represented in SWE-Bench. We believe these will be strong motivators to adapt the benchmarks/method proposed here. Additionally, we plan to launch a benchmark website similar to swebench.com to allow for easy result tracking and comparison.\n\n**Can you include a direct comparison to ExecutionAgent?**  \nWe first want to highlight that ExecutionAgent is not capable of setting up specific (historic) states of repositories. It can thus not be used directly for benchmark generation and is limited to our repository setting in Table 3. Directly compared on 25 random repositories considered for SWA-Bench and using GPT4o-mini for both methods, SetupAgent (ours) succeeds on 13, requiring 29.2 s on average (12 min total), while ExecutionAgent succeeds on only 9, requiring on average 2700 seconds (18h total). We thank the reviewer for suggesting this experiment and will include it in the revised version of our paper.\n\n**Can SetupAgent be expanded to handle more complex setups?**  \nWhile SetupAgent can already extract complex setup steps, a key limitation is that our dockerized execution environment does not support running docker instances inside this docker environment. We believe this is an exciting item for future work but out of scope here given the involved engineering challenges. We note that such a version of the benchmark would also be incompatible with existing SWE-Bench evaluation harnesses.\n\n**Conclusion**  \nWe hope to have addressed the Reviewer's concerns, look forward to their reply, and remain happy to answer follow-up questions.", "claims_and_evidence": "A few areas could benefit from additional validation. \nFor example, \" SETUPAGENT ensures high correctness and historical fidelity\", while the 95% pass-rate threshold and test-level granularity are good proxies, it seems no human validation or reproducibility check.", "methods_and_evaluation_criteria": "- The paper tackles the challenge of limited, manually curated benchmarks like SWE-Bench.\n- Integration of LLMs for interpreting semi-structured repo files: test execution provides concrete, verifiable feedback to supervise LLM-generated commands.\n- The authors balance automation with correctness through iterative feedback-driven correction and 95% test pass threshold in the validation phase", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "Benchmark quality relies on automated test results, which may miss subtle correctness issues (e.g., partial fixes, untested behaviors).\n\nNo human evaluation or baseline “gold set” is used to further validate the correctness of the generated tasks.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader scientific literature on code generation benchmarks and LLM-based code agents.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "Strengths \n- The paper goes beyond reporting benchmark creation—it analyzes failure modes, conducts ablation studies, and investigates performance correlations with task characteristics.\n- Designed to scale to thousands of benchmark instances without human intervention.\n- The system can be reused for future benchmark updates with new repositories or tasks.\n\nWeakness\n- No qualitative or manual evaluation is done to assess the correctness or realism of generated benchmark instances. The correctness relies entirely on test suite outcomes, which may miss semantic issues or partial correctness.\n- Evaluation focuses on few agents.\n- The system may not support projects requiring non-trivial external setup, and the automatically generated test case may be trivial.", "other_comments_or_suggestions": "Consider randomly sampling a subset (e.g., 50 instances) of generated benchmark tasks for manual review to verify: (1) The issue and patch are meaningfully connected. (2)The extracted test suite is valid and reflects the issue. (3) The task is realistic and unambiguous.\nThis would boost confidence in the correctness and representativeness of the dataset.\n\nA direct head-to-head comparison (in terms of runtime, accuracy, coverage, and granularity) with EXECUTIONAGENT would solidify SETUPAGENT’s claimed superiority. Even a small-scale test (e.g., on 10 shared repos) would be informative.\n\nConsider including an example of how external researchers can integrate the benchmark into their own code agent evaluation pipelines.\n\nAdd more baseline models of SWE-bench to test your benchmarks. (https://www.swebench.com/#verified)", "questions_for_authors": "- How do you plan to encourage the research community to adopt and test their models on your benchmarks?\n- How to integrate SETUPAGENT with existing agent frameworks (e.g., SWE-Agent) to enable seamless evaluation?\n- How do you ensure that the generated benchmarks reflect realistic and meaningful developer tasks beyond just passing/failing the generated test cases(maybe trivial)?\n- Can SETUPAGENT be extended to support projects with more complex or non-standard setups?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K32rSEOsTx", "reviewer": "Reviewer_msS6", "summary": "The paper addresses the problem of automatically creating repository-level execution benchmarks for software engineering tasks. The authors first describe SetupAgent, an LLM-powered agentic framework to setup the execution environment for any Python repository. This is then used to create SWA- and SWEE-Bench, increasing diversity of problems represented and minimizing contamination against models like GPT-4o. Evaluations show similar level of performance on SWE-Bench as these benchmarks with SWA-Bench showing slightly lower performance for problems after the models’ knowledge cut-off.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review, acknowledging the value of our contribution to the community and the quality of our analysis. Below, we address their remaining concerns.\n\n**Can you evaluate the presence and correctness of fail-to-pass tests more rigorously?**  \nWe first want to highlight that we already filter the dataset to only include instances where we extracted F2P tests by executing the test suite in the original state with test changes applied and in the fixed state with test changes applied. We have additionally conducted a manual review of the extracted tests (see response to Reviewer FoWC) and find them to be of high quality for 90% of instances.\nFinally, we have rerun the test execution and extraction 10 times for SWA to identify possibly flaky tests and extract 509 shared instances across all 10 runs (compared to 535 from the original run). We traced the remaining discrepancy down to two main causes: randomly parameterized tests and genuinely flaky tests. We will create a version of both the SWA and SWEE datasets with these filtered out but expect this to have minimal impact on our results and note that similar behaviour is present in SWE-Bench (see e.g., https://arxiv.org/pdf/2407.21787).\n\n**Have you evaluated SetupAgent both as a whole and with respect to its individual components?**  \nWe evaluate SetupAgent in Section 5.2, discussing its overall performance in Table 3 and conducting an ablation study in Table 4, demonstrating the importance of all its components. We have now additionally conducted a human evaluation, discussed in the response to Reviewer FoWC.\n\n**Can you analyse the issues discussed in SWE-Bench-Verified in context of your benchmarks?**  \nGiven the similarity in the benchmark generation pipeline, we expect similar findings to those for SWE-Bench(-Verified). However, the two main review/filtering criteria in the creation of SWE-Verified, namely, the completeness of the issue description and the specificity of added tests, only aim to filter out instances that can not be solved without additional information. This is crucial to assess the absolute performance of code generation systems but will affect all evaluated systems to a similar extent and thus have only marginal impact on a comparative analysis. We note that such a comparative analysis is sufficient to select among different systems for a given use case or guide their development. Similarly, we focus our analysis on the differences in the relative performance between models and compare our benchmarks to SWE-Full and not SWE-Verified.\nFinally, we have conducted a similar human review as for the creation of SWE-Verified on 30 random samples of SWA, which we discuss in the response to Reviewer FoWC. We are happy to include this discussion in the next revision of our work.\n\n**Can you discuss the differences between SWE, SWA, and SWEE? Are all needed?**  \nThe main difference is the set of considered projects. While SWE(E) focuses on libraries that typically expose functionality to other programs and thus tend to have more stable, well-tested APIs, SWA focuses on applications (e.g., locust or xonsh) that are directly used by humans and often have (graphical) UIs. As we show in Table 6, the performance of some models (GPT4o-min) is quite consistent across these different tasks, while others (GPT4o) varies significantly. We have conducted additional experiments (see response to Reviewer XqFN) where we see significant (3x) differences across benchmark for SWE-Agent v1 and DeepSeek V3. We believe this demonstrates the value of more diverse benchmarks in this highly active field. \n\n**Comment on Section 3.1**  \nWe introduced the notation in Section 3.1 to formalize our plain English (and sometimes incomplete for brevity's sake) descriptions of different settings. However, we are happy to move this formalization and its usages to the appendix.\n\n**Conclusion**  \nWe hope to have addressed the Reviewer's concerns, look forward to their reply, and remain happy to answer follow-up questions.", "claims_and_evidence": "1. SetupAgent is a very useful contribution to the community allowing researchers to turn any Python repository into a benchmark instance. However, this system is not adequately evaluated for accuracy or reliability. While the authors run tests reproduce original issues, I believe that the system needs to evaluated deeper to check the presence, correctness, reliability and reproducibility of fail-to-pass tests. A thorough evaluation of the system with this in mind is missing from the paper. Further, SetupAgent incorporates multiple LLM-guided steps, and as far as I can tell, no attempts were made to vet the LLM outputs. While the system proposal is useful, I think the authors should take more steps towards improving the transparency of the robustness and reliability of SetupAgent.\n2. The paper claims higher diversity in SWEE- and SWA-Bench compared to SWE-Bench. These are backed by statistical studies in Section 4. \n3. Finally, the statistical significance of performance of various models on various benchmarks are also shown. In some cases the p-values are very high, e.g., Table 8 has many p-values > 0.1. Nevertheless, I commend the authors for being transparent about these results.", "methods_and_evaluation_criteria": "I have specified a brief commentary on the creation of SetupAgent above. Besides that, I think the authors also miss to comment on the merits and comparisons of their efforts to SWE-Bench Verified [1]. It is now known that the original SWE-Bench has many issues at the systemic and problem semantic levels, and SWE-Bench Verified was an effective annotation effort to address such concerns. Given that SetupAgent leads to a similar benchmark as SWE-Bench, I think that the authors should also check if the same concerns are present in SWA- and SWEE-Bench.\n\n[1] https://openai.com/index/introducing-swe-bench-verified/", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "I have answered this in other parts of my review.", "relation_to_broader_scientific_literature": "The paper is related to the evaluation of LLMs as agents in software engineering. The most relevant work in this direction is SWE-Bench [1].\n\n[1] Jimenez, Carlos E., et al. \"Swe-bench: Can language models resolve real-world github issues?.\" *arXiv preprint arXiv:2310.06770* (2023).", "essential_references_not_discussed": "I have touched upon this above. The problems raised and addressed in the SWE-Bench Verified work are not mentioned in this work. I think this is essential for the adoption of SetupAgent, SWA- and SWEE-Bench just as SWE-Bench Verified is now the default benchmark for evaluation.", "other_strengths_and_weaknesses": "1. I believe that SetupAgent solves an important problem of automating the creation of execution environments but this needs to be done in a more rigorous way. I have elaborated on this above.\n2. I think that Section 3.1 is completely unnecessary. It obfuscates a pretty simple setup and does not add much value. I recommend the authors to remove this section and all the following mathematical notations.\n3. I did not understand the difference between SWA- and SWEE-Bench. Specifically, I’m not sure about what makes SWA-Bench unique.  I do not understand this statement: “Code Agents develop software applications that suffer from different types of bugs compared to libraries due to architectural and structural differences.” Could you elaborate or maybe show an example?\n4. I’m not sure if this should be a major or minor concern: I do not see much difference in model performances across SWE- and SWEE-/SWA-Bench (Table 5, 6). Am I missing something from these results?", "other_comments_or_suggestions": "Line 375 - e → We", "questions_for_authors": "1. Can you concretely clarify the difference between SWA- and SWEE-Bench? Are both needed? If not, which one is better? This will help researchers understand what’s useful for their use case.\n2. How would you judge the accuracy and reliability of SetupAgent? Have you evaluated the various components of SetupAgent? This will help increase the trust in the system.\n3. What is the status of issues raised in SWE-Bench Verified in SWEE- and SWA-Bench? Did you take any steps to address these issues? If not, what should researchers using your benchmarks be mindful of?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nnbybnd59Z", "reviewer": "Reviewer_XqFN", "summary": "This works presents SETUPAGENT - an automated LLM-based approach to generate repository-level coding benchmarks from a list of github repos. It enables creation of larger and diverse benchmarks suitable for evaluating software engineering agents, and automates steps such as dependencies setup, test execution, result parsing, etc. \n\nAuthors use SETUPAGENT to create 2 new benchmarks: SWEE-Bench (extended version of SWE-Bench encompassing more diverse repositories) and SWA-Bench (focusing on applications rather than libraries). \n\nAuthors highlight the distributional differences of SWEE and SWA to SWE-Bench (issue description quality, fix complexity, etc.) and highlight possible contamination of models by showing difference in SWE agent success rates before and after model’s knowledge cutoff dates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable and positive review, highlighting both the novelty of our work and the value it brings to the community. Below, we address their remaining questions:\n\n**Can you extend the evaluation of the created benchmarks?**  \nWe have added three additional agents/methods (SWE-agent v1, ZeroShot-BM25, ZeroShot-Oracle) and four additional models (Haiku3.5, Llama 3.3 70B, Qwen 2.5 72B, DeepSeek V3) to our evaluation. See the results below\n\n**Model comparison with ACR**\n| |SWA|SWEE|SWE|\n|-|-|-|-|\n|GPT4o-mini| 8.4% | 9.0% | 8.2% | \n|GPT4o| 10.2% | 15.1% | 16.6% | \n|Haiku| 10.8% | 12.9% | 13.6% | \n|Llama3.3 70B| 8.8% | 10.8% | 12.5% | \n|Qwen2.5 72B *| 3% | 2%| 4% | \n|DeepSeek V3 *| 8% | 13% | 26% | \n*only evaluated on a random subset of 100 instances\n\n**Agent comparison with GPT4o-mini**\n| |SWA|SWEE|SWE|\n|-|-|-|-|\n|AutoCodeRover v2| 8.4% | 9.0% | 8.2% | \n|Openhands| 3.9% | 4.4% | 4.6% | \n|SWE-Agent v1|2.6% | 8.9% | 8.2% | \n|ZeroShot (Oracle) | 0.9% | 2.2% | 2.8% | \n|ZeroShot (BM 25) | 1.3% | 2.8% | 1.5% | \n\nWhile most of the additional results are in line with what we observed so far, we want to highlight that there are some interesting behaviours. In particular, DeepSeek V3 performs much better (2-3x) on SWE than on both other benchmarks, similarly SWE-Agent v1 performs much worse on SWA than on both SWEE and SWE. This highlights the value of diverse benchmarks, enabling a holistic evaluation of new models and methods.\n\n**Where do you observe the performance differences mentioned in the Abstract?**  \nE.g., in Table 6, we show that the performance of GPT4o is 40% lower on SWA compared to SWE. We observe even bigger differences in the results reported above.", "claims_and_evidence": "Yes \n\n- The claim of 40% lower agent success rates on SWEE and SWA compared to SWE claim is unclear. Table 5 and 6 show very similar success rates for all 3 benchmarks, only some difference between SWEE and SWA with one of the models.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "None", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Overall, the LLM-based approach to generate SWE benchmarks is novel and very valuable. This work contributes 2 new benchmarks, extending the SWE-Bench & addresses its limitations (diversity, potential contamination of models.).", "essential_references_not_discussed": "Nil", "other_strengths_and_weaknesses": "Weakness: \n- Authors have shown experiments with 2 models (GPT4o and 4omini), any performance differences of SWE agents using opensource models on SWE, SWA, and SWEE will strengthen the value of the work. \n- Although authors point out distributional differences of SWE with SWEE & SWA, a deeper understanding (based on multiple attributes) of type of new diverse samples present in SWEE and SWA (not present in SWE) would be welcome.", "other_comments_or_suggestions": "None", "questions_for_authors": "See claims & weakness sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Meta-Review of “Automated Benchmark Generation for Repository-Level Coding Tasks”**

**Summary of Paper**  
This paper introduces **SetUpAgent**, a system that automatically configures historically accurate execution environments and extracts test suites for repository-level coding tasks, thereby enabling the creation of large-scale, real-world benchmarks without extensive human intervention. Using SetUpAgent, the authors build two new benchmarks—**SWA-Bench** (focusing on software applications) and **SWEE-Bench** (covering a broad range of less-popular Python repositories)—and compare them against the existing SWE-Bench. They claim that these new benchmarks are both more diverse and more difficult, revealing performance gaps of up to 40% for certain code agents and suggesting that model success rates on SWE-Bench may give a misleading picture of real-world performance.

---

## Strengths

1. **Addresses a Real Bottleneck in Benchmark Creation**  
   - Prior repository-level benchmarks rely on heavy manual curation or resort to aggressive filtering that discards complex projects. By contrast, SetUpAgent automates repository installation and test extraction, making it feasible to scale to hundreds of repositories.

2. **Broader and More Diverse Benchmarks**  
   - SWE-Bench includes only a small set of popular Python libraries (12 repositories). The newly introduced SWA-Bench (focusing on applications) and SWEE-Bench (focusing on diverse, less-popular projects) include hundreds of repositories and PRs, providing a richer distribution of tasks.
   - Additional experiments (provided in the rebuttal) show model-dependent gaps among SWA, SWEE, and SWE.  

3. **Evidence of Distributional Differences**  
   - The authors show that issue descriptions, fix complexity, and repository popularity vary notably between the new benchmarks and SWE-Bench. Several code-generation models exhibit substantially lower success on SWA/SWEE (up to 40% using AutoCodeRover on SWA-Bench), highlighting that performance gains on smaller or older benchmarks may not generalize.

4. **Thorough Empirical Analysis**  
   - The paper includes:  
     - An ablation study of SetUpAgent, measuring which steps (e.g., CI/CD file parsing, iterative improvements) contribute most.  
     - A discussion of potential data contamination for some models when popular repositories are used.

---

## Weaknesses and Discussion Points

1. **Concerns About Difference Magnitude and Baselines**  
   - Two reviewers felt the performance differences from SWE-Bench were not always large in the main experiments, and that stronger or more numerous baselines could have highlighted the difficulty gap more convincingly. The authors’ rebuttal adds comparisons with more code agents and LLMs.
   - Some reviewers question whether the community needs a benchmark that is not very different from existing one.

2. **Cost and Practicality**  
   - Running large-scale repository-level benchmarks can be expensive (authors mention thousands of dollars for just a few model evaluations). At least one reviewer worried that the community might be deterred by computational overheads. While the authors argue cost is inevitable at realistic scale, it remains a real practical concern for adoption.

3. **Comparison to SWE-Bench Verified**  
   - SWE-Bench Verified was introduced to mitigate known issues with the original SWE-Bench. Some reviewers wanted more direct discussion of how SetUpAgent’s automatically generated tasks compare in quality to a carefully validated variant such as SWE-Bench Verified.

---

## Recommendation

On balance, authors contributions likely outweigh the reservations. I recommend acceptance.
Decision: Accept (poster)