SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
Authors: Niels Mündler
Mark Niklas Mueller
Jingxuan He
Martin Vechev
Venue: NeurIPS 2024 poster (2024)
Keywords: language model
test generation
code agent
Primary area: natural_language_processing
TL;DR: LLM based Code Agents are suitable for generating software tests in large and complex code bases, introducing a new paradigm for test generation and additional metrics for Code Agent performance.

Abstract: Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests.  We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench.

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
Niels Mündler', Mark Niklas Müller! Jingxuan He', Martin Vechev!
Department of Computer Science, ETH Zurich
2 LogiieStaraai
{niels muendle,, mark. mue.er, jingxuan.her martin. vertin..echee)eeint ethz. markelogicstar..i.
Abstract
Rigorous software testing is crucial for developing and maintaining high-quality code, making automated generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. How- ever, while code generation with Large Language Models (LLMs) an extraor- dinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to for- malize user issues into test cases. To this end, we propose novel benchmark based on popular GitHub repositories, containing real-world issues, ground truth bug-fixes, and golden tests. We find that LLMM generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code gen- eration, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing dual metric for analyzing systems designed for code repair. Finally, find that generated tests code fixes, doubling the precision of SWE SWE-AGENT We release data and code at githtb...boooo com/ logic www.ssttarraaaiiaae ai/SWT-Bench
effective filter for proposed
Introduction
As the complexity of software systems increases, rigorous testing is becoming more important than ever to ensure their reliability and correctness. However, while large portion of these tests aims to reproduce previously reported issues (Kang al. 2023). such issue reproduction often disliked by professional developers (Straubinger & Fraser 2023). Therefore, automatic generation of tests reproducing such issues from informal natural language descriptions
promising path toward
improving both code quality and developer productivity. Finally, generated tests can be leveraged as formal specifications boost the effectiveness of automatic code repair tools (Chen al., (2023a).
However, while automatic code generation, in particular using Code Agents, is an extremely active research (i.e. Yang al. (2024); Tao al. (2024); Zhang al. (2024); [Bouzenia (BuueeBo
al. (2024b); OpenDevin (2024); Bouzenia et al. (2024a): Schäfer et al. (2024); Alshahwan et al.) (2024a)), there comparatively little work investigating automatic test generation directly. Indeed, while prior work has proposed methods based on symbolic execution (Lukasczyk & Fraser 2022), specialized transformers (Tufano al.) 22000). and general-purpos. LLMs (Li et al. [2033] Alshahwan et al.) 2024b, Kang et al. [023]] 2024; Chen et al.] (023b). Code Agents have not been considered in this context, and even less work applicable to the issue reproduction setting. Finally, large-scale, di- verse cest-generation datasets are lacking for Python, which is one of the most popular programming languages at the time of writing (TIOBE 2024 PYPL 2024) and focus of Code Agent research.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task inputs
Generated Tests
Pre PR
Post PR
Lsdalid currently allows trailing newlines but only alphanumeric characters
isVa Lid ("name\n"" False
Fail
Pass
cm
should accepted.
isVa ("nane") = True
Pass
Pass
Codebase (Pre PR)
isVa Lid ("nane") == False
Fail
Fail
Figure 1: Evaluation of an SWT-BENCH instance. Given an issue description in natural language and the corresponding codebase, the task is generate tests that reproduce the issue. We considered a test to reproduce the issue if it fails on the codebase before the pull request (PR) is accepted, i.e., before the golden patch applied. but passes after. We call this a fail-to-pass test PP)
A Benchmark for Test Generation In this work, propose SWT-BENCHE novel and com- prehensive dataset for test generation with the goal of issue reproduction in Python. SWT-BENCH contains over 900 samples, each consisting of a GitHub issue, golden code patch resolving the issue by adjusting the code, and a set of golden reference tests, obtained by transforming the popular SWE-BENCH (Jimenez al.) 2023) from code repair to test generation. We leverage the fact that any code repair task can be transformed into a test generation task, even in the absence of golden tests, by utilizing the golden code patch for evaluation. Concretely, for every generated test, we determine whether it reproduces the described issue, by checking whether it fails on the original repository but passes after the golden patch is applied. The golden reference tests, used in SWE- BENCH for the evaluation of code repair performance, are solutions in this test generation setting. We illustrate this evaluation process of SWT-BENCH in Fig. [ Further, we report the coverage of the code modified by the golden patch as more fine- ame-igrined evaluation metric for generated tests.
Benchmarking Test Generation Methods We evaluate various existing test generation ap- proaches on SWT-BENCH, including directly prompting state-of-the-art LLMs to generate tests for the given issue, state-of-the-arr issue reproduction method LIBRO (Kang al. 2023), and dif- ferent Code Agents adapted to the task of test generation (anng et al. 20224] ZZhang et al. 2024 Aider 2024). Interestingly, we find that despite being designed for code repair, the Code Agent SWE-AGENT outperforms non-agent methods at test generation, both reproducing more issues and achieving higher coverage, and generally find all agents to perform strongly in both areas. How- ever, we still observe significant complementarity between the different approaches, with an ideal ensemble of the best four methods solving 71% more samples than the best single method. Further, while the performance on code repair and test generation is generally correlated, this does not hold on a per-sample basis. This indicates that reproducing an issue with test and fixing this issue are distinct tasks of different difficulty. Finally, we find that generated tests can serve as strong signal for the correctness of proposed code fixes, with SWE-AGEN achieving over twice the precision on fixes that pass self-generater tests that failed before the fix was applied.
Key Contributions
Our key contributions are:
We introduce SWT-BENCH, a new benchmark for test-based issue reproduction based on an tensive dataset of real-world software repositories, user issues, code patches, and test cases ($3).
We propose to adapt Code Agents to the task of test generation for issue reproduction ($4)
We provide an extensive evaluation of SWT-BENCH, and demonstrate that, while issue reproduc-
tion is generally hard, Code Agents perform well, outperforming prior methods (55).
2 Related Work
Code Datasets Over recent years, a varicty of code datasets such as HvmanEval (Chen et al,, 2021), APPS (Hendrycks et al. 202T), and MBPP (Austin al. (02T) have been proposed assess the capabilities of code synthesis and repair systems (Lin et al. 2017 Li et al. 2022). However, these largely focus interview-sty style coding challenges function-level code synthesis and do not capture the complexity of real-world codebases. Further, they have been shown to often include
insufficient test cases to properly assess the correctness of the generated code (Liu et al. 2023a).
Recently, range of repository-level code-generation benchmarks (Liu et al., 2023b Jain et al. 2024) including the popular SWE-BENCH (Jimenez et al. 2023) have emerged, as modern L.MMs

---

began to saturate the simpler function-level benchmarks. However, none of these benchmarks were designed to assess test generation.
The only dataset for reproducing bugs based on real-world issues, Defects4J (Just et al. 2014), focuses on Java, is outdated, limited in size, and contains only short bug descriptions rather than detailed issue reports. In contrast, SWT-BENCH is based on Python, which is better supported by modern Code Agents, contains more recent issue reports, and is significantly larger.
Automated Unit Test Generation Many approaches have been suggested to automate (unit) test
generation leveraging symbolic execution [Lukasczyk & Fraser
2022), specialized transformers
(Tufano et al. 2020), and general purpose LLMs (Li et al. [0033]
Alshahwan et al. 2024b Kang
al.
2023 [ Tutano al. 2020 Kang et al. 2024 Schäfee al.
2024 Alshahwan al. 2024a
Chen et al. 2023b). Depending on their focus, they can be used to increase test coverage (Alshah-
wan et al. 2024b| Schäfer al. 2024), find edge cases (Lukasczyk & Fraser 2022), or reproduce
reported issues (Kang et al. (023). Issue-reproducing tests are especially interesting, as they can be used validate automatically generated code repair candidates and thus improve the precision of
code repair systems (Chen et al. (023a). However, most test ast-generation approaches are not appli- cable issue reproduction. We therefore evaluate the most recent applicable method, LIBRO (Kang et al. 2023). and range of other LLM-based baselines on SWT-BENCH.
Code Agents Over the last year, LLMs have been equipped with tools observe and interact with their environment over multiple turns and preserve a state across these turns (Wang et al. (024). These so-called agents have proven successful on a range of complex tasks, including code repair and synthesis (Bouzenia et al. (2024b) Openn 2024, Zhang et al.) 2024] Yang et al. 2024: Tao al.] [2224] Bouzenia al.j 20224a| Aider 2024). Such Code Agents can typically search, read, and
edit code using an agent computer interface (ACI) (Yang et al. 2024). In this work, we leverage such Code Agents for generating issue-reproducing tests by changing their instructions.
3 Benchmarking Test Generation
In this section, we outline the structure of the proposed benchmark, SWT-BENCH, and how we leverage it to measure the capabilities of LLMSs and Code Agents for test generation.
3.1 Notation and Definitions
We first introduce the notation to describe codebases, their test suites, and changes to these codebases in the form of patches. We denote codebase R after applying patch X R X. Several patches can be applied sequentially, i.e. R a X o Y is the codebase I after applying a first patch X and then second one Y. When a code patch X is applied R. a set of tests T can be used check the correctness of the applied patch.
A single test R. We consider a test to fail if an error is thrown during its execution, e.g., an AssertionErro o or ValueError. Such test errors frequently occur if R lacks or misimplements the functionality targeted by the test. They can also occur due to other reasons, such as incorrect syntax or formatting of the test s. Conversely, test passes when running the test triggers no error. We define this process as an execution function: exec(s, Rt) E F).
either pass (P) fail (F) after execute it within the context of codebase
We consider a test to reproduce described issue of R, which is resolved by patch X if it fails on the original codebase (i.e. exec(s, R) = F) but passes on the patched codebase (i.e. exec(s, R X) = P). We denote these fail-to-pass tests with F + P and define F + F, P P. and P P F tests similarly. If test transitions from failing on R to any state on Ro,,, we denote it as F x and vice versa for x -> F. Further,
consider set of tests T to be successful at reproducing the issue I, if it contains at least one F P test and no x - F test, or equivalently Es E T, exee(s, R) =
F A Vs E T, exco(s, s, R X) P.
3.2 Benchmark Overview
To construct SWT-BENCH, we leverage the same underlying data as SWE-BENCH (Timenez al., 2023) and summarize its three-stage construction process here for completeness.
3

---

1. Scrape total of ~0 000 pull requests (PRs) from 12 popular open-source Python reposi- tories from GitHub.
2. Filter PRs to only include those that were merged, resolved a GitHub issue, and made changes to at least one test file.
3. Filter PRs to feature at least one F - P test, removing PRs that result in installation or runtime errors.
This results 294 task instances, each consisting GitHub issue, a golden patch X fixing issue, and set of golden reference tests 7"
Table Characterization of different attributes of SWT-BENCH instance.
However, we find that for 311 instances, the golden patch can not be evaluated without er- rors does fix the described issue reliably, i.e. some tests of T fail on R o X The main reasons are flaky test suites, e.g., django cases where НTTP requests sometimes return 500 Internal Server Error although the re- lated code was not changed, erroneous test suite setup, e.g., the test suite tool tox not allowing external tools invoked in the sphinx setup, and time outs, c.g., when slow tests in the sympy li- brary run. We exclude these, leaving a total of 983 instances in SWT-BENCH. To enable cheaper evaluation LITE, a subset of 276 issues, corresponding to SWE-BENCH-LITE
Mcan
Max
Issue Text
Words
318.0
8756
Codebase
Files Lines
1563.0
2757
337K
772K
F
0.0
4
F
183
P
116.7
4837
Existing Tests
P-+
1.5
6017
Total
123.2
4842
Coverage
70.1%
100%
F
2.0
958
P P
0.9
339
Added
2.9
Golden Tests
create SWT-BENCH-
Removed
0.3
104
Files Edl.
1.5
15
31.8
summarize key statistics of SWT-BENCI Table D and show its repository composition in Fig. 2 While issue descriptions are average only 318 words long, the longest one reaches 756 words. Generally, repository complexity is high with average over 1 500 files and over 300 000 lines of code. Many repositories feature large test suites of 120 and up 4 800 tests, already covering 70% of the lines in the to-be-patched code. Most of these existing tests are unaffected by the golden patch with basically no F-> P and only 1.5 P - F tests on average. The golden tests remove on average 0.3 tests and add another 2.9 new test cases, of which roughly two- thirds are F + P. The test patches edit on average 31.8 lines in 1-2 files. Due to the filtering for unresolved issues during dataset curation, no golden tests are F + F or P-
pylan
django
Rude
aplin eepreets
pofrett
Figure 2: SWT-BENCH instances over GitHub repositories.
Distribution of
3.3 Metrics
We propose two main metrics evaluate the test generation performance of any method; Success rate (8) and change coverage (C), described below. We further introduce the necessary but insuffi- cient property of patch well-formedness (W).
Success Rate The success rate S measures the portion of instances where the generated tests T reproduced the issue according to the definition in $3.1 i.c. at least one test in transitions from failing to passing and none fail after applying the patch. This is the most important performance measure, the presence of F' P and the absence of X + tests are key for test-driven de- velopment and automatic code generation. We further report the portion of instances for which at least one Pail-to-Pass. (F + P), Fail-to-Any (F x), and Pass-to-Pass (P-+ P) was generated. While F x tests, i.e., all tests that fail on the original codebase, are not necessarily desirable, only F tests can result the reproducing F test, whereas p-+ X tests can never reproduce an issue. As F > can further be identified without knowledge of the golden code patch, generation methods can aim to always produce an F x test. Finally, generated well-formed and valid, but unrelated tests.
tests indicate that the model

---

Change Coverage Coverage is an important metric to determine what portion of codebase is tested. While path coverage measures this optimally, the exponential number of paths makes infeasible in practice. We thus follow common practice, and instead measure line coverage. As we aim to specifically test the code described in the issue text, we consider only the coverage of the changes made by the golden code patch. Further, we observe that patches may include portions of non-executable lines, c.g. documentation or configuration files, and exclude them. Specifically, we consider all lines that are executed by the original test suite T or the golden test suite 7 on both I and It X to be executable, and track coverage of such executable lines. Finally, we consider both the coverage of removed (including modified) lines of code in the original codebase and added (including modified) lines of code in the patched codebase, illustrated in Fig. 3
C50
C+3
diget
& digictaFrent E'dee."
EighttoFront macch)
ir digictiereees: return Polie
trastitel (ewl(c pronti trrdi seutl) printi Tho eagst") return tratllted
Formally, given the number of times CR (!) E 7,20 a specific line of code was executed when running the test suite T codebase R. we define the exe- cutable lines of the patch X as
I
o X"
T UT
T
AC
x² = E X, CPR(l) (1) + CT. (1) > 0} X² - X., Cf (1) C/.x (1) 0}
Figure 3: Illustration of change coverage AC of the generated tests T, given the test suite T of the original code base R, the golden patch X', and the golden tests T"
where X, and Xa are the lines added and removed by patch X, respectively, and T are the golden tests. Finally, we obtain the change coverage of the gener- ated tests T as
H1 X2 C. (1) > CRaON + E x + x
CTRUT (1) >
RR (() T
ACT-
Where X and T are clear from context, we drop them for notational clarity. If none of the lines modified by the golden patch are executed by any test, i.e., [ + |x = O, we exclude this instance from our coverage analysis (1% of cases).
Patch Well-Formedness Many LLMs struggle to generate well-formed code patch files (Jimenez et al. 2023) and the methods we investigate employ different approaches to mitigate this issue. To assess them, we additionally measure the patch applicability W as the portion of instances for which a well-formed patch was generated. We define W as the portion of instances for which the generated patch X can be applied to the original codebase It without errors. Since well-formedness necessary for any test to be executed, it always exceeds S, F- P, and related rates.
Automatic Test Generation
We first discuss how the test generation task differs from code repair, before introducing novel code diff format based on these insights that is optimized for fault tolerance. Finally, we propose a range of test generation methods based on directly querying LLMs and leveraging Code Agents.
4.1 Test Generation vs Code Repair
Automatic test generation is closely related to code repair: Instead of predicting a patch X that fixes the described issue and is then evaluated using a golden test I** we aim to predict reproducing tests T which are then evaluated on both the original state of the codebase R and the state after applying the golden code patch X*. However, there are some key differences between the two tasks: First, adapting an existing test suite to reproduce an issue typically only requires adding new tests. Concretely, 71% of golden tests in SWT-BENCH only add new test functions, with another 28% modifying existing functions, and only 1% removing functions. Second, testing permits and requires a more granular analysis. While fixed code is either correct and passes all test cases or incorrect when failing any of them, generated tests can be correct but irrelevant to the issue (P- P), call relevant code but fail to expose the precise bug (increase in coverage), reproduce the issue with varying comprehensiveness on edge cases (F PP with varying coverage), or fail in different ways.
5

---

demo/file.py
demo/file.py
+++ demo/file.py
2 rewrite
60-4,5 +4,5 ee
3 1
def test_euclideande b):
def test_euclidean(a, b):
assert euclidean 0)
assert euclidean (100, 10) == 10
assert euclidean 100, 10) 10
assert euclidean
assert euclidean(1, 1)
7 end diff
Figure Comparison of the default unified diff format (left) and our fault-tolerant version (right).
4.2 A Code Diff' Format for Automatic Test Generation
Code changes are typically represented in the unified diff format, i.e., in the git patch and diff' format. While using this format to represent code changes is both precise and human-readable, is very sensitive to misspecifications, requiring, c.g., the exact line numbers of code changes to be specified and specific code snippets (including all to-be-changed lines) to be repeated verbatim. As a result, many LLMs struggle to produce well-formed patch files (Jimenez et al. (023). Even when loosening the strict diff requirements and code, GPT-4 only succeeded in 48% of cases, resulting in only 10 correctly reproduced issues.
the generated diff to best-fit part of the
To alleviate this issue, propose adjusted patch format optimized for LLM generation that is easier to adhere to and more robust. Specifically, our custom diff format allows entire functions classes be inserted, replaced, deleted, given the full function or class definition and (fault- tolerant) location in the code. We show an example in Fig. comparing it to the unified diff format. Based on whether the model wants to rewrite an existing function insert new function, the provided code is then substituted or inserted at the code location. This format is particularly well suited for test generation which usually only requires adding test functions. We provide a more formal description of this format in App. and demonstrate its effectiveness in S
4.3 Direct LLM Generation of Tests
We consider four baselines for test generation: Direct zero-shot prompting with the unified patch format (ZEROSHOT), zero-shot prompting with our novel patch format (ZEROSHOTPLUs), select- ing the best of 5 patches using oracle (Pass @ 5), and the sate-of-the-an test generation method, LIBRO (Kang et al.] 2023). which uses a range of heuristics to pick the most promising among multiple generated tests. In all methods, the LLM instructed to add tests reproduce and cover the described issue in the codebase. We describe these methods below, deferring further details to App. E
ZEROSHOT prompts the model with the issue description, subset the codebase retrieved using BM-25 (Robertson & Zaragoza 2009), and instructions to generate patch file in unified diff format.
This method corresponds to the LLM-only baseline in SWE-BENCH (Jimenez et al. 2023).
ZEROSHOTPLU'S is similar to ZEROSHOT but leverages our custom diff format, discussed in $4.2 which is optimized for LLMs and robustness to minor specification errors.
PASS @ uses our ZEROSHOTPLUS prompting scheme to generate 5 proposal tests and then uses an oracle pick the best one. While this of course not practical in real-world setting, it allows assess the potential of the LLM to generate good test cases given an effective selection mechanism.
LIBRO (Kang et al. 2023), is the current ofat--of-thh-ar for LLM-based test generation. Similar to PASS @ it generates multiple proposal tests using ZEROSHOTPLUS prompting. However, instead of using oracle, it combines multiple heuristics select the best test cases. In particular, it runs all generated tests and then selects the one inducing an error that is most similar to the problem description. This permits not only checking whether generated diff is well-formed and the pro- posed test fails on the original codebase but also selecting the most relevant test case, As LIBRO was originally proposed for Java, we adapt
Python setting, detailed in App. B
4.4 Code Agents for Test Generation
L.LM-based agents are systems that take actions based on I. L.M-generated text, providing tools to observe and interact with their environment over multiple turns and preserve some state across these

---

turns. In the case of Code Agents, they can typically search, read, and edit code using an agent computer interface (ACI) (Yang et al. 2024). Recent work has shown that such Code Agents are particularly effective for complex repository-level code synthesis and repair tasks, outperforming unaided L. by a significant margin (Bouzenia al,, 2024b] OOeenDevin] [024] /Zhagg et al.) 2024 Yang et al., 2024 [Tao et al. 2024). In this work, we leverage Code Agents for automatic test generation by adjusting their instructions. Specifically, we adapt SWE-AGENT (Yang et al., 2024).
AIDER (Aider
2024), and AUTOCODEROVER (Zhang al. 2024).
SWE-AGENT Yang et al., 2024 provides the LLM direct access to (augmented) command line tools and processes the output of these tools to be more easily parseable by an LLM. In particular, they provide special tools for searching, viewing, and editing files. Beyond initial instructions, they provide little guardrails or structure for the LLM and let it interact with a limited shell environment.
AIDER (Aider 2024) performs a repository indexing step guide file selection and then includes all selected files in the next prompts. Further, model-generated summaries and reflections on previous actions are leveraged to augment the context. Before edit is applied, it undergoes validation via static analysis and repository test cases using project-specific evaluation harnesses.
AUTOCODEROVER (Zhang et al. 2024) separates the code repair task into two distinct stages. In the first stage, the LLM is tasked with collecting all required context for the task at hand. To this end, it is equipped with range of advanced code search and navigation tools, allowing it, e.g., to retrieve class signatures, function definitions, or surrounding code snippets. Once the LLM believes it has gathered sufficient context, it proceeds the second stage. There, the L.MM is tasked with generating the actual code patch in a single step, retrying only if the patch can not be applied.
Adapting Code Agents for Test Generation As SWE-AGENT, AIDER, and AUTOCODEROVER were designed for program repair, we adapt their system and instruction prompts to focus on creating high-quality test cases. We find that the underlying L.MSs are capable of following these changed instructions and successfully generate test cases for up to 87% of issues. Typically, the instruction changes were as simple as replacing phrases like "solve the issue" with "create unit tests that cover the issue". We provide a more detailed description of the used prompts in App. B
We experiment with instructing SWE-AGENT explicitly to execute the generated test cases before submitting them. We call this variant S SWE-AGENT from 15.9% to 18.5% (see Table Note we do not provide any information on how to run the tests. This contrasts the LIBRO setting, in which the test execution commands are assumed to be known.
and find that this increases the success rate S
5 Experimental Evaluation
We leverage BWTCHENCH to compare the performance of different test generation methods and underlying LLMs ( (5.2). their relation with the code repair setting ($5.3), and the impact of instance
characteristics 5..4. We further explore hyperparameter ablations in App. C
5.1 Experimental Setup
We consider GPT-4 (gpt-4- 106-prev OpenAl 2023), GPT-4o mini (gpt-4o-mini-2024-07)
2024), Claude 3.0 Haiku (Anthropic, 2023). Claude 3.5 Sonnet (Anthropic (024)), Mis- tral Large 2 (Team 2024b) (served via the Mistral AI API), and Mixtral 7x22b (Team served by Together Al TogetherAl 2023), as underlying LLMs, using GPT-4 unless indicated otherwise. 0 for all zero-shot methods and agents and at / = 0.7 for LIBRO and 5. For SWE-AGENT, AIDER, and AUTOCODEROVER, we use their default settings, restricting the number of API calls to 20, reflection steps to 3, and interaction rounds to 10, re- spectively. For LIBRO we sample tests. Due to budget constraints, we focus our evaluation on SWT-BENCH-LITFE. In App. C we explore and justify this choice of hyperparameters in detail.
We sample at temperature
5.2 Automatic Test Generation
Comparing Test Generation Methods We compare test generation performance in Table 2 where all methods have access only to the issue description and the original codebase. We ob- serve that using the original git code-diff format, ZEROSHOT only generates well-formed patches
7

---

for 48.6% of issues. Using our novel test-specific code-diff format (ZEROSHOTPLUS) boosts this rate to 89.5% yielding an almost 3x increase success rate (S) to 9.4%. While pick- ing the best among five generated tests (PASS@5) even yields an S of 20.3%, the heuris- ties employed by LIBRO can only convert about half of this gap into an of 14.1%, still beating AUTOCODEROVER and ATDER which achieve an S of 9.1% and 12.7% respectively. SWE-AGENT, however, outperforms LI- BRO at 8 of 15.9%, increased to 18.5%, when instructed to check its generated tests (SWE-AGENT+) This stronger performance is significant at 0.1%. Interestingly, SWE-AGEN produces fewer - x tests than AIDER and LIBRO despite having much higher applicability and yielding a higher S.
Table 2: Rate of well-formed patches (W), successful tests (S), potentially reproducing initially failing tests (F - x), reproducing fail-to- pass tests (F + P), and correct but unhelpful pass- to pass tests (P + P), in %.
Method
W
S
F
F-P
P-P
GOLDEN
100.0
100.0
100.0
100.0
11.2
PASS@5
93.1
20.3
62.7
22.1
7.2
ZEROSHOT
48.6
3.6
38.8
5.8
3.6
ZEROSHOTPLUS
89.5
9.4
55.4
10.1
7.2
We conclude that general-purpose Code Agents already perform as well as domain-specific test generation methods, with simple test-specific adjustments providing significant improvements.
LIBRO
92.0
14.1
60.1
15.2
7.2
AUTOCODEROVER
47.1
9.1
43.8
9.1
7.6
AIDER
66.7
12.7
57.6
17.0
8.7
SWE-AGENT
87.3
48.2
16.7
9.8
SWE- SWE-AGENT+
85.5
18.5
46.4
19.2
10.1
Coverage of Generated Tests We analyze the change coverage AC of the generated tests, i.e.,
Table Change Coverage AC [%] as defined in 3.3 aggregated over all instances, S in- stances and non instances (-S).
the portion of executable golden patch code that is covered by the generated tests, in Table 3 Across all methods, we observe a significantly higher coverage on successful instances (ACS), indicat- ing that coverage indeed correlated with test quality but more granular than S. Interestingly, SWE-AGENT+ achieves notably higher coverage on successful instances than SWE-AGENT high- lighting the impact of providing agents with more rest-generation-specffe tools to identify promising tests. Further, LIBRO achieves lower coverage than most Code Agents, most likely consequence of preferring shorter tests.
Method
AC
ACs
AC
GOLDEN
72.0
72.0
PASS@5
31.3
65.6
22.5
ZEROSHOT ZEROSHOTPL.US
7.6
34.9
6.6
21.5
76.7
15.7
LIBRO
23.8
64.2
17.0
AUTOCODEROVER
17.9
61.3
13.6
AIDER
27.8
59.5
23.1
SWE-AGENT
26.5
64.7
19.1
SWE-AGENT4
27.6
69.4
18.0
Model Effect We compare the effect of different underlying LLMs for SWE-
Table 4: Comparison of different underlying LLMs for SWE-AGENT, all in %.
AGENT in Table 4 We observe that only S but even applicability (W) is highly sensitive to the underlying LLM's performance, with Haiku, GPT-4o mini, and Mixtral achieving significantly lower performance than GPT-4. More capable models like Claude 3.5 Sonnet and Mis- Large perform on par, with the lat- ter even outperforming GPT-4.
Model
W
S
F x
ACall
Mistral L.
76.1
16.3
51.4
23.0
GPT-4
87.3
15.9
48.2
26.5
CI. 3.5 Sonnet
67.8
12.3
59.1
30.3
GPT-4o mini
71.0
9.8
36.2
20.9
CI. 3.0 Haiku
20.3
2.5
6.9
3.0
Mixtral 8x22B
3.3
0.7
1.8
0.9
5.3 Code Repair and Test Generation
Table 5: Performance of ZEROSHOTPLUS, given the test file to change, none (-), the golden (v) or an in- correct (X) code patch, and the files retrieved via BM-25 (r), or modified by the golden (v) or incorrect patch (X).
Test Generation for a Given Code Patch To assess the effectiveness of au- tomatic test generation testing specific, provided fixes, we investigate the effect of providing a (possibly incorrect) code patch, the files it changed, and the test file to be modified instead the files re-
Test File
Mod. Files
Patch
W
F+x
AC
87.8
52.3
12.5
/
85.5
10.5
61.0
18.4
X
75.0
10.5
50.0
13.2
trieved with BM25, reporting results in
76.7
15.1
59.3
17.6
8

---

Table in %. We use ZEROSHOTPLUS to generate incorrect patches, resampling I5 times and excluding instances where we could not generate an incorrect but applicable patch, reducing the sample size to n 172. Providing the test files to change almost doubles S from 8.1% to 15. 1 1,, pulling even with SWE-AGENT. We observe that meanwhile providing a code patch and the files it changed has a much smaller impact, increasing S only to 10.5% for both the golden patch and an incorrect patch. This highlights the importance of retrieving the correct context for generating relevant tests. Meanwhile, GPT-4 is able to leverage the correct patch, and to improve the coverage increase of the relevant lines by almost 50%, from 12.5% to 18.4%.
Filtering Code Fixes with Generated Tests State-of-the-ar code generation methods only solve around 20% of cases on SWE-BENCH-LINE (Jimenez et al. 2023). Without suitable tests to distinguish correct from incorrect fixes, the overhead from manual testing (Yang al.. 2008 would thus outweigh any benefits from automatic code generation. To address this issue, we use SWE- AGENT to generate both bug fixes and tests, in a similar manner to thhh et al. (2023a). We then filter the generated bug fixes, retaining only those where all generated tests are F-P or P- While only achieving 20% recall, this more than doubles the precision of SWE-AGENT to 47.8%, making it significantly more practically useful, highlighting the importance of test generation, and opening avenue to transferring the results from Chen et al. (2023a) towards more complex and realistic scenarios with more expensive inference and evaluation steps.
Correlation of Test Generation and Code Repair We analyze the overlap between solved instances of SWE-BENCH and SWT- showing results in Table 6. We observe that the overlap is small for both methods, with no statistical evidence of correlation (p-values of 80.4% and 72.8% for ZAROSHOTPL.US and SWE-AGENT, respectively, under the null hypothesis of independence and uniform hardness), indicating that generating tests and fixes are distinct tasks of different difficulties. We explore this relationship in more detail in App. D
Table 6: Overlap in solved instances of SWE-BENCH and SWT-BENCH.
SWT
SWE
Overlap
Value [S}
ZEROSHOTPLUS SWE-AGENT
26
16
80.4%
72.8%
5.4 Test Generation Success and Instace Characteristics
Effect of Issue Description Length We investigate the relationship between issue description length and test gener- ation performance in Fig. S We ob- serve general trend that issues with longer descriptions are easier to gener- ate tests for, with all methods achieving a higher for longer descriptions, how- ever tending to slightly decrease for very long descriptions. This is likely due the increased amount of infor- mation available in longer descriptions, while too-long descriptions may contain many distractors and make it difficult extract relevant information for the LLM. SWE-AGENT+, which actively summarizes context, limiting file content and reducing history, is least sensitive to issue description length, achieving approximately the same S for all but the shortest lengths.
% SSeccess Rote (8)
20.0-
17.5-
ZeedShotPlu
15.0-
LIBRO
12.5-
Cod:oooe
10.0-
7.5
Aidkr
SWE-Agur++
0.0-
3al
300
Figure 5: Distribution of success rate (S) across issue de- scription lengths in # tokens
Effect of Data Contamination As SWT-BENCH is based on historic GitHub issues, they may be contained in the pre-training data of the LLMs we use. To investigate this issue, we con- ducted an experiment compar- ing the performance of ZEROSHOTPLLCS on all issues created after the Knowledge Cutoff (KC) of GPT-4 (April 2023) to random subset of the same size of instances created before, and report the results in Table 7 While we observe small performance difference, we can not confirm its statistical significance (p as 37%) due to the low number of samples created after the KC. Further,
Table 7: Performance of ZFROSHOTPL.US on PRs before/after
GPT-4 knowledge cutoff (KC 30th April 2023) in %.
PR created
W
S
F+x
F+P
P+P
ACall
before KC
83
56.6
6.0
42.2
8.4
4.8
35.9
after KC
83
47.0
4.8
39.8
4.8
3.6
35.9

---

all methods in Table 2 use the same underlying LLM and should thus benefit from any potential contamination to similar degree, allowing for fair comparison between different methods.
Method Complimentarity We consider four di- verse methods from $5.2 and analyze the overlap in the instances for which they are able to generate suc- cessful tests. We show the results in Fig. 6 While the best-performing approach, SWE-AGENT+, alone is able to solve 51 instances, the combination of all four approaches is able to solve 87 instances, highlighting the benefit of exploring a variety of approaches for test generation.
LIBRO
Akder
RoverrdeRover
SWE Agentt+
25
35
10
6
51
1
3
10
16
7
3
6 Limitations and Future Work
Figure 6: Overlap in instances solved by the four best performing methods.
While our novel SWT-BENCH covers a wide range of real-world issues, it has several limitations: It is lim- ited to Python, which may limit the generalizability of our findings to other programming languages. Second, the dataset is based on popular GitHub repositories, which may be representative of common software development practices and does preclude the generation of a private holdout test set. Finally, the dataset limited bug reproduction and issues that can be easily covered by adding test cases and does not measure edge case detection or global coverage increase.
Further, discussed in $5.4. most issues in SWT-BENCH have been created before the knowledge cutoff of state-of-the-arr models, posing a risk for data contamination. One approach to address this issue is to create rolling version of SWT-BENCH, based only the most recently created GitHub issues. However, this comes at the cost of direct comparability of results and increased cost for reproducing results for all baselines on a changing evaluation set.
Addressing these limitations would be an interesting direction for future work. As concrete starting points, we found several common errors even in the best performing method SWE-AGENT that could be addressed through specialized monitoring: Adding passing tests that do not reproduce the given issue, getting stuck in loops after generating inapplicable edit commands, failing to execute the test environment correctly and adding tests with syntax errors or using invalid variables.
Conclusion
We proposed SWT- SWT-BENCH, novel benchmark for generating reproducing tests from GitHub issue descriptions and the corresponding code bases. SWT--ENCC leverages the dataset underlying the popular SWE-BENCH which additionally contains golden patch fixing the described issue. We judge whether a generated test reproduces the described issue by checking whether the test fails before applying this golden patch and succeeds afterward. We measure both the rate of such fail-to- pass tests and the coverage of the golden patch, providing corresponding evaluation harness. We evaluated a variety of LLM-based test generation methods including Code Agents on SWT-BENCH and found that Code Agents already outperform other approaches with only minor adaptations for test-generation task. Finally, we demonstrated the great potential of generated tests to serve as signal for the correctness of code fixes, i.e., we double the precision of Code Agents by filtering the generated patches to only those that cause a previously failing celf-generated test to pass.
10

---

References
Aider. Aider is SOTA for both SWE Bench and SWE Bench Lite, Jun 2024.
Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improve- ment using large language models at meta. CoRR, abs/2402.09171,, 2024a. doi: 10.48550/ARXIV. 2402.09171.
Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improve- ment using large language models at meta. CoRR, abs/2402.09171 2024b. doi: 10.48550/ARXIV.
2402.09171.
Anthropic. Introducing Claude, Mar 2023.
Anthropic. Introducing Claude 3.5 Sonnet, Jun 2024.
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv preprint, abs/2108.07732. 2021.
Islem Bouzenia, Premkumar TT. Devanbu, and Michael Pradel. Repairagent: An autonomous, Ilm- based agent for program repair. CoRR, abs/2403.17134 2024a. doi: 10.48550/ARXIV.2003..
17134.
Islem Bouzenia, Premkumar T. Devanbu, and Michael Pradel. Repairagent: An autonomous, Ilm- based agent for program repair. CoRR, 2024b.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In Proc. of ICLR, 2023a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, 2021.
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. Chatunitest: A framework for llmebased test generation. arXiv e-prints, 2023b.
Dan Hendryeks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In NeurlPS Datasets and Benchmarks, 2021.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2e: Turning any github repository into programming agent environment. In ICLR 2024, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? ArXiv preprint, abs/2310.067777 2023.
René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: database of existing faults to enable controlled testing studies for java programs. In Proc. of ISSTA, 2014. doi: 10.1145/261338, 2628055.
Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Exploring llmebased general bug reproduction. In Proc. of ICSE, 2023. doi: 10. 10.1099/CSF486199222. 00194.

---

Sungmin Kang, Juyeon Yoon, and Shin Yoo. LLM-powere test case generation for detecting tricky bugs. ArXiv preprint, abs/2404. 10304. 2024.
Tsz On Li, Wenxi Zong. Yibo Wang, Haoye Tian, Ying Wang, Shing- Bhing-Chi Cheung, and Jeff Kramer. Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In Proc. of ASE, 2023, doi: 10. A09/A/5362299 20..00022000000
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. Science, 378(6624), 2022. doi: 10. 10.1126/sciencc.aagll..
Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. Quixbugs: multi-lingual program repair benchmark set based on the quixey challenge. Proc. of SPLASH, 2017. doi: 10.1145/133932.3113344
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Proc. of NeurIPS, 2023a.
Tianyang Liu, Canwen Xu, and Julian J. McAuley, Repobench: Benchmarking reoee code auto-completion systems. CoRR, abs/2306.0300 2023b. doi: 10:4855/ARXXV.2366.00000
Stephan Lukasczyk and Gordon Fraser. Pynguin: Automated unit test generation for python. In Proc. of ICSE, 2022. doi: 10.1 (0.11455/510444.3116222
OpenAI. GPT-4 technical report. ArXiv preprint, abs/2303.08774, 2023.
OpenAl. GPT-4o mini: advancing cost-efficient intelligence, Jul 2024.
OpenDevin. Opendevin: Code less, make more, 2024.
PYPL. Pypl popularity of programming language index, Aug 2024, URL httpss://web http://bb22444001000/////tttt/// http://ppw...ituuu...ooPPP.....
Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4), 2009. doi: 10.156//150000001
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Trans. Software Eng., 50(1), 2024. doi: 10.1109/TSE.22233333399
Philipp Straubinger and Gordon Fraser. survey on what developers think about testing. In 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023, Florence, Italy, October 9-12, 2023, pp. 80-90. IEEE, 2023. doi: 10.1 10.1000955559994...2022... 20230.000 URL
https://doi.. wrg/10...000oo.. 1101 2.199/ISSEE99BA 2823.00027
Wei Tao, Yucheng Zhou, Wenqiang Zhang, and Yu Cheng. MAGIS: Ilm-based multi-agent frame- work for github issue resolution. CoRR, 11s22403.1792,, 2024, doi: 10.48550/ARXIV.22003.
17927.
MistralAI Team. Cheaper, Better, Faster, Stronger Continuing push the frontier of AI and making it accessible to all., Apr 2024a.
MistralAI Team. Large enough, Jul 2024b.
TIOBE. Tiobe index for august 2024, Aug 2024. URL https://wwb.archive.orgg// 28220888702585//httpp:: //wpttttt::::/../.t tiobe. www.e...o/ttotttttt.ooooooo wiwtom/ttibeiteeeeeeoooo index/
TogetherAI. Together AI API, 2023.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. ArXiv preprint, abx/2009.05617, 2020.
12

---

Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 2024.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent Computer Interfaces Enable Software Engineering Language Models, 2024.
Ye Yang, Mei He, Mingshu Li, Qing Wang, and Barry W. Boehm. Phase distribution of software development effort. In Proc. of ESEM, 2008. doi: 10. I 104144461414004.1441011.
Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. CoRR, abs/2404.05427, 2024, doi: 10.45550AARXX..2000052222
13

---

diff
path or filename
< "rewrite" or "insert" >
< line number / EOF BOF
function to rewrite or insert
end diff
< repeat as necessary
Figure 7: The Custom Diff format for ZEROSHOTPLUS
A Formalization of Custom Prompt Format for ZEROSHOTPLUS
We introduce a custom prompt format for language models aid them with patch generation in the zero-shot setting. The format is visualized in Fig. 7 similar to how it is provided to the language model. full example of applying the format on two files is part of the full prompt of ZEROSHOT- PLUS in Figs. and [2
A diff block must start and end with diff and end diff respectively. The first line inside the block must specify an existing file for rewrites and may point to a new file in the case of insertion. Next, the language model specifies whether it intends to rewrite function. If no exact match of the function name is found, we employ a fuzzy search using the line number EOF/BOF as indicator for where look for the existing functions. EOF and BOF particularly useful for inserting new functions. We note that diff blocks can be repeated an arbitrary number of times.
existing function
insert new
B Adapting LIBRRO to our Setting
Kang et al. (Kang et al. 2023) originally proposed LIBRO for an evaluation in a pass@k setting. There, is useful to rank all generated tests improve performance pass@1 1, we drop ranking components irrelevant for the top-1 test in our reimplementation. Further, LIBRO includes heuristics for importing missing dependencies and inserting tests into correct classes. While effective in Java, this is superfluous for Python, where tests can be added outside classes and dependency imports are (empirically) correctly generated by the LLM. We thus also drop these components.
1. As we only consider
LIBRO clusters cases based on whether the generated execution trace matches the issue descrip- tion. As exact matching is often not applicable for the unstructured issue descriptions, we measure the similarity between the error message and the issue description by extracting the execution trace of the generated test cases and querying the same LLM used for test generation to judge whether they relate to the same issue. Depending on its answer, we obtain two clusters and choose the shortest. result of the preferred cluster.
C Hyperparameter Ablations
C.1 Ablation on number of LIBRO samples
We perform ablation study by varying the number of samples used for the LIBRO baseline. The result is presented in Fig. Sa LIBRO'S performance improves as more samples are considered, however the gains of additional samples marginal from around the 5 samples use by default. As shown in section App. G to enable comparison at similar cost to Code Agents, we settle for samples.
C.2 Ablation on Interaction Rounds for Code Agents
We analyze the number of interactions required for each Agent to submit a solution and plot the results in Fig. 8b We observe increasing interaction rounds improve performance until saturation at 5-10 iterations (we use 20 as a default) with the only exception being AutoCodeRover, which still gains performance up to the maximum of 20 iterations we consider.
14

---

Table Comparison of ZEROSHOPPLUS for different on GPT-4 (95%
n 25).
T
W
S
F+x
F+P
P-P
Acall
0.0
89.5
9.4
55.4
10.1
7.2
21.5
0.2
89.3 + 0.3
8.8 + 0.5
56.2
9.6
6.0 0.2
21.8 0.3
89.9 0.5
10.1
5.6
1..9
6.1
22.3
0.7
89.3 + 0.4
8.8 + 0.4
55.5
9.7
6.1 0.5
21.1 0.6
SWI--GGAT
SWI-AGEET AETOCYORROVER
15
AIDER
W
10
20
10
Samples
Calls
Calls
(a) Ablation of W and by num- LIBRO samples.
(b) Ablation of W and S ende agent number of needed
API calls until solution submission.
Figure 8: Ablation the number of samples and API calls for LIBRO and code agents resp.
C.3 Ablation Temperature
We run ZEROSHOTPLUS
using GPT-4 with 25 samples and analyze the performance variation for
temperature range from greedy decoding (T' 0). used for ZEROSHOT, ZEROSHOTPLUS and the agent settings, to 7" = 0.7, the setting used in LIBRO. The results are presented in Table g We observe a tendency of decreased performance and increasing variance all metrics for increasing T. Moreover we observe minimal variability among several runs of the test environment at = 0, however much smaller than the variability due to temperature.
D Distribution over Repositories
We compare the success rate of SWE- AGENT for test and fix generation across repositories in Fig. g We observe that,
S Saccess Rate (8)
50-
20-
while SWE-AGENT obtains similar suc- cess rates in both settings in three reposi- tories, success rates vary strongly on most other repositories. Indeed, there are five repositories where test generation fails en- tirely while code repair fails on three and on only two of these both fail. Manually inspecting instances from the repositories where test generation fails shows variety of reasons, astropy usually features complex computations where unit test creation is difficult, requests features highly complex code base, flask has extremely long golden test lengths indi- cating particularly challenging testing problems. For pylint generated tests all P P making them correct but unhelpful.
30
SWR-Bmeh
SWI-Bend
10-
d-
astropy
djugo
nutphellb
pefint
npyred
resneds
vellat
waber
splite
Spupy
sarray
Figure 9: Distribution of success rates across reposito- ries for SWE-AGENT
E Full prompts
ZEROSHOT, ZEROSHOTPL.US and LIBRO The full prompt for ZEROSHOT is displayed in Figs. T0 and TT The full prompt for ZEROSHOTPLLUS and LIBRO displayed in Figs. and [3 Except for the way we include files, all lines are changed with respect to the setting in SWE-BENCH
15

---

Table 9: Cost of different LLMs running SWE-AGENT on SWT-BENCH Lite in USD
Model
GPT-4
Haiku
Mixtral 8x22B
Mistral Large
Sonnet
GPTT-4o mini
Cost
290.71
10.28
67.90
211.34
263.13
8.43
Table 10: Cost of running different methods on SWT- Lite using GPT-4 in USD
Method
ZEROSHOT
ZEROSHOTPLUS
PASS@5
LIBRO
Cost
82.13
80.70
403.65
420.14
Method
AIDER
AUTOCODEROVER
SWE-AGENT SWE-AGENT+
Cost
256.10
368.40
290.71
478.21
This includes in particular the demonstration of the unified diff format on example. In the setting for Table 5 we add the lines highlighted boldface.
SWE-AGENT and SWE-AGENT+
The prompt for SWE-AGENT and SWE-AGENT+ is shown
in Fig. 14 Changes with respect to the prompt of (Jimenez et al. 2023) are highlighted in boldface. The additional changes for SWE-AGENT+
highlighted in green.
AIDER We only minimally adapt the provided evaluation harness for AIDER on SWE-BENCH In this harness, AIDER is provided with single initial user prompt based on the user issue, while the entire agent workflow remains unchanged. We provide the entire prompt in Fig. and highlight our change in boldface.
RoVEER AUTOCODEROVER (Zhang al. 2024 leverages number of prompts that
are provided to the model in different phases of the code/test generation process. We adapt the key prompts and display them in Fig. [5] Changes are highlighted in boldface. Further, we change every occurrence of "bug location" in the original prompts to "relevant location". We further add function the ACI that allows inserting code new files and fetching the entire code (capped the first 100 lines) of any file.
F Licenses of used Code
We adapt code from the following projects in our work and include the respective licenses:
1. SWE-BENCH (Jimenez et al.
2023 MIT License
2. SWE-AGENT Yang et al.
2024): MIT License
3. AIDER (Aier 2024): Apache License 2.0
4. AUToCODEROVER (Zhang et al. 2024): GNU General Public License
For all licenses of the repositories used in SWT-BENT BENCH, refer to Jiminez al. (Jimenez al., 2023 which contains detailed list with licenses for each repository included.
G
Computational cost
There is cost to both running inference on Language Models and on evaluation their predictions on the test suites of the repositories. Since the evaluation can be performed on a consumer grade machine in reasonable time, we focus on the cost inferred from LLM inference. We report the cost for each setting Tables 9 and [TO] displaying the average cost of full inference on SWT-BENCH Lite for each model and method. The difference between cost of PASS @ and LIBRO is just the additional filtering step incurred by LIBRO.
https://giihub..oo//aaa http:///ww....oouuuuuouoceeo/eee
16

---

Table 11: Average execution time per instance
Method ZEROSHOTPLUS
LIBRO
SWE-AGENT
SWE-AGENT+ AUTOCODEROVER
12.6s
2m52s
3m42s
4m25s
5mls
H
Execution times
We run the different methods from Table 2 5 instances and compute the average execution time. For all LLMs we consider, part of the execution time directly related to the number tokens digested and generated (see Table TO). For methods that require interaction with execution en- vironment however, time usually dominated by setting up such an environment in a clean and reproducible manner (i.e. dockerized). We list results execution times in Table TT and observe that all methods except zero-shot inference take between 3-5 minutes per instance, where we can observe small trade off due many-tum interactions in Code Agents versus single-shot execution in LIBRO. Given these small differences however, we believe execution time to be of limited prac- tical relevance issues can be processed in the background, similar continuous integration, in response to raised user issues
17

---

The following text contains Further, you are provided with file contents several files in the repository contain relevant code (in <code> brackets). It may be necessary to use code from third party dependencies or files not contained in the attached documents however. Your task is to identify the issue and implement test case that verifies text.
issue (in <issue/> brackets) posted at repository.
proposed solution to this issue. More details at the
<issue>
user issue comes here </issue>
retrieval results or oracle files come here
Please generate test cases that check whether implemented solution resolves the issue of the user the top, within <issue/> brackets). 11 Present the test cases unified diff formatting.
12
The general format of diff is the unified output format, described follows. The unified output format starts with two-line header, which looks Like this:
16 ... from-fill
17 ++ to-file
Next
hunks di differences; each hunk differ. Unified format hunks look Like this:
where the files
20
21 @ from- fww-ffile-lie Line-numbers to- file-le-lie numbers 00
Line- mrom-eihher-file
23 Line eiih-eiter--file
24
25 If a hunk contains just one line, only its start line number appears. Otherwise its line numbers look like 'start,count', An empty hunk is considered to start at the line that follows the hunk.
26
27 If a hunk and its context contain two or more lines, its line numbers look like 'start, count', Otherwise only its end line number appears. An empty hunk is considered to end the Line that precedes the hunk.
29 The lines common to both files begin with space character. The lines that actually differ between the two files have one of the following indicator characters in the Left print column:
31
'+' line was added here
file.
32 line was removed here from the first file.
33
Insertion only be done at the end beginning of the file, indicated EOF or BOF follows.
respectively.
35
36 As an example for a diff, consider the following two versions of the same file, once before and once after a change.
The original version of the file
38 [start of demo/test_filee.py
39 def test_euclidean(a, b): assert euclidean 0) = 0
40 2
41 assert euclidean
assert (uclidean
43 assert euclidean
44 6
45 7 @pytest.mark. paraakkkarammerriee("" b, expected", [[0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1)
})
def tedtaggd(aa, b):
47 9
assert (a, b) = expected
48 10
[end of demo/file.pyy
Figure 10: Part the Prompt for ZEROSHOT on SWT-BENCH
18

---

The diff for fix in function euclidean and adds the function gcd is as follows This diff changes the first file into the second file. diff
... a/demo/tile.py
+++ a/demo/tile.py
-4,4 +4,5 euclidean(1, 0)
assert euclidean (1, 1) 1
assert euclidean (100, 10) == 10
@pytest. mark. param b, expected", [(0, 0), (0. 8, 1), 1)
13 @ -9,2 + Ce
14 assert ged(a, b) == expected
+@pytest. mark. parametrize("a, b, expected", [(B, 0), (0. (1, 8, 1), (1, 1). (100, 10, 10)1)
17 +def test lcm(a, b):
assert lam(a, b) == expected
20
21
22 The new version of the file is follows. [start of demo/file.pp
def eusiidecii,,,,,,
25 assert @
26 3
assert euclidean (0, 1) ==
27 4
assert @uclidean 0) ==
assert (1,
29
assert euclidean 100,, 10) 10
30
31 8 @pytest. mark.. parametrize("a, b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1)
}]
def test gad(a, b):
39 assert gad(a, expected
34 11
35 12 @pytest. mark. parametrize("a, b, expected", [(0, 9, 0), (0, 1). (1, 8, 1), (1, 1, 1), (100, 10, 10)))
36 def temt_ b):
37 14 assert lcm(a, b) = expected
38 15
[end of demo/file.pyl
41 you can see, you need to indicate the approximate Line numbers, function name and path and file name you want to change, 42 but there can be as many independent blocks of changes as need. You also apply changes to several files.
Apply much reasoning as you please and see necessary. The format of the solution fixed and has to follow the custom diff format.
44 Make sure to implement only test cases and don't try to fix the issue itself.
Figure 11: Part the Prompt for ZEROSHOT on SWT-BENCK
19

---

The following text contains user issue <issue/> brackets) posted repository. Further, you are provided with file contents of several files in the repository that contain relevant code (in <code> brackets). It may be necessary to use code from files not contained in the attached documents however.
third party dependencies Your task is to identify issue and implement test case that verifies proposed solution to this issue. More details at the of this text.
<issue>
issue here
</issue>
The following patch has been proposed to fix the issue described in the user issue (in <issue/> brackets). The patch might give you hint on how to write a covering test for the issue, but you should not assune that the patch is correct. It might be that provided patch
should check whether the patch resolves the issue. issue. spteeeppophhppppeeee patch</patch>
correct, your
retrieval results or oracle files come here
10
Please generate test cases that check whether implemented solution resolves the issue of the user (at the top, within <issue/> brackets). 13 Present the test cases as a diff (custon format, explained below).
14
15 The general format of diff is as follows.
"ccusoomddff
diff
18 <path/filename>
19 < "rewrite" or "insert" >
rough line number EOF / B0F
insert function that should be added rewritten
22 end
23 < repeat blocks of diff as necessary
24
Insertion only be done at the end beginning of the file, indicated by EDF or BOF respectively.
26
27 As an example for a diff, consider the following two versions of the same file, once before and once after change.
The original version of the file was follows.
29 [start of cemo/test_file.pp
30 1 def test_euclidean(a, b):
31 2
assert euclidean(0, 0) = 0
assert euclidean
(uclidean
34
assert euclidean(1,
35
6
36 m b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1)
})
def gad(a, b):
18 9
assert ga, b) = expected
39 10
[end of demo/file.pyl
Figure 12: Part I of the Prompt for ZEROSHOTPLUS on SWT-BENCH
20

---

end diff diff The diff for fix in function euclidean and adds the function gcd is fol lows. This diff changes first file into the second file.
custom-dir diff
diff
demo/file.pp
rewrite
def test euclideants b):
assert @uclidean(0, 0) ==
assert @uclidean(0,
11
assert euclidean(1,
12
assert euclidean(1, 1) =
13
assert euclidean (60,, 10) == 10
16 demo/file.py
17 insert
18 EOF
pytest.mark parametrizee"", b, expected", [(0, 0), (0, (1, 8, 1), 1, 1),
(100, 10, 1))
20 def test_lcm(a, b):
21
assert lcm(a, b) == expected
end diff
24 The new version the file follows
25 [start of demo/file.pp
26 1 def test_euclidean(a, b):
assert (U
28 assert euclidean
19 assert euclidean (1, ==
30 5
assert euclidean 1) = 1
6
assert euclidean [10,, 10) 10
33 taytestt.irkkkprammriirtt"" b, expected", I(0, 0, 0), (0, 1, 11, (1, 0, 1), (1, 1, 1)
1)
34 9 def test_gcd(a,, b):
35 10
assert (a, expected
37 faytest.marke parametrizee"a, b, expected", [(0, 0, 0), (8, 1), (1, 0, 1), (1, 1, 1), (100, 18, 10)1)
13 def temt_ b):
assert lcm(a, expected
10
41 [end of demo/file.pyl
42
As you can path and file name you
need to indicate the approximate Line numbers, functian name and the change,
44 but there can be as many independent blocks changes as you need. You may also apply changes to several files.
Apply much reasoning fixed and has follow the custom diff format.
you please and necessary. The format of the solution is
Make sure implement only test cases don't try to fix the itself.
Figure 13: Part 2 of the Prompt for ZEROSHOTPL.US SWT-BENCH
21

---

We have received following issue within our repository. Here's the issue text:
ISSUE:
(issue)
INSTRUCTIONS:
Now, you re going to create unit tests that cover the issue. In other words, you should write unit tests that fail in the current state of the repositorybut, will pass when the issue has been resolved. Essentially, you'll ll want to write unit test that reproduces the described
Your terminal session has started and you're in the repository's root directory. You can any bash commands the special interface to help you. Edit all the files you need to and run any checks or tests that you want.
Remember, YOU CAN ONLY ENTER ONE COMMAND AT A TIME. You should always wait for feedback after every command.
When you' re satisfied with all of the changes you've made, you submit your changes to the code base simply running submit command.
10 Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with "python <script- namet.paaaappppp py"
12 NOTE ABOUT EDIT COMMAND: Indentation really matters! When editing a file, make sure to insert appropriate indentation before each line!
IMPORTANT TIPS:
Always by trying replicate bug that issues discusses.
16
If the issue includes code for reproducing the bug, we recommend that you re- implement that in your environment, and run it to make sure you can reproduce the bug.
Then start trying to fix it. when you think you've fixed bug, re-run the bug reproduction script to make sure that the bug has indeed been fixed.
18
19
If the reproduction script does not print anything when it successfully runs, we recommend adding a ("int("S completed successfully, errors. command the end the file, 50 that you can be sure that the script indeed ran fine all the way through.
21
22
you command and doesn't work, try running different command. command that not work once will work the second time unless you modify it!
24
25 3. If you open file and need to get to area around specific line that is not in the first 180 lines, say line 583, don't just use the scroll_ down command multiple times. Instead, the goto 583 command. It's much quicker.
26
27 4. If the bug reproduction script requires inputting/readiig specific file, such as and you'd like to understand how to input that file, conduct search in the existing repo code, to whether someone else has already done that. this by running command: find_ file "buggy-input.ppg If that doesn't 't work, use the linux 'find' command.
ouggy-input..om
78
29 5. Always make sure to look at the currently (which appears right after the currently open file). The currently open file might be in different directory than the working directory! Note that some commands, such as 'create', open files, so they might change the current open file.
file and the current working directory
30
6. When editing files, it is easy to accidentally specify wrong line number or to write code incorrect indentation. Always check the code after you issue an to make sure that reflects what you wanted to accomplish. If didn't, t, issue another command to fix it.
7. After having applied your changes and before submitting, make sure pytest and check if the code *fails* as expected due to the issue description. If it doesn't, revisit your changes and adapt accordingly.
Figure 14: The Prompt for SWE-AGENT on SWT-BENCH
22

---

You are software developer maintaining large project.
You working
issue submitted your project.
The issue contains description marked between <issue> and </issue>.
Your task is to invoke a few search API calls to gather information about relevant code the
lines, then write unit tests to capture the described behaviour Aesue.Ideally. the unit tests should fail before the bug is fixed or the requested feature is added,
pass after. you but just capture the behaviour described in the issue by creating appropriate test cases.
trying solve the bug itself,
You are a software developer maintaining a large project.
You are working
issue submitted to your project.
The issue contains description marked between <issue> and </issue>.
You ultimate goal to write one or more unit tests that capture this Ideally
unit should fail before the bug fixed or the requested feature is added, and pass after.Note you are not trying to solve the bug itself, but just capture the behaviour described in the issue creating appropriate test cases.
Write more unit tests for the issue, based the retrieved context.
You can import necessary libraries.
Return the tests patch in the format below.
Within "<filex</file>", replace with actual file path.
Within "<rriginal></originaa""" replace with the original code snippet from the program.
11
12 Within <<patchedb<<patchhdd=> replace .... with the fixed version of the original code. When adding orignal code and updated code, pay attention to indentation, as the code is in Python.
You can write multiple modifications if needed.
14
15
# modification
<files..
19 <patched=.... </ptedchee..pppthhee
20
21 # modification
<files...
23 poriginals...ooriiginnn
24 satthedd....ppatccheee
modification
28
Figure 15: The Prompt for AUTOCODEROVER SWT-BENCH
23

---

Below real GitHub issue from a popular GitHub repository.
The issue was filed some time ago.
The repo has been checked out at the commit that existed at the moment the issue was filed.
you already familiar with this repo, cautious!
You are working with an version the repo!
Filenames, directory names, file contents, etc may be different than what you re used to
Propose changes to update repo reproduce the problem below.
You're going to create unit tests that cover the issue. In other words, you should current state of the repository
write unit tests that fail but will pass when the issue has been resolved. Essentially, you'll want write unit test that reproduces the described issue.
11
{issue}
Figure 16: The Prompt for AIDER on SWT-BENCH
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, as described in $
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations thoroughly in 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
There are no theoretical results included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe our approach in detail and publish all code, data, and instructions required to reproduce our results. All experimental details can be found in $ and full details on used prompts are laid out in App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release all code, data, and instructions required reproduce results online at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.cc/ pabLEc//guioss//ooouuuissSPPooo, for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
7 //nips. wwtpsccccpbbllie httpgguiddee/ooouumiimissss
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We describe key experimental details in prompts for LLM interaction are provided in App.
$5 Further details on the used
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We add results of tests of statistical significance in the text accompanying experiments in $5 We further provide temperature ablation with error bars App.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ys]]
Justification: We outline the cost of the significant parts of experiments (mostly domi- nated by the cost of LLM inference) in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the lines
NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research conforms the NeurIPS Code of Ethics. In particular, did not conduct research involving humans, used only publicly available data open source data with permissive licenses, and believe that our research has negative societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular upplications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators have included the licenses of all used code in the submitted code together with the package. The code adjusted and used for the experiments this paper was licensed with permissive licenses, that we respect and redistribute with out adjusted code.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
29

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, [aaperswi com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Ye]]
Justification: We provide detailed description of our dataset creation process in $ and distribute this description as well as license information alongside the dataset at https: //github. com/ logic- star-all aat-aat/swwwwwa--- ai/swt-bench
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
30

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "eT3TZiHa0V", "reviewer": "Reviewer_PhkB", "summary": "Authors present a new benchmark for the SWE tasks of generating tests corresponding to an issue in a codebase with unit tests. They propose to repurpose the SWE-bench dataset for this task. They also evaluate LLM based prompting and agentic approaches (SWE-Agent and AutoCodeRover) with metrics like code change coverage and fail to pass rate that are formally defined and measured.", "questions": "- Lines 328-334: You mention about the impact on precision and recall, can you share more details of the method used and the final correctness rate with synthetic tests generated by the Agent?\n- What are some future directions you would recommend to improve the performance of LLM Agents on SWT-bench and SWE-bench based on your study? A future work recommendation would greatly help the completeness of the paper.", "limitations": "Authors have briefly discussed limitations of their work in the main paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work focuses on generating test cases corresponding to user issues with a codebase with existing unit tests. This problem has been not studied in such detail in my knowledge. Authors have presented this problem with sufficient motivation.\n- While authors adapt an existing code fix benchmark (SWE-bench) for their task (SWT-bench), their contributions in the form of task formulation, corresponding new metrics and exhaustive analysis of the benchmark could be key to future research on this topic.\n- Authors also adapt popular LLM approaches like SWE-Agent and ACR for their benchmark and show their effectiveness on generating tests. The demonstrated impact automated test generation has on automated code fixing is particularly promising. Their results include exhaustive analysis of the performance of different methods on the proposed benchmark.\n- The paper is very well written and clear to follow.", "weaknesses": "- Arguably the most significant impact of automatic test generation would be on automated code repair/generation. While authors describe this aspect of their work (Lines 328-334), providing more details and large-scale experiments could significantly strengthen the perceived utility of this work. \n- CodeT by Chen et al https://arxiv.org/abs/2207.10397 is a key related work that was missed in the related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer PhkB for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the importance and novelty of the problem we study, the importance of our contributions, the extent of our analysis, and the quality of our exposition. Below we address their remaining questions.\n\n**Can you provide more details on how the generated tests improve the precision of code repair agents? And can you conduct more large-scale experiments in this direction**  \nUsing SWE-Agent, we generate tests and code patches independently for all instances. We then run the generated code patches against the generated tests and retain only those code patches where the tests fail before the patch is applied and succeed after. Evaluating the resulting patches against the ground truth tests shows a precision of 47.5%, significantly higher than the precision of unfiltered patches (19.2%). How to best leverage tests directly for code generation rather than simply selection is an active topic of research with many competing approaches. We therefore leave this as a promising future work item.\n\n**How does your work relate to CodeT by Chen et al.?**  \nCodeT focuses on selecting the best implementation(s) from a large set of proposals (100) by generating large numbers of tests and using execution agreement as a quality metric to select correct ones. However, while this approach is effective for function-level, HumanEval-style problems, we focus on significantly more challenging repository-level synthesis. There, neither sampling nor executing such large numbers of different tests and implementations is feasible due to the substantial computational and monetary (when using APIs) cost. In particular, ZeroShot sampling of 100 tests and repairs would cost around $15k for one evaluation of SWT-Bench-Lite. Therefore, we do not compare to CodeT directly. We will include a corresponding discussion in our related work section.\n\n**Can you discuss future directions for improving the performance of LLM Agents on SWT-bench and SWE-bench?**  \nWe recommend future work to further explore the relationship between patch generation and test generation to improve the performance for both tasks. CodeT [1] is an interesting prior work in this direction. However, it relies on a large number of candidate implementations and tests to find agreements, which is too costly for real-world repositories. Therefore, future work might consider developing more cost-effective approaches.\n\nWe have further conducted a detailed analysis of common errors for our best-performing method SWE-Agent+ and will gladly add the findings to the paper for the camera-ready version together with concrete examples of their occurrence. We find the most common issues of the agent to be adding passing tests that do not reproduce the issue, getting stuck in a loop after making inapplicable edits or encountering syntax errors, failing to execute the test environment during test creation, and adding tests with syntax errors. Future work may consider addressing these issues through specialized modules or additional monitoring.\n\nWe are happy to include this analysis, discussion and recommendations for future work in the camera-ready version of the submission.\n\n**Conclusion**  \nWe hope to have been able to address all of the reviewers questions and remain happy to answer any follow up questions they might have.\n\n**References**  \n[1] Chen et al. 2022, CodeT: Code Generation with Generated Tests", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fDcQUhsXfg", "reviewer": "Reviewer_i5fN", "summary": "This paper re-purposes SWE-Bench, a previous benchmark on repository-level code generation, to SWT-Bench, a benchmark for test generation by asking LLMs to generate tests to reproduce the issues for various GitHub repos and see if such tests can capture the bugs before the gold-standard patches are applied and whether the generated tests will pass after the patches are applied (i.e., Fail -> Pass). Experiments are conducted with multiple baselines and results show that SWE-Agent, the previously proposed method for code repair, achieves the best performance for test generation.", "questions": "Please respond to the concerns in the \"Weakness\" section.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. This paper targets on test generation, which is a relatively under-explored area and could be useful for increasing popular coding agents;  \nS2. This work thoroughly studies the relation between the tasks of code repair and test generation, and show how we can turn a code repair dataset into a test generation dataset using SWE-Bench -> SWT-Bench as an example;  \nS3. The designed metrics to evaluate the generated tests (i.e., Fail-to-Pass rate, Change Coverage, and Patch Applicability) are quite interesting and reasonable, which could be useful for further work on automatic test generation evaluation.", "weaknesses": "I think the main weakness of this paper is over-claiming. While it is a clever idea to re-purpose code repair benchmarks as SWE-Bench into a test generation benchmark, there are quite a lot limitations of this method for the conclusions on the resulting dataset to claim \"state of the art software tester\". The main purpose of software testing is not to reproduce the issues, but given the specifications, crafting inputs and the constraints on the outputs to makes sure that the code to be tested has the expected behavior given different inputs. To this end, I believe there is a large gap between the ability that SWT-Bench is testing, and the actual test generation ability needed for the job of software testers. To make such a claim, I believe at least a few software testing benchmarks (e.g., https://swtesting.techconf.org) need to be used and the results to be reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer i5fN for their review and valuable questions. We are happy to hear they consider the problem we study important and under-explored, our work thorough and interesting and our results impactful. Below, we address their remaining concern.\n\n**Is your paper over-claiming and implying that SWT-Bench measures all abilities of human software testers?**  \nWhile our title may be placative, it only implies that Code Agents perform better than other automated systems on the studied task. In this case, this is the task of reproducing an issue from a description in the form of a test. We fully agree with the reviewer that there is a big gap between the skills SWT-Bench is testing and the full spectrum of abilities required by human software testers. We want to highlight that we do not make any such claims anywhere in our paper. Instead, we explicitly focus on reproducing tests (see e.g. the Conclusion (Line 408) and the Introduction (Line 35)). We will make it even more explicit in the revision that reproducing issues is the only measured skill of multiple software testing skills.\n\n**How important is the measured capability of reproducing issues?**  \nWhile reproducing a described issue or testing a specific functionality is certainly not the sole purpose of software testing, we firmly believe it is a very important one. In particular, it is an aspect of software testing that is difficult to solve without powerful NLP processors such as LLMs. The task of issue reproduction requires translating natural language descriptions of these issues/intended functionalities and into formalized definitions of the desired behavior in the form of unit tests. This is in stark contrast to the task of, i.e increasing coverage of a given (subset of a) codebase, or fuzzing, where the target is already formally defined. Finally, our approach permits test generation for test-driven development, as it can generate tests for unimplemented functionality, i.e. when the raised user issue is a feature request. We thus find it important and promising to study this (relatively under-explored) aspect of software testing.\n\n**Conclusions**  \nWe hope to have been able to address the reviewer’s concerns regarding the claims made by our paper and would like to respectfully ask them to reconsider their evaluation given their own positive assessment of our work. We are happy to answer follow-up questions and would greatly appreciate it if the reviewer could highlight specific sections where they believe we overclaim.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HneUpOBBYB", "reviewer": "Reviewer_Ky1F", "summary": "This paper focuses on automatic test generation using large language models (LLMs) and code agents. The authors introduce a benchmark called SWT-BENCH, which aims to analyze the performance of LLMs and agents in generating unit tests given a task description. The key contributions include:\n\n1) Creating the SWT-BENCH testing benchmark, which contains over 1700 samples from real-world GitHub issues.\n\n2) Benchmarking various LLMs and agent systems. This includes evaluating direct LLM prompting, existing test generation tools (e.g., LIBRO), and adapted code agents (e.g., SWE-AGENT, AUTOCODEROVER). \n\n3) Introducing a new format for LLM-based code editing.", "questions": "1) Have you looked into leveraging the failure modes of the generated test cases to improve code repair or code generation tasks?\n\n2) LLMs for code generation seem to be very stochastic by nature. You mention that the failure modes on the benchmark vary depending on the underlying method used. To what extent do you think this variation is explained simply by the stochasticity of the method? For example, if you ran swe-bench 5 times, would that variation disappear?\n\n3) How do you think would your method generalize in a private codebase that isn't available online?", "limitations": "The contamination problem with LLMs is, in my opinion, very important and under-discussed. Due to GitHub data being public, a lot of the data already exists in the training data for LLMs, which can render some conclusions less impressive. This is especially true in the code repair and test generation areas. I think it would be very helpful to include some analysis of the contamination problem. This could be done by using a held-out dataset in a similar way to how the benchmark below does it:\n\nhttps://livecodebench.github.io/", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) LLMs are stochastic black boxes by nature, and building experimental testbeds is crucial to advance the field. They can provide training data for the models as well as relevant signals for improvements. I think this benchmark is a very valuable contribution to the field as it will enable more algorithmic and agentic advances on the topic.\n\n2) This benchmark does a good job at testing several methods such as LLMs, agents, and modifications to existing agentic systems.\n\n3) The performance analysis of various methods is quite detailed, and the correlation or lack thereof between methods is very interesting.", "weaknesses": "1) The scope of the dataset could be expanded further. In its current form, it only addresses Python codebases with associated GitHub PR issues. The distribution of testing problems encountered by real-world software engineers is likely quite different from those represented in this testbed.\n\n2) A private test set that is not part of the publicly available training data would strengthen the claims that could be made when improvements are observed on this benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Ky1F for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the potential impact of our work as well as the extensiveness of our empirical evaluation and its analysis. Below we address their remaining questions.\n\n\n**Can the scope of the benchmark be expanded to address other repositories and programming languages?**  \nOur approach can be applied to any programming language and can be easily extended to other repositories, including private projects if Github issues and pull requests were used. However, setting up the corresponding execution environments with historically correct dependency versions and parsing of all test results requires some manual effort. Given the already substantial cost of evaluating the whole benchmark, we decide to leave this to future work targeting specific languages or repositories (please also see our answer in the main reply). Further, we want to highlight that most currently used benchmarks such as HumanEval ([1], 164 instances) and the concurrent Software Test benchmark TestEval ([2], 210 instances), focus on much smaller sets of standalone functions in one or few languages, which are much less representative of real-world issues than the popular Github repositories our benchmark is built on.\n\n**Can you add a private test/holdout set to avoid contamination?**  \nHaving a fully private holdout set is challenging for agent evaluation, as this would require the party hosting this benchmark to execute submitted methods, which can incur substantial costs. Alternatively, the issue descriptions and codebase details would need to be shared with submitters, which would defeat the purpose of a private holdout set. Besides these practical challenges, the code underlying any such holdout set would still be available online, making it challenging to guarantee that it was not used for training. A previously entirely private repository, circumventing the latter issue, is difficult to obtain. Sticking to online code, a rolling benchmark set of recent instances could be used, similar to live code bench. However, this has the issue that reported numbers can not be compared directly and rerunning all methods on the current version would be necessary when evaluating the benchmark. We thus leave this variation to future work.\n\n**Can the failure modes of the test generation be leveraged to improve code repair?**  \nWe believe that test generation provides a valuable alternative perspective on the capabilities of Code Agents and help drive developments in the field. It can for example unveil repositories where Agents struggle to leverage even the existing tests, highlighting the need for further improvements in Agent tooling. We find several such cases, especially in the repository django, where executing the test suite itself is non-trivial but would benefit Code Agents for validating their changes. Even when test suites are executed, we find that the model often wrongly considers unrelated failures as related to the given issue. Other failures, such as getting stuck in editing loops seem universal and their resolution would benefit both code repair and test generation. We leave a detailed exploration of these directions for future work.\n\n**To what extent does the stochasticity of LLMs explain the variance in failure modes and performance between methods?**  \nWe first want to highlight that we use greedy decoding for most methods (except Libro and pass@k) which is deterministic up to floating point errors. Further, since the submission, more powerful cheaper models have been released, allowing us to investigate the variance across multiple runs with GPT4o-mini. There, we obtain the results shown in Table 3 of the attached PDF, showing 95% confidence intervals obtained with 5 runs for a range of temperatures. We observe very low variance, which is interestingly dominated by the flakiness of tests (despite execution in the same docker), present even for greedy decoding (temperature 0).\n\n**Can you comment on the generalizability of your results to private codebases that are not available online?**  \nBased on manual inspections of agent traces we find that models typically don’t show signs of memorization. In a few cases, the model tries to assess files that do not exist, pointing to either hallucinations or outdated memorization. Overall, we observe that models make use of the provided tooling to navigate and explore the code base. Therefore we conjecture that our results should transfer well to private codebases. We further confirm this by comparing the performance of ZeroShotPlus on all instances created after the training data cutoff of GPT4 to a random subset of instances created before. We observe no statistically significant difference (p-value of 37% using Students t-test).\n\n**Conclusion**  \nWe hope we were able to address all of the reviewer's remaining questions and remain happy to answer any follow-up questions they might have.\n\n**References**  \n[1]  OpenAI 2021, Evaluating Large Language Models Trained on Code  \n[2] Wang et al. 2024, TESTEVAL: Benchmarking Large Language Models for Test Case Generation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xs4LZqlMpQ", "reviewer": "Reviewer_pxq3", "summary": "This paper introduces SWT-BENCH, a novel benchmark for evaluating automated test generation capabilities of AI models, particularly Code Agents. The authors adapt existing code repair datasets and methods to the task of test generation, proposing new metrics such as fail-to-pass rate and change coverage. Their experiments reveal that Code Agents, originally designed for code repair, outperform methods specifically designed for test generation. The best-performing method, SWE-AGENT+, achieves an 11.1% success rate in generating relevant fail-to-pass tests. The study also demonstrates significant complementarity between different approaches, with an ensemble of methods solving 70% more samples than the best single method. Additionally, the authors show that generated tests can serve as a strong signal for the correctness of proposed code fixes. While the study is limited to Python and may have some selection biases, it suggests that Code Agents have significant potential for automated test generation, potentially improving both software quality and developer productivity.", "questions": "1. Given the computational constraints that prevented a full statistical analysis, could you provide more details on the variability of your results? Even with limited runs, could you estimate confidence intervals for your main findings?\n\n2. The paper lacks a detailed error analysis. Could you provide examples of common failure modes for the different methods, particularly for SWE-AGENT+? How do these failure modes relate to the characteristics of the issues or repositories?\n\n3. How sensitive are the Code Agents' performances to different hyperparameters? Did you explore variations in the number of interaction rounds or temperature settings? If so, what insights can you share about optimizing Code Agents specifically for test generation?\n\n4. The paper doesn't compare AI-generated tests to human-generated ones. Have you considered conducting even a small-scale comparison with professional developers? This could provide valuable context for understanding the practical impact of your methods.", "limitations": "1. Limited Statistical Analysis: Due to computational constraints, the authors were unable to perform a full statistical analysis. This limits the confidence in the reported results and makes it difficult to assess the robustness and reproducibility of the findings.\n\n2. Lack of Detailed Error Analysis: The paper doesn't provide an in-depth analysis of the types of errors made by different methods. This limits understanding of where and why the methods fail, which could be crucial for further improvements.\n\n3. Python-Centric Approach: The study focuses exclusively on Python, which may limit the generalizability of the results to other programming languages.\n\n4. Absence of Human Baseline: There's no comparison between AI-generated tests and human-generated tests, making it challenging to assess the practical impact and efficiency gains of these methods in real-world scenarios.\n\n5. Limited Hyperparameter Exploration: The paper doesn't thoroughly explore the impact of different hyperparameters on the performance of Code Agents, potentially missing opportunities for optimization.\n\n6. Lack of Runtime Performance Analysis: While computational costs are mentioned, there's no detailed analysis of execution times for different methods, making it difficult to assess practical trade-offs.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\n- Introduces SWT-BENCH, a novel benchmark for test generation, adapting existing code repair datasets to a new task.\n- Proposes the use of Code Agents for test generation.\n- Develops new metrics (fail-to-pass rate and change coverage) specifically for evaluating test generation.\n\nQuality:\n- Comprehensive evaluation of multiple methods, including baselines and state-of-the-art approaches.\n- Rigorous experimental setup with detailed reporting of methodologies and results.\n- Great analysis of results, including complementarity between methods and correlation with code repair performance.\n\nClarity:\n- Well-structured paper with clear explanations of complex concepts.\n- Effective use of figures and tables to illustrate key points and results.\n- Detailed appendices providing full prompts and additional experimental details.\n\nSignificance:\n- Demonstrates the potential of Code Agents for automated test generation, a critical area in software development.\n- Shows that generated tests can effectively validate code fixes, potentially improving code quality processes.\n- Provides a new benchmark (SWT-BENCH) that could drive further research in this area.\n- Highlights the complementarity of different approaches, suggesting potential for ensemble methods in this domain.", "weaknesses": "1. Limited statistical analysis: The authors acknowledge they were unable to perform a full statistical analysis due to computational costs. This limits the confidence in the reported results. It would be great if they can conduct a power analysis to determine the minimum number of runs needed for statistical significance.\n\n2. Lack of error analysis: The paper doesn't provide a detailed analysis of the types of errors made by different methods. Maybe they should categorize and quantify common error types for each method. Provide qualitative examples of generated tests, both successful and unsuccessful. Analyze how error types correlate with issue complexity or repository characteristics.\n\n3. Limited exploration of hyperparameters: The paper doesn't discuss the impact of different hyperparameters on the performance of Code Agents. To improve: Conduct an ablation study on key hyperparameters (e.g., number of interaction rounds, temperature). Provide insights on how to optimize Code Agents specifically for test generation.\n\n4. Insufficient comparison to human performance: The paper lacks a comparison to human-generated tests. To address this: Include a small-scale study with professional developers generating tests for a subset of issues. Compare the quality, coverage, and time efficiency of AI-generated vs. human-generated tests.\n\n5. Narrow focus on Python: The study is limited to Python, potentially limiting generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pxq3 for their detailed review, insightful questions, and helpful suggestions. We are happy to hear they appreciate the originality, quality, clarity, and significance of our work and in particular our comprehensive and rigorous evaluation across methods and models. Below, we address their remaining questions.\n\n**Can you conduct additional experiments to establish statistical significance and/or report confidence intervals?**  \nPlease see the main response for a detailed answer. In short, we have added experiments and statistical analysis demonstrating a high statistical significance of our results.\n\n**Can you conduct an error analysis, categorizing common errors made by different methods, and provide qualitative examples? Do these error types correlate with issue complexity or repository characteristics?**  \nWe have conducted a detailed analysis of common errors for our best-performing method SWE-Agent+ and will gladly add the findings to the paper for the camera-ready version together with concrete examples of their occurrence. We find the most common issues of the agent to be adding tests that do not reproduce the issue (either due to unrelated failure or non-failure) and incorrectly classifying the result as relevant, getting stuck in a loop after making inapplicable edits or encountering syntax errors, failing to execute the test environment after test creation, and adding tests with missing fixtures or missing indentation, rendering the test function non-executable. Further, we want to point the reviewer to Section 5.4, where we already investigate correlations of success with repositories, issue description length, and code repair success. \n\n**Can you conduct an ablation study on key hyperparameters of the employed code agents?**  \nIn the attached PDF, we provide ablation studies on the temperature used for decoding, the number of interaction rounds for various agents, and the Libro sample size (see Table 3 and Figure 1a,b). In agreement with prior work, we consistently observe the best performance for greedy decoding (t=0). We further observe increasing interaction rounds improve performance until saturation at 5-10 iterations (we use 20 as a default) with the only exception being AutoCodeRover, which still gains performance up to the maximum of 20 iterations we consider. Similarly, Libro’s performance improves as more samples are considered, however saturating also around the 5 samples we use by default. We will include these ablations confirming the choices of our hyperparameters in the updated appendix.\n\n**Can you conduct a small-scale study with professional developers to establish a baseline in terms of quality, coverage, and time efficiency?**  \nWe believe a human study of statistically significant sample size to be infeasible as a baseline given that many of the issues at hand are complex and challenging to understand, with the original underlying issues taking up to a year to resolve. Further, as all ground truth tests were written by humans, we believe that most professional developers should be able to solve this task given sufficient (substantial) time. We want to highlight that such human baselines are uncommon for code benchmarks, even at the much simpler function-synthesis level, due to the above-mentioned substantial time requirements. Finally, we believe that establishing such human baselines, possibly across skill levels, and focusing on simpler benchmarks could be an interesting work on its own and leave it to future work.\n\n**Can you discuss the applicability of your approach beyond Python?**  \nPlease see our answer in the main response.\n\n**Can you include an analysis of execution times for different methods?**  \nFor all LLMs we consider, part of the execution time is directly related to the number of tokens digested and generated (see Tables 7 and 8 of the main submission). For methods that require interaction with an execution environment however, time is usually dominated by setting up such an environment in a clean and reproducible manner (i.e. dockerized). We list results on execution times below and observe that all methods except zero-shot inference take between 3-5 minutes per instance, where we can observe a small trade off due to many-turn interactions in Code Agents versus single-shot execution in LIBRO. Given these small differences however, we believe execution time to be of limited practical relevance as issues can be processed in the background, similar to continuous integration, in response to raised user issues.\n\n| Method | Execution Time |\n|---|---|\n| ZeroShotPlus | 12.6s |\n| LIBRO | 2m53s |\n| SWE-Agent | 3m42s |\n| SWE-Agent+ | 4m25s |\n| AutoCodeRover | 5m1s |\n\n\n**Conclusion**  \nWe hope to have been able to address the reviewer's concerns, in particular regarding the statistical significance and hyperparameter sensitivity of our approach, remain happy to answer their follow-up questions and look forward to their reply.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I9bJVcNSiE", "reviewer": "Reviewer_ueFW", "summary": "In this paper, the authors propose a novel benchmark for evaluating the capability of LLMs and Code Agents in generating test cases from GitHub issues. The benchmark includes real-world issues, patches, and golden tests from popular Python repositories. The authors claim that Code Agents outperform traditional test generation methods and highlight the potential of these agents in improving software quality and developer productivity.", "questions": "How does the benchmark ensure that it does not include tasks already present in pre-trained DNN datasets?\n\nWhy were significant existing works on LLM-based test generation not included in the paper?\n\nWhat measures are in place to ensure the long-term relevance and maintenance of the SWT-BENCH benchmark?\n\nHow might the results differ if the benchmark included tasks in programming languages other than Python?\n\nCan the authors discuss the potential societal impacts of the widespread adoption of Code Agents in software testing, particularly concerning developer employment?\n\n\nPlease see the above to clarify any misunderstandings and add additional results.", "limitations": "The paper lacks discussion on various limitations, including the limitations of the metrics used and potential data contamination across models.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper proposes a new benchmark for test generation, addressing a gap in current research.\n\nThe approach leverages real-world data from GitHub, making the benchmark relevant and practical.", "weaknesses": "The data contamination problem. The proposed benchmark is collected from GitHub, which might have been exposed to the pretraining corpus of LLMs.\n\nThe authors overlook substantial existing work on test case generation using LLMs.\n\nThere is no discussion on the long-term relevance and maintenance of the benchmark, which is crucial for its sustained utility.\n\nThe benchmark is focused solely on Python, ignoring other programming languages. \n\nThe experiments are conducted on only three LLMs (GPT-4, Claude-3, Mixtral 7x22B), which is insufficient for a comprehensive evaluation of the benchmark.\n\nThe paper fails to address the broader societal impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ueFW for their critical perspective and for raising interesting questions, which we address below.\n\n**Can you discuss the effect of possible data contamination on your results and how it might be mitigated?**  \nWe thank the reviewer for raising this point. In short, we see no statistically significant difference in performance (p=37%) for issues from before and after the data cutoff and thus conclude that contamination is not affecting our results significantly. Please see the main response for more details.\n\n**Why were significant existing works on LLM-based test generation not included in the paper?**  \nWe discuss automated test generation (including using LLMs) in our related work (Lines 71-75). Due to space constraints, this discussion focuses on the most important related works and, e.g., omits [1,2,3,4] which are not applicable to our setting as they only consider coverage and generating passing tests [1,3,4], require a large number of tests and fixes to be generated [2], and the to be tested code to be specified [3,4]. We are happy to include these in the next revision using the additional space. If the reviewer could provide a list of works they believe to be relevant, we would be happy to review and include them.\n\n**How can the long-term relevance and maintenance of SWT-Bench be ensured?**  \nWe agree that long-term relevance is an important topic, actively discussed in the community. We believe SWT-Bench is particularly well positioned to ensure long-term relevance. In particular, new instances can be easily created from new repositories as well as new issues in already used repositories. However, such a rolling benchmark also has disadvantages (comparability of scores, cost of reevaluation). Regardless of these aspects, we believe our initial results showing the promise of Code Agents to already be valuable irrespective of any long-term maintenance of SWT-Bench.\n\n**Can you discuss the relevance of your results on Python and how they might generalize to other languages?**  \nPlease see our main response for a discussion on Python’s relevance and how our approach applies to other languages. Further, it is common in the field to focus on a single language, especially when addressing a new task, be that Java [3,4], Kotlin [1], or Python [2,5]. \n\n**Can you evaluate your benchmark on more than three LLMs?**  \nWe first want to highlight that while we only considered 3 LLMs (the, at the time, best proprietary model (GPT4), a model balancing cost and performance (Claude 3 Haiku), and a strong open-weight model (Mixtral 8x22B)), we consider a wide range of different agents based on the strongest available model. Given the challenging nature and thus low resulting performance even of GPT4, we refrained from considering weaker models. We want to highlight that all other reviewers specifically highlighted the quality of our extensive evaluation. Finally, since the original submission, stronger models have been released and we have conducted additional experiments on GPT4o-mini, Claude 3.5 Sonnet, and Mistral Large 2 using SWE-Agent, reporting results in Table 2 of the attached PDF. We observe that results follow general model capability as expected.\n\n**Can you discuss the potential societal impact of widespread adoption of Code Agents in software testing, particularly concerning developer employment?**  \nTesting and bug reproduction are often neglected among professional developers due to a lack of extrinsic and intrinsic motivation [6]. We thus believe a more widespread adaptation of Code Agents in software testing has the potential to not only improve code quality but also developer productivity and satisfaction. Regarding the potential to displace human developers, benchmarks such as SWT-Bench show that current Code Agents are still far from matching or even outperforming human developers. Instead, they show that human supervision is still essential to leverage current Code Agents. Finally, while we believe that benchmarks such as ours can help drive progress toward better AI systems and thus increased automation, we do not believe our work specifically to have an outsized societal impact beyond the general developments in Generative AI. We are happy to include this discussion in the revised version of the paper.\n\n**Can you discuss the various limitations of your work including the metrics used and potential data contamination?**  \nWe first want to highlight that we discuss a range of limitations in “Section 6: Limitations”. Regarding data contamination, we are happy to include the new results presented in the attached PDF and discussed in the global response in the paper. We would like to ask the reviewer to explain which further “various limitations, including … of the metrics used” they have in mind.\n\n**Conclusion**  \nWe hope we were able to address the reviewer’s concerns and questions. We would further like to ask the reviewer, where they see flaws in the soundness and presentation of our work given that all other reviewers rate them as good or excellent. Finally, we would like to ask the reviewer to reconsider if their score is justified given the practical relevance and high potential impact they attest our work with weaknesses being shared by most other work in the space.\n\n**References**  \n[1] Alshahwan et al. 2024, Automated Unit Test Improvement using Large Language Models at Meta  \n[2] Chen et al. 2022, CodeT: Code Generation with Generated Tests  \n[3] Schäfer et al. 2023, An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation  \n[4] Chen et al. 2024, ChatUniTest: A Framework for LLM-based Test Generation  \n[5] Wang et al. 2024, TESTEVAL: Benchmarking Large Language Models for Test Case Generation  \n[6] Straubinger and Fraser 2023, A Survey on What Developers Think About Testing", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their detailed, insightful, and overwhelmingly positive reviews.
We are encouraged to see that the reviewers consider the problem we investigate under-studied and important (PhkB, i5fN, pxq3, ueFW), our benchmark impactful (PhkB, i5fN, Ky1F, pxq3, ueFW), our empirical evaluation and its analysis exhaustive and interesting (PhkB, i5fN, Ky1F, pxq3), and our exposition clear and well written (PhkB, pxq3). Below, we address remaining shared questions, before going into more detail in reviewer-specific responses.

Since the original submission of this work, we have fully dockerized our evaluation, improved the parsing of test results to cover additional edge cases, and conducted experiments on more models (GPT4o-mini, Claude 3.5 Sonnet and Mistral Large 2, all released after the submission), methods (Aider), and ablation settings (sampling temperature, agent interaction rounds, multiple runs). Please see the attached PDF for these updated and extended results, referred to in some of our responses.

**Statistical significance of results (Ky1F, pxq3)**  
To confirm the statistical significance of our results, we have conducted a dependent t-test for paired samples to compute the statistical significance of SWE-Agent+ having higher performance than LIBRO. We find that SWE-Agent+ has higher performance (Fail-to-Pass rate) with a p-value of $7 \times 10^{-6}$, indicating strong statistical significance of this main result. We further want to highlight that most of the considered approaches use greedy decoding and are thus deterministic up to floating point errors. Finally, we conduct additional experiments with GPT4o-mini using sampling at a range of temperatures and consistently find small 95% confidence intervals of around $\pm$0.2% over 5 runs of ZeroShotPlus, demonstrating only small sensitivity to inference randomness even when not using greedy decoding. Further details are provided in Table 3 of the attached PDF. We believe these results further strengthen our conclusions and are happy to include them in the main paper.

**Relevance of Python results and generalization to other languages (Ky1F, pxq3, ueFW)**  
First, we want to highlight that Python is an extremely popular language, often ranked #1 by a wide margin in corresponding surveys [1][2]. Thus, we believe that our results on Python are already highly relevant. Further, our approach to designing a testing benchmark can be applied to any programming language where Github repositories with suitable issues and pull requests can be sourced. However, setting up the corresponding execution environments with historically correct dependency versions and test results parsing requires some manual effort. Given the already substantial cost of evaluating the whole benchmark of 1700 instances, we decide to leave this to future work targeting specific languages or issue types.

**What is the effect of possible data contamination on SWT-Bench and how can it be addressed? (Ky1F, ueFW)**  
As SWT-Bench is based on historic GitHub issues, they could be included in the pre-training data of the LLMs underlying the methods we investigate. To assess this effect, we conducted an experiment comparing the performance (of ZeroShotPlus) on all issues created after the training data cutoff of GPT4 (April 2023) to a random subset of instances created before of the same size and include the results in the attached PDF in Table 4. We observe that for the issues created before the cutoff, only one more sample is solved compared to those created after, leading to no statistically significant difference in performance (p-value of 37% using Students t-test). This agrees well with the findings on SWE-Bench. We are happy to include a corresponding discussion and these results in the revised version of our paper. We further note that all methods we investigate should benefit from memorization to a similar extent and hence contamination should not affect their ranking and our conclusion that Code Agents perform surprisingly well.

One approach to address this contamination issue is to create a rolling version of SWT-Bench, based only on the most recent issues. However, this comes at the cost of direct comparability of results and increased cost for reproducing results for all baselines on a changing evaluation set. 

**Conclusion**  
We hope to have been able to address the reviewers’ questions and look forward to the discussion period.

**References**  
[1] TIOBE Index for August 2024, [https://www.tiobe.com/tiobe-index/](https://web.archive.org/web/20240807025036/https://www.tiobe.com/tiobe-index/)  
[2] PYPL PopularitY of Programming Language, [https://pypl.github.io/PYPL.html](https://web.archive.org/web/20240806100838/https://pypl.github.io/PYPL.html)

Meta-review: This paper has a very wide spread of scores from 3 (reject) to 8 (strong accept). Unfortunately, two reviewers who suggest accept and strong accept did not participate in discussions. The issues raised by reviewer who suggested reject do not seem significant to me apart from data contamination (see below). After reading the paper, reviews, rebuttal, and substantial discussions between authors and reviewers, I believe that the paper should be accepted.

Summary of Strengths:

- Novel Benchmark: The paper introduces a new benchmark (SWT-Bench) for evaluating automated test generation, particularly using Code Agents and LLMs. This is recognized as filling a gap in current research.
- Practical Relevance: The benchmark uses real-world data from popular Python repositories, making it relevant and practical for testing the capabilities of LLMs in generating tests.
- Detailed Analysis: The evaluation is comprehensive, with multiple metrics (e.g., fail-to-pass rate, code coverage) and methods compared, showing the potential impact of automated test generation on software quality and developer productivity.

Weaknesses identified:

- Limited Statistical Analysis: The statistical analysis is seen as insufficient, particularly due to the computational constraints that prevented a full analysis. This limitation affects the confidence in the reported results.
- Focus on Python: The benchmark focuses solely on Python, which raises concerns about the generalizability of the results to other programming languages.
- Potential Data Contamination: The benchmark might include data from GitHub that was part of the pre-training corpus for LLMs, potentially contaminating the evaluation results.
- Limited Exploration of Hyperparameters: The paper does not fully explore the impact of different hyperparameters on the performance of Code Agents, missing potential optimization opportunities.

Authors' Response to Weaknesses:

- Statistical Analysis: Additional experiments were conducted to demonstrate the statistical significance of their results, which were reported in the rebuttal.
- Python Focus: The authors explained that while the current work focuses on Python, their approach is applicable to other programming languages, and expanding to other languages is left for future work.
- Data Contamination: The authors provided additional experiments to assess the effect of data contamination and found no significant impact on their results. However these experiments are based on small data sizes. This remains a potential issue.
- Hyperparameter Exploration: An ablation study on key hyperparameters was conducted and shared in the rebuttal, providing insights into optimizing the agents.

In terms of weaknesses, I think that only data contamination has not been fully addressed and is possibly concerning with respect to paper's results.
Decision: Accept (poster)