SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

Authors: niels  mündler
mark niklas müller
jingxuan  he
martin  vechev
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:47:23

Abstract: Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests.  We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench.

Main paper: 
 introduction: 
 As the complexity of software systems increases, rigorous testing is becoming more important than ever to ensure their reliability and correctness. However, while a large portion of these tests aims to reproduce previously reported issues (Kang et al., 2023) , such issue reproduction is often disliked by professional developers (Straubinger & Fraser, 2023) . Therefore, automatic generation of tests reproducing such issues from informal natural language descriptions is a promising path toward improving both code quality and developer productivity. Finally, generated tests can be leveraged as formal specifications to boost the effectiveness of automatic code repair tools (Chen et al., 2023a) . However, while automatic code generation, in particular using Code Agents, is an extremely active research area (i.e. Yang et al. (2024) ; Tao et al. (2024) ; Zhang et al. (2024) ; Bouzenia et al. (2024b) ; OpenDevin (2024); Bouzenia et al. (2024a) ; Schäfer et al. (2024) ; Alshahwan et al. (2024a) ), there is comparatively little work investigating automatic test generation directly. Indeed, while prior work has proposed methods based on symbolic execution (Lukasczyk & Fraser, 2022) , specialized transformers (Tufano et al., 2020) , and general-purpose LLMs (Li et al., 2023; Alshahwan et al., 2024b; Kang et al., 2023 Kang et al., , 2024;; Chen et al., 2023b) , Code Agents have not been considered in this context, and even less work is applicable to the issue reproduction setting. Finally, large-scale, diverse test-generation datasets are lacking for Python, which is one of the most popular programming languages at the time of writing (TIOBE, 2024; PYPL, 2024) and a focus of Code Agent research.Figure 1 : Evaluation of an SWT-BENCH instance. Given an issue description in natural language and the corresponding codebase, the task is to generate tests that reproduce the issue. We considered a test to reproduce the issue if it fails on the codebase before the pull request (PR) is accepted, i.e., before the golden patch is applied, but passes after. We call this a fail-to-pass test (F → P ).A Benchmark for Test Generation In this work, we propose SWT-BENCH, a novel and comprehensive dataset for test generation with the goal of issue reproduction in Python. SWT-BENCH contains over 1 900 samples, each consisting of a GitHub issue, a golden code patch resolving the issue by adjusting the code, and a set of golden reference tests, obtained by transforming the popular SWE-BENCH (Jimenez et al., 2023) from code repair to test generation. We leverage the fact that any code repair task can be transformed into a test generation task, even in the absence of golden tests, by utilizing the golden code patch for evaluation. Concretely, for every generated test, we determine whether it reproduces the described issue, by checking whether it fails on the original repository but passes after the golden patch is applied. The golden reference tests, used in SWE-BENCH for the evaluation of code repair performance, are solutions in this test generation setting. We illustrate this evaluation process of SWT-BENCH in Fig. 1 . Further, we report the coverage of the code modified by the golden patch as a more fine-grained evaluation metric for generated tests. began to saturate the simpler function-level benchmarks. However, none of these benchmarks were designed to assess test generation.The only dataset for reproducing bugs based on real-world issues, Defects4J (Just et al., 2014) , focuses on Java, is outdated, limited in size, and contains only short bug descriptions rather than detailed issue reports. In contrast, SWT-BENCH is based on Python, which is better supported by modern Code Agents, contains more recent issue reports, and is significantly larger.Automated Unit Test Generation Many approaches have been suggested to automate (unit) test generation leveraging symbolic execution (Lukasczyk & Fraser, 2022) , specialized transformers (Tufano et al., 2020) , and general purpose LLMs (Li et al., 2023; Alshahwan et al., 2024b; Kang et al., 2023; Tufano et al., 2020; Kang et al., 2024; Schäfer et al., 2024; Alshahwan et al., 2024a; Chen et al., 2023b) . Depending on their focus, they can be used to increase test coverage (Alshahwan et al., 2024b; Schäfer et al., 2024) , find edge cases (Lukasczyk & Fraser, 2022) , or reproduce reported issues (Kang et al., 2023) . Issue-reproducing tests are especially interesting, as they can be used to validate automatically generated code repair candidates and thus improve the precision of code repair systems (Chen et al., 2023a) . However, most test-generation approaches are not applicable to issue reproduction. We therefore evaluate the most recent applicable method, LIBRO (Kang et al., 2023) , and a range of other LLM-based baselines on SWT-BENCH.Code Agents Over the last year, LLMs have been equipped with tools to observe and interact with their environment over multiple turns and preserve a state across these turns (Wang et al., 2024) . These so-called agents have proven successful on a range of complex tasks, including code repair and synthesis (Bouzenia et al., 2024b; OpenDevin, 2024; Zhang et al., 2024; Yang et al., 2024; Tao et al., 2024; Bouzenia et al., 2024a; Aider, 2024) . Such Code Agents can typically search, read, and edit code using an agent computer interface (ACI) (Yang et al., 2024) . In this work, we leverage such Code Agents for generating issue-reproducing tests by changing their instructions. 
benchmarking test generation: 
 In this section, we outline the structure of the proposed benchmark, SWT-BENCH, and how we leverage it to measure the capabilities of LLMs and Code Agents for test generation.We first introduce the notation to describe codebases, their test suites, and changes to these codebases in the form of patches. We denote a codebase R after applying patch X as R • X. Several patches can be applied sequentially, i.e. R • X • Y is the codebase R after applying a first patch X and then a second one Y . When a code patch X is applied to R, a set of tests T can be used to check the correctness of the applied patch. A single test s can either pass (P) or fail (F) after we execute it within the context of codebase R. We consider a test to fail if an error is thrown during its execution, e.g., an AssertionError or ValueError. Such test errors frequently occur if R lacks or misimplements the functionality targeted by the test. They can also occur due to other reasons, such as incorrect syntax or formatting of the test s. Conversely, a test passes when running the test triggers no error. We define this process as an execution function: exec(s, R) ∈ {P, F }.We consider a test s to reproduce a described issue I of R, which is resolved by patch X if it fails on the original codebase (i.e. exec(s, R) = F ) but passes on the patched codebase (i.e. exec(s, R • X) = P ). We denote these fail-to-pass tests with F → P and define F → F , P → P , and P → F tests similarly. If a test transitions from failing on R to any state on R•X, we denote it as F → × and vice versa for × → F . Further, we consider a set of tests T to be successful at reproducing the issue I, if it contains at least one F → P test and no × → F test, or equivalently ∃s ∈ T, exec(s, R) = F ∧ ∀s ∈ T, exec(s, R • X) = P .To construct SWT-BENCH, we leverage the same underlying data as SWE-BENCH (Jimenez et al., 2023) and summarize its three-stage construction process here for completeness.1. Scrape a total of ∼ 90 000 pull requests (PRs) from 12 popular open-source Python repositories from GitHub.2. Filter PRs to only include those that were merged, resolved a GitHub issue, and made changes to at least one test file.3. Filter PRs to feature at least one F → P test, removing PRs that result in installation or runtime errors.This results in 2 294 task instances, each consisting of a GitHub issue, a golden patch X * fixing the issue, and a set of golden reference tests T * . We summarize key statistics of SWT-BENCH in Table 1 and show its repository composition in Fig. 2 . While issue descriptions are on average only 318 words long, the longest one reaches 8 756 words. Generally, repository complexity is high with on average over 1 500 files and over 300 000 lines of code. Many repositories feature large test suites of > 120 and up to 4 800 tests, already covering 70% of the lines in the to-be-patched code. Most of these existing tests are unaffected by the golden patch with basically no F → P and only 1.5 P → F tests on average. The golden tests remove on average 0.3 tests and add another 2.9 new test cases, of which roughly twothirds are F → P . The test patches edit on average 31.8 lines in 1-2 files. Due to the filtering for unresolved issues during dataset curation, no golden tests are F → F or P → F .We propose two main metrics to evaluate the test generation performance of any method; Success rate (S) and change coverage (C), described below. We further introduce the necessary but insufficient property of patch well-formedness (W).The success rate S measures the portion of instances where the generated tests T reproduced the issue according to the definition in §3.1, i.e. at least one test in T transitions from failing to passing and none fail after applying the patch. This is the most important performance measure, as the presence of F → P and the absence of × → F tests are key for test-driven development and automatic code generation. We further report the portion of instances for which at least one Fail-to-Pass (F → P ), Fail-to-Any (F → ×), and Pass-to-Pass (P → P ) was generated. Coverage is an important metric to determine what portion of a codebase is tested. While path coverage measures this optimally, the exponential number of paths makes it infeasible in practice. We thus follow common practice, and instead measure line coverage. As we aim to specifically test the code described in the issue text, we consider only the coverage of the changes made by the golden code patch. Further, we observe that patches may include portions of non-executable lines, e.g. documentation or configuration files, and exclude them. Specifically, we consider all lines that are executed by the original test suite S or the golden test suite T * on both R and R • X * to be executable and track coverage of such executable lines. Finally, we consider both the coverage of removed (including modified) lines of code in the original codebase and added (including modified) lines of code in the patched codebase, illustrated in Fig. 3 .Formally, given the number of times C R S (l) ∈ Z ≥0 a specific line of code l was executed when running the test suite S on codebase R, we define the executable lines of the patch X asX * r = {l ∈ X r | C R S (l) + C R S∪T * (l) > 0} X * a = {l ∈ X a | C R•X S (l) + C R•X S∪T * (l) > 0}where X r and X a are the lines added and removed by patch X, respectively, and T * are the golden tests. Finally, we obtain the change coverage of the generated tests T as∆C X T = |{l ∈ X * r | C R S∪T (l) > C R S (l)}| + |{l ∈ X * a | C R•X S∪T (l) > C R•X S (l)}| |X * r | + |X * a |.Where X and T are clear from context, we drop them for notational clarity. If none of the lines modified by the golden patch X are executed by any test, i.e., |X * r | + |X * a | = 0, we exclude this instance from our coverage analysis (1% of cases).Patch Well-Formedness Many LLMs struggle to generate well-formed code patch files (Jimenez et al., 2023) and the methods we investigate employ different approaches to mitigate this issue. To assess them, we additionally measure the patch applicability W as the portion of instances for which a well-formed patch was generated. We define W as the portion of instances for which the generated patch X can be applied to the original codebase R without errors. Since well-formedness is necessary for any test to be executed, it always exceeds S, F → P , and related rates. 
automatic test generation: 
 We first discuss how the test generation task differs from code repair, before introducing a novel code diff format based on these insights that is optimized for fault tolerance. Finally, we propose a range of test generation methods based on directly querying LLMs and leveraging Code Agents.Automatic test generation is closely related to code repair: Instead of predicting a patch X that fixes the described issue and is then evaluated using a golden test T * , we aim to predict reproducing tests T which are then evaluated on both the original state of the codebase R and the state after applying the golden code patch X * . However, there are some key differences between the two tasks: First, adapting an existing test suite to reproduce an issue typically only requires adding new tests. Concretely, 71% of golden tests in SWT-BENCH only add new test functions, with another 28% modifying existing functions, and only 1% removing functions. Second, testing permits and requires a more granular analysis. While fixed code is either correct and passes all test cases or incorrect when failing any of them, generated tests can be correct but irrelevant to the issue (P → P ), call relevant code but fail to expose the precise bug (increase in coverage), reproduce the issue with varying comprehensiveness on edge cases (F → P , with varying coverage), or fail in different ways.1 --demo/file.py 2 +++ demo/file.py 3 @@-4,5 +4,5 @@ 4 def test _ euclidean(a, b): Code changes are typically represented in the unified diff format, i.e., in the git patch and diff format. While using this format to represent code changes is both precise and human-readable, it is very sensitive to misspecifications, requiring, e.g., the exact line numbers of code changes to be specified and specific code snippets (including all to-be-changed lines) to be repeated verbatim. As a result, many LLMs struggle to produce well-formed patch files (Jimenez et al., 2023) . Even when loosening the strict diff requirements and fuzzy-matching the generated diff to a best-fit part of the code, GPT-4 only succeeded in 48% of cases, resulting in only 10 correctly reproduced issues.To alleviate this issue, we propose an adjusted patch format optimized for LLM generation that is easier to adhere to and more robust. Specifically, our custom diff format allows entire functions or classes to be inserted, replaced, or deleted, given the full function or class definition and (faulttolerant) location in the code. We show an example in Fig. 4 , comparing it to the unified diff format.Based on whether the model wants to rewrite an existing function or insert a new function, the provided code is then substituted or inserted at the code location. This format is particularly well suited for test generation which usually only requires adding test functions. We provide a more formal description of this format in App. A and demonstrate its effectiveness in §5.We consider four baselines for test generation: Direct zero-shot prompting with the unified patch format (ZEROSHOT), zero-shot prompting with our novel patch format (ZEROSHOTPLUS), selecting the best out of 5 patches using an oracle (PASS@5), and the state-of-the-art test generation method, LIBRO (Kang et al., 2023) , which uses a range of heuristics to pick the most promising among multiple generated tests. In all methods, the LLM is instructed to add tests to reproduce and cover the described issue in the codebase. We describe these methods below, deferring further details to App. E.ZEROSHOT prompts the model with the issue description, a subset of the codebase retrieved using BM-25 (Robertson & Zaragoza, 2009) , and instructions to generate a patch file in unified diff format. This method corresponds to the LLM-only baseline in SWE-BENCH (Jimenez et al., 2023) .ZEROSHOTPLUS is similar to ZEROSHOT but leverages our custom diff format, discussed in §4.2, which is optimized for LLMs and robustness to minor specification errors.PASS@5 uses our ZEROSHOTPLUS prompting scheme to generate 5 proposal tests and then uses an oracle to pick the best one. While this is of course not practical in a real-world setting, it allows us to assess the potential of the LLM to generate good test cases given an effective selection mechanism.LIBRO (Kang et al., 2023) , is the current state-of-the-art for LLM-based test generation. Similar to PASS@5 it generates multiple proposal tests using ZEROSHOTPLUS prompting. However, instead of using an oracle, it combines multiple heuristics to select the best test cases. In particular, it runs all generated tests and then selects the one inducing an error that is most similar to the problem description. This permits not only checking whether a generated diff is well-formed and the proposed test fails on the original codebase but also selecting the most relevant test case. As LIBRO was originally proposed for Java, we adapt it to our Python setting, as detailed in App. B.LLM-based agents are systems that take actions based on LLM-generated text, providing tools to observe and interact with their environment over multiple turns and preserve some state across these turns. In the case of Code Agents, they can typically search, read, and edit code using an agent computer interface (ACI) (Yang et al., 2024) . Recent work has shown that such Code Agents are particularly effective for complex repository-level code synthesis and repair tasks, outperforming unaided LLMs by a significant margin (Bouzenia et al., 2024b; OpenDevin, 2024; Zhang et al., 2024; Yang et al., 2024; Tao et al., 2024) . In this work, we leverage Code Agents for automatic test generation by adjusting their instructions. Specifically, we adapt SWE-AGENT (Yang et al., 2024) , AIDER (Aider, 2024), and AUTOCODEROVER (Zhang et al., 2024) .SWE-AGENT (Yang et al., 2024) provides the LLM direct access to (augmented) command line tools and processes the output of these tools to be more easily parseable by an LLM. In particular, they provide special tools for searching, viewing, and editing files. Beyond initial instructions, they provide little guardrails or structure for the LLM and let it interact with a limited shell environment.AIDER (Aider, 2024) performs a repository indexing step to guide file selection and then includes all selected files in the next prompts. Further, model-generated summaries and reflections on previous actions are leveraged to augment the context. Before an edit is applied, it undergoes validation via static analysis and repository test cases using project-specific evaluation harnesses.AUTOCODEROVER (Zhang et al., 2024) separates the code repair task into two distinct stages. In the first stage, the LLM is tasked with collecting all required context for the task at hand. To this end, it is equipped with a range of advanced code search and navigation tools, allowing it, e.g., to retrieve class signatures, function definitions, or surrounding code snippets. Once the LLM believes it has gathered sufficient context, it proceeds to the second stage. There, the LLM is tasked with generating the actual code patch in a single step, retrying only if the patch can not be applied.Adapting Code Agents for Test Generation As SWE-AGENT, AIDER, and AUTOCODEROVER were designed for program repair, we adapt their system and instruction prompts to focus on creating high-quality test cases. We find that the underlying LLMs are capable of following these changed instructions and successfully generate test cases for up to 87% of issues. Typically, the instruction changes were as simple as replacing phrases like "solve the issue" with "create unit tests that cover the issue". We provide a more detailed description of the used prompts in App. E.We experiment with instructing SWE-AGENT explicitly to execute the generated test cases before submitting them. We call this variant SWE-AGENT+ and find that this increases the success rate S from 15.9% to 18.5% (see Table 2 ). Note we do not provide any information on how to run the tests. This contrasts the LIBRO setting, in which the test execution commands are assumed to be known. 
experimental evaluation: 
 We leverage SWT-BENCH to compare the performance of different test generation methods and underlying LLMs ( §5.2), their relation with the code repair setting ( §5.3), and the impact of instance characteristics ( §5.4). We further explore hyperparameter ablations in App. C.We consider GPT-4 (gpt-4-1106-preview OpenAI 2023), GPT-4o mini (gpt-4o-mini-2024-07-18 OpenAI 2024), Claude 3.0 Haiku (Anthropic, 2023), Claude 3.5 Sonnet (Anthropic, 2024), Mistral Large 2 (Team, 2024b) (served via the Mistral AI API), and Mixtral 7x22b (Team 2024a served by Together AI TogetherAI 2023), as underlying LLMs, using GPT-4 unless indicated otherwise. We sample at temperature t = 0 for all zero-shot methods and agents and at t = 0.7 for LIBRO and PASS@5. For SWE-AGENT, AIDER, and AUTOCODEROVER, we use their default settings, restricting the number of API calls to 20, reflection steps to 3, and interaction rounds to 10, respectively. For LIBRO we sample 5 tests. Due to budget constraints, we focus our evaluation on SWT-BENCH-LITE. In App. C we explore and justify this choice of hyperparameters in detail.Comparing Test Generation Methods We compare test generation performance in Table 2 where all methods have access only to the issue description and the original codebase. We observe that using the original git code-diff format, ZEROSHOT only generates well-formed patches for 48.6% of issues. Using our novel test-specific code-diff format (ZEROSHOTPLUS) boosts this rate to 89.5% yielding an almost 3x increase in success rate (S) to 9.4%. While picking the best among five generated tests (PASS@5) even yields an S of 20.3%, the heuristics employed by LIBRO can only convert about half of this gap into an S of 14.1%, still beating AUTOCODEROVER and AIDER which achieve an S of 9.1% and 12.7% respectively. Test Generation for a Given Code Patch To assess the effectiveness of automatic test generation at testing specific, provided fixes, we investigate the effect of providing a (possibly incorrect) code patch, the files it changed, and the test file to be modified instead of the files retrieved with BM25, reporting results in We analyze the overlap between solved instances of SWE-BENCH and SWT-BENCH, showing results in Table 6 . We observe that the overlap is small for both methods, with no statistical evidence of correlation (p-values of 80.4% and 72.8% for ZEROSHOTPLUS and SWE-AGENT, respectively, under the null hypothesis of independence and uniform hardness), indicating that generating tests and fixes are distinct tasks of different difficulties. We explore this relationship in more detail in App. D. We investigate the relationship between issue description length and test generation performance in Fig. 5 . We observe a general trend that issues with longer descriptions are easier to generate tests for, with all methods achieving a higher S for longer descriptions, however tending to slightly decrease for very long descriptions. This is likely due to the increased amount of information available in longer descriptions, while too-long descriptions may contain many distractors and make it difficult to extract relevant information for the LLM. SWE-AGENT+, which actively summarizes context, limiting file content and reducing history, is least sensitive to issue description length, achieving approximately the same S for all but the shortest lengths. As SWT-BENCH is based on historic GitHub issues, they may be contained in the pre-training data of the LLMs we use. To investigate this issue, we conducted an experiment comparing the performance of ZEROSHOTPLUS on all issues created after the Knowledge Cutoff (KC) of GPT-4 (April 2023) to a random subset of the same size of instances created before, and report the results in Table 7 . While we observe a small performance difference, we can not confirm its statistical significance (p ≈ 37%) due to the low number of samples created after the KC. Further, all methods in Table 2 use the same underlying LLM and should thus benefit from any potential contamination to a similar degree, allowing for a fair comparison between different methods. Method Complimentarity We consider four diverse methods from §5.2 and analyze the overlap in the instances for which they are able to generate successful tests. We show the results in Fig. 6 . While the best-performing approach, SWE-AGENT+, alone is able to solve 51 instances, the combination of all four approaches is able to solve 87 instances, highlighting the benefit of exploring a variety of approaches for test generation. 
limitations and future work: 
 While our novel SWT-BENCH covers a wide range of real-world issues, it has several limitations: It is limited to Python, which may limit the generalizability of our findings to other programming languages. Second, the dataset is based on popular GitHub repositories, which may not be representative of common software development practices and does preclude the generation of a private holdout test set. Finally, the dataset is limited to bug reproduction and issues that can be easily covered by adding test cases and does not measure edge case detection or global coverage increase.Further, as discussed in §5.4, most issues in SWT-BENCH have been created before the knowledge cutoff of state-of-the-art models, posing a risk for data contamination. One approach to address this issue is to create a rolling version of SWT-BENCH, based only on the most recently created GitHub issues. However, this comes at the cost of direct comparability of results and increased cost for reproducing results for all baselines on a changing evaluation set.Addressing these limitations would be an interesting direction for future work. As concrete starting points, we found several common errors even in the best performing method SWE-AGENT+ that could be addressed through specialized monitoring: Adding passing tests that do not reproduce the given issue, getting stuck in loops after generating inapplicable edit commands, failing to execute the test environment correctly and adding tests with syntax errors or using invalid variables.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We describe all key experimental details in §5. Further details on the used prompts for LLM interaction are provided in App. E.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We proposed SWT-BENCH, a novel benchmark for generating reproducing tests from GitHub issue descriptions and the corresponding code bases. SWT-BENCH leverages the dataset underlying the popular SWE-BENCH which additionally contains a golden patch fixing the described issue. We judge whether a generated test reproduces the described issue by checking whether the test fails before applying this golden patch and succeeds afterward. We measure both the rate of such fail-topass tests and the coverage of the golden patch, providing a corresponding evaluation harness. We evaluated a variety of LLM-based test generation methods including Code Agents on SWT-BENCH and found that Code Agents already outperform other approaches with only minor adaptations for the test-generation task. Finally, we demonstrated the great potential of generated tests to serve as a signal for the correctness of code fixes, i.e., we double the precision of Code Agents by filtering the generated patches to only those that cause a previously failing self-generated test to pass. We introduce a custom prompt format for language models to aid them with patch generation in the zero-shot setting. The format is visualized in Fig. 7 similar to how it is provided to the language model. A full example of applying the format on two files is part of the full prompt of ZEROSHOT-PLUS in Figs. 11 and 12 .A diff block must start and end with diff and end diff respectively. The first line inside the block must specify an existing file for rewrites and may point to a new file in the case of insertion. Next, the language model specifies whether it intends to rewrite an existing function or insert a new function. If no exact match of the function name is found, we employ a fuzzy search using the line number or EOF/BOF as an indicator for where to look for the existing functions. EOF and BOF are particularly useful for inserting new functions. We note that diff blocks can be repeated an arbitrary number of times.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We add results of tests of statistical significance in the text accompanying experiments in §5. We further provide a temperature ablation with error bars in App. C.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We outline the cost of the significant parts of our experiments (mostly dominated by the cost of LLM inference) in App. G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Our research conforms to the NeurIPS Code of Ethics. In particular, we did not conduct research involving humans, used only publicly available data open source data with permissive licenses, and believe that our research has no negative societal impact.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators have included the licenses of all used code in the submitted code together with the package. The code adjusted and used for the experiments in this paper was licensed with permissive licenses, that we respect and redistribute with out adjusted code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
29: 
 The lines common to both files begin with a space character. The lines that actually differ between the two files have one of the following indicator characters in the left print column: 30 31 '+' A line was added here to the first file. 32 '-' A line was removed here from the first file. The diff for fix in function euclidean and adds the function gcd is as follows.3 This diff changes the first file into the second file.4 ```diff 5 ---a/demo/file.py 6 +++ a/demo/file.py 7 @@ -4,4 +4,5 @@ 8 assert euclidean(1, 0) == 1 9 assert euclidean(1, 1) == 1 10 + assert euclidean(100, 10) == 10 11 12 @pytest.mark.parametrize("a, b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1) ])13 @@ -9,2 +10,6 @@ 14 assert gcd(a, b) == expected 15 16 +@pytest.mark.parametrize("a, b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1), (100, 10, 10)])17 +def test _ lcm(a, b): 18 + assert lcm(a, b) == expected 19 + 20 ``2 1 22The new version of the file is as follows.23 [start of demo/file.py] 24 1 def test _ euclidean(a, b): 25 2 assert euclidean(0, 0) == 0 26 3 assert euclidean(0, 1) == 1 27 4 assert euclidean(1, 0) == 1 28 5 assert euclidean(1, 1) == 1 29 6 assert euclidean(100, 10) == 10 30 7 31 8 @pytest.mark.parametrize("a, b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1) ])32 9 def test _ gcd(a, b): 33 10 assert gcd(a, b) == expected 34 11 35 12 @pytest.mark.parametrize("a, b, expected", [(0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 1), (100, 10, 10)]) 42 but there can be as many independent blocks of changes as you need. You may also apply changes to several files.43 Apply as much reasoning as you please and see necessary. The format of the solution is fixed and has to follow the custom diff format.44 Make sure to implement only test cases and don't try to fix the issue itself. The following text contains a user issue (in <issue/> brackets) posted at a repository. Further, you are provided with file contents of several files in the repository that contain relevant code (in <code> brackets). It may be necessary to use code from third party dependencies or files not contained in the attached documents however. Your task is to identify the issue and implement a test case that verifies a proposed solution to this issue. More details at the end of this text. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
 
 Appendix: Kang et al. (Kang et al., 2023) originally proposed LIBRO for an evaluation in a pass@k setting. There, it is useful to rank all generated tests to improve performance at k > 1. As we only consider pass@1, we drop ranking components irrelevant for the top-1 test in our reimplementation. Further, LIBRO includes heuristics for importing missing dependencies and inserting tests into the correct classes. While effective in Java, this is superfluous for Python, where tests can be added outside classes and dependency imports are (empirically) correctly generated by the LLM. We thus also drop these components.LIBRO clusters test cases based on whether the generated execution trace matches the issue description. As exact matching is often not applicable for the unstructured issue descriptions, we measure the similarity between the error message and the issue description by extracting the execution trace of the generated test cases and querying the same LLM used for test generation to judge whether they relate to the same issue. Depending on its answer, we obtain two clusters and choose the shortest result of the preferred cluster.We perform an ablation study by varying the number of samples used for the LIBRO baseline. The result is presented in Fig. 8a . LIBRO's performance improves as more samples are considered, however the gains of additional samples are marginal from around the 5 samples we use by default.As shown in section App. G, to enable comparison at similar cost to Code Agents, we settle for 5 samples.We analyze the number of interactions required for each Agent to submit a solution and plot the results in Fig. 8b . We observe increasing interaction rounds improve performance until saturation at 5-10 iterations (we use 20 as a default) with the only exception being AutoCodeRover, which still gains performance up to the maximum of 20 iterations we consider. 21.5 0.2 89.3 ± 0.3 8.8 ± 0.5 56.2 ± 0.5 9.6 ± 0.5 6.0 ± 0.2 21.8 ± 0.3 0.4 89.9 ± 0.5 10.1 ± 0.6 56.8 ± 0.6 10.9 ± 0.6 6.1 ± 0.5 22.3 ± 0.4 0.7 89.3 ± 0.4 8.8 ± 0.4 55.5 ± 0.8 9.7 ± 0.5 6.1 ± 0.5 21.1 ± 0.6 We run ZEROSHOTPLUS using GPT-4 with 25 samples and analyze the performance variation for a temperature range from greedy decoding (T = 0), used for ZEROSHOT, ZEROSHOTPLUS and the agent settings, to T = 0.7, the setting used in LIBRO. The results are presented in Table 8 . We observe a tendency of decreased performance and increasing variance in all metrics for increasing T . Moreover we observe a minimal variability among several runs of the test environment at T = 0, however much smaller than the variability due to temperature. We compare the success rate of SWE-AGENT for test and fix generation across repositories in Fig. 9 . We observe that, while SWE-AGENT obtains similar success rates in both settings in three repositories, success rates vary strongly on most other repositories. Indeed, there are five repositories where test generation fails entirely while code repair fails on three and on only two of these both fail. Manually inspecting instances from the repositories where test generation fails shows a variety of reasons, astropy usually features complex computations where unit test creation is difficult, requests features a highly complex code base, flask has extremely long golden test lengths indicating particularly challenging testing problems. For pylint generated tests are all P → P making them correct but unhelpful.ZEROSHOT, ZEROSHOTPLUS and LIBRO The full prompt for ZEROSHOT is displayed in Figs. 10 and 11. The full prompt for ZEROSHOTPLUS and LIBRO is displayed in Figs. 12 and 13 . Except for the way we include files, all lines are changed with respect to the setting in SWE-BENCH. This includes in particular the demonstration of the unified diff format on an example. In the setting for Table 5 we add the lines highlighted in boldface.The prompt for SWE-AGENT and SWE-AGENT+ is shown in Fig. 14 . Changes with respect to the prompt of (Jimenez et al., 2023) are highlighted in boldface.The additional changes for SWE-AGENT+ are highlighted in green.AIDER We only minimally adapt the provided evaluation harness for AIDER on SWE-BENCH1 . In this harness, AIDER is provided with a single initial user prompt based on the user issue, while the entire agent workflow remains unchanged. We provide the entire prompt in Fig. 16 and highlight our change in boldface.AUTOCODEROVER AUTOCODEROVER (Zhang et al., 2024) leverages a number of prompts that are provided to the model in different phases of the code/test generation process. We adapt the key prompts and display them in Fig. 15 . Changes are highlighted in boldface. Further, we change every occurrence of "bug location" in the original prompts to "relevant location". We further add a function to the ACI that allows inserting code in new files and fetching the entire code (capped at the first 100 lines) of any file.We adapt code from the following projects in our work and include the respective licenses:1. SWE-BENCH (Jimenez et al., 2023) For all licenses of the repositories used in SWT-BENCH, we refer to Jiminez et al. (Jimenez et al., 2023) , which contains a detailed list with licenses for each repository included.There is cost to both running inference on Language Models and on evaluation their predictions on the test suites of the repositories. Since the evaluation can be performed on a consumer grade machine in reasonable time, we focus on the cost inferred from LLM inference. We report the cost for each setting in Tables 9 and 10 , displaying the average cost of a full inference on SWT-BENCH Lite for each model and method. The difference between cost of PASS@5 and LIBRO is just the additional filtering step incurred by LIBRO. We run the different methods from Table 2 on 5 instances and compute the average execution time.For all LLMs we consider, part of the execution time is directly related to the number of tokens digested and generated (see Table 10 ). For methods that require interaction with an execution environment however, time is usually dominated by setting up such an environment in a clean and reproducible manner (i.e. dockerized). We list results on execution times in Table 11 and observe that all methods except zero-shot inference take between 3-5 minutes per instance, where we can observe a small trade off due to many-turn interactions in Code Agents versus single-shot execution in LIBRO. Given these small differences however, we believe execution time to be of limited practical relevance as issues can be processed in the background, similar to continuous integration, in response to raised user issues 1The following text contains a user issue (in <issue/> brackets) posted at a repository. Further, you are provided with file contents of several files in the repository that contain relevant code (in <code> brackets). It may be necessary to use code from third party dependencies or files not contained in the attached documents however. Your task is to identify the issue and implement a test case that verifies a proposed solution to this issue. More details at the end of this text.2 3 <issue> 4 user issue comes here 5 </issue> 6 7 retrieval results or oracle files come here 8 9 Please generate test cases that check whether an implemented solution 10 resolves the issue of the user (at the top, within <issue/> brackets). 11 Present the test cases in unified diff formatting.12 13 The general format of a diff is the unified output format, described as follows.14 The unified output format starts with a two-line header, which looks like this: 26 27 If a hunk and its context contain two or more lines, its line numbers look like 'start, count'. Otherwise only its end line number appears. An empty hunk is considered to end at the line that precedes the hunk.Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/paul-gauthier/aider-swe-benchWe have received following issue within our repository. Here's the issue text: 


Decision: This paper has a very wide spread of scores from 3 (reject) to 8 (strong accept). Unfortunately, two reviewers who suggest accept and strong accept did not participate in discussions. The issues raised by reviewer who suggested reject do not seem significant to me apart from data contamination (see below). After reading the paper, reviews, rebuttal, and substantial discussions between authors and reviewers, I believe that the paper should be accepted.

Summary of Strengths:

- Novel Benchmark: The paper introduces a new benchmark (SWT-Bench) for evaluating automated test generation, particularly using Code Agents and LLMs. This is recognized as filling a gap in current research.
- Practical Relevance: The benchmark uses real-world data from popular Python repositories, making it relevant and practical for testing the capabilities of LLMs in generating tests.
- Detailed Analysis: The evaluation is comprehensive, with multiple metrics (e.g., fail-to-pass rate, code coverage) and methods compared, showing the potential impact of automated test generation on software quality and developer productivity.

Weaknesses identified:

- Limited Statistical Analysis: The statistical analysis is seen as insufficient, particularly due to the computational constraints that prevented a full analysis. This limitation affects the confidence in the reported results.
- Focus on Python: The benchmark focuses solely on Python, which raises concerns about the generalizability of the results to other programming languages.
- Potential Data Contamination: The benchmark might include data from GitHub that was part of the pre-training corpus for LLMs, potentially contaminating the evaluation results.
- Limited Exploration of Hyperparameters: The paper does not fully explore the impact of different hyperparameters on the performance of Code Agents, missing potential optimization opportunities.

Authors' Response to Weaknesses:

- Statistical Analysis: Additional experiments were conducted to demonstrate the statistical significance of their results, which were reported in the rebuttal.
- Python Focus: The authors explained that while the current work focuses on Python, their approach is applicable to other programming languages, and expanding to other languages is left for future work.
- Data Contamination: The authors provided additional experiments to assess the effect of data contamination and found no significant impact on their results. However these experiments are based on small data sizes. This remains a potential issue.
- Hyperparameter Exploration: An ablation study on key hyperparameters was conducted and shared in the rebuttal, providing insights into optimizing the agents.

In terms of weaknesses, I think that only data contamination has not been fully addressed and is possibly concerning with respect to paper's results.

Reviews: review: paper_summary: paper_summary: Authors present a new benchmark for the SWE tasks of generating tests corresponding to an issue in a codebase with unit tests. They propose to repurpose the SWE-bench dataset for this task. They also evaluate LLM based prompting and agentic approaches (SWE-Agent and AutoCodeRover) with metrics like code change coverage and fail to pass rate that are formally defined and measured.
strength_weakness: strength_weakness: - Arguably the most significant impact of automatic test generation would be on automated code repair/generation. While authors describe this aspect of their work (Lines 328-334), providing more details and large-scale experiments could significantly strengthen the perceived utility of this work. 
- CodeT by Chen et al https://arxiv.org/abs/2207.10397 is a key related work that was missed in the related work.
questions: questions: - Lines 328-334: You mention about the impact on precision and recall, can you share more details of the method used and the final correctness rate with synthetic tests generated by the Agent?
- What are some future directions you would recommend to improve the performance of LLM Agents on SWT-bench and SWE-bench based on your study? A future work recommendation would greatly help the completeness of the paper.
limitations: limitations: Authors have briefly discussed limitations of their work in the main paper.
paper_summary: paper_summary: This paper re-purposes SWE-Bench, a previous benchmark on repository-level code generation, to SWT-Bench, a benchmark for test generation by asking LLMs to generate tests to reproduce the issues for various GitHub repos and see if such tests can capture the bugs before the gold-standard patches are applied and whether the generated tests will pass after the patches are applied (i.e., Fail -> Pass). Experiments are conducted with multiple baselines and results show that SWE-Agent, the previously proposed method for code repair, achieves the best performance for test generation.
strength_weakness: strength_weakness: I think the main weakness of this paper is over-claiming. While it is a clever idea to re-purpose code repair benchmarks as SWE-Bench into a test generation benchmark, there are quite a lot limitations of this method for the conclusions on the resulting dataset to claim "state of the art software tester". The main purpose of software testing is not to reproduce the issues, but given the specifications, crafting inputs and the constraints on the outputs to makes sure that the code to be tested has the expected behavior given different inputs. To this end, I believe there is a large gap between the ability that SWT-Bench is testing, and the actual test generation ability needed for the job of software testers. To make such a claim, I believe at least a few software testing benchmarks (e.g., https://swtesting.techconf.org) need to be used and the results to be reported.
questions: questions: Please respond to the concerns in the "Weakness" section.
limitations: limitations: N/A
paper_summary: paper_summary: This paper focuses on automatic test generation using large language models (LLMs) and code agents. The authors introduce a benchmark called SWT-BENCH, which aims to analyze the performance of LLMs and agents in generating unit tests given a task description. The key contributions include:

1) Creating the SWT-BENCH testing benchmark, which contains over 1700 samples from real-world GitHub issues.

2) Benchmarking various LLMs and agent systems. This includes evaluating direct LLM prompting, existing test generation tools (e.g., LIBRO), and adapted code agents (e.g., SWE-AGENT, AUTOCODEROVER). 

3) Introducing a new format for LLM-based code editing.
strength_weakness: strength_weakness: 1) The scope of the dataset could be expanded further. In its current form, it only addresses Python codebases with associated GitHub PR issues. The distribution of testing problems encountered by real-world software engineers is likely quite different from those represented in this testbed.

2) A private test set that is not part of the publicly available training data would strengthen the claims that could be made when improvements are observed on this benchmark.
questions: questions: 1) Have you looked into leveraging the failure modes of the generated test cases to improve code repair or code generation tasks?

2) LLMs for code generation seem to be very stochastic by nature. You mention that the failure modes on the benchmark vary depending on the underlying method used. To what extent do you think this variation is explained simply by the stochasticity of the method? For example, if you ran swe-bench 5 times, would that variation disappear?

3) How do you think would your method generalize in a private codebase that isn't available online?
limitations: limitations: The contamination problem with LLMs is, in my opinion, very important and under-discussed. Due to GitHub data being public, a lot of the data already exists in the training data for LLMs, which can render some conclusions less impressive. This is especially true in the code repair and test generation areas. I think it would be very helpful to include some analysis of the contamination problem. This could be done by using a held-out dataset in a similar way to how the benchmark below does it:

https://livecodebench.github.io/
paper_summary: paper_summary: This paper introduces SWT-BENCH, a novel benchmark for evaluating automated test generation capabilities of AI models, particularly Code Agents. The authors adapt existing code repair datasets and methods to the task of test generation, proposing new metrics such as fail-to-pass rate and change coverage. Their experiments reveal that Code Agents, originally designed for code repair, outperform methods specifically designed for test generation. The best-performing method, SWE-AGENT+, achieves an 11.1% success rate in generating relevant fail-to-pass tests. The study also demonstrates significant complementarity between different approaches, with an ensemble of methods solving 70% more samples than the best single method. Additionally, the authors show that generated tests can serve as a strong signal for the correctness of proposed code fixes. While the study is limited to Python and may have some selection biases, it suggests that Code Agents have significant potential for automated test generation, potentially improving both software quality and developer productivity.
strength_weakness: strength_weakness: 1. Limited statistical analysis: The authors acknowledge they were unable to perform a full statistical analysis due to computational costs. This limits the confidence in the reported results. It would be great if they can conduct a power analysis to determine the minimum number of runs needed for statistical significance.

2. Lack of error analysis: The paper doesn't provide a detailed analysis of the types of errors made by different methods. Maybe they should categorize and quantify common error types for each method. Provide qualitative examples of generated tests, both successful and unsuccessful. Analyze how error types correlate with issue complexity or repository characteristics.

3. Limited exploration of hyperparameters: The paper doesn't discuss the impact of different hyperparameters on the performance of Code Agents. To improve: Conduct an ablation study on key hyperparameters (e.g., number of interaction rounds, temperature). Provide insights on how to optimize Code Agents specifically for test generation.

4. Insufficient comparison to human performance: The paper lacks a comparison to human-generated tests. To address this: Include a small-scale study with professional developers generating tests for a subset of issues. Compare the quality, coverage, and time efficiency of AI-generated vs. human-generated tests.

5. Narrow focus on Python: The study is limited to Python, potentially limiting generalizability.
questions: questions: 1. Given the computational constraints that prevented a full statistical analysis, could you provide more details on the variability of your results? Even with limited runs, could you estimate confidence intervals for your main findings?

2. The paper lacks a detailed error analysis. Could you provide examples of common failure modes for the different methods, particularly for SWE-AGENT+? How do these failure modes relate to the characteristics of the issues or repositories?

3. How sensitive are the Code Agents' performances to different hyperparameters? Did you explore variations in the number of interaction rounds or temperature settings? If so, what insights can you share about optimizing Code Agents specifically for test generation?

4. The paper doesn't compare AI-generated tests to human-generated ones. Have you considered conducting even a small-scale comparison with professional developers? This could provide valuable context for understanding the practical impact of your methods.
limitations: limitations: 1. Limited Statistical Analysis: Due to computational constraints, the authors were unable to perform a full statistical analysis. This limits the confidence in the reported results and makes it difficult to assess the robustness and reproducibility of the findings.

2. Lack of Detailed Error Analysis: The paper doesn't provide an in-depth analysis of the types of errors made by different methods. This limits understanding of where and why the methods fail, which could be crucial for further improvements.

3. Python-Centric Approach: The study focuses exclusively on Python, which may limit the generalizability of the results to other programming languages.

4. Absence of Human Baseline: There's no comparison between AI-generated tests and human-generated tests, making it challenging to assess the practical impact and efficiency gains of these methods in real-world scenarios.

5. Limited Hyperparameter Exploration: The paper doesn't thoroughly explore the impact of different hyperparameters on the performance of Code Agents, potentially missing opportunities for optimization.

6. Lack of Runtime Performance Analysis: While computational costs are mentioned, there's no detailed analysis of execution times for different methods, making it difficult to assess practical trade-offs.
paper_summary: paper_summary: In this paper, the authors propose a novel benchmark for evaluating the capability of LLMs and Code Agents in generating test cases from GitHub issues. The benchmark includes real-world issues, patches, and golden tests from popular Python repositories. The authors claim that Code Agents outperform traditional test generation methods and highlight the potential of these agents in improving software quality and developer productivity.
strength_weakness: strength_weakness: The data contamination problem. The proposed benchmark is collected from GitHub, which might have been exposed to the pretraining corpus of LLMs.

The authors overlook substantial existing work on test case generation using LLMs.

There is no discussion on the long-term relevance and maintenance of the benchmark, which is crucial for its sustained utility.

The benchmark is focused solely on Python, ignoring other programming languages. 

The experiments are conducted on only three LLMs (GPT-4, Claude-3, Mixtral 7x22B), which is insufficient for a comprehensive evaluation of the benchmark.

The paper fails to address the broader societal impact.
questions: questions: How does the benchmark ensure that it does not include tasks already present in pre-trained DNN datasets?

Why were significant existing works on LLM-based test generation not included in the paper?

What measures are in place to ensure the long-term relevance and maintenance of the SWT-BENCH benchmark?

How might the results differ if the benchmark included tasks in programming languages other than Python?

Can the authors discuss the potential societal impacts of the widespread adoption of Code Agents in software testing, particularly concerning developer employment?


Please see the above to clarify any misunderstandings and add additional results.
limitations: limitations: The paper lacks discussion on various limitations, including the limitations of the metrics used and potential data contamination across models.

Comments: title: Reply to Reply to Reply to ueFW
Reply to Reply to ueFW
Response by Reviewer ueFW
Reply to Reply of i5fN
Thanks for the response
Reviewer responses to authors (Aug 7 - Aug 13)
Thank you
Response to the AC
Questions for authors
Author-Reviewer discussion (Aug 7 - Aug 13)
comment: We thank the reviewer for their quick reply and hope that we could successfully address all points they did not have follow-up question on.

**Effect of Data Contamination on Applicability for GPT4**  
While the applicability is indeed higher for issues created before the KC for GPT4, both F2P rate and Coverage are lower, which are much stronger indicators for memorization. Applicability solely measures whether the LLM can respect the required diff format, while F2P and Coverage measure the correctness of the generated tests. While memorization would lead to a correct test (higher F2P and coverage) it would not necessarily improve applicability as the test would not have been included in the training data in the right diff format. Further, we want to highlight that only 11 samples after the KC were available, making any conclusions based on these results statistically questionable (as highlighted in our previous reply).
Dear authors

Thanks for your response. 

I find that the Applicable (pre-KC) for GPT-4-turbo-preview are largely different with post-KC. Can you clarify that?
We thank the reviewer for acknowledging our rebuttal, engaging in the discussion, and raising their score. We would like to address their comments below.

**Can you provide the data contamination results for each model?**  
Below we report results on SWT-bench Lite for all considered models, split by whether the issue was created before or after the corresponding Knowledge Cutoff (KC). The results correspond to Table 2 in the global attached PDF, comparing the model performances using SWE-Agent. Note that for our previous results on GPT4, we considered the full SWT-bench to ensure a sufficiently large number of samples from after the KC is available. Due to time constraints, this is not possible here and the smaller size of SWT-bench Lite results in much fewer instances being available after the KC. In fact, many of the recent models have a KC later than the latest instance (15 Aug 2023) in the full SWT-Bench (see the table below). For the only model with sufficiently early KC (Mixtral 8x22B), the overall F2P rate is so low that no meaningful comparison is possible (2/190 F2P instances before KC and 0/89 after).

| Model                 	| KC          | PR created   |   n | Applicable   |F2P   | Coverage   |
|---------------------------|-------------|--------------|-----|--------------|-------|------------|
| GPT-4 Preview 1106    	| 30 Apr 2023 | before KC	| 268 | 76.9     	| 16.4  | 18.1   	|
|                       	|             | after KC 	|  11 | 54.5     	| 18.2  | 20.0   	|
| Mistral Large 2       	| 31 Jan 2024 | before KC	| 279 | 4.3      	| 0.7   | 0.2    	|
|                       	|             | after KC 	|   0 | -        	| - 	| -      	|
| Claude 3.5 Sonnet     	| 30 Apr 2024 | before KC	| 279 | 60.2     	| 12.2  | 22.1   	|
|                       	|             | after KC 	|   0 | -        	| - 	| -      	|
| GPT-4o mini  (2024-07-18) | 31 Oct 2023 | before KC	| 279 | 53.8     	| 9.7   | 13.0   	|
|                       	|             | after KC 	|   0 | -        	| - 	| -      	|
| Claude 3.0 Haiku      	| 31 Aug 2023 | before KC	| 279 | 6.8      	| 2.9   | 1.9    	|
|                       	|             | after KC 	|   0 | -        	| - 	| -      	|
| Mixtral 8x22B         	| 30 Sep 2021 | before KC	| 190 | 2.6      	| 1.1   | 0.0    	|
|                       	|             | after KC 	|  89 | 2.2      	| 0.0   | 1.4    	|


**Why did you not discuss CodeT in the paper?**  
As discussed in detail in our response to Reviewer PhkB, CodeT is not applicable to our setting, and we thus omitted it in our literature review due to space constraints. However, we will make sure to include it with a suitable discussion in the next revision of our paper, using the extra space of the camera ready version.

**Why did you not consider popular Open-Source Code Models like CodeLlama?**  
During our experimentation, we indeed evaluated CodeLlama-70B and WizardCoder-Python-34B. However, we found they were not capable of following the desired diff format or agent instructions. Instead, they produced incoherent, incorrect, and degenerated outputs. As they thus consistently yielded close-to-zero F2P rates, we have excluded them from the reported results. We do not believe that comparing such low F2P rates would be interesting or meaningful, but are happy to include results for these models in the appendix of a revised version. Since both the strongest models (e.g. GPT4) alone (ZeroShotPlus) and slightly weaker models (e.g. Claude 3.0 Haiku) in an agent framework struggle with SWT-Bench, we believe it is most interesting to focus on comparing different agent frameworks using the best available base models.

**Conclusion**  
We hope to have been able to address the reviewers concerns and are looking forward to their response.
Thank you for the responses, and sorry for the late response due to a flurry of proposals and review dues. I have increased my overall assessment as some of my concerns have been addressed. While others are shown in below.

**Data contamination** Thanks for your experiments for GPT-3-preview 1106, the experiments do not address my concern. At least, I need to mention the data contamination for all models by directly providing previous inference results and dividing them based on the KC time.

**Existing works not included in the paper?** Thanks for your response. But I need to mention that omitting CodeT is crazy.

**Other languages?** The response does not address my concern.

**Other LLMs** I would recommend providing experiments for OpenCodeInterpreter, DeepSeek-Coder, XwinCoder, CodeLlama, WizardCoder, and Starcoder2 family in open-source LLMs for a new benchmark.
We thank the reviewer for their response and engaging in the discussion. We are happy to modify the title of our submission (in accordance with the submission guidelines) to follow the reviewer’s suggestion and avoid over-claiming.  

We propose to adjust the title to **Can Code Agents Reproduce Real-World GitHub Issues?**.

 As promised in the initial rebuttal, we will also revise the text to make the scope of our work more explicit. We are happy to answer any followup questions or suggestions they might have.
I'd like to thank the authors for the response. 

> While our title may be placative, it only implies that Code Agents perform better than other automated systems on the studied task.

I think this is where the major discrepancy is: the studied task is **GitHub issue reproduction**, which is quite far from what people would normally think of the work of **software testers**. Thus I still think the title is over-claiming and somewhat misleading, and I'm afraid I could not give higher score for this work.

Respectfully,  
Reviewer i5fN
Dear Submission15335 reviewers,

We appreciate your reviews. We have only two days left for the author-reviewer discussions. Please read all reviews and author responses. It is helpful if you respond to authors' rebuttal and address any remaining questions or concerns with the authors. At a minimum, please acknowledge that you have read the author rebuttal. If your evaluation of the paper has changed, please adjust your scores and reviews as necessary.

Thank you.
Thank you for your responses.
We thank the AC for taking such an active role in chairing the discussion and are happy to answer their questions below:

**What are golden tests?**  
Exactly as the AC suggests, these are the tests that were added in the pull request that “resolved” the GitHub issue underlying the SWT-Bench instance. At least one of these tests fails on the original state of the codebase, before the golden code patch is applied, and passes after. They are thus always a valid (gold standard) solution for the SWT-Bench task. We will make this more clear in Line 32 where they are first introduced.

**Can you discuss what can cause none of the lines modified by the golden code patch to be covered by the test suite?**  
The main reason for this is that Python’s _trace_ module by default is unable to measure coverage in subprocesses. If (part of) a test suite is executed using such independent processes, coverage can not be measured accurately and no execution of the patched code is reported. Meanwhile, threading based concurrency is handled correctly. We would like to highlight that this issue comes from the task instance and Python’s _trace_ module, but not the test generation method. The measured coverage is thus still comparable across different test generation methods and computed over the remaining 57% of SWT-Bench instances.

**Can you clarify what a _valid_ patch is in Section 5.2?**  
A valid patch is one that can be successfully/without error applied to a codebase, i.e. that has the correct format. As generated tests are added to the code base in the form of a patch, we reused the term “valid patch”, currently defined implicitly in Line 170. We will follow the AC’s suggestion and rename them to “well-formed tests” and introduce this terminology explicitly to improve clarity. As many models struggle to produce valid/well-formed (i.e. applicable without error) test patches in the default unidiff standard, the rate of valid test patches is a relevant metric in particular when comparing our novel (ZeroShotPlus) with the standard unidiff patch format (ZeroShot).

**What is a fail-to-any (F -> X) test and why is it “good” to measure this?**  
A fail-to-any test is one that fails on the original codebase and either passes or fails on the codebase with the golden patch applied. While some F->X tests might be undesirable as the AC suggests, we consider the rate of F->X tests as a useful metric for assessing a test agent’s overall capabilities for two main reasons. First, creating an F->X test is a prerequisite for generating a fail-to-pass (F->P) test, which is ultimately the desired outcome. Therefore, the rate of F->X serves as an intermediate metric for our final F->P metric. Second, given that the agent has access to the original version of the codebase and the issue report, it can leverage these two inputs to self-verify if a test fails on the codebase before proposing the test. The rate of F->X tests reflects the agent’s ability to perform this self-verification. We will add this discussion in the corresponding section.

**Conclusion**  
We hope to have been able to address the AC’s questions and remain happy to answer any follow-up they might have.
Dear authors,

Thank you for your rebuttal.
I have couple questions about your submission that may clarify certain aspects of your paper and facilitate the discussion with reviewers.

Page 3:
"Golden Tests" -> are these tests that were added in the patch under consideration?

Page 4:
"If none of the lines modified by the golden patch X are executed by any test, i.e., |X ∗r | + |X ∗166 a | = 0, we exclude this instance from our coverage analysis (43.1% of the cases)." - can you elaborate on this? How does the patch fix an issue if none of the tests execute any lines in it? Is the patch deleting erroneous code? 43% seems like a large percentage of cases for such occurrences.

Page 7, table 2:
What is "valid patch" in section 5.2? Is it a test case patch that is valid? You are not talking about a code patch that fixes the issue, right? How is the validity checked? Maybe consider renaming the "valid patch", since it might be misleading for readers when test change is named "patch" the same way as code fix change is named "patch".

What is fail-to-any tests (F→×)? Why are they considered to be good? Can you elaborate? Isn’t there a risk that LLM generates a meaningless test that fails both on initial codebase and on the patched codebase and therefore does not contribute a valid test?

Thank you
Dear Submission15335 reviewers,

We appreciate your reviews as we move into the Author-Reviewer discussions phase (Aug 7 - Aug 13).
Please read all reviews and author responses carefully. 
Please address any remaining questions or concerns with the authors and respond to their rebuttal as needed. Authors need time to respond to your messages, so please post your responses as soon as possible, so there is time for back-and-forth discussion with the authors. At a minimum, please acknowledge that you have read the author rebuttal. Based on the material provided, please adjust your scores and reviews as necessary.

Dear Submission15335 authors,

We appreciate your rebuttal. Please continue the dialogue with the reviewers during this discussion phase.

This message and thread are visible to both reviewers and authors. If you need to respond privately, adjust the "Readers" setting accordingly.