TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark

## Abstract

Abstract Code generation models can help improve many common software tasks ranging from code completion to defect prediction. Most of the existing benchmarks for code generation LLMs focus on code authoring or code completion.
Surprisingly, there has been far less effort dedicated to benchmarking software testing, despite the strong correlation between well-tested software and effective bug detection.
To address this gap, we create and release TestGenEval , a large-scale benchmark to measure test generation performance. Based on SWEBench, TestGenEval comprises 68,647 tests from 1,210 code and test file pairs across 11 well-maintained Python repositories. It covers initial tests authoring, test suite completion, and code coverage improvements.
Test authoring simulates the process of a developer writing a test suite from scratch, while test completion mimics the scenario where a developer aims to improve the coverage of an existing test suite.
We evaluate several popular models, with sizes ranging from 7B to 405B parameters.
Our detailed analysis highlights TestGenEval ’s contribution to a comprehensive evaluation of test generation performance. In particular, models struggle to generate high-coverage test suites, with the best model, GPT-4o, achieving an average coverage of only 35.2%. This is primarily due to models struggling to reason about execution, and their frequent assertion errors when addressing complex code paths.

### 1 Introduction

Software testing is a critical component of the software development process. A high-quality test suite can be instrumental in finding inconsistencies between a system’s specifications and its implementation. Test suites ideally execute all code paths (high code coverage), and catch regressions in the code under test that a developer might introduce (high mutation score) Fraser & Arcuri (); Veloso & Hora (). However, writing high quality tests can be time-consuming Beller et al. (; ) and is often either partially or entirely neglected.

As a result there has been extensive research on automated test generation, including both
classical Fraser & Arcuri (); Brandt & Zaidman (); Baldoni et al. () and neural-based methods  Dinella et al. (); Watson et al. (); Touvron et al. (); OpenAI (). However, despite this growing area of research, unit test generation benchmarks remain limited in size and scope Chen et al. (); Bhatia et al. (). While existing benchmarks capture test generation abilities on simple, typically self-contained programs, there is an absence of large scale test generation benchmarks corresponding to real world use cases. Other related benchmarks Jain et al. () report performance on adjacent tasks such as generating equivalence tests rather than standard unit tests, which also differs from the real world use case. Current benchmarks report pass@k, with few reporting code coverage and none reporting mutation score, despite mutation score being most correlated with real fault detection Just et al. (); Papadakis et al. ().

Existing benchmarks also do not measure test completion capabilities, despite many code completion benchmarks existing Liu et al. (); Zhuo et al. (). Test completion can be used to add tests to an already existing unit test file and improve overall coverage. This is important for IDE auto-completion features, where given a part of a test file an the code under test, the goal is to add more tests. Test completion is also measured by many state of the art software testing models Rao et al. (); Nie et al. (); Dinella et al. (); Tufano et al. (), yet a benchmark that measures test completion doesn’t exist.

Figure: Figure 1: An overview of TestGenEval. We start with a GitHub code file and generate a test suite with an LLM. Then we execute the generated test suite and measure the proportion of lines in the code file that are executed (code coverage). We also inject synthetic bugs into the code and measure the proportion of synthetic bugs detected by the generated test suite (mutation score).
Refer to caption: extracted/5893052/figures/testgeneval_overview.png

Motivated by this, we introduce TestGenEval with two tasks 1) full file unit test generation and 2) test completion (see Section 2.2 for more details). Our benchmark consists of real world projects, with each source file containing an average 1,157 lines of code (LOC) and each test file containing an average of 943 LOC. TestGenEval consists of 68,647 tests from 1,210 unique code-tests file pairs. For fast iteration in low-compute settings, we also provide a smaller version of the benchmark TestGenEvalLite, which approximates all the metrics computed in TestGenEval (see Appendix E for more details).
TestGenEvalLite includes 160 code-tests file pairs, file unit test generation, and test completion tasks.
It was sampled to be representative of the full TestGenEval: the repositories and other statistics are similar in TestGenEvalLite and TestGenEval (see Appendix A for more details and Appendix G for statistical significance tests).

We find that models struggle to generate high quality test suites (Section 3.1). The best performing model—GPT-4o—has an average coverage of 35.2% and a mutation score of 18.8%. Generating tests for large scale projects is significantly harder than generating tests for self-contained problems; this is reflected by significantly lower scores compared to existing benchmarks such as TestEval Wang et al. (), where top models achieve nearly 100% line coverage. Test completion (Section 3.2) is significantly easier than test generation, reflected by the high pass@5 rates of the best performing models. However, models struggle to add coverage to a complete test suite, with top models adding less than 1% coverage when generating the last test for an existing file.

We perform extensive quantitative and qualitative analysis of all results (Section 4). We measure correlation between TestGenEval and other popular benchmarks, the correlation between the test generation and test completion tasks, and the correlation between models for each task. We also perform an analysis of errors, and measure the effects of sampling more and context size on TestGenEval performance (Section 4.1). We also examine cases where TestGenEval can discriminate between highly performing models (Section 4.2). More analysis can be found in Appendix F.

We provide all the code for our benchmark at [https://figshare.com/s/51171ae97cd21d233d4f](https://figshare.com/s/51171ae97cd21d233d4f), including detailed instructions on how to run our benchmark, and even extend it. We also provide a website with all model generations for TestGenEval. We hope that this will enable the community to use TestGenEval and further build upon our work.

Our contribution are as follows:

- •
We release a benchmark for partial and full test suite generation on a realistic set of 1,210 snippets in 11 repositories. We use coverage and mutation score metrics to evaluate the value of the generated test suites
- •
We evaluate various prominent open and closed-source code generation models on our benchmark, and show that, for large scale repositories, models struggle to generate high coverage test suites
- •
We release docker images allowing to easily run code from these 11 repositories and evaluate scores on our benchmark

### 2 TestGenEval

Figure: Figure 2: A pipeline describing the creation of TestGenEval. We start with docker images of the SWEBench dataset and instrument them with coverage and mutation score dependencies. Then we extract code test file pairs by performing a heuristic match on filename. Finally, we filter out tests that don’t have coverage on the code under test and run in 60 seconds.
Refer to caption: extracted/5893052/figures/data_pipeline.png

TestGenEval consists of 1,210 code test file pairs from 11 large, well-maintained repositories (3,523-78,287 stars). We use these file pairs to construct two testing tasks: 1) unit test completion for the first, last and additional tests and 2) full file unit test generation. Our benchmark is easy to run and extend, as we have docker containers for each version of each repository with coverage and mutation testing dependencies installed. For both tasks we use execution based metrics, including pass@1, pass@5 along with code coverage improvement, and mutation score improvement compared to the gold (human written) tests. Code and test files in TestGenEval are long in length (on average 782 LOC per code file and 677 LOC per test file) and high coverage (median coverage of 60.4%). Detailed statistics of TestGenEval can be found in Appendix A.

#### 2.1 Benchmark Construction

We construct TestGenEval by adapting the SWEBench dataset to software testing tasks. Figure 2 shows the full set of steps to construct TestGenEval. We apply these exact steps to SWEBenchLite to construct TestGenEvalLite.

Modify SWEBench images: We start with the docker images provided by SWEBench (^1^11https://github.com/aorwall/SWE-bench-docker). We first modify the images that did not build manually, by installing appropriate dependencies and modifying requirements files so every test suite executes. Next, we install both coverage and mutation testing dependencies and modify the test commands to run with coverage instrumentation.

Extract test file pairs: After we have the execution environment for each pull request version built, we next extract code test file pairs from the code and test files run in the PR. Specifically, we extract code test file pairs by performing a heuristic match on filenames, and filtering out pull requests that do not meet our heuristic. With each file pair we perform program analysis on the test file to extract the first test, last test as context for our test completion settings. We later validate for each file pair that the gold tests cover some part of the code under test.

Filter paired tests: Next, we run code coverage for all the gold test settings (first, last, and extra) to filter out repositories where contexts were extracted incorrectly or partial test files do not run. We also filter out tests that take longer than 60 seconds to run to ensure TestGenEval runs efficiently.

#### 2.2 Tasks

Figure 5 shows an example of both testing tasks. TestGenEval consists of 2 separate tasks: test generation and test completion.

Test generation: The goal of test generation is to generate an entire test suite given the class under test. We provide the necessary inputs to the model as part of the prompt (see Appendix C for full details). Our test generation task aligns with real world unit testing; unit testing in practice involves writing tests for large code files in complex projects.

Test completion: The goal of test completion is to generate the next test in an existing test suite given an existing test suite and the file under test. Test completion is measured by many software testing models, and can be applied to in IDE tools, however there is no benchmark for this task. The test completion task aligns with different stages in the development life cycle; first test completion mirrors a developer starting their test suite, last test completion mirrors the finishing of their test suite and extra test completion measures whether language models can add an additional test to a test suite a developer thinks is complete. This setup is in line with current test completion work Rao et al. (); Nie et al. ().

Figure: Figure 3: Full test suite generation

#### 2.3 Properties of TestGenEval :

**Table 1: Comparison of TestGenEval against existing test generation benchmarks. TestGenEval mirrors the real world task of unit test generation, including file level, human written tests. TestGenEval includes both individual test generation and full test suite generation, and is the only benchmark to measure mutation score.**
| Dataset | Size | File level | Human tests | Test suite | Mutation score |
| --- | --- | --- | --- | --- | --- |
| TestEval | 210 | ✗ | ✓ | ✗ | ✗ |
| SWT-Bench | 1762 | ✓ | ✓ | ✗ | ✗ |
| R2E | 246 | ✓ | ✗ | ✓ | ✗ |
| TestGenEval | 1210 | ✓ | ✓ | ✓ | ✓ |

Table 1 outlines some of the properties that differentiate TestGenEval from existing test generation benchmarks. We explain each property in depth.

File level test generation: Real world unit testing involves reasoning over complex files, generating tests for a given class under test. Unlike existing benchmarks that deal with a small, self contained programs, TestGenEval includes code and tests from large scale, highly starred projects.

Human-written tests: Existing benchmarks such as R2E Jain et al. () measure the ability of an LLM to generate equivalence tests. While models that generate equivalence tests have the advantage of high coverage and ease of scaling to large amounts of data, they often look different from developer-written tests Kambhamettu et al. (). Measuring the ability of LLMs to generate equivalence tests is an adjacent, but different task from measuring unit test generation capabilities. TestGenEval is the first large-scale test-generation benchmark using human-written tests.

Test suite generation: The goal of unit test generation is to generate high-quality test suites, which is correlated to high coverage and mutation score for the code under test.
However, existing methods such as TestEval Wang et al. () and SWT-Bench Mündler et al. () only measure the ability of an LLM to generate an individual test method rather than an entire test suite. We propose complementing individual test completion tasks with the broader test suite generation task.

Mutation score: Unlike existing benchmarks, we report mutation score in conjunction with coverage. The mutation score is empirically far more correlated with bug detection capabilities Just et al. (); Papadakis et al. () and much harder to hack; in order to achieve high mutation score tests must be able to discriminate non-buggy code from code with synthetic bugs introduced.

### 3 Evaluation

We evaluate a selection of models on TestGenEval to better understand how models of different sizes and families (see Table 2 for list, and Section D.1 for more details about model choices). The HuggingFace checkpoints of all open source models used are also available at Appendix B. We prompt each model with the maximum context window size possible, otherwise truncate the starting tokens to fit the prompt in the context window.

We report results for all models in both the full test generation (Section 3.1) and test completion tasks (Section 3.2) on TestGenEval. For test suite generation we report pass@1 (if any of the tests in the generated test suite pass), all pass@1 (if the generated test suite passes), coverage (coverage of passing tests), and mutation score (proportion of synthetic bugs introduced to code caught by test suite). For test completion we report pass@1 and pass@5 (whether generated test passes), along with coverage improvement from adding the generated test. More detailed descriptions and our full set of metrics can be found at Section D.2. Our full set of results for TestGenEval and results for TestGenEvalLite can be found in Appendix E. We also perform statistical significance tests and report 95% confidence intervals in Appendix G.

#### 3.1 Test generation performance of various models

**Table 2: Full test suite generation. All results shown for temperature=0.2. Larger models generally perform better at test suite generation, however all models struggle to achieve high coverage and mutation score.**
| Model | All Pass@1 | Pass@1 | Coverage | Mutation |
| --- | --- | --- | --- | --- |
| Small Models |  |  |  |  |
| CodeLlama 7B | 3.2% | 4.1% | 1.2% | 0.5% |
| Gemma 9B | 3.5% | 42.1% | 20.2% | 9.0% |
| Llama 3.1 8B | 3.1% | 30.5% | 14.1% | 6.8% |
| Medium Models |  |  |  |  |
| DeepSeekCoder 16B | 23.0% | 63.7% | 28.2% | 12.1% |
| Gemma 27B | 7.4% | 57.7% | 30.1% | 14.6% |
| Codestral 22B | 26.8% | 72.7% | 33.0% | 14.2% |
| Large Models |  |  |  |  |
| CodeLlama 70B | 14.0% | 22.7% | 7.0% | 2.5% |
| Llama 3.1 70B | 7.8% | 60.7% | 30.6% | 15.4% |
| Flagship Models |  |  |  |  |
| GPT-4o | 7.5% | 64.0% | 35.2% | 18.8% |
| Llama 3.1 405B | 17.7% | 73.1% | 35.0% | 16.4% |

Table 2 shows pass@1, all pass@1 coverage and mutation score for small, medium and large models. All three smaller models perform significantly worse than their larger counterparts, with a large difference (42.6% for Llama 3.1 models and 15.6% for Gemma models in pass@1 performance). All pass@1 penalizes how verbose a model is: DeepSeekCoder 16B on average generates a test suite with 106 lines of code and 16 methods, while Llama 3.1 70B generates an average of 324 lines of code and 36 methods. This intuitively makes sense, as the more verbose a model is, the more likely it is to generate a test with errors.

Coverage and mutation score remain low across all models. For coverage, GPT-4o performs the best, but still covers only 35.2% of the lines of code tested in TestGenEval.
The mutation score is even lower than the coverage, which implies that tests generated by these models cannot catch all induced bugs. Despite generating passing tests for a smaller proportion of programs, GPT-4o achieves higher coverage than Llama 3.1 405B, meaning that GPT-4o generates higher quality tests on average (also reflected by the higher mutation score of GPT-4o compared to Llama 3.1 405B).

#### 3.2 Test completion performance of various models

**Table 3: Pass rates for different models in first, and last settings. Pass@1 is for temperature=0.2, Pass@5 is for temperature=0.8. Codestral 22B outperforms all models across pass@1 and pass@5 (other than first test completion pass@5), solving between 60-70% of all tasks.**
|  | First | Last |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Pass@1 | Pass@5 | +Cov | Pass@1 | Pass@5 | +Cov |
| Small Models |  |  |  |  |  |  |
| CodeLlama 7B | 4.2% | 19.8% | 6.7% | 6.9% | 24.0% | 0.0% |
| Gemma 9B | 8.4% | 18.0% | 6.7% | 21.4% | 46.4% | 0.1% |
| Llama 3.1 8B | 14.4% | 33.3% | 12.8% | 32.0% | 54.3% | 0.1% |
| Medium Models |  |  |  |  |  |  |
| DeepSeekCoder 16B | 18.6% | 47.0% | 19.0% | 17.0% | 62.8% | 0.2% |
| Gemma 27B | 12.7% | 33.7% | 13.6% | 32.2% | 62.2% | 0.1% |
| Codestral 22B | 38.3% | 61.7% | 24.0% | 50.4% | 74.3% | 0.4% |
| Large Models |  |  |  |  |  |  |
| CodeLlama 70B | 0.5% | 30.2% | 11.2% | 0.9% | 50.7% | 0.0% |
| Llama 3.1 70B | 19.3% | 46.4% | 18.7% | 35.0% | 61.9% | 0.5% |
| Flagship Models |  |  |  |  |  |  |
| GPT-4o | 31.9% | 63.5% | 26.9% | 32.6% | 66.6% | 0.5% |
| Llama 3.1 405B | 32.1% | 57.7% | 21.6% | 42.6% | 72.3% | 0.3% |

Table 3 shows pass@1, pass@5 and coverage improvement for first and last settings. Information on extra test completion can be found in Appendix E). Model performance generally increases as more of the test file is provided as context (extra pass@5 is higher than both last pass@5 and first pass@5). Similar to the full test setting, larger models tend to outperform smaller ones. An outlier is Llama 3.1 8B, with a significantly higher pass@5 in all settings than its smaller model counterparts. Codestral 22B performs the best at generating passing tests, with a pass@5 of 74.3% on the last test completion setting.
Models face challenges in augmenting coverage for existing human-written test-suites, whereas they can more readily add coverage when no tests are initially present. In the final test completion setting, all models generate virtually no new coverage, primarily testing computation paths that have already been covered.

### 4 Analysis

We perform a quantitative and qualitative analysis of all results. This includes correlation with other benchmarks, correlation between settings and models, common model errors, effects of samples on pass@k, effects of context window size along with a qualitative analysis of differentiating problems between Codestral, GPT-4o and Llama 405B. Our complete analysis can be found in Appendix F.

#### 4.1 Quantitative Analysis

Correlation with other benchmarks: We measure correlation between TestGenEval and HumanEval (code generation benchmark), TestEval (test generation benchmark), and MMLU (natural language benchmark). We find a weak positive correlation between TestGenEval scores and other benchmarks. For HumanEval outliers include Gemma 9B (performs better at test generation than expected given code generation performance) and CodeLlama (performs worse at test generation than expected). For TestEval, the main outlier is Gemma 9B, with reasonable test generation performance on TestGenEval, but very poor performance on TestEval (it fails to properly follow the prompt format for TestEval).
Code-specific models such as Codestral and DeepSeekCoder are not targetting MMLU and unsurprisingly rank much better on test generation than on MMLU.
Full details and plots of benchmark correlation can be found in Section F.1.

Correlation between settings and models: We measure the correlation between all settings in TestGenEval and models within each setting. The full test generation is less correlated with test completion performance (0.75-0.85 correlation coefficient) than the test completion settings are with each other (0.85-1.00 correlation).
The extra and last test completion settings are nearly perfectly correlated with each other, indicating that the final test of a test suite has little impact on what models generate. Models correlation within a setting is relatively low (highest correlation between predictions performance is 0.3), however models in the same family (i.e. Llama 3.1 8B, Llama 3.1 70B and Llama 3.1 405B) are more correlated than models across families. Full details and plots of setting and model correlation can be found in Section F.2.

Effect of number of samples: We measure the effect of k on our coverage and pass@k metrics used in full test suite generation and test completion tasks on Llama 3.1 8B. For full test suite generation we sample 20 examples and measure coverage@k. For test completion we sample 100 examples and measure pass@k. For full test suite generation, we find that coverage seems to gradually increase, with no plateau in the first 20 generations. For test completion, we find that the first 5 samples improve performance the most, and performance gains seem to plateau after k=20 (the gain between k=20 and k=100 is minimal). Full details of this analysis can be found in Section F.3.

Effect of context window on TestGenEval performance: We measure the effect of context length for both our test generation and test completion tasks. For test generation, we find that coverage only slightly improves with additional context; even seeing part of the code under test is enough for the model to generate a test suite (can mock inputs to various methods in the partial file). However for test completion context is more important, with pass@5 increasing with more context up until around 32k tokens where benefits decay. To complete tests, one must understand existing tests and their relationship with the code under test. Having the entire file helps contextualize what existing tests are testing, improving the performance of completed tests. Full details of this analysis can be found in Section F.4.

Most common model errors: We parse the execution logs of all models on TestGenEval and extract errors from the output of running tests. Across all models, the most common error is not generating tests with asserts; the models often fail to follow the prompt and generate code rather than tests.
Larger models such as Llama 405B and GPT-4o follow the prompt much more accurately. For them, the most frequent errors are due to poor reasoning about code and test execution, as shown by a large number of assertion and timeout errors.
Hallucination is also a problem for flagship models, with models incorrectly invoking the class under test (name and type errors). Full details of our error analysis can be found in Section F.5.

#### 4.2 Qualitative Analysis

We perform a qualitative error analysis of our top three models (GPT-4o, Llama 405B and Codestral 22B), describing cases where TestGenEval discriminates between these models (one model solves a problem, other models do not). These examples are simplified, and the full examples can be found in our website. Further qualitative examples of common error types can be found in Section F.6 and examples of frontier problems (problems solved by at most one model and no models) in Section F.7.

##### 4.2.1 Example 1 - Test setup (only solved by GPT-4o)

Figure: Listing 1: Query set class for managing data from database.

Our first example involves a QuerySet class that manages records returned from a database. The class has no database dependencies. The initialization takes a model and query, which can also not be set if no database is being used. This is hard to test because it involves either mocking or creating a complex model object.

Figure: Listing 2: Cut GPT-4o generated test

GPT-4o instantiates the QuerySet with no parameters (the simplest possible version of the test). It then tests all code methods, with all of the generated tests passing on the code under test.

Figure: Listing 3: Cut Llama 3.1 405B generated test

This is not the case with both Llama 3.1 405B and Codestral 22B. Both models ignore the empty case entirely and instead hallucinate invocations or imports of models in Django (instead these models should have mocked the model object and tested the null case). All models failing to properly mock the class under test’s dependencies lead to low coverage of the class under test, despite GPT-4o generating passing tests all when other models fail.

##### 4.2.2 Example 2 - Incorrectly mocking objects (only solved by Llama 3.1 405B)

Figure: Listing 4: Migration detection class to automatically detect changes needed between project states.

Our second example involves a MigrationAutodetector class that takes in two project states and measures the changes between the states passed in to automatically detect code that should be migrated.

Figure: Listing 5: Cut Llama 3.1 405B generated test

Llama 3.1 405B generates passing test for the changes method, providing an empty graph and model, meaning the only difference between the from and to states is the addition of the model. The assert passes, as the method only outputs one change.

Figure: Listing 6: Cut GPT-4o generated test

GPT-4o attempts to mock the ProjectState class (passing instance checks of the ProjectState class). However, it misses the line in initialization where the models attribute of from_state is accessed. This leads the setup to fail, and the MigrationAutodetector initialization fails. Codestral 22B makes a similar error, where it incorrectly mocks the ProjectState class and misses the model attribute.

##### 4.2.3 Example 3 - Handling class dependencies (only solved by Codestral 22B)

Figure: Listing 7: Database classes to handle lookups and comparisons in queries.

Our final example deals with lookups and comparisons that could be applied when searching a database (for example exact match, greater than, less than, etc.). The file under test has many classes and subclasses such as Exact, which depends on FieldGetDbPrepValueMixin.

Figure: Listing 8: Cut GPT-4o generated test

GPT-4o and Llama 3.1 405B both hallucinate class invocations, rather than mocking or using the classes provided in context. F is not a correct class to pass as lhs (missing the required output field).

Figure: Listing 9: Cut Codestral 22B generated test

Unlike GPT-4o and Llama 3.1 405B, Codestral 22B is able to understand the file under test. Codestral 22B both imports and instantiates valid field under test that is also imported in the file under test and correctly instantiates the Exact class that has a dependency on the field.

### 5 Limitations

We outline potential limitations with TestGenEval. A more detailed set of limitations can be found in Appendix H.

Overfitting to SWEBench repositories: A potential limitation is that our benchmark is adapted from SWEBench and as a result risks models overfitting to this specific dataset. Currently, this does not seem to be a major issue, as model performance is low across the board. Even once models can achieve high coverage, there is the significantly harder task of achieving high mutation score (actually catching synthetic bugs introduced into the code under test). These multiple levels of difficulty and numerous tasks help mitigate the risk of a model overfitting to any one task specifically.

Data contamination: There is also a risk of data contamination in the pretraining data of models. To further understand data contamination, we measure perplexity of 10 randomly selected tests in TestGenEval for Llama 3.1 8B and common frequent, and non recent code from GitHub with similar lengths. We find that the perplexity of this common GitHub code is lower than the 10 tests from TestGenEval (1.6 vs 2.0), indicating that data is unlikely to be contaminated. This is further supported by the low performance of all models on TestGenEval across the board.

Compute cost of mutation score: One other limitation is the compute cost of computing mutation score. Each synthetic bug we introduce to the code under test, requires an additional test suite execution. However, our results show that coverage and mutation score are highly correlated. Setting a timeout of one hour per mutation testing run, we only timeout on 20% of files, and get an average uncertainty of 1.1%. We provide an option to run TestGenEval without mutation, enabling those who lack compute to still benefit from TestGenEval.

### 6 Related Work

Test Generation Benchmarks: Recently, there has been more effort to evaluate language models software testing capabilities, however these benchmarks typically still consist of small, self-contained projects. Common code generation benchmarks such as HumanEvalFix Chen et al. (), MBPP Plus Austin et al. () and APPS Hendrycks et al. () can easily be adapted to test generation tasks. TestEval Wang et al. () measures test generation capabilities for LeetCode problems of varying difficulties. Bhatia et al. () measure test generation performance for modular code without external dependencies. While these benchmarks provide important execution metrics, the small size of these problems and solutions does not mirror realistic test generation.

There are also repository level test completion benchmarks, however these benchmarks often lack realistic execution metrics. TeCo Nie et al. () and ConTest Villmow et al. () provides a benchmark for completing unit tests, but only measures lexical metrics such as BLEU Papineni et al. () and ROUGE scores Lin (). ATLAS Watson et al. () and TOGA Dinella et al. () provide large scale benchmarks but for completing assertions rather than generating entire tests. SWT-Bench Mündler et al. () provides a benchmark for test method generation targeted as bug fixing PRs. Their task is also adjacent but measuring a specialized part of software testing (generating tests that fail on code prior to PR and pass after PR).

More recently, there has been efforts to create executable code benchmarks. SWEBench Jimenez et al. () measures the ability of language models to generate patches for failing PR tests. R2E Jain et al. () provides a collection of 400 repositories that can be executed, however they leverage equivalence test harnesses which look structurally different than human written unit tests. CruxEval Gu et al. () provides a large set of executable code snippets, however measures execution reasoning ability, which is a subset of the test generation task. TestGenEval provides a large scale executable environment to measure test generation and test completion performance.

Large Language Models for Code Generation: Large language models (LLMs) have shown promising results on many software engineering tasks, including software testing Rao et al. (); Black et al. (); Touvron et al. (). Top open source models include Llama 3.1 Dubey et al. (), DeepSeekCoder DeepSeek-AI et al. () and CodeStral (^2^22https://mistral.ai/news/codestral/). Closed source models including GPT-4-o (^3^33https://openai.com/index/hello-gpt-4o/) and Claude 3.5 Sonnet (^4^44https://www.anthropic.com/news/claude-3-5-sonnet) have also shown state of the art performance on a wide variety of code generation benchmarks. We measure performance for top open source models and GPT-4o, the current state of the art closed source model.

### 7 Conclusion

We introduce TestGenEval, the first file-level benchmark for test generation and test completion. We release a lite version consisting of 160 code-test file pairs and a full benchmark comprising 1,210 file pairs from real open-source projects.
Considering real-world settings, we employ coverage and mutation score to evaluate the models in our benchmark, as these metrics are closely related to the real-world quality of test suites.
We perform a comprehensive evaluation of both open and closed source models on TestGenEval. Additionally, we conduct a thorough error analysis, revealing that models struggle with reasoning about execution, frequently making assertion errors and failing to generate tests that run within the time limit.
Overall, we believe TestGenEval provides a complementary dataset to existing test generation datasets, offering a more challenging and larger-scale version of current benchmarks.

### 8 Ethics statement

As models continue to grow in size and test generation capabilities improve, it is important to consider the broader impacts of test generation. While generally test generation is an important task for ensuring high quality software, generated tests can be used to assert that buggy code is correct (as seen by the oracle problem). All generated tests should therefore be checked and approved by developers. Furthermore, automated test generation techniques have the potential to automate quality assurance jobs, which can also potentially lead to negative societal impacts. We recommend practitioners be cognizant of the impacts of test generation work when building and deploying new techniques.

### 9 Reproducibility statement

We took extensive efforts to ensure TestGenEval is reproducible. We release code to run and extend TestGenEval along with all docker images for each project in SWEBench. We release a website to view model generated tests for qualitative analysis. Our code and website is available at [https://figshare.com/s/51171ae97cd21d233d4f](https://figshare.com/s/51171ae97cd21d233d4f). We designed TestGenEval’s code to be easy to extend; all projects have individual docker images to run them, enabling those looking to extend TestGenEval to simply create a new docker container and hook into our existing code.

## Appendix

### Appendix A Benchmark Statistics

This section contains more details on benchmark statistics. Both our TestGenEval and TestGenEvalLite splits consist of 11 repositories shown in Table 4.

**Table 4: Benchmark repository information and datapoint distribution.**
| Repository | URL | # Stars | # Lite | # Full |
| --- | --- | --- | --- | --- |
| astropy/astropy | [https://github.com/astropy/astropy](https://github.com/astropy/astropy) | 4325 | 3 | 45 |
| django/django | [https://github.com/django/django](https://github.com/django/django) | 78287 | 66 | 451 |
| matplotlib/matplotlib | [https://github.com/matplotlib/matplotlib](https://github.com/matplotlib/matplotlib) | 19756 | 7 | 72 |
| mwaskom/seaborn | [https://github.com/mwaskom/seaborn](https://github.com/mwaskom/seaborn) | 12264 | 2 | 12 |
| pallets/flask | [https://github.com/pallets/flask](https://github.com/pallets/flask) | 67203 | 1 | 2 |
| pydata/xarray | [https://github.com/pydata/xarray](https://github.com/pydata/xarray) | 3523 | 3 | 45 |
| pylint-dev/pylint | [https://github.com/pylint-dev/pylint](https://github.com/pylint-dev/pylint) | 5206 | 2 | 17 |
| pytest-dev/pytest | [https://github.com/pytest-dev/pytest](https://github.com/pytest-dev/pytest) | 11719 | 12 | 67 |
| scikit-learn/scikit-learn | [https://github.com/scikit-learn/scikit-learn](https://github.com/scikit-learn/scikit-learn) | 59057 | 20 | 192 |
| sphinx-doc/sphinx | [https://github.com/sphinx-doc/sphinx](https://github.com/sphinx-doc/sphinx) | 6264 | 2 | 70 |
| sympy/sympy | [https://github.com/sympy/sympy](https://github.com/sympy/sympy) | 12657 | 42 | 239 |

#### A.1 TestGenEval

We report detailed statistics for the TestGenEval. These include the coverage of various files, unique generations, correlations between different settings and MMLU.

Figure: (a) TestGenEval code lengths
Refer to caption: extracted/5893052/figures/swebench/code_length_distribution.png

Figure 6(a), Figure 6(b) and Figure 6(c)
show the distribution of code lengths, test lengths, and repositories across TestGenEval. Tests and code files are thousands of tokens long, as the benchmark is large in size. The repository distribution mirrors that of SWEBench and the TestGenEvalLite split repo distribution is also similar.

Figure: (a) TestGenEval code methods
Refer to caption: extracted/5893052/figures/swebench/code_method_dist.png

Figure 7(a), and Figure 7(b)
show the distribution of code methods and test methods. Generally, both code and tests methods are quite extensive across file pairs (average of 57 test methods and 58 code methods per file pair).

Figure: Figure 8: TestGenEval gold test coverage
Refer to caption: extracted/5893052/figures/swebench/coverage_violin_plots.png

Figure 8 provides violin plots of the coverage data of gold tests in TestGenEval. Coverage is generally high across all settings (first test alone adds on average 40% coverage) and by the end of the test suite, file level coverage is approximately 80%.

#### A.2 TestGenEvalLite

We perform the same analysis on the TestGenEvalLite split of our benchmark. Many of the distributions and trends observed in TestGenEval hold here too.

Figure: (a) TestGenEvalLite code lengths
Refer to caption: extracted/5893052/figures/swebenchlite/code_length_distribution.png

Figure 9(a), Figure 9(b), and Figure 9(c) show the distribution of code lengths, test lengths, and repositories in the TestGenEvalLite split. Both the distribution of repositories and lengths of code and test files approximately mirror those present in full TestGenEval. The long lengths even in the TestGenEvalLite split require models that support long context lengths.

Figure: (a) TestGenEvalLite code methods
Refer to caption: extracted/5893052/figures/swebenchlite/code_method_dist.png

Figure 7(a), and Figure 7(b)
show the distribution of code methods and test methods for TestGenEvalLite. Results mirror those of TestGenEval; both code and test files in this file have many methods (average of 58 test methods and 50 code methods for each file pair).

Figure: Figure 11: TestGenEvalLite gold test coverage
Refer to caption: extracted/5893052/figures/swebenchlite/coverage_violin_plots.png

Figure 11 provides violin plots of the coverage data of gold tests in the TestGenEvalLite split. TestGenEvalLite coverage distributions mirror those of TestGenEval.

### Appendix B Model URLs

We used GPT-4o on 08/29/2024. Table 5 shows the model name and model URL for all open source models on HuggingFace.

**Table 5: Model Names and URLs**
| Model Name | Model URL |
| --- | --- |
| CodeLlama (7B) | [https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf](https://huggingface.co/meta-llama/CodeLlama-7b-Instruct-hf) |
| CodeLlama (70B) | [https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf](https://huggingface.co/meta-llama/CodeLlama-70b-Instruct-hf) |
| Llama 3.1 (8B) | [https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) |
| Llama 3.1 (70B) | [https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct) |
| Llama 3.1 (405B) | [https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-405B-Instruct) |
| Codestral (22B) | [https://huggingface.co/mistralai/Codestral-22B-v0.1](https://huggingface.co/mistralai/Codestral-22B-v0.1) |
| DeepSeekCoder-Lite (16B) | [https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct) |
| Gemma-2 (9B) | [https://huggingface.co/google/gemma-2-9b-it](https://huggingface.co/google/gemma-2-9b-it) |
| Gemma-2 (27B) | [https://huggingface.co/google/gemma-2-27b-it](https://huggingface.co/google/gemma-2-27b-it) |

### Appendix C Prompts

For each setting we prompt with different contexts. We provide the code under test as part of the prompt for all settings. We also provide the preamble of the test file (imports and test setup) for first test completion, the entire test file minus the last test for last test completion and the entire test file for extra test completion. For our full file test generation setting, we only prompt with the code under test. Since we use a large variety of models, our prompts are slightly different for each class of model.

Figure: Figure 12: Prompt for the full test suite generation setting. For models that use specific format for chat, we format this appropriately to chat format.

Figure: Figure 13: Prompt for the test completion settings (first, last, extra). For models that use specific format for chat, we format this appropriately to chat format.

### Appendix D Experiment Setup

We outline the metrics used for evaluation for each of our tasks.

#### D.1 Models

We evaluate a selection of models on TestGenEval to better understand how models of different sizes and families. We include a mixture of small (less than 9B params), medium (between 9B and 27B params), large (approximately 70B params) and flagship (greater than 70B params) models. This includes open source models (CodeLlama, Llama 3, DeepSeekCoder 2, Codestral, and Gemma 2) of varying sizes and GPT-4o, a state of the art closed source model.

We choose the Llama and Gemma families of models to understand the effects of size on model performance in conjunction with their high scores on code benchmarks such as HumanEval. We also include DeepSeekCoder, CodeLlama and Codestral due to their code specialization. Finally, we include GPT-4o due to its state of the art performance on numerous code generation benchmarks.

#### D.2 Metrics

We report metrics for both our test generation (Section D.2.1) and test completion (Section D.2.2) tasks.

##### D.2.1 Test Generation Metrics

All pass@1: All pass@1 measures if the generated test suite passes when run on the code under test. This penalizes more verbose models, as the likelihood of an error increases with each additional test generated in a test suite.

Pass@1: Pass@1 measures if any test in the generated test suite passes when run on the code under test. We include this to not penalize models that generate longer test suites.

Coverage: Coverage measures the proportion of lines in the file under test executed by the test suite. Ideally, a high quality test suite should execute a large percentage of the lines in the code under test.

Coverage@pass: Coverage@pass measures the coverage of only the passing tests. Models with high coverage@pass may not successfully generate tests for many of the problems in TestGenEval, but for the ones they do, they obtain high coverage.

Mutation score: The main limitation of coverage and pass@1 is that they can be potentially gamed (a model can invoke all methods in the class under test without testing anything and achieve 100% coverage and 100% pass@1). To compute mutation score we inject synthetic bugs into the code under test. We choose which bugs we inject in accordance with best practices Derezinska & Halas (); Offutt et al. (). We then measure the percentage of bugs detected by the test suite (should pass on the original code and fail on the buggy code). Unlike other metrics, mutation score is much harder to game, however it is compute costly, as we have to execute the entire test suite for each bug. We rank by coverage, as models still have relatively low coverage across the board and it is more compute efficient for the community to run (we allow for TestGenEval to be run omitting mutation score).

Mutation score@pass: Mutation score@pass measures the the mutation score of only passing tests. Mutation score@pass is similar to coverage@pass, measuring how the quality of generated tests, rather than their coverage of all problems in TestGenEval.

##### D.2.2 Test Completion Metrics

Pass@k: Pass@k measures if any of k test generated pass when added to the existing test suite for the code under test. We rank by this metric, as coverage improvement is near 0 for 2/3 settings for TestGenEval.

Avg pass@k: Average pass@k measures the average proportion of k generated tests that pass. This provides an idea of how many erroneous tests a model will generate before generating a correct test.

Coverage improvement: Coverage improvement measures the change in line coverage when adding the generated test. Ideally, newly added tests should improve overall code coverage. We choose not to report mutation score improvement here, due to computational cost and already near 0 coverage improvement of generated tests.

Coverage improvement@pass: Coverage improvement@pass measures the change in line coverage for only problems where generated tests pass the test suite. High coverage improvement@pass indicates that a model may be good at generating high coverage test completion, but could still struggle with generating passing test completions for all problems.

### Appendix E Results

We report our full set of results on both TestGenEval and TestGenEvalLite (results across both splits are similar). We also report complete test completion results for the extra setting and add the average pass@k metric.

#### E.1 TestGenEval

We report performance on both the test generation and test completion tasks for TestGenEval.

##### E.1.1 Test Generation

**Table 6: Full test suite generation. All results shown for temperature=0.2. We report all pass@1, pass@1, coverage, coverage of only passing tests, mutation score and mutation score of only passing tests.**
| Model | All@1 | Pass@1 | Cov | Cov@Pass | Mut | Mut@Pass |
| --- | --- | --- | --- | --- | --- | --- |
| Small Models |  |  |  |  |  |  |
| CodeLlama 7B | 3.2% | 4.1% | 1.2% | 30.3% | 0.5% | 11.0% |
| Gemma 9B | 3.5% | 42.1% | 20.2% | 47.9% | 9.0% | 21.5% |
| Llama 3.1 8B | 3.1% | 30.5% | 14.1% | 46.3% | 6.8% | 22.4% |
| Medium Models |  |  |  |  |  |  |
| DeepSeekCoder 16B | 23.0% | 63.7% | 28.2% | 44.3% | 12.1% | 19.1% |
| Gemma 27B | 7.4% | 57.7% | 30.1% | 52.3% | 14.6% | 25.5% |
| Codestral 22B | 26.8% | 72.7% | 33.0% | 45.4% | 14.2% | 19.7% |
| Large Models |  |  |  |  |  |  |
| CodeLlama 70B | 14.0% | 22.7% | 7.0% | 32.4% | 2.5% | 11.4% |
| Llama 3.1 70B | 7.8% | 60.7% | 30.6% | 50.4% | 15.4% | 25.6% |
| Flagship Models |  |  |  |  |  |  |
| GPT-4o | 7.5% | 64.0% | 35.2% | 54.9% | 18.8% | 29.4% |
| Llama 3.1 405B | 17.7% | 73.1% | 35.0% | 47.9% | 16.4% | 22.5% |

Table 6 shows full test suite performance for TestGenEval. Smaller models struggle to generate any tests that pass on complex code files present in TestGenEval. Codestral 22B can generate a passing test in 72.7% of all problems, however still has lower coverage and mutation score than GPT-4o. This indicates despite generating tests for more problems, the tests generated by Codestral 22B tend to be lower quality than those generated by GPT-4 (indicated by a approximately 10% lower coverage of passing tests).

##### E.1.2 Test Completion

**Table 7: Pass rates for different models in first, last, and extra completion settings. Pass@1 is for temperature=0.2, Pass@5 is for temperature=0.8.**
|  | First | Last | Extra |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Pass@1 | Pass@5 | Pass@1 | Pass@5 | Pass@1 | Pass@5 |
| Small Models |  |  |  |  |  |  |
| CodeLlama 7B | 4.2% | 19.8% | 6.9% | 24.0% | 5.2% | 22.9% |
| Gemma 9B | 8.4% | 18.0% | 21.4% | 46.4% | 18.9% | 46.7% |
| Llama 3.1 8B | 14.4% | 33.3% | 32.0% | 54.3% | 31.8% | 53.8% |
| Medium Models |  |  |  |  |  |  |
| DeepSeekCoder 16B | 18.6% | 47.0% | 17.0% | 62.8% | 15.5% | 61.7% |
| Gemma 27B | 12.7% | 33.7% | 32.2% | 62.2% | 31.7% | 66.8% |
| Codestral 22B | 38.3% | 61.7% | 50.4% | 74.3% | 48.3% | 75.7% |
| Large Models |  |  |  |  |  |  |
| CodeLlama 70B | 0.5% | 30.2% | 0.9% | 50.7% | 0.5% | 52.8% |
| Llama 3.1 70B | 19.3% | 46.4% | 35.0% | 61.9% | 36.4% | 61.2% |
| Flagship Models |  |  |  |  |  |  |
| GPT-4o | 31.9% | 63.5% | 32.6% | 66.6% | 30.4% | 63.5% |
| Llama 3.1 405B | 32.1% | 57.7% | 42.6% | 72.3% | 42.4% | 74.5% |

Table 7 shows the pass rates across first, last, and extra test completion settings. Extra test completion setting results mirror those of last test completion, with Codestral 22B outperforming existing models.

**Table 8: Coverage improvements, including coverage improvements for passing tests, and average pass@5 for first, last, and extra completion settings. Models struggle to improve coverage, and in general report substantially worse average pass@5 than pass@5.**
|  | First | Last | Extra |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | +Cov | +Cov@P | Avg@5 | +Cov | +Cov@P | Avg@5 | +Cov | +Cov@P | Avg@5 |
| Small Models |  |  |  |  |  |  |  |  |  |
| CodeLlama 7B | 6.7% | 33.9% | 4.7% | 0.0% | 0.1% | 5.7% | 0.0% | 0.1% | 5.3% |
| Gemma 9B | 6.7% | 37.3% | 8.2% | 0.1% | 0.2% | 17.4% | 0.1% | 0.2% | 16.5% |
| Llama 3.1 8B | 12.8% | 38.5% | 10.5% | 0.1% | 0.2% | 23.1% | 0.0% | 0.1% | 23.5% |
| Medium Models |  |  |  |  |  |  |  |  |  |
| DeepSeekCoder 16B | 19.0% | 40.4% | 16.4% | 0.2% | 0.3% | 22.2% | 0.1% | 0.1% | 21.9% |
| Gemma 27B | 13.6% | 40.2% | 13.2% | 0.1% | 0.2% | 28.0% | 0.1% | 0.1% | 29.0% |
| Codestral 22B | 24.0% | 39.1% | 31.2% | 0.4% | 0.5% | 43.2% | 0.2% | 0.2% | 43.5% |
| Large Models |  |  |  |  |  |  |  |  |  |
| CodeLlama 70B | 11.2% | 37.5% | 7.5% | 0.0% | 0.0% | 14.2% | 0.0% | 0.0% | 15.4% |
| Llama 3.1 70B | 18.7% | 40.5% | 18.4% | 0.5% | 0.7% | 29.5% | 0.1% | 0.2% | 29.0% |
| Flagship Models |  |  |  |  |  |  |  |  |  |
| GPT-4o | 26.9% | 42.4% | 32.4% | 0.5% | 0.7% | 31.5% | 0.2% | 0.3% | 28.8% |
| Llama 3.1 405B | 21.6% | 37.5% | 31.3% | 0.3% | 0.4% | 40.9% | 0.1% | 0.2% | 41.4% |

Table 8 shows the average pass@5 rates along with coverage improvements for all settings. Coverage improvement for last and extra settings is near 0; it is hard to improve coverage for a near complete test suite. Average pass@5 is interestingly lower for GPT-4o, despite higher coverage, meaning that it would take more attempts to generate a passing test completion, but the quality of the completion is better.

Figure: (a) First setting
Refer to caption: extracted/5893052/figures/swebench/unique_gens_first.png

Figure 14(a), Figure 14(b), Figure 14(c) show the number of unique generations for each model across temperatures 0.2 and 0.8. Of all models, CodeLlama models are the least diverse, with not a big difference between 0.2 and 0.8 temperature, while Llama 3.1 generates far more unique solutions.

#### E.2 TestGenEvalLite

We also report our full set of results on our TestGenEvalLite split. We find that results generally mirror those of TestGenEval.

##### E.2.1 Test Generation

**Table 9: Full test suite generation. All results shown for temperature=0.2**
| Model | All@1 | Pass@1 | Cov | Cov@Pass | Mut | Mut@Pass |
| --- | --- | --- | --- | --- | --- | --- |
| Small Models |  |  |  |  |  |  |
| CodeLlama 7B | 3.8% | 4.4% | 2.1% | 47.5% | 0.8% | 18.3% |
| Gemma 9B | 7.5% | 45.0% | 19.6% | 43.6% | 8.1% | 17.9% |
| Llama 3.1 8B | 5.0% | 29.4% | 12.1% | 41.3% | 5.1% | 17.9% |
| Medium Models |  |  |  |  |  |  |
| DeepSeekCoder 16B | 24.4% | 64.4% | 27.3% | 42.5% | 11.6% | 18.0% |
| Gemma 27B | 11.9% | 58.8% | 28.4% | 48.3% | 12.8% | 22.1% |
| Codestral 22B | 30.0% | 71.9% | 31.7% | 44.1% | 12.3% | 17.3% |
| Large Models |  |  |  |  |  |  |
| CodeLlama 70B | 18.8% | 25.6% | 8.0% | 32.8% | 1.8% | 7.7% |
| Llama 3.1 70B | 10.0% | 60.6% | 30.3% | 50.0% | 15.1% | 25.4% |
| Flagship Models |  |  |  |  |  |  |
| GPT-4o | 5.0% | 55.0% | 29.7% | 54.0% | 14.7% | 27.3% |
| Llama 3.1 405B | 18.1% | 74.4% | 34.1% | 45.9% | 14.7% | 19.9% |

Table 9 shows pass@1, all pass@1 coverage and mutation score for small, medium and large models. Similar to TestGenEval, smaller models perform significantly worse than their larger counterparts. All pass@1 penalizes how verbose a model is, thus less verbose models such as Codestral 22B have a much higher all pass@1 than verbose models such as GPT-4o, which generate many tests.

Coverage and mutation score remain low across all models. Similar to TestGenEval, GPT-4o and Llama 3.1 405B performs highly. Mutation score is even lower than coverage, implying that models are not able to catch all induced bugs.

##### E.2.2 Test Completion

**Table 10: Pass rates for different models in first, last, and extra completion settings. Pass@1 is for temperature=0.2, Pass@5 is for temperature=0.8.**
|  | First | Last | Extra |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | Pass@1 | Pass@5 | Pass@1 | Pass@5 | Pass@1 | Pass@5 |
| Small Models |  |  |  |  |  |  |
| CodeLlama 7B | 6.9% | 25.0% | 7.5% | 22.5% | 6.9% | 21.9% |
| Gemma 9B | 6.9% | 18.1% | 18.8% | 45.0% | 25.0% | 45.6% |
| Llama 3.1 8B | 14.4% | 41.2% | 35.0% | 56.2% | 31.9% | 51.2% |
| Medium Models |  |  |  |  |  |  |
| DeepSeekCoder 16B | 18.8% | 48.8% | 20.6% | 60.0% | 13.1% | 60.6% |
| Gemma 27B | 16.2% | 35.6% | 31.9% | 63.7% | 30.6% | 63.1% |
| Codestral 22B | 43.1% | 67.3% | 51.2% | 75.0% | 47.5% | 75.0% |
| Large Models |  |  |  |  |  |  |
| CodeLlama 70B | 1.2% | 26.9% | 0.0% | 61.9% | 0.0% | 56.9% |
| Llama 3.1 70B | 21.9% | 48.1% | 36.2% | 59.4% | 41.2% | 64.4% |
| Flagship Models |  |  |  |  |  |  |
| GPT-4o | 36.2% | 66.9% | 26.9% | 60.6% | 26.2% | 61.3% |
| Llama 3.1 405B | 33.8% | 60.0% | 38.8% | 68.1% | 40.6% | 71.2% |

**Table 11: Coverage improvements, including coverage improvements for passing tests, and average pass@5 for first, last, and extra completion settings. Models struggle to improve coverage, and in general report substantially worse average pass@5 than pass@5.**
|  | First | Last | Extra |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | +Cov | +Cov@P | Avg@5 | +Cov | +Cov@P | Avg@5 | +Cov | +Cov@P | Avg@5 |
| Small Models |  |  |  |  |  |  |  |  |  |
| CodeLlama 7B | 9.3% | 37.3% | 5.8% | 0.0% | 0.1% | 5.0% | 0.0% | 0.0% | 5.3% |
| Gemma 9B | 7.9% | 43.7% | 6.8% | 0.1% | 0.2% | 17.1% | 0.1% | 0.1% | 16.1% |
| Llama 3.1 8B | 16.3% | 39.5% | 10.7% | 0.0% | 0.0% | 26.4% | 0.1% | 0.1% | 23.8% |
| Medium Models |  |  |  |  |  |  |  |  |  |
| DeepSeekCoder 16B | 20.6% | 42.3% | 17.9% | 0.1% | 0.1% | 21.5% | 0.1% | 0.1% | 20.1% |
| Gemma 27B | 15.5% | 43.6% | 14.5% | 0.1% | 0.1% | 29.1% | 0.0% | 0.0% | 27.5% |
| Codestral 22B | 26.4% | 39.4% | 32.5% | 0.2% | 0.3% | 48.0% | 0.1% | 0.1% | 43.8% |
| Large Models |  |  |  |  |  |  |  |  |  |
| CodeLlama 70B | 8.7% | 33.3% | 7.0% | 0.0% | 0.0% | 17.0% | 0.0% | 0.0% | 15.7% |
| Llama 3.1 70B | 18.8% | 39.6% | 16.9% | 0.1% | 0.2% | 29.1% | 0.1% | 0.1% | 32.5% |
| Flagship Models |  |  |  |  |  |  |  |  |  |
| GPT-4o | 28.2% | 42.2% | 34.4% | 0.3% | 0.6% | 28.8% | 0.3% | 0.5% | 28.1% |
| Llama 3.1 405B | 20.9% | 35.2% | 33.4% | 0.2% | 0.4% | 41.3% | 0.1% | 0.2% | 43.9% |

Table 10 shows pass@1, and pass@5 for all settings. Similar to TestGenEval split, model performance generally increases as more of the test file is provided as context (extra pass@5 is higher than both last pass@5 and first pass@5). Table 11 shows the average pass@5 and coverage improvement across all three settings. Similar to TestGenEval, models struggle to add coverage to an existing test suite, performing best at first test generation, where there is no coverage (or minimal coverage from test setup) and thus it is easiest to add new coverage. For the last and extra test generation testing, the model generates nearly 0 new coverage, testing already tested functions, rather than attempting to test new functions. This represents a frontier of test generation (improving coverage of an already high coverage test suite). Similar to TestGenEval, Codestral has a higher average pass@5 than GPT-4o, despite GPT-4o generating tests with higher coverage.

Figure: (a) First setting
Refer to caption: extracted/5893052/figures/swebenchlite/unique_gens_first.png

Figure 15(a), Figure 15(b), Figure 15(c) show the number of unique generations for each model across temperatures 0.2 and 0.8. Similar to the full TestGenEval, CodeLlama generates the fewest unique solutions, while models such as Llama 3.1 8B generate a large number of unique solutions.

### Appendix F Analysis

#### F.1 Correlation with other benchmarks

##### F.1.1 TestGenEval

Figure: (a) Correlation with HumanEval
Refer to caption: extracted/5893052/figures/swebench/humaneval_corr.png

We perform correlation analysis between TestGenEval in our full test generation setting and other benchmarks. We select two benchmarks for comparison: HumanEval (a widely used code generation benchmark) and TestEval (a existing test generation benchmark).

Figure 16(a) shows correlation between TestGenEval and HumanEval. Correlation between HumanEval (a code generation task) and TestGenEval is higher with a correlation coefficient of 0.86. The major outlier here is CodeLlama 7B with a very low TestGenEval and HumanEval score. Another outlier is Gemma 9B, with a lower HumanEval score than expected given its TestGenEval score. The moderate correlation illustrates that there is still information added by adding TestGenEval, but that it does indeed correlate with code generation performance.

Figure 16(b) shows correlation between TestGenEval and TestEval, a smaller scale test generation benchmark. Generally all models tend to perform, well on TestEval, however Gemma 9B performs significantly worse than expected. This is because the instruct model does not follow the prompt provided by TestEval authors, instead generating code snippets. Additionally, there does is minimal differentiation between models, with some models performing highly on TestEval (Llama 3.1 70B and CodeLlama 70B), but significantly worse on TestGenEval.

Figure 16(c) shows correlation between TestGenEval and MMLU, a state of the art natural language processing benchmark. Code generation models such as Codestral and DeepSeekCoder perform significantly worse on MMLU than on TestGenEval. Otherwise, the performance is generally correlated.

##### F.1.2 TestGenEvalLite

Figure: (a) Correlation with HumanEval
Refer to caption: extracted/5893052/figures/swebenchlite/humaneval_corr.png

Figure 17(a), Figure 17(c), and Figure 17(b) display the correlations with HumanEval, MMLU, and TestEval datasets, respectively, in the TestGenEvalLite dataset. Similar to TestGenEval, we find that HumanEval has the highest correlation of any benchmark, however there are some outliers such as GPT-4o and Llama 3.1 8B having a higher HumanEval score. MMLU correlation is less strong, with code models such as Codestral and DeepSeekCoder scoring significantly lower on MMLU than on TestGenEval. Finally, TestEval has the lowest correlation, with Gemma 9B failing to follow the TestEval prompt.

#### F.2 Correlation between settings and models

##### F.2.1 TestGenEval

Figure: (a) Full-first correlation
Refer to caption: extracted/5893052/figures/swebench/full_first_corr.png

Figure 18(a), Figure 18(b), and Figure 18(c) show the correlations between the full dataset and the first, last, and extra subsets, respectively, in the TestGenEval. Our full test generation setting has a relatively high correlation with other settings, however it is not a perfect correlation, with some models performing better on one setting than others.

Figure: (a) First-last correlation
Refer to caption: extracted/5893052/figures/swebench/first_last_corr.png

Figure 19(a), Figure 19(b), and Figure 19(c) show the correlations between the test completion settings. First and last and first and extra test completion settings have a highly positive but not perfect correlation. Interestingly, last test generation and extra test generation have perfect correlation, indicating that models performing well in one setting will in the other setting.

Figure: Figure 20: TestGenEval full model correlation
Refer to caption: extracted/5893052/figures/swebench/full_model_corr.png

Figure: (a) TestGenEval first model correlation
Refer to caption: extracted/5893052/figures/swebench/first_model_corr.png

Figure 20 shows the correlation of all models in their ability to solve the TestGenEval full test generation setting. Models in the same families have the highest correlation (Gemma 9B and Gemma 27B for example, and Llama 3.1 8B and Llama 3.1 70B). Gemma 27B and Codestral 22B also are very correlated in their ability to solve problems in the full test suite generation setting.

Figure 21(a), Figure 21(b), and Figure 21(c) show the correlations of all models in their ability to solve first, last, and extra test completion tasks in TestGenEval. Trends from the model predictions in the full test suite generation continue here, with models in the same family most correlated in their ability to solve tasks. Llama 3.1 405B and Codestral 22B also share some similarities in their first, last and extra test settings problems solved, as well.

##### F.2.2 TestGenEvalLite

Figure: (a) Full-first correlation
Refer to caption: extracted/5893052/figures/swebenchlite/full_first_corr.png

Figure 22(a), Figure 22(b), and Figure 22(c) show the correlations between the full test generation and the first, last, and extra completion settings, respectively, in the TestGenEvalLite split. Similar to TestGenEval, correlation remains high between the full test suite generation and test completion settings. However, there are still cases where a model is better at one setting than others, indicating there is information gained by having all settings.

Figure: (a) First-last correlation
Refer to caption: extracted/5893052/figures/swebenchlite/first_last_corr.png

Figure 23(a), Figure 23(b), and Figure 23(c) show the correlations between the test completion settings. The first test completion setting is highly correlated with both the last and extra test completion settings. Last test completion is nearly perfectly correlated with extra test completion performance. We hypothesize this is because the tasks are very similar, with the only difference being one additional test added to the context.

Figure: Figure 24: TestGenEvalLite full model correlation
Refer to caption: extracted/5893052/figures/swebenchlite/full_model_corr.png

Figure: (a) TestGenEvalLite first model correlation
Refer to caption: extracted/5893052/figures/swebenchlite/first_model_corr.png

Figure 24 shows the correlation the correlation of all models in their ability to solve the TestGenEval full test generation setting. Similar to TestGenEval, models in the same family (Llama 3.1, Gemma) exhibit the highest correlation between different sizes, larger models solve more of problems than smaller models, but still solve similar types of tasks.

Figure 25(a), Figure 25(b), and Figure 25(c) show the correlation of all models in their ability to solve the TestGenEvalLite first, last, and extra test completion settings. Similar to the full test generation, models in the same families are highly correlated in their ability to solve similar problems. Codestral is also more correlated with Llama 3.1 8B than other models for the extra test completion task.

#### F.3 Effect of Sampling on Coverage and Pass@k

Figure: (a) Coverage@k vs k (full test generation)
Refer to caption: extracted/5893052/figures/swebench/k_effect_full.png

Figure 26(a) and Figure 26(b) show how performance in the full test generation and first, last and extra test completion settings changes as more tests are sampled. For full test suite generation, we find that coverage at k gradually increases, with no noticeable plateau in the first 20 generations. The curve is slightly steeper at the beginning, but still looks linear as k approaches 20. Sampling 20 generations leads to approximately a 10% increase in overall coverage (coverage@20 is 23.3%).

For test completion, we find that pass@k is most substantial in the first 5 tests, but slowly increases after. Pass@100 performance is substantially higher than pass@1 and pass@5 performance, indicating models are capable of solving test completion tasks, but with many attempts. The plot curve is steep initially, and then decays in value as more samples are added (first test completion plateaus around 80% pass@k and last and extra test completion plateaus around 90% pass@k).

#### F.4 Effect of Context on Coverage and Pass@5

We measure the effect of context length on both coverage for our full test generation setting and for pass@5 for our test completion setting.

Figure: (a) Coverage by context length (full test generation)
Refer to caption: extracted/5893052/figures/swebench/full_context_len.png

Figure 27(a) and Figure 27(b) show the effect of context length on both our coverage
(full test generation) and on pass@5 (test completion). For full test suite generation, context does not help much; seeing part of a large file with many classes is enough to generate covering tests. Since tests can mock inputs in the partial test file, this partial context is sufficient to generate large test suite. However, for test completion context size helps significantly more. This is because to complete the next test in a file one needs knowledge of both the code under test and the existing test file. Full context ensures that the mapping between existing tests and the code under test is clear, improving performance of completing additional tests.

#### F.5 Quantitative Error Analysis

##### F.5.1 TestGenEval

Figure: Figure 28: Different errors made on TestGenEval by Codestral 22B (the best model on test completion). Despite high performance, Codestral struggles to reason about execution (large number of assertion and timeout errors).
Refer to caption: extracted/5893052/figures/swebench/best_error_types_hist.png

We bucket errors based off the logs from executing tests. Specifically for setting and generation we scrape the logs and extract the Python error class. Figure 28 shows the most frequent errors made by the best performing model for test completion, Codestral 22B: specifically assertion, no assertion, timeout, indentation and name errors. As evidenced by the large number of assertion and timeout errors Codestral 22B struggles to reason about code execution. There are also cases where Codestral 22B generates tests that do not contain an assert and simply invoke the method under test. We filter these out and label such cases as an error.

Figure: (a) Top 5 errors across all models
Refer to caption: extracted/5893052/figures/swebench/all_error_types_hist.png

Figure 29(a), Figure 29(b), and Figure 29(c) show examples of the most common errors made by all models in TestGenEval. We find that all models struggle the most with following the prompt, generating tests that lack assertions. After this error, the most common error types have to do with code execution (assertion, timeout errors), model hallucination (name error) and formatting (indentation error).

##### F.5.2 TestGenEvalLite

Figure: Figure 30: Different errors made on TestGenEvalLite by Codestral 22B. Results are similar to TestGenEval.
Refer to caption: extracted/5893052/figures/swebenchlite/best_error_typeslite_hist.png

Figure 30 shows the most frequent errors made by our best performing model Codestral 22B. Error counts remain the same as TestGenEval.

Figure: (a) Top 5 errors across all models
Refer to caption: extracted/5893052/figures/swebenchlite/all_error_typeslite_hist.png

Figure 31(a), Figure 31(b), and Figure 31(c) show examples of the most common errors made by all models in TestGenEvalLite. Similar to TestGenEval, we find that all models struggle the most with following the prompt, generating tests that lack assertions and making both execution and hallucination errors.

#### F.6 Qualitative Error Analysis

We manually examine errors made by Codestral (the top model for our test completion task) and highlight examples for the most popular errors for TestGenEval. Generally, models struggle with reasoning about execution (high amount of assertion errors) and hallucination (leads to timeout and value errors).

Assertion Errors: A common type of error is the model generating tests with incorrect asserts. This is challenging, as models often miss nuances in code execution and methods under test themselves.

Figure: Listing 10: Codestral first test method completion and code method

This example shows a case where the code under test requires setup (creating directories), but the model fails to add the required setup (either mocking the os methods or creating the required file in test path so that the import is recognized).

Figure: Listing 11: Codestral last test method completion

An example where Codestral asserts that two objects that have identical labels are not equal. This is a case where Codestral misunderstands the code under test, which does not include min_maxima_ratio in the labels.

No Assert Errors: Another common type of error is no assert errors. We filter out generated tests that do not contain asserts.

Figure: Listing 12: Codestral extra test completion (no asserts)

Figure: Listing 13: Codestral first test completion (no asserts)

Timeout Errors: Another common type of error is generating tests that lead to a degenerative state (infinite loops, timeout errors).

Figure: Listing 14: Codestral extra test method completion

This example shows a case where Codestral incorrectly mocks a request handler, by putting self.server = self.MockServer() in the class MockServer rather than in a __init__ method in MockRequestHandler. The cleanup_headers method requires that the headers are well formed to properly close the connection, and thus never actually closes the connection.

Figure: Listing 15: Codestral first test method completion

This shows a similar example with mocking. Because the mock will continuously return a non None object when called by get_traceback_data in a loop (tb = tb.tb_next), the model needs to mock the traceback correctly and ensure that non null is returned.

Value Errors: Another common type of error is generating tests that invoke functions with incorrect arguments (a value error) or missing some precondition to the function.

Figure: Listing 16: Codestral last test method completion

This example shows a case where Codestral misses an implementation detail of localtime in the code under test. Specifically, localtime cannot be applied on a naive datetime, instead a correct test would use the .localize method to convert a datetime object to an object with a timezone.

Figure: Listing 17: Codestral first test method completion

This shows a different example. Specifically, the input ’django.conf’ is not a valid package that can be passed to the get_paths method. The code under test shows a way to get valid paths (invoking the apps.get_app_configs method, but instead of following this, the model hallucinates a non-existant package).

Figure: Listing 18: Codestral first test method completion

This shows a different example. Specifically, the input ’django.conf’ is not a valid package that can be passed to the get_paths method. The code under test shows a way to get valid paths (invoking the apps.get_app_configs method, but instead of following this, the model hallucinates a non-existant package).

#### F.7 Frontiers of TestGenEval

We also show some code snippets on the frontier of TestGenEval. Specifically we show two examples of code snippets that no model can generate tests for and code snippets where only one model can generate tests.

No Solve Cases: Below are two cases where no model is capable of generating tests.

Figure: Listing 19: No solve problem (class with many nested methods)

The above class is an example of a code file that no model is capable of generating tests for. This is because generating tests for such a class requires understanding complex execution (we only show one of many methods for brevity, but all methods involve complex execution sequences of methods). Furthermore, generating a unit test for such a class requires intricate knowledge of the code under test or complex mocking of all intermediate classes.

Figure: Listing 20: No solve problem (dealing with IO/Files)

The above code file is another case where models fail to generate valid tests. The most common issue across models for this class is incorrectly testing the output file, as the method under test writes its output to a file. Checking this output requires intricate knowledge of the content written to the file, or complex mocking of IO operations. Ideally a model generating a unit test for this class would either need increased context, or proper mocking of file writing.

One Solve Cases: Below are two cases where only one model is capable of generating tests (frontier problems for TestGenEval).

Figure: Listing 21: One solve problem (only GPT-4o)

This example highlights a case where in order to write tests for the main class (QuerySet) the model must understand a large number of classes defined in the code under test. GPT-4 successfully generates tests for methods that return these additional classes.

Figure: Listing 22: One solve problem (only GPT-4o)

This example highlights a case where in order to write tests for the main class (QuerySet) the model must understand a large number of classes defined in the code under test. GPT-4 successfully generates tests for methods that return these additional classes.

Figure: Listing 23: One solve problem (only Codestral 22B)

This shows another example of a one solve problem. The challenge here is the subclassing of the CreateModel class. Codestral is able to successfully invoke these subclasses and come up with valid asserts.

### Appendix G Statistical Tests

We run pairwise comparisons for both TestGenEval and TestGenEvalLite splits of our data.

#### G.1 TestGenEval

**Table 12: 95% confidence interval for pass@1 along with noise related statistics for each of TestGenEval settings.**
| Setting | 95% Int. | No Solve | Tau- | Sig-Noise |
| --- | --- | --- | --- | --- |
| Test generation | 3.1% | 4.7% | 6.4% | 2.70 |
| Extra test completion | 3.5% | 12.0% | 11.2% | 1.21 |
| First test completion | 3.7% | 25.9% | 7.8% | 1.87 |
| Last test completion | 4.4% | 10.5% | 10.4% | 1.27 |

Figure: (a) TestGenEval acc by problem
Refer to caption: extracted/5893052/figures/swebench/TestGenEval_full_figplt_histogram_accs.png

Figure 32(a) and Figure 32(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our full test generation setting.

Figure: (a) TestGenEval cov by problem
Refer to caption: extracted/5893052/figures/swebench/TestGenEval_full_coverage_hist.png

Figure: (a) TestGenEval acc by problem
Refer to caption: extracted/5893052/figures/swebench/TestGenEval_extra_figplt_histogram_accs.png

Figure 34(a) and Figure 34(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our extra test completion setting.

Figure: (a) TestGenEval acc by problem
Refer to caption: extracted/5893052/figures/swebench/TestGenEval_last_figplt_histogram_accs.png

Figures Figure 35(a) and Figure 35(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our extra test completion setting.

Figure: (a) TestGenEval acc by problem
Refer to caption: extracted/5893052/figures/swebench/TestGenEval_first_figplt_histogram_accs.png

Figure 36(a) and Figure 36(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our first test completion setting.

**Table 13: Model performance for the full test suite generation setting. Coverage refers to the coverage of generated tests.**
| Model | Coverage | Win Rate | Elo |
| --- | --- | --- | --- |
| GPT-4o | 35.2% | 79.6% | 1280.6 |
| Llama 3.1 405B | 35.0% | 72.8% | 1221.2 |
| Codestral 22B | 33.0% | 65.8% | 1161.2 |
| Llama 3.1 70B | 30.6% | 66.3% | 1168.8 |
| Gemma 27B | 30.1% | 69.8% | 1196.5 |
| DeepSeekCoder 16B | 28.2% | 53.4% | 1086.1 |
| Gemma 9B | 20.2% | 41.2% | 996.7 |
| Llama 3.1 8B | 14.1% | 27.1% | 899.1 |
| CodeLlama 70B | 7.0% | 7.3% | 641.9 |
| CodeLlama 7B | 1.3% | 1.4% | 347.8 |

**Table 14: Model performance for the first test completion setting. Pass@1 refers to if any of the generated tests pass.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 38.3% | 80.4% | 1288.1 |
| Llama 3.1 405B | 32.1% | 73.7% | 1231.1 |
| GPT-4o | 31.9% | 71.8% | 1220.4 |
| Llama 3.1 70B | 19.3% | 52.6% | 1085.9 |
| DeepSeekCoder 16B | 18.6% | 51.2% | 1082.6 |
| Llama 3.1 8B | 14.4% | 41.7% | 1025.1 |
| Gemma 27B | 12.7% | 37.6% | 1003.0 |
| Gemma 9B | 8.4% | 25.9% | 912.0 |
| CodeLlama 7B | 4.2% | 13.9% | 774.6 |
| CodeLlama 70B | 0.5% | 1.6% | 377.2 |

**Table 15: Model performance for the last test completion setting. Pass@1 refers to if any of the generated tests pass.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 50.4% | 79.1% | 1258.6 |
| Llama 3.1 405B | 42.6% | 72.1% | 1195.1 |
| Llama-3.1 70B | 35.0% | 62.1% | 1122.9 |
| GPT-4o | 32.6% | 58.0% | 1103.5 |
| Gemma 27B | 32.2% | 57.3% | 1105.4 |
| Llama 3.1 8B | 32.0% | 57.6% | 1096.5 |
| Gemma 9B | 21.4% | 40.8% | 994.0 |
| DeepSeekCoder 16B | 17.0% | 33.4% | 944.7 |
| CodeLlama 7B | 6.9% | 13.9% | 758.6 |
| CodeLlama 70B | 0.9% | 2.2% | 420.7 |

**Table 16: Model performance for the extra test completion setting. Pass@1 refers to if any of the generated tests pass.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 48.3% | 77.9% | 1262.1 |
| Llama 3.1 405B | 42.4% | 72.9% | 1216.4 |
| Llama 3.1 70B | 36.4% | 65.6% | 1160.7 |
| Llama 3.1 8B | 31.8% | 59.0% | 1121.8 |
| Gemma 27B | 31.7% | 57.8% | 1127.3 |
| GPT-4o | 30.4% | 56.5% | 1111.2 |
| Gemma 9B | 18.9% | 38.2% | 998.5 |
| DeepSeekCoder 16B | 15.5% | 32.2% | 958.2 |
| CodeLlama 7B | 5.2% | 11.6% | 744.5 |
| CodeLlama 70B | 0.5% | 1.0% | 299.2 |

#### G.2 TestGenEvalLite

**Table 17: 95% confidence interval along with noise related statistics for each of TestGenEvalLite settings.**
| Setting | 95% Int. | No Solve | Tau- | Noise |
| --- | --- | --- | --- | --- |
| Test generation | 5.6% | 0.6% | 6.2% | 1.16 |
| Extra test completion | 6.9% | 11.2% | 11.9% | 0.31 |
| First test completion | 5.6% | 22.5% | 6.9% | 0.87 |
| Last test completion | 7.5% | 8.8% | 10.0% | 0.13 |

Figure: (a) TestGenEvalLite acc by problem
Refer to caption: extracted/5893052/figures/swebenchlite/TestGenEvalLite_full_figplt_histogram_accs.png

Figure 37(a) and Figure 37(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our full test generation setting.

Figure: (a) TestGenEvalLite cov by problem
Refer to caption: extracted/5893052/figures/swebenchlite/TestGenEvalLite_full_coverage_hist.png

Figure: (a) TestGenEvalLite acc by problem
Refer to caption: extracted/5893052/figures/swebenchlite/TestGenEvalLite_extra_figplt_histogram_accs.png

Figure 39(a) and Figure 39(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our extra test completion setting.

Figure: (a) TestGenEvalLite acc by problem
Refer to caption: extracted/5893052/figures/swebenchlite/TestGenEvalLite_last_figplt_histogram_accs.png

Figure 40(a) and Figure 40(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our extra test completion setting.

Figure: (a) TestGenEvalLite acc by problem
Refer to caption: extracted/5893052/figures/swebenchlite/TestGenEvalLite_first_figplt_histogram_accs.png

Figure 41(a) and Figure 41(b) show the average model accuracies by problem, along with the min elo required to solve each problem for our first test completion setting.

**Table 18: Model performance for the full test suite generation setting. Coverage refers to the coverage of generated tests.**
| Model | Coverage | Win Rate | Elo |
| --- | --- | --- | --- |
| Llama 3.1 405B | 35.3% | 77.3% | 1245.2 |
| Codestral 22B | 34.9% | 69.1% | 1176.7 |
| GPT-4o | 32.9% | 74.6% | 1223.3 |
| Llama 3.1 70B | 31.3% | 66.8% | 1171.7 |
| Gemma 27B | 29.4% | 64.8% | 1148.5 |
| DeepSeekCoder 16B | 27.6% | 48.6% | 1045.2 |
| Gemma 9B | 20.2% | 38.1% | 968.0 |
| Llama 3.1 8B | 12.4% | 17.1% | 783.9 |
| CodeLlama 70B | 8.0% | 10.9% | 708.8 |
| CodeLlama 7B | 2.1% | 4.1% | 528.6 |

**Table 19: Model performance for the first test completion setting. Pass@1 refers to if any of the generated tests pass.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 43.1% | 81.7% | 1283.1 |
| GPT-4o | 36.2% | 72.7% | 1212.3 |
| Llama 3.1 405B | 33.8% | 71.7% | 1195.2 |
| Llama 3.1 70B | 21.9% | 53.6% | 1080.2 |
| DeepSeekCoder 16B | 19.4% | 48.9% | 1053.7 |
| Gemma 27B | 16.2% | 42.7% | 1012.9 |
| Llama 3.1 8B | 14.4% | 38.3% | 987.5 |
| CodeLlama 7B | 6.9% | 20.2% | 829.0 |
| Gemma 9B | 6.9% | 19.3% | 822.1 |
| CodeLlama 70B | 1.2% | 4.0% | 524.1 |

**Table 20: Model performance for the last test completion setting. Pass@1 refers to if any of the generated tests pass. We omit CodeLlama 70B because it is not able to complete any tests on first generation.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 51.2% | 79.2% | 1575.4 |
| Llama 3.1 405B | 40.0% | 68.0% | 1478.3 |
| Llama 3.1 70B | 37.5% | 64.1% | 1453.6 |
| Llama 3.1 8B | 36.9% | 63.9% | 1453.5 |
| Gemma 27B | 33.1% | 57.6% | 1424.4 |
| GPT-4o | 28.1% | 51.0% | 1373.3 |
| DeepSeekCoder 16B | 20.6% | 39.9% | 1310.2 |
| Gemma 9B | 19.4% | 36.5% | 1277.5 |
| CodeLlama 7B | 7.5% | 16.0% | 1103.5 |
| CodeLlama 70B | 0.0% | 0.0% | -2449.8 |

**Table 21: Model performance for the extra test completion setting. Pass@1 refers to if any of the generated tests pass. We omit CodeLlama 70B because it is not able to complete any tests on first generation.**
| Model | Pass@1 | Win Rate | Elo |
| --- | --- | --- | --- |
| Codestral 22B | 48.1% | 78.1% | 1552.9 |
| Llama 3.1 405B | 42.5% | 72.6% | 1501.1 |
| Llama 3.1 70B | 41.9% | 71.4% | 1494.1 |
| Llama 3.1 8B | 33.8% | 60.3% | 1419.8 |
| Gemma 27B | 31.9% | 56.4% | 1411.4 |
| GPT-4o | 26.2% | 48.8% | 1350.0 |
| Gemma 9B | 25.6% | 47.8% | 1349.1 |
| DeepSeekCoder 16B | 13.8% | 28.6% | 1221.7 |
| CodeLlama 7B | 6.9% | 13.5% | 1058.2 |
| CodeLlama 70B | 0.0% | 0.0% | -2358.4 |

### Appendix H Limitations

Variation due to prompt and temperature Model performance is highly dependant on the prompt and temperature used Gao et al. (); Wei et al. (). A potential limitation of our work is that we primarily focused on 0-shot performance, asking each model to generate the entire test suite or to complete a test given the code under test. To mitigate this we provide our prompt in Appendix C and also provide the model with the correct imports to the code under test to enable prompted models to generate the test suite successfully.

Overfitting to SWEBench repositories: Another potential limitation is that our benchmark is adapted from SWEBench and as a result risks models overfitting to this specific dataset. Currently, this does not seem to be a major issue, as model performance is low across the board. Even once models can achieve high coverage, a significantly harder task of achieving high mutation score (actually catching synthetic bugs introduced into the code under test). These multiple levels of difficulty and numerous tasks help mitigate the risk of a model overfitting to any one task specifically.

Data contamination: There is also a risk of data contamination in the pretraining data of models. To further understand data contamination, we measure perplexity of 10 randomly selected tests in TestGenEval for Llama 3.1 8B and common frequent, and non recent code from GitHub with similar lengths. We find that the perplexity of this common GitHub code is lower than the 10 tests from TestGenEval (1.6 vs 2.0), indicating that data is unlikely to be contaminated. This is further supported by the low performance of all models on TestGenEval across the board.

File level context: TestGenEval currently works at the file level. This means that the context given to the model is the source file, imports and for the test completion setting a partial test file. While in many cases, this context is sufficient sometimes there are cross file dependencies that are not captured, making the context provided to the model limited. However, we believe this is not a major limitation, as models can mock objects which they lack context for. Furthermore, expanding to repository level context is out of context window for most state-of-the-art models.

Additional baselines: One other limitation is the lack of agentic baselines. This might be a promising future work direction in overcoming some of the limitations of the context windows of current models, and could potentially perform better than baselines showed. However, the benchmark is still highly valuable to the community, as the first file level test generation benchmark.

Compute cost of mutation score: One other limitation is the compute cost of computing mutation score. Each synthetic bug we introduce to the code under test, requires an additional test suite execution. However, our results show that coverage and mutation score are highly correlated. Setting a timeout of one hour per mutation testing run, we only timeout on 20% of files, and get an average uncertainty of 1.1%. We provide an option to run TestGenEval without mutation, enabling those who lack compute to still benefit from TestGenEval.

Misc Limitations: Quantitative metrics such as code coverage and mutation score can approximate the quality of generated tests, however they do not perfectly measure the quality of generated tests. For example, a test could be very hard to read, while still achieving high coverage and mutation score. Additionally, all test generation benchmarks assume the code under test is correct. Generated tests may fail on the code under test, while exposing bugs in the code under test (a phenomenon known as the oracle problem). Despite this, current test generation approaches are still useful in catching regressions or bugs introduced in future versions of the code under test.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: - What is the key difference between the newly created benchmark and the evaluation sets used in prior work on ML for test generation/completion (e.g., Rao et al. 2023, Nie et al. 2023, Dinella et al. 2022, Tufano et al. 2020)? Namely, Rao et al. 2023 had a pretty large dataset for pre-training, and evaluated on the task of generating first/last/additional tests. Both Rao et al. 2023 and Nie et al. 2023 also used runtime metrics (compile and pass).

- line 142, "we next extract code test file pairs from the code and test files run in the PR": do you only consider the tests that appeared in PRs in SWEBench? Or all the tests in those repositories?

- line 189, "This setup is in line with current test completion work Rao et al. (2023); Nie et al. (2023).": this may not be true. The "test completion" in Rao et al. 2023 and Nie et al. 2023 was statement-level completion, but yours is method-level completion, which sounds more like the "test method generation" task in Rao et al. 2023.

- Which mutation tool and mutants did you use? And how many mutants do you generate for each example?

- For test suite generation task, consider renaming "Pass @ 1" to "Any Pass @ 1" to reduce confusion.

- line 361, "the most common error is not generating tests with asserts": but will that lead to a runtime error, and if not, how do you detect them? Also, do you count the generated tests without asserts as pass or not pass?
soundness: 3
strengths: + Included the measurement of code coverage and mutation score as part of the evaluation metrics, which are important metrics for measuring the quality of generated tests.

+ Conducted a comprehensive set of experiments using LLMs of different sizes and different configurations.

+ Performed an interesting analysis on the current LLMs' performance on TestGenEval, including the correlation with existing benchmarks and common error types.

+ Paper is well-written and easy to follow.
summary: This paper introduces a new LLM for test generation benchmark, TestGenEval. It builds on the same repositories as SWEBench, but focuses on extracting the test files and code-test pairs. TestGenEval studies two tasks: test suite generation (generating an entire test suite given a codebase) and test completion (generating test methods to a partial test suite). The generated tests are not only compared against human-written tests based on similarity, but also based on code coverage and mutation score, which are more important metrics for test quality. This paper benchmarked several recent LLMs across different sizes on TestGenEval, and found that generating and completing test suites are challenging tasks for current LLMs.
weaknesses: - Not sure if it is the best to include only the tests appeared in the PRs used by SWEBench; this may miss the other tests that did not appear in those PRs (i.e., don't have recent bug-fixing changes), making the data collection process biased.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why is the massive Llama 405B selected in the evaluation rather than other better-performing models like Claude and Gemini?
- What effort does the benchmark made to ensure the test matches the conventional definition of a unit test?
- Why isn't any agents/tools evaluated in the evaluation?
soundness: 2
strengths: - The introduction of mutation score improves on previous benchmarks and ensures a more proper benchmark
- The work introduces the most comprehensive benchmark dedicated to test generation so far
- The work evaluated on a large set of models, even on a very expensive 405B Llama
- Extensive quantitative/correlation analysis, contamination check, and also detailed qualitative analysis, which shows the good quality of the dataset
summary: This paper constructs TestGenEval, a comprehensive and proper benchmark for testing unit test generation abilities of existing LLMs. The benchmark unprecedentedly incorporates mutation scores into their evaluation metric. It also considers multiple generation scenarios including full, first, last, and extra test generation. 10 models including a massive Llama 405B is evaluated on the benchmark and detailed results are given in the main paper and the appendix. 
This paper is leaning towards the rejection side since 1. the author did not make effort to evaluate if the generated test matches the definition of a unit test and merely add in mutation score to evaluate further the test effectiveness, and 2. the model selected for evaluation is questionable.
weaknesses: ### Major
- The evaluation runs the 405B Llama model, which is quite impressive. However, it is unclear why is Llama model evaluated but not other closed-source ones (which are much easier to run comparing to Llama 405B) ones like Claude and Gemini.
- The paper acknowledges that test generated by the LLM can be very different from what human might write. However, there is no procedure in the paper to guarantee that the model generates actually “unit tests” as deemed by human developers (except for the prompt). This means that the model can just generate one big complicated test case for all of the generation scenarios considered (full, first, last, extra), which is against the purpose of unit tests in the first place.
- The task of test generation is generally more solvable by agents or by providing example usages. However, the paper only evaluates on vanilla models and didn’t evaluate even the simple case of adding dependencies into the prompt related to the task.

### Misc
- Citations are not wrapped in brackets (Like Jain et al. when discussing R2E), which is mildly annoying

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. I would like to know what is the cost involved in executing TestGenEval? Including length of prompt, required GPU memory, running time and more cost analysis?

2. How do the authors ensure the quality of the dataset, including but not limited to possible leakage of data and semantic correctness of the given code?
soundness: 3
strengths: 1. Continuous Improvement of Test Suites: TESTGENEVAL supports not only the generation of test suites from scratch but also the completion of existing test suites, which is particularly important for test optimization in continuous integration and continuous deployment (CI/CD) processes.

2. Evaluating and Comparing Different Models: TESTGENEVAL provides a standardized environment to evaluate and compare various code generation models, including both open-source and proprietary models, helping researchers and practitioners understand the performance of different models in real-world software testing tasks.

3. Test Generation for Real-World Use Cases: TESTGENEVAL is built based on real-world projects, meaning it is closer to actual development testing needs rather than being limited to academic research or toy problems.
summary: The theme of this paper is the introduction and evaluation of a large-scale benchmark named TESTGENEVAL, which is designed to measure the performance of test generation and test completion. 

TESTGENEVAL is constructed based on the SWEBench dataset and includes 68,647 test cases from 1,210 code and test file pairs across 11 well-maintained Python repositories. 

This benchmark aims to fill the gap in existing code generation language model (LLM) benchmarks that focus less on software testing, despite the strong correlation between well-tested software and effective bug detection. 

TESTGENEVAL covers tasks such as initial test authoring, test suite completion, and code coverage improvement, aiming to comprehensively assess test generation performance. 

The paper also evaluates several popular models with parameter sizes ranging from 7B to 405B and provides a detailed analysis of TESTGENEVAL's contribution to the evaluation of test generation performance.
weaknesses: 1. Dependence on Prompts and Temperature Settings:
Model performance is highly dependent on the prompts and temperature parameters used. The paper focuses mainly on 0-shot performance, asking each model to generate an entire test suite or complete a test given the code under test, which may limit the model's performance.

2. Risk of Data Contamination:
There is a risk of data contamination in the pre-training data of models. Although the paper suggests that data contamination is unlikely by comparing the perplexity of tests in TESTGENEVAL with common GitHub code of similar length, it remains a potential issue.
In addition, since TESTGENEVAL is adapted from the SWEBench dataset, there is a risk of models overfitting to this specific dataset. 

3. Computational Cost:
The computational cost of calculating mutation scores is high. Each synthetic bug introduced to the code under test requires an additional test suite execution.

4. Assumptions of Test Generation:
All test generation benchmarks assume that the code under test is correct. Generated tests may fail on the code under test while exposing bugs in the code under test (a phenomenon known as the oracle problem).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Pleased address the concerns in **Weakness**.
soundness: 3
strengths: * A standard, real-world project-based benchmark for test generation is crucial and useful for advancing software testing research.
* TestGenEval is the first benchmark to incorporate mutation score as a metric, a commonly used measure in software engineering to assess the robustness of test cases.
* It offers Docker images instrumented with coverage and mutation score, ensuring the reusability of the artifacts.
* TestGenEval is also the first benchmark to evaluate file-level test completion, which aligns well with the real-world software development workflows.
* The authors perform a comprehensive quantitative analysis of the results, demonstrating the correlations and effects of various components within the benchmark.
summary: This paper introduces TestGenEval, 
a novel benchmark designed to evaluate neural test generation models on real-world Python projects. 
TestGenEval is built upon 11 popular Python repositories selected from the patch generation benchmark SWE-Bench.
It assesses neural test generation from two key perspectives: test generation from scratch and test completion based on some existing test functions. 
Both aspects, particularly test completion, are valuable for real-world software engineering applications, such as IDE plugins. 
The authors evaluate several popular LLMs on TestGenEval, 
pointing out that even the best-performing model GPT-4o struggles with achieving high code coverage.
weaknesses: * This paper lacks details on mutation score calculation.
  The authors do not provide specifics on how mutation score is measured in the benchmark. 
  In traditional software engineering, mutation testing is creating mutants of the code under test to evaluate the robustness of the test cases in detecting these modifications. 
  However, the paper does not clarify the process used to construct program mutants for test generation, 
  which is a non-trivial task that requires further clarification.


* The authors claim that real-world unit testing involves reasoning over complex *files*, 
  generating tests for a given *class under test*. 
  The authors also point out that the other related benchmark only measure the individual test method rather than an entire test suite is a weakness. 
  This claim is not well supported with reference nor evidence, 
  and is not commonly adopted by the software engineering community. 
  Unit testing aims to validate the correctness of the smallest building block (and hence, an "unit") in a complex software project.
  In modern software engineering, the "units" of software are often functions in the codebase.
  If these functions are tested/verified to be correct, then the composition should be correct without further testing.
  Therefore, the best practices often suggest to write functions that are independent and de-coupled.
  The more recent and popular programming languages, for example Rust, often abandon class and favor function composition
  over inheritance.
  Are these languages not related to "real-world unit testing"?

* The authors claim that real-world unit testing requires reasoning over complex *files* and generating tests for a given *class under test*. 
  They argue that other benchmarks' focus on individual test methods,
  rather than entire test suites, is a weakness. 
  However, this claim is not well-supported by references or evidence and is not commonly adopted by the software engineering community.

  Unit testing aims to validate the smallest building blocks of software, or so-called "units," 
  which are often functions rather than entire classes. 
  In modern software engineering, if individual functions are tested and verified to be correct, 
  the composition of them is expected to be correct without additional testing. 
  Consequently, best practices emphasize writing independent and de-coupled functions.

  Furthermore, more recent programming languages, such as Rust, 
  favor function composition over inheritance, reflecting a shift away from class-based design. 
  This raises the question of whether these languages fall outside the scope of "real-world unit testing" as claimed by the authors, 
  despite their increasing relevance in software development.

* The lack of quantitative analysis results in the main text is a significant weakness. 
  While the authors conduct five different quantitative analyses, all results are placed in the Appendix. 
  If an analysis is mentioned in the main text, it would be more effective to present the corresponding results alongside the discussion. 
  For space consideration, the authors could include only the most important analyses and results in the main text, 
  while moving less critical details to the Appendix. 
  This approach should also be applied to qualitative analyses for better readability and impact.


* Minor writing issue: the authors cite all references without using parentheses, regardless of context. 
  In some cases, using `\citep` instead of `\cite` would enhance the reading experience by making the citations more contextually appropriate. 
  Adjusting this would improve the flow and clarity of the text.

Meta-review: This paper introduces a new benchmark to support the research on code generation using large language models (LLMs) for real-world software engineering applications. Specifically, the benchmark covers tasks including producing initial test cases, test suite completion, and code coverage improvement, with the goal of comprehensive assessment of test generation performance. The paper evaluates multiple LLMs on this benchmark and finds that even the best-performing models struggles with achieving high code coverage.

The reviewers' were generally positive about the paper, but also raised a number of questions. The author rebuttal answered most questions satisfactorily. One reviewer did not respond to the rebuttal and the corresponding response looks good to me.

Therefore, I recommend accepting the paper and strongly encourage the authors' to incorporate all the discussion in the camera copy to further improve the paper.

Decision: Accept (Poster)