TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
Authors: Kush Jain
Gabriel Synnaeve
Baptiste Roziere
Venue: ICLR 2025 Poster (2025)
Keywords: test generation
software engineering
language models
Primary area: datasets and benchmarks
TL;DR: a new benchmark for measuring test generation ability on large scale projects

Abstract: Code generation models can help improve many common software tasks ranging from code completion to defect prediction. Most of the existing benchmarks for code generation LLMs focus on code authoring or code completion. Surprisingly, there has been far less effort dedicated to benchmarking software testing, despite the strong correlation between well-tested software and effective bug detection. To address this gap, we create and release TestGenEval, a large-scale benchmark to measure test generation performance. Based on SWEBench, TestGenEval comprises 68,647 tests from 1,210 code and test file pairs across 11 well-maintained Python repositories. It covers initial tests authoring, test suite completion, and code coverage improvements. Test authoring simulates the process of a developer writing a test suite from scratch, while test completion mimics the scenario where a developer aims to improve the coverage of an existing test suite. We evaluate several popular models, with sizes ranging from 7B to 405B parameters. Our detailed analysis highlights TestGenEval's contribution to a comprehensive evaluation of test generation performance. In particular, models struggle to generate high-coverage test suites, with the best model, GPT-4o, achieving an average coverage of only 35.2\%. This is primarily due to models struggling to reason about execution, and their frequent assertion errors when addressing complex code paths.

Published as a conference paper at ICLR 2025
TESTGENEVAL: A REAL WORLD UNIT TEST GENER- ATION AND TEST COMPLETION BENCHMARK
Kush Jain Gabriel Synnaeve 2, Baptiste Rozière²
1 Carnegie Mellon University, FAIR, Meta Al aedjainj@andree cmu. edu
ABSTRACT
Code generation models can help improve many common software tasks rang- ing from code completion to defect prediction. Most of the cxisting benchmarks for code generation LLMs focus on code authoring or code completion. Surpris- ingly, there has been far less effort dedicated benchmarking software testing, despite the strong correlation between well-tested software and effective bug de- tection. To address this gap, create and release TESTGENEVAL, large-scale benchmark to measure test generation performance. Based on SWEBench, TEST- GENEVAL comprises 68,647 tests from 1,210 code and test file pairs across 11 well- maintanedd Python repositories. It covers initial tests authoring, test suite completion, and code coverage improvements. Test authoring simulates the pro- cess of a developer writing a test suite from scratch, while test completion mimics the scenario where developer aims to improve the coverage of an existing test suite. We evaluate several popular models, with sizes ranging from 7B to 405B parameters. Our detailed analysis highlights TESTGENEVAL's contribution to comprehensive evaluation of test generation performance. In particular, mod- els struggle to generate high-coverage test suites, with the best model, GPT-4o, achieving average coverage of only 35.2%. This is primarily due to models struggling to reason about execution, and their frequent assertion errors when ad- dressing complex code paths.
INTRODUCTION
Software testing is critical component of the software development process. A high-quality test suite can be instrumental finding inconsistencies between a system's 's specifications and its imple- mentation. Test suites ideally execute all code paths (high code coverage), and catch regressions in
the code under test that a developer might introduce (high mutation score) (Fraser & Arcuri [013] Veloso & Horal 2022). However, writing high quality tests can be tmee-cosumiig (Beller al. 2015a b) and is often either partially or entirely neglected.
As result there has been extensive research automated test generation, including both classi- cal (Fraser & Arcuri 2011
Brandt & Zaidman 2022) Baldoni et al., 2018) and neural-based meth-
ods (Dinella et al.) 2022 Watson al.] 2020 Touvron al. 2023b) OpenAl (023). However, despite this growing area of research, unit test generation benchmarks remain limited in size and scope (Chen et al. 202T Bhatia et al., 2024). While existing benchmarks capture test generation abilities on simple, typically self-contained programs, there is an absence of large scale test gen- eration benchmarks corresponding real world use cases. Other related benchmarks (Jain al. 2024) report performance on adjacent tasks such generating equivalence tests rather than standard unit tests, which also differs from the real world use case. Current benchmarks report pass @ k, with few reporting code coverage and none reporting mutation score, despite mutation score being most correlated with real fault detection (Just et al. 2014 Papadakis et al. 2018).
Existing benchmarks also do not measure test completion capabilities, despite many code comple- tion benchmarks existing (Liu et al., 2023] Zhuo et al., (2024). Test completion can be used to add tests already existing unit test file and improve overall coverage. This is important for IDE auto-completion features, where given a part of test file an the code under test, the goal is to add
more tests. Test completion is also measured by many state of the art software testing models (Rao

---

Published as a conference paper at ICLR 2025
GitHub code
LLM generated tests
Execute and evaiuaate LLM generated tests
Coverage
Mulation score
clase testclassiniitteet. Testtiaae:
reveras(lat);
class Uttie:
eovert r..ersel122.39
eeveran(lat))
11: def reveraz()st)
12: retern lani:::3] mutated
14:
return letel
Xle
return lenst
Xumarrt 12,3,3]
132,11
Figure 1: An overview of TESTGENEVAL. We start with GitHub code file and generate test suite with an LLM. Then we execute the generated test suite and measure the proportion of lines in the code file that are executed (code coverage). We also inject synthetic bugs into the code and measure the proportion of synthetic bugs detected by the generated test suite (mutation score).
al. 2023 Nie al.
2023 Dinella al. test completion doesn't I exist.
2022 Tufano al. 2020), yet benchmark that measures
Motivated by this, we introduce TESTGENEVAL with two tasks full file unit test generation and 2) test completion (see Section 2.2 for more details). Our benchmark consists of real world projects, with each source file containing an average 1,157 lines of code (LOC) and each test file containing an average of 943 LOC. TESTGENEVAL consists of 68,647 tests from 1,210 unique code-tests file pairs. For fast iteration in Low-compute settings, we also provide a smaller version of the benchmark TEST- GENEVALLITE, which approximates all the metrics computed in TESTGENEVAL (see Appendix for more details). FESTGRNEVALLLLTE includes 160 code-tests file pairs, file unit test generation, and test completion tasks. It sampled to be representative of the full TESTGENEVAL: the repos- itories and other statistics are similar in TESTGENEVVLLLTE and TESTGENEVAL (see Appendix for more details and Appendix G for statistical significance tests).
F
We find that models struggle to generate high quality test suites (Section The best performing model- GPT-4o 1aa--sssssst an average coverage of 35.2% and mutation score of 18.8%. Generating tests for large scale projects is significantly harder than generating tests for self-contained prob- lems; this is reflected by significantly lower scores compared existing benchmarks such as TestE- val (Wang et al.) 2024). where top models achieve nearly 100% line coverage. Test completion (Section 3.2) is significantly easier than test generation, reflected by the high pass@5 rates of the best performing models. However, models struggle to add coverage to a complete test suite, with top models adding less than 1% coverage when generating the last test for an existing file.
We perform extensive quantitative and qualitative analysis of all results (Section 4.. We measure correlation between TESTGENEVAL and other popular benchmarks. the correlation between the test generation and test completion tasks, and the correlation between models for each task. We also perform an analysis of errors, and measure the effects of sampling more and context size on TEST- GENEVAL performance (Section 4.I. We also examine cases where TESTGENEVAL can discrimi- nate between highly performing models (Section 4.2). More analysis can be found in Appendix F
We provide all the code for our benchmark at /tttps:/fiisshiee 61171ae97cd21om d2 33 including detailed instructions on how to run our benchmark, and even extend it. We also provide website with all model generations for TestGENEvAl EVLL We hope that this will enable the community to use TESTGENEVAL and further build upon our work.
com/s/ s//
Our contribution are as follows:
We release a benchmark for partial and full test suite generation on realistic set of 1,210 snippets in 11 repositories. We coverage and mutation score metrics evaluate the value of the generated test suites
We evaluate various prominent open and closed-source code generation models benchmark, and show that, for large scale repositories, models struggle to generate high coverage test suites
We release docker images allowing to easily run code from these 11 repositories and eval- uate scores our benchmark
2

---

Published as a conference paper at ICLR 2025
Modify SWEBench images
2
Extract test file pairs
3
Filter paired tests
broken images
houristic
covarage computes all sottings
add coverage, mutation
and imports code under lest
60 seconds
Figure 2: A pipeline describing the creation of TESLL. We start with docker images of the SWEBench dataset and instrument them with coverage and mutation score dependencies. Then we extract code test file pairs by performing a heuristic match on filename. Finally, we filter out tests that don't have coverage on the code under test and run in 60 seconds.
2 TESTGENEVAL
TESTGENEVAL consists of 1,210 code test file pairs from 11 large, well-maintainels repositories (3.523-78,287 stars). We use these file pairs to construct two testing tasks: 1) unit test completion for the first, last and additional tests and 2) full file unit test generation. Our benchmark is easy to run and extend, as we have docker containers for each version of each repository with coverage and mutation testing dependencies installed. For both tasks we use execution based metries. including pass @ 1, pass@ 5 along with code coverage improvement, and mutation score improvement compared to the gold (human written) tests. Code and test files in TEsTGenEVAl. are long in length (on average 782 LOC per code file and 677 LOC test file) and high coverage (median coverage of 60.4%). Detailed statistics TESTGENEVAL can be found in Appendix A
2.1 BENCHMARK CONSTRUCTION
We construct TESTGENEVAL by adapting the SWEBench dataset to software testing tasks. Figure 2 shows the full set of steps to construct TESTGENEVAL. We apply these exact steps to SWEBench- Lite to construct TESTGENEVALLITE.
Modify SWEBench images: We start with the docker images provided by SWEBench We first modify the images that did not build manually, by installing appropriate dependencies and modifying requirements files so every test suite executes. Next, we install both coverage and mutation testing dependencies and modify the test commands to run with coverage instrumentation.
Extract test file pairs: After have the execution environment for each pull request version built, we next extract code test file pairs from the code and test files run in the PR. Specifically, we extract code test file pairs by performing heuristic match on filenames, and filtering out pull requests that do not meet our heuristic. With each file pair we perform program analysis on the test file to extract the first test, last test as context for our test completion settings. We later validate for each file pair that the gold tests cover some part of the code under test.
Filter paired tests: Next, we run code coverage for all the gold test settings (first, last, and extra) to filter out repositories where contexts were extracted incorrectly or partial test files do not run. We also filter out tests that take longer than 60 seconds to run to ensure TESTGENEVAL runs efficiently.
2.2 TASKS
Figure shows an example of both testing tasks. TESTGENEVAL consists of 2 separate tasks: test generation and test completion.
The goal of test generation to generate an entire test suite given a file under test. We provide the necessary inputs the model part of the prompt (see Appendix Coor full details). Our test generation task aligns with real world unit testing; unit testing in practice involves writing tests for large code files in complex projects.
Test generation:
Test completion: The goal of test completion is to generate the next test in an existing test suite given an existing test suite and the file under test. Test completion measured by many software
3

---

Published as a conference paper at ICLR 2025
class Character:
class Character:
def leval (sel)fff self. +
def level_up (sels): self, 1-
def damage (self, health); selt,liesath heath
def damage (salf, hezlth): welf,healte Ith heath
f TeatGenEvel test prefix
IestGenBval Pull: full test suite generation
def test cest_cheracter_.ett
tess.chaaracter.settpp/
def wtw__cchaccteeeeeeeeeeeee
def Les lesi_chharallerrleeeel..u
TestGenBvall teat corpletion
der test_character.domm
ser best character damage (self):
Figure 3: Full test suite generation
Figure 4: Test completion (last)
Figure 5: Two software testing tasks (and highlighted model generations). Full test suite generation requires knowledge of code under test setup, along with meaningful assert statements. Test comple- tion requires understanding the code under test and current test generate an additional test method (first, last, and extra test).
testing models, and can be applied in IDE tools, however there no benchmark for this task. The test completion task aligns with different stages in the development life cycle; first test completion mirrors developer starting their test suite, last completion mirrors the finishing of their test suite and extra test completion measures whether language models can add an additional test to test suite developer thinks is complete. This setup is in line with Rao et al. (2023), which models test generation at the method level.
2.3 PROPERTIES OF TESTGENEVAL:
Table 1:
Comparison of TESTGENEVAL against existing test generation benchmarks. TEST- GENEVAL mirrors the real world task of unit test generation, including file level, human written tests. TESTGENEVAL includes both individual test generation and full test suite generation, and is the only benchmark to measure mutation score.
Dataset
Size
File level
Human tests
Test suite
Mutation score
TESTEVAL
210
SWT-BENCH
1762
V
X
X
R2E TESTGENEVAL
246
X
1210
/
Table I outlines some of the properties that differentiate TESTGENEVAL from existing test genera- tion benchmarks. We explain each property depth.
File level test generation: Real world unit testing involves reasoning over complex files, generating tests for given file under test. Unlike existing benchmarks that deal with small, self contained programs, TESTGENEVAL includes code and tests from large scale, highly starred projects.
Human-written tests: Existing benchmarks such as R2E (Jain et al. 2024) measure the ability of an LLM to generate equivalence tests. While models that generate equivalence tests have the advantage of high coverage and case of scaling to large amounts of data, they often look different from developer written tests (Kambhamettu al. 2022). Measuring the ability of LLMs generate equivalence tests is an adjacent, but different task from measuring unit test generation capabilities. TesTGenEVal is the first large-scale sestageneration benchmark using human-writ written tests.

---

Published as a conference paper at ICLR 2025
Test suite generation: The goal of unit test generation is to generate high-quality test suites, which correlated to high coverage and mutation score for the code under test. However, existing methods such as TestEval (Wang et al., 2024) and SWT-Bench (Mündler et al. 2024) only measure the ability of an L.M to generate an individual test method rather than an entire test suite. We propose complementing individual test completion tasks with the broader test suite generation task.
Mutation score: Unlike existing benchmarks, we report mutation score in conjunction with cov- erage. The mutation score is empirically far more correlated with bug detection capabílities (Just et al. 2014) Papadakis et al. 2018) and much harder to hack; in order to achieve high mutation score tests must be able discriminate non-buggy code from code with synthetic bugs introduced.
EVALUATION
We evaluate selection of models on TESTGENEVAL. to better understand how models of different sizes and families (see Table [ for list, and Appendix [D.]] for more details about model choices) perform at test suite generation and test completion. The HuggingFace checkpoints of all open source models used are also available at Appendix B We prompt each model with the maximum context window size possible, otherwise truncate the starting tokens to fit the prompt in the context window.
We report results for all models in both the full test generation (Section 3.1 and test completion tasks (Section on TESTGENEVAL For test suite generation we report any pass@ (if any of the tests in the generated test suite pass), all pass @ (if the generated test suite passes), coverage (coverage of passing tests), and mutation score (proportion of synthetic bugs introduced to code caught by test suite). For test completion with coverage improvement from adding the generated test. More detailed descriptions and our full set of metrics can be found Appendix D.2 Our full set of results for TESTGENEVAL and results for TESTGENEVALLITE can be found in Appendix E
report pass@ and pass @ (whether generated test passes), along
We also perform statistical significance tests
and report 95% confidence intervals in Appendix
3.1 TEST GENERATION PERFORMANCE OF VARIOUS MODELS
Table 2: Full test suite generation. All results shown for temperature=0.2. Larger models generally perform better test suite generation, however all models struggle to achieve high coverage and mutation score.
Model
All @ 1
Any @
Coverage
Mutation
Small Models
CodeLlama 7B
3.2%
4.1%
1.2%
0.5%
Gemma 9B
3.5%
42.
20.2%
9.0%
Llama 3.1 8B
3.1 1.1%
30.5%
14.1%
6.8%
Medium Models
DeepSeckCoder 16B
23.0%
63.7%
28.2%
12.1%
Gemma 27B Codestral 22B
7.4%
57.7%
30.1% 10.1%
14.6%
26.8%
72.7%
33.0%
14.2%
Large Models
CodeLlama 70B
14.0%
22.7% 60.7%
7.0%
2.5%
Llama 3.1 70B
7.8%
30.6%
15.4%
Flagship Models
GPT-4o Llama 3.1 405B
7.5%
64.0% 73.1 %
35.2%
18.8 %
17.7%
35.0%
16.4%
Table 2 shows any pass@1 1, all pass@1 coverage and mutation score for small, medium and large models. All three smaller models perform significantly worse than their larger counterparts, with
5

---

Published as a conference paper at ICLR 2025
large difference (42.6% for Llama 3.1 models and 15.6% for Gemma models in any pass@) performance). All pass@1 penalizes how verbose a model is: DeepSeekCoder 16B on average generates test suite with 106 lines of code and 16 methods, while Llama 3.1 70B generates an average of 324 lines of code and 36 methods. This intuitively makes sense, as the more verbose a model is, the more likely it is generate test with errors.
Coverage and mutation score remain low across all models. For coverage, GPT-4o performs the best, but still covers only 35.2% of the lines of code tested in TESTGENEVAL. The mutation score is even lower than the coverage, which implies that tests generated by these models cannot catch all induced bugs. Despite generating passing tests for smaller proportion of programs, GPT-4o achieves higher coverage than Llama 3.1 405B, meaning that GPT-4o generates higher quality tests on average (also reflected by the higher mutation score of GPT-4o compared to I.lama 3.1 405B).
3.2 TEST COMPLETION PERFORMANCE OF VARIOUS MODELS
Table 3: Pass rates for different models first, and last settings. Pass for temperature=0.2,
Pass fiss@5 5 is for temperature=0... Codestral 22B outperforms all models across pass @ I and pass @ (other than first test completion 5) solving between 60-70% of all tasks.
First Pass@5
Last
Model
Pass @
+Cov
Pass @
Pass @5
+Cov
Small Models
Cama 7B
4.2%
19.8%
6.7%
6.9%
24.0%
0.0%
Gemma 9B
8.4%
18.0%
6.7%
21.4%
46.4%
0.1 %
Llama 3.1 8B
14.4%
33.3%
12.8%
32.0%
54.3%
0.1%
Medium Models
DeepSeekCoder 16B
18.6%
47.0%
19.0%
17.0%
62.8%
0.2%
Gemma 27B Codestral 22B
12.7%
33.7%
13.6%
32.2%
62.2%
0.1%
38.3% %
61.7%
24.0%
50.4%
74.3%
0.4%
Large Models
Codel lama 70B
0.5%
30.2%
11.2% 18.7%
0.9%
50.7%
0.0%
Ilama 3.1 70B
19.3%
46.4%
35.0%
61.9%
0.5%
Flagship Models
GPT-4o
31.9%
63.5%
26.9%
32.6%
66.6%
0.5 %
Llama 3.1 405B
32.1%
57.7%
21.6%
42.6%
72.3%
0.3%
Table 3 shows pass@1, p and coverage improvement for and last settings. Information on extra test completion can be found in Appendix Eb. Model performance generally increases as more the test file is provided as context (extra pass @5 is higher than both last pass@5 and first pass @5). Similar to the full test setting, larger models tend outperform smaller ones. An outlier is Llama 3.1 8B, with a significantly higher pass@; 5 in all settings than its smaller model counterparts. Codestral 22B performs the best at generating passing tests, with pass pass@5 of 74.3% the last test completion setting. Models face challenges in augmenting coverage for existing human-written test- suites, whereas they can more readily add coverage when tests are initially present. the final test completion setting, all models generate virtually no new coverage, primarily testing computation paths that have already been covered.
4 ANALYSIS
We perform a quantitative and qualitative analysis all results. This includes correlation with other benchmarks, effects of samples @ass@k, effects of context window size along with qualitative analysis of differentiating problems between Codestral, GPT-4o and Llama 405B. Our complete analysis can be found in Appendix

---

Published as a conference paper at ICLR 2025
4.1 QUANTITATIVE ANALYSIS
We perform an analysis of model correlation with other benchmarks (Section 4.1.1 along with the effects of sampling more (Section (.1.2) and increasing context window (Section 4.1.3) on model performance. Details correlation between models and settings and common model errors can be found in Appendix F.2 and Appendix F.5 respectively.
4.1.1 CORRELATION WITH OTHER BENCHMARKS
lestoentvall covert aga091 Vos Humanitvaa passegg
Nostienbval coverapeeg) vnstivall patsap11
64-
sodellams
coce
small
medium
large
large tagohip
Tagship
15
WestDenEvvll Coverage::
TesiClenEval coverageg1
(a) Correlation HumanEval
(b) Correlation with TestEval
Figure 6: Correlation with HumanEval (a SOTA code generation benchmark), and TestEval (a SOTA test generation benchmark). We find that there is a weak positive correlation between HumanEval and TESTGENEVAL, and with the exception of Gemma 9B there is similar weak positive correla- tion between TestEval and TESTGENEVAL.
Figure 6a and and Figure 6b display correlation between TESTGENEVAL, HumanEval (a code genera- tion benchmark), and TestEval (a test generation benchmark). We find a weak positive correlation between TESTGENEVAL scores and other benchmarks. For HumanEval outliers include Gemma 9B (performs better at test generation than expected given code generation performance) and CodeL- lama (performs worse at test generation than expected). For TestEval, the main outlier is Gemma 9B, with reasonable test generation performance on TESTGENEVAL, but very poor performance on TestEval (it fails to properly follow the prompt format for TestEval). Full details and comparison against MMLU can be found in Appendix F.
4.1.2 EFFECT OF NUMBER OF SAMPLES
coverageblc vs k for full test peneration setting
passsik first. last, extra settings
Frst
List
72
Extm
23
30
2.5
30.5
22.5
13.2
27.5
200
23
380
(a) Coverage@k k (full test generation)
(b) Pass @k vs (test completion)
Figure 7: Effect of sampling more tests for all settings on Llama 3.1 8B. Coverage @k seems to gradually increase. Pass@k k also increases more for lower k values and seems to plateau after k-20.

---

Published as a conference paper at ICLR 2025
Figure 7a and Figure show how performance in the full test generation and first, last and extra test completion settings changes as more tests are sampled for Llama 3.1 8B. For full test suite generation sample 20 examples and measure coverage @k For test completion measure pass@k @k. For full test suite generation, we find that coverage seems to gradually increase, with no plateau in the first 20 generations. For test completion, improve performance the most, and performance gains seem to plateau after k=20 (the gain between k=20 and k=100 minimal). Full details of this analysis can be found in Appendix F3
sample 100 examples and
find that the first 5 samples
4.1.3
EFFECT OF CONTEXT WINDOW ON TESTGENEVAL PERFORMANCE
coverage VS context length for full sctting
passg5 vs context length for first, lst, and extra scttings
345
Lest
Extra
22003 +2063
.0:00
00043
130000
120030
2006 80000
B0000
14:000
12900
(a) Coverage by context length (full test generation)
P by context length (test completion)
Figure 8: Effect of context window for both our full setting (coverage of generated test suite) and for our test completion setting (pass @ for our first, last and extra completion settings). We find that context length generally helps test completion, however for test generation, even receiving parts of the file under test (measured by a lower context window) seems to effective.
Figure Sa and Figure &b show the effect context length on both our coverage (full test generation) and pass 5 (test completion). For test generation, we find that coverage only slightly improves with additional context; even seeing part of the code under test is enough for the model to generate a test suite (can mock inputs to various methods in the partial file). However for test completion con- text is more important, with pass @5 increasing with more context up until around 32k tokens where benefits decay. To complete tests, one must understand existing tests and their relationship with the code under test. Having the entire file helps contextualize what existing tests are testing, improving the performance of completed tests. Full details of this analysis can be found Appendix F.4
4.2 QUALITATIVE ANALYSIS
We highlight a case where TESTGENEVAL discriminates between our top three models (GPT-4o, Llama 405B and Codestral 22B). Additional examples comparing these three models can found in Appendix [.6 Additionally, qualitative examples of common error types can be found in Ap- pendix F.7 and examples of frontier problems (problems solved by at most one model and no mod- els) in Appendix
class QuerySet (AltersData):
def Liiit (seLf, madel=Hone, query=Nene, ...):
def zepr (sell):
return "ell Aran (self, claff class ...
nome
data)
Listing 1: Query set class for managing data from database.
Our example involves a QuerySet class that manages records returned from a database. The class has database dependencies. The initialization takes model and query, which can also not be set if no database is being used. This is hard to test because it involves either mocking or creating complex model object.
8

---

Published as a conference paper at ICLR 2025
def gest_querysett_c (self): queryaet = Qveryšet model=-onaal
assertEqual (repr (querysett, "sQueryšet (J>*)
Listing 2: Cut GPT-4o generated test
GPT-4o instantiates the QuerySet with no parameters (the simplest possible version of the test), It then tests all code methods, with all of the generated tests passing the code under test.
from django.da import connection, modela
Lest_query queryuueeeeeeee set_init (sell):
model seddels.xodell. qa querySet (model=model)
velf.assertaqual (gs.nodel, model)
Listing 3: Cut Llama 3.1 405B generated test
This is not the case with both Llama 3.1 405B and Codestral 22B, Both models igmore the empty case entirely and instead hallucinate invocations imports of models in Django (instead these models should have mocked the model object and tested the null case). All models failing to properly mock the class under test's dependencies lead to low coverage of the source file, despite GPT-4o generating passing tests all when other models fail.
LIMITATIONS
We outline potential limitations with TESTGENEVAL. A more detailed set limitations can be found in Appendix H
Overfitting to SWEBench repositories: A potential limitation is that our benchmark is adapted from SWEBench and result risks models overfitting to this specific dataset. Currently, this does not seem to be a major issue, as model performance is low across the board. Even once models can achieve high coverage, there is the significantly harder task of achieving high mutation score (actually catching synthetic bugs introduced into the code under test). These multiple levels of difficulty and numerous tasks help mitigate the risk of model overfitting to any task specifically.
There also a risk of data contamination in the pretraining data of models. To further understand data contamination, we measure perplexity of 10 randomly selected tests in TESTGENEVAL for Llama 3.1 8B and common frequent, and non recent code from GitHub with similar lengths. We find that the perplexity of this common GitHub code is lower than the 10 tests from TESTGENEVAL (1.6 vs 2.0), indicating that data is unlikely to be contaminated. This is further supported by the low performance of all models on TESTGENEVAL across the board.
Data contamination:
Compute cost of mutation score: One other limitation is the compute cost of computing mutation score. Each synthetic bug we introduce the code under test, requires an additional test suite execution. However, our results show that coverage and mutation score are highly correlated. Setting a timeout of one hour per mutation testing run, we only timeout on 20% of files, and get an average uncertainty of 1.1%. We provide option to run TESTGENEVAL without mutation, enabling those who lack compute to still benefit from TESTGENEVAL.
6 RELATED WORK
Test Generation Benchmarks: Recently, there has been more effort to evaluate language mod- els software testing capabilities, however these benchmarks typically still consist of small, self- contained projects. Common code generation benchmarks such as HumanEvalFix (Chen et al. 2021), MBPP Plus (Austin et al. 2021) and APPS (Hendrycks et al.) 2021) can easily be adapted to test generation tasks. TestEval Waag et al. (024) measures test generation capabilities for Leet- Code problems of varying difficulties. Bhatia et al. (2024) measure test generation performance for modular code without external dependencies. While these benchmarks provide important execution metrics, the small size of these problems and solutions does not mirror realistic test generation.

---

Published as a conference paper at ICLR 2025
There are also repository level test completion benchmarks, however these benchmarks often lack realistic execution metrics. TeCo (Nie et al., 2023) and ConTest (Villmow et al. 2021 provides
benchmark for completing unit tests, but only measures lexical metrics such BLEU Papineni
et al.
2002) and ROUGE scores (Lin 2004). ATLAS (Watson et al.] 2020) and TOGA (Dinella
al.
2022) provide large scale benchmarks but for completing assertions rather than generating
entire tests. SWT-Bench (Mündler et al.) 2024) provides a benchmark for test method generation targeted as bug fixing PRs. Their task also adjacent but measuring a specialized part of software testing (generating tests that fail on code prior to PR and pass after PR).
More recently there has been efforts create executable code benchmarks. SWEBench (Jimenez et al.] 2222) measures the ability of language models to generate patches for failing PR tests. R2E (Jain et al. 2024) provides collection of 400 repositories that can be executed, however they leverage equivalence test. harnesses which look structurally different than human written unit tests. CruxEval (Gu et al. 2024) provides a large set of executable code snippets, however measures execution reasoning ability, which is subset of the test generation task. TESTGENEVAL provides a large scale executable environment to measure test generation and test completion performance.
Large Language Models for Code Generation: Large language models (LLMs) have shown promising results on many software engineering tasks, including software testing (Rao et al. 2023 Black et al. 2022 Touvron al. 2023a). Top open source models include Llama 3.1 (Dbeyy al. 2024). Seek (D (DeepSeek-A) Al et al. 2024] and CodeStral [ Closed source models includ- ing GPT-4-0 F and Claude 3.5 Sonnet have also shown state of the art performance
wide variety of code generation benchmarks. We measure performance for top open source models and GPT-4o, the current state of the closed source model.
7
CONCLUSION
We introduce TEST GENEVAL, the first file-level benchmark for test generation and test completion. We release a lite version consisting of 160 code-test file pairs and full benchmark comprising 1 1,210 file pairs from real open-source projects. Considering real-world settings, we employ coverage and mutation score to evaluate the models in our benchmark, as these metrics are closely related to the real-world quality of test suites. We perform a comprehensive evaluation of both open and closed source models on TESTGENEVAL. Additionally, we conduct thorough error analysis, revealing that models struggle with reasoning about execution, frequently making assertion errors and failing to generate tests that run within the time limit. Overall, we believe TESTGENEVAL provides complementary dataset to existing test generation datasets, offering a more challenging and larger- scale version of current benchmarks.
8 ETHICS STATEMENT
As models continue grow in size and test generation capabilities improve, it is important sider the broader impacts of test generation. While generally test generation is an important task for ensuring high quality software, generated tests can be used to assert that buggy code correct (as seen by the oracle problem). All generated tests should therefore be checked and approved by developers. Furthermore, automated test generation techniques have the potential to automate qual- ity assurance jobs, which can also potentially lead to negative societal impacts. We recommend practitioners be cognizant of the impacts of test generation work when building and deploying new techniques.
9 REPRODUCIBILITY STATEMENT
We took extensive efforts to ensure TESTGENEVAL is reproducible. We release code to run and extend TESTGENEVAL along with all docker images for each project in SWEBench. We release website to view model generated tests for qualitative analysis. Our code and website is available
10

---

Published as a conference paper at ICLR 2025
We designed TESTGENEVAL' 's code to be easy to extend; all projects have individual docker images to run them, enabling those looking to extend TESTGENEvAL VAL to simply create new docker container and hook into existing code.
10 ACKNOWLEDGMENTS
The authors would like thank Ori Yoran and Pierre Chambon for their feedback on the paper, as well as Claire Le Goues for her support and mentorship. We also thank the CodeGen team for their extensive support and feedback throughout the project. Last but not least, we give special thanks to Mei A an outstanding canine software engineering researcher, for providing support and motivation throughout this paper.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. arXiv:abs/2108.07733 2021.
Roberto Baldoni, Emilio Coppa, Daniele Cono D'Elia, Camil Demetreseu, and Irene Finocchi. A Survey of Symbolic Execution Techniques. ACM Computing Survey, 51(3):50-88, 2018.
Moritz Beller, Georgios Gousios, Annibale Panichella, and Andy Zaidman. When, how, and why developers (do not) in their ides. In Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering, ESEC/FSE *15, pp. 179-190, 2015a.
Moritz Beller, Georgios Gousios, and Andy Zaidman. How (much) do developers test? In Interna- lional Conference on Software Engineering, ICSE "15 pp. 559-562, 2015b.
Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. Unit test generation using gen- erative A comparative performance analysis of autogeneration tools, 2024. URL https:
//
10622
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang. Michael Pieler, USVSN Sai Prashanth, Shivan- shu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. arXiv:abs/2204.067455 2022.
Carolin Brandt and Andy Zaidman. Developer-centric test amplification: The interplay between automatic generation human exploration. Empirical Software Engineering, 27(4), 2022. ISSN 1382-3256.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, 1lya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code.
1tt.:aab/2110.0333/2 2021.
Sourav Deb, Kush Jain, Rijnard van Tonder, Claire Le Goues, and Alex Groce. Syntax is all you need: A universal- universal-language approach to mutant generation. Proc. ACM Softw. Eng., (FFE)), July 2024. doi: 10. 145/3643756. URL httpt:ttt///........ :ttpss://do. .acilorg//i1 0ttp//11..11454437777777
11

---

Published as a conference paper at ICLR 2025
DeepSeek-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Rumxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang. Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao. Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder--v2 Breaking the barrier of closed-source models in code intelligence, 2024. URL httts://aaxiii org/ abs 2200..22446..0 11931
Anna Derezinska and Konrad Halas. Experimental evaluation of mutation testing approaches to python programs. In 2014 IEEE Seventh International Conference on Software Testing, Verifica- tion and Validation Workshops, pp. 156-164, 2014. doi: 10. 10.1109/CSTT:22144.44
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu Lahiri. Toga: A neural method for test oracle generation. In International Conference on Software Engineering, ICSE '22, pp- 2130-2141, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail. Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasami. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Rantala-Ycary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh. Manohar Paluri, Marcin Kardas. Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami. Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De
12

---

Published as a conference paper at ICLR 2025
Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker. Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute. Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshninarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj. Igor Molybog, Igor Tufanov. Irina-Eleer Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher,
Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen. Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usumier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chemoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang. Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar. Shugiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe. Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vitito, Zef Rosnbrick, Zhaoduo Wen. Zhenyu Yang, and Zhiwei Zhao The llama 3 herd of models, 2024. URL
abw/2407 21783
Gordon Fraser and Andrea Arcuri. Evosuite: Automatic test suite generation for object-oriented software. In Joint Meeting the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering, ESEC/FSE 11, pp. 416-419, 2011.
Gordon Fraser and Andrea Arcuri. Whole test suite generation. Engineering, 39(2):276-29) 2013. doi: 10. 10.1109/TSE.2012.14
IEEE Transactions on Software
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrievallauggmentd generation for large language models: A survey, 2024. UtLpttts::///tt arxiv. wwwpr/wwwwrgabb////eeee
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arxiv preprint arXiv:2401.03066 2024.
13

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In NeurlPS Datasets and Benchmarks, 2021.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and lon Stoica. R2e: Turning any github repository into programming agent environment. In ICML, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE--hench Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024. BtLhttps: http:ttsp//:/nne /ttp://///ppereieee..
net/ mee//fornme Eorum? VIF 8yNQM66
René Just, Darioush Jalali, Laura Inozemtseva, Michael D Emst, Reid Holmes, and Gordon Fraser. Are mutants a valid substitute for real faults in software testing? In Symposium on Foundations of Software Engineering, FSE '44 pp. 654-665, 2014. URL https: org/ 1145/
2635868 2635929
Rajeswari Hita Kambhamettu, John Billos, Tomi Oluwaseun-Apo, Benjamin Gafford, Rohan Pad- hye, and Vincent J. Hellendoorn. On the naturalness of fuzzer-generated code. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, pp. 506-510, New York, NY, USA. 2022. Association for Computing Machinery. ISBN 9781450393034, doi: 10.11457324842.33227922 URL htllhttp: wwg./o..ggo.ooo 114.::..111532255113 3527972
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Conference Text Summarization Branches Out, pp. 74-81, 2004.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023. URL
wttpo////a.///....... 93091
Nicls Mündler, Mark Niklas Müller, Jingxuan Hc, and Martin Vechev, Code agents are state of the art software testers, 2024. URL httphtttts/////ttet org/ abs 220. 12952
Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. Learning deep semantics for test completion. In International Conference on Software Engineering, ICSE '23, pp. 2111-2123, 2023.
A. Jefferson Offutt, Ammei Lee, Gregg Rothermel, Roland H. Untch, and Christian Zapf. An experimental determination of sufficient mutant operators. ACM Trans. Softw: Eng. Methodol., 5 (2):99-1188 118, apr 1996. ISSN 1049-331X. doi: 10.1145/227007.227611 URL //dos.
OpenAl. Gpt-4 technical report, 2023.
Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In International Conference on Software Engineering, ICSE '18 pp. 537-548, 2018. doi: 10.1145/3 188155.3180111
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In ACL, pp. 311-318. ACL, 2002.
N. Rao, K. Jain, U. Alon, C. Goues, and V. J. Hellendoorn. Cat-lm training language models on aligned code and tests. In 2023 38th IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE), pp. 409 420, Los Alamitos, CA, USA, sep 2023. IEEE Computer Soci- ety. doi: 10.1 10911199//552229..... 2023.00199 URL https: doi brr/1001 1ttp:/1.11119//55522222 202.:22222222220.222222222 00193
Ana B. Sánchez, Pedro Delgado-Pérez, Inmaculada Medina-Buld and Sergio Segura. Mutation testing in the wild: findings from github. Empirical Softw. Engg.. 27(6), November 2022. ISSN 1382-3256. doi: 10. 10.10077/510644422---1000 00664-022-1077--
URL /
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models. arXiv:abs/2307.022 092888 2023b.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers. CoRR, abs/2009 006617,7,0977,... 2020.
Victor Veloso and Andre Hora. Characterizing high-quality test methods: first empirical study. In Proceedings of the 19th International Conference on Mining Software Repositories, MSR '22, pp. 265 269, New York, NY, USA. 2022. Association for Computing Machinery. ISBN 9781 9781450393034.
doi: 10.1145//52444233229022
URL https::/dci..rg////
3524842 3529092
Johannes Villmow, Jonas Depoix, and Adrian Ulges. ConTest: A Unit Test Completion Benchmark featuring Context. In Workshop on Natural Language Processing for Programming, pp. 17-25, August 2021.
Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2024. http:tttt///axi...aaaaa org/abs/////aaa.aoooooo.o../s.sa 2406 04531
Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. On learning meaningful assert statements for unit test cases. CoRR, abs/2002.05800, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. httht::t///aaet
org. hbtp://bb//2000 11912
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong. Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL htt.http::/// /ttt:i///xxivii org/ ab5/ 15877
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Benchmark Statisties
17
A.1 TESTGENEVAL
17
A.2 LESTGENÉVALLITE
17
Model URLs
21
Prompts
22
D Experiment Setup
24
D.1 Models
24
D.2 Metrics
24
D.2.1
Test Generation Metries
24
D.2.2
Test Completion Metries
25
E Results
26
E.1
TESTGENEVAL
26
E.I.I Test Generation
26
E.1.2
Test Completion
26
E.2
TESTGeNEVAl LiTE
28
E.2.1 Test Generation
28
E.2.2
Test Completion
28
F
Analysis
30
F.1
Correlation with other benchmarks
30
F.I.1
TESTGENEVAL
30
F11.2
TESTGENEVAL LITE
31
F2
Conelation between settings and models
32
F2.1
TESTGENEVAL
32
F2.2
TESTGENEVALLITE
32
F3
Eifect of Sampling Coverage and Pass
34
F4
Effect of Context Coverage and Pass
35
1:5
Quantitative Error Analysis
35
F.5.1
TESTGENEVAL
35
F.5.2
TESTGENEVAL LITE
37
R6
Qualitative Model Comparison
38
F.6.1
Example Test setup (only solved by GPT-40)
F.6.2
Example 2 Incorrectly mocking objects (only solved by Ilama 3. 405B)
38
F.6.3
Example Handling class dependencies (only solved by Codestral 22B
39
F7
Qualitative Eror Analysis
40
F8
Frontiers of TESTGENEVAL
42
Statistical Tests
47
TESTGENEVAL
47
TESTGENEVAL
49
H Jimitations
53
16

---

Published as a conference paper at ICLR 2025
A
BENCHMARK STATISTICS
This section contains more details on benchmark statistics. Both our TESTGENEVAL and TEST- GENEVALLITE splits consist 1l repositories shown Table
Table 4: Benchmark repository information and datapoint distribution.
Repository
URL
# Stars
Lite
# Full
astropy/astropy
https//gtthubbcoomm/sttooy/attry
4325
45
django/django
http://gtthbb.commddjnnggggggggg
78287
66
451
matplotiib/matllotibb
http://gghhllccoom/mapppptiiitnt
19756
72
mwaskom/seaborn
https//gtttbbbcoommmwvvaaaaaaaao
12264
12
pallets/flask
https:/ggthhb.ccomm/ppllrttsssst
67203
pydata/xarray
https//gtthubb.oom//yydddaaxuuyr
3523
pylint-dev/pyling
https//ggtlubbccoo//yyyyttytyyyp
5206
2
17
pytest-dev/pytest
https//gtttbbbcomm/yyttte-vvvvys
11719
12
scikit-leam/scikit--ee
https//ggghuubccoom//ssstffaaala
59057
20
192
sphinx-doc/sphinx
6264
sympy/sympy
https:/giilbbbcomm//yyyyyyyyyyy
12657
42
239
A.1 TESTGENEVAL
We report detailed statistics for the TESTGENEVAL. These include the coverage of various files, unique generations, correlations between different settings and MMLU.
Figure 9a Figure and Figure show the distribution of code lengths, test lengths, and reposito- ries across TESTGENEVAL. Tests and code files are thousands of tokens long, as the benchmark is large in size. The repository distribution mirrors that of SWEBench and the TESTGENEVALLITE split repo distribution is also similar.
Figure 10a and Figure 10b show the distribution of code methods and test methods. Generally, both code and tests methods are quite extensive across file pairs (average of 57 test methods and 58 code methods per file pair).
Figure provides violin plots of the coverage data gold tests TESTGENEVAL. Coverage is generally high across all settings (first test alone adds average 40% coverage) and by the end of the test suite, file level coverage approximately 80%.
A.2 TESTGENEVALLITE
We perform the analysis on the TESTGINEEVLLLLTE split of our benchmark. Many of the distributions and trends observed in TESTGENEVAL hold here too.
Figure 12a Figure 12b and Figure 12c show the distribution of code lengths, test lengths, and repositories in the "ESTGENEVALLITE split. Both the distribution of repositories and lengths of code and test files approximately mirror those present in full TESTGENEVAL. The long lengths even in the TESTGENEVALLITE split require models that support long context lengths.
Figure [0a and Figure [ob show the distribution of code methods and methods for TEST- LIEE Results mirror those of TESTGENEVAL: both code and test files in this file have many methods (average of 58 test methods and code methods for each file pair).
Figure 14 provides violin plots of the coverage data of gold tests in the TEETGEEEEVALLLTE split. TESTGENEVALLITE coverage distributions mirror those of TESTGENEVAL.
17

---

Published as a conference paper at ICLR 2025
Code length distribucion
length distrisution
303
230
150
153
100
102
Length tokens)
Length takeral
(a) TESTGENEVAL code lengths
(b) TESTGENEVAL test lengths
Repository Distribution
400
300
estropy/sstrppy
gangonfange
mik mouslom/ssatoo
palletsflask
pydata/rarra
pylint devpllle
prtest-dvipyieet
scikt wikd-leamstekkriian Reama
sphinx Boci-da
sympytbympp
Repository
(c) TESTGENEVAL repo distribution
Figure 9: Code and test lengths across TESTGENEVAL. Both the code and test files TEST- GENEVAL from real world projects and thus are often 10,000+ tokens long. Also shows the distribution of repositories TESTGENEVAL.
Methed Distribution
"iist Methed Boution
200
253
303
250
120
882
sWithodt
(a) TESTGENEVAL code methods
(b) TESTGENEVAL test methods
Figure 10: Distribution code and test methods for VESTGENEVAL.
18

---

Published as a conference paper at ICLR 2025
100
Last
Coverage Percentago
80
40
20
First Test
Last-1 Test
Last Test
Figure 11: TESTGENEVAL gold test coverage
length
length
25
10
20000
23000
50033
(a) TESTGENEVALLITE code lengths
(b) LESTGENEVALLITE test lengths
Repository Distribution
60-
30
20-
10
Bangordaagg
scikit baabrrckteaamm
Symps/sympy
priest deulpt-ee
antropyistropy
matplatiimatsiaati
menslomsteesto
gutlne docisphins
pylntt deupylin
prdatsmarryy
aallstorfaak
Repository
(c) TESTGENEVALLITE repo distribution
Figure 12: Distributions in the TESTGENEVALLITE dataset
19

---

Published as a conference paper at ICLR 2025
Code Methed Ditributtion
"lst Methcd Dist-ibution
Text Nertice
Iest Mathods
23
20
10
100
200
300
200
500
#dothoots
antctrecs
(a) TESTGENEVALLITE code methods
(b) TESTGENEVALLITE test methods
Figure 13: Distribution of code and test methods for TESTGENEVAL.
100
Last Test
Coverage Percentage
40
First Tast
Last-1 Tast
Last Tast
Figure 14: TESTGENEVALLITE gold test coverage
20

---

Published as a conference paper at ICLR 2025
B MODEL URLS
We used GPT-4o on 08/29/2024. Shoos the model name and model URL for all open source models on HuggingFace.
Table 5: Model Names and URLs
Model Name
Model URL
CodeLlama (7B)
-
CodeLlama (70B)
hf
http:///aggaaaaicocccc//mm///ila Llama 3.1 (405B)
Llama 3.1 (8B)
Llama 3.1 (70B)
Codestral (22B)
DeepSeckCodder-Lie Cider-Lite (16B)
Gemma-2 2 (9B)
http://wwgggghhaaeoooooggggmmmmg
htsp:h/wggggabbbcoooooo////mmmaz Gemma-2 (27B)
21

---

Published as a conference paper at ICLR 2025
C PROMPTS
For each setting we prompt with different contexts. We provide the code under test part of the prompt for all settings. also provide the preamble of the test file (imports and test setup) first test completion, the entire test file minus the last test for last test completion and the entire test file for extra test completion. For our full file test generation setting, we only prompt with the code under test. Since use large variety of models, our prompts slightly different for each class of model.
System Prompt
are expert Python automated testing assistant. Your job to generate test file gi ven a code file.
Instructions
Below is code file:
" "python (code_sre)
The code file is called: (code filename]
Your job is to output a corresponding unit test file that obtains high coverage and invokes the code under test.
Here are some examples how import fcode_ (you should use these as reference)
" "pyyhon (imports)
Each unit test must be function starting with test.. Include all your test imports and setup before your first test. Do the tests in the file, just output series of tests. Do not include main method to run the tests.
Only output the unit test Python file in this format:
"pyyhon
Unit test Python code (file level)
Figure 15: Prompt for the full suite generation setting. For models that use specific format for chat, we format this appropriately to chat format.
22

---

Published as a conference paper at ICLR 2025
System Prompt
You are an expert Python software testing assistant. Your job is to complete the next test given code file an some existing test context.
Instructions
Below is a code file: 'pythoo (code_sre)
And the current unit test file ''yython (test_arc) '1
Your jod write the Python code the next test in the file. Ideally your next test should improve coverage of the existing unit test file for the code file.
Only output the next unit test, preserve indentation and formatting. Do not output anything else. Format like this:
"pythhn Next unit test Python code
Prompt for the test completion settings (first, last, extra). For models that use specific format for chat, we format this appropriately to chat format.
Figure 16:
23

---

Published as a conference paper at ICLR 2025
D EXPERIMENT SETUP
We outline the metrics used for evaluation for each of our tasks.
D.1 MODELS
We evaluate selection of models on TESTGENEVAL. to better understand how models of different sizes and families perform at test suite generation and test completion. We include a mixture small (less than 9B params), medium (between 9B and 27B params), large (approximately 70B params) and flagship (greater than 70B params) models. This includes open source models (CodeLlama,, Llama 3, DeepSeekCoder 2, Codestral, and Gemma 2) of varying sizes and GPT-4o, state of the art closed source model.
We choose the Llama and Gemma families of models to understand the effects of size model performance in conjunction with their high scores on code benchmarks such as HumanEval. We also include CoepSeekCoder, CodeLlama and Codestral due their code specialization. Finally, include GPT-4o due to its state of the art performance on numerous code generation benchmarks.
D.2 METRICS
We report metrics for both our test generation (Appendix D.2.1) and test completion (Ap- pendix D.2.2 tasks.
D.2.1 TEST GENERATION METRICS
All pass@1. 1: All pass@1 1 measures if the generated test suite passes when the code under test. This penalizes more verbose models, as the likelihood of an error increases with each additional test generated in suite.
Any pass@ I: Any pass@1 measures if any test in the generated test suite passes when run on the code under test. We include this to not penalize models that generate longer test suites.
Coverage: Coverage measures the proportion of lines in the file under test executed by the test suite. Ideally, high quality test suite should execute large percentage of the lines in the code under test.
Coverage@ pass: Coverage@ pass measures the coverage of only the passing tests. Models with high coverage pass may not successfully generate tests for many of the problems in TEST- GENEVAL, but for the ones they do, they obtain high coverage.
Mutation score: The main limitation of coverage and pass@ I that they can be potentially gamed (a model can invoke all functions in the file under test without testing anything and achieve 100% coverage and 100% pass@1 I). To compute mutation score we inject synthetic bugs into the code under test. We then measure the percentage of bugs detected by the test suite (should pass on the original code and fail on the buggy code). Unlike other metrics, mutation score much harder to game, however it is compute costly, we have to execute the entire test suite for each bug. We rank by coverage, as models still have relatively low coverage across the board and it is more compute (we allow for TESTGENEVAL to be run omitting mutation score).
efficient for the community
We use cosmic-ray"l
F to generate mutants and use the default set mutation operators This default of operators follows best practices defined by the mutation testing community (Derezinkka & mutation testing
Halas 2014; Offftt et al.) 1996). Cosmic ray has 565 stars and commonly research (Sánchez al. [2022] Deb al., (2224). We choose to not filter out mutation operators to achieve the most granular results possible (with no filtering there is only 1 1.06% uncertainty in mutation score results).
Mutation score@ pass: Mutation Score@ measures the the mutation score of only passing tests. Mutation score@ pass is similar to coverage pass, measuring how the quality of generated tests, rather than their coverage of all problems in TESTGENEVAL.
https///................//////et operators
24

---

Published as a conference paper at ICLR 2025
D.2.2 TEST COMPLETION METRICS
@ Pass@k measures any of k test generated pass when added the existing suite for the code under test. We rank by this metric, as coverage improvement is near for 2/3 settings for TESTGENEVAL.
Avg pass@k: Average pass@k measures the average proportion of k generated tests that pass. This provides an idea of how many erroneous tests model will generate before generating a correct test.
Coverage improvement: Coverage improvement measures the change in line coverage when adding the generated test. Ideally, newly added tests should improve overall code coverage. We choose not to report mutation score improvement here, due to computational cost and already near 0 coverage improvement generated tests.
Coverage improvement @ pass: Coverage improvement @ pass measures the change in line cover- age for only problems where generated tests pass the test suite. High coverage improvement pass indicates that model may be good at generating high coverage test completion, but could still struggle with generating passing test completions for all problems.
25

---

Published as a conference paper at ICLR 2025
E RESULTS
We report full set of results both TESTGENEVAL and TESTGENEVALLITE (results across both splits are similar). We also report complete test completion results for the extra setting and add the average pass@k metric.
E.1 TESTGENEVAL
We report performance on both the test generation and test completion tasks for TESTGENEVAL.
E.1.1 TEST GENERATION
Table 6: Full test suite generation. All results shown for temperature=0.2.. We report all pass @ 1, any pass@1, coverage, coverage of only passing tests, mutation score and mutation score of only passing tests.
Model
All@1
Any@1
Cov
Cov@ Pass
Mut
Mut@ Pass
Small Models
CodeLlama 7B
3.2%
4.1 %
1.2%
30.3%
0.5%
11.0%
Gemma 9B
3.5%
42. %
20.2% 14.1%
47.9%
9.0%
21.5%
Llama 3.1 8B
3.1%
30.5%
46.3%
6.8%
22.4%
Medium Models
CoepSeekCoder 16B
23.0%
63.7%
28.2%
44.3%
12.1%
19.1%
Gemma 27B
7.4%
57.7%
30.1% %
52.3%
14.6%
25.5%
Codestral 22B
26.8 %
72.7%
33.0%
45.4%
14.2%
19.7%
Large Models
CodeLlama 70B
14.0%
22.7% 60.7%
7.0% 30.6%
32.4%
2.5%
11.4%
Llama 3.1 70B
7.8%
50.4%
15.4%
25.6%
Flagship Models
GPT-4o Llama 3.1 405B
7.5%
64.0%
35.2%
54.9%
18.8%
29.4%
17.7%
73.1 %
35.0%
47.9%
16.4%
22.5%
Table Sone@Oosooe shows full test suite performance for TESTGENEVAL. Smaller models struggle to generate any tests that pass on complex code files present TESTGENEVAL. Codestral 22B can generate a passing test in 72.7% of all problems, however still has lower coverage and mutation score than GPT-4o. This indicates despite generating tests for more problems, the tests generated by Codestral 22B tend to be lower quality than those generated by GPT-4 (indicated by a approximately 10% lower coverage of passing tests).
E.1.2 TEST COMPLETION
shows shows the pass rates across first, last, and extra test completion settings. Extra test comple- tion setting results mirror those last test completion, with Codestral 22B outperforming existing models.
Table
Table shows the average @ rates along with coverage improvements for all settings. Cov- erage improvement for last and extra settings is near O; it is hard to improve coverage for near complete test suite. Average pass@5 is interestingly lower for GPT-4o, despite higher coverage, meaning that it would take attempts to generate passing test completion, but the quality of the completion better.
Figure 17a Figure 17b Figure 17c show the number of unique generations for each model across temperatures 0.2 and 0.8. Of all models, CodeLlama models are the least diverse, with not big difference between 0.2 and 0.8 temperature, while Llama 3.1 generates far more unique solutions.
26

---

Published as a conference paper at ICLR 2025
Table 7: Pass rates for different models in first, last, and extra completion settings. Pass @ temperature=0.2, P is for temperature=0.8.
for
First
Last
Extra
Model
Pass
Pass@5
Pass@1
P
Pass 1
P
Small Models
CodeLlama 7B
4.2%
19.8%
6.9%
24.0%
5.2%
22.9%
Gemma 9B
8,4%
18.0%
21.4%
46.4%
18.9%
46.7%
Llama 3.1 8B
14.4%
33.3%
32.0%
54.3%
31.8%
53.8%
Medium Models
DeepSeekCoder 16B
18.6%
47.0%
17.0% 32.2% 50.4 %
62.8%
15.5%
61.7%
Gemma 27B Codestral 22B
12.7%
33.7%
62.2%
31.7%
66.8%
38.3%
61.7%
74.3%
48.3%
75.7%
Large Models
CodeLlama 70B
0.5%
30.2%
0.9% 35.0%
50.7% 61.9%
0.5% 36.4%
52.8%
Llama 3.1 70B
19.3%
46.4%
61.2%
Flagship Models
GPT-4o
31.9%
63.5%
32.6%
66.6%
30.4%
63.5%
Llama 3.1 405B
32.1% %
57.7%
42.6%
72.3%
42.4%
74.5%
Table Coverage improvements, including coverage improvements for passing tests, and average pass @5 for first, last, and extra completion settings. Models struggle to improve coverage, and in general report substantially worse average pass@ than pass@5.
First
Last
Extra
Model
+Co
Avg
+Cov
+Cov@P
Avg@S
+Cov
+Cov @
Avg
Small Models
CodeLLama 7B
6.7%
33.9%
4.7%
0.0%
0.1%
5.7%
0.0%
0.1%
5,3%
Gemma
6.7%
37.3%
8.2%
0.1%
0.2%
17.4%
0.1%
0.2%
16.5%
Llama 3.1
12.8%
38.5%
10.5%
0.1%
0.2%
23.1%
0.0%
0.1%
23.5%
Medium Models
DeepSSec Coder 16B
19.0%
40.4%
16.4%
0.2%
0.3%
22.2%
0.1%
0.1
21.9%
Gemmna 27B
13.6%
40.2%
13.2%
0.1%
0.25:
28.0%
0.1%
0.1%
29.0%
Codestral 22B
24.0%
39.1%
31.2%
0.4%
0.5%
43.2%
0.2%
0.2%
43.5%
Large Models
CodeLlama 70B
11.2%
37.5%
7.5%
0.0%
0.0% 0.7%
14.2%
0.0%
0.0%
15.4%
Llama 3.1 70B
18.7%
40.5%
18.4%
0.5%
29.5%
0.1%
0.2%
29.0%
F Flagship Models
GPT-4o
26.9%
42.4%
32.4%
0,5 %
0.7% 0.45%
31.5%
0.2%
0.3%
28.8%
Llama 3.1 405B
21.6%
37.5%
31.3%
0.3%
40.9%
0.1%
0.2%
41.4%
(a) First setting
(b) Last setting
(c) Extra setting
Figure 17:
Unique generations by model and temperature for first, last and extra completion settings (out of maximum of 5 generations).
27

---

Published as a conference paper at ICLR 2025
E.2
TESTGENEVALLITE
We also report our full set results on our TESTGENEVALLITE split. We find that results generally mirror those of TESTGENEVAL.
E.2.1
TEST GENERATION
Model
All@1
Pass@1
Cov
Cov @ Pass
Mut
Mut @ Pass
Small Models
CodeL.lama 7B
3.8%
4.4%
2.1%
47.5%
0.8%
18.3%
Gemma 9B
7.5%
45.0%
19.6%
43.6%
8.1%
17.9%
Llama 3.1 8B
5.0%
29.4%
12. 1%
41.3%
5.1%
17.9%
Medium Models
CeepSeckCoder 16B
24.4%
64.4%
27.3%
42.5%
11.6%
18.0%
Gemma 27B
11.9%
58.8%
28.4%
48.3%
12.8%
22.1%
Codestral 22B
30.0% %
71.9%
31.7%
44.1%
12.3%
17.3%
Large Models
Cade 70B
18.8%
25.6% 60.6%
8.0%
32.8% 50.0%
1.8%
7.7%
Llama 3.1 70B
10.0%
30.3%
15.1 %
25.4%
Flagship Models
GPT-4o Llama 3.1 405B
5.0%
55.0%
29.7% 34.1 %
54.0% 45.9%
14.7% 14.7%
27.3%
18.1%
74.4%
19.9%
Table Full test suite generation. All results shown for temperature=0.2
Table 9 shows pass@1, 1, all pass@ I coverage and mutation score for small, medium and large mod- els. Similar TESTGENEVAL. smaller models perform significantly worse than their larger coun- terparts. All pass @ penalizes how verbose model is, thus less verbose models such as Codestral 22B have much higher all pass@1 than verbose models such as GPT-4o, which generate many tests.
Coverage and mutation score remain low across all models. Similar TESTGENEVAL, GPT-4o and Llama 3.1 40SB performs highly. Mutation score is even lower than coverage, implying that models are not able catch all induced bugs.
E.2.2 TEST COMPLETION
Table 10 shows pass @ and pass @5 for all settings. Similar to TESTGENEVAL split, model perfor- mance generally increases as more of the test file is provided as context (extra pass @5 is higher than shows the average pass @5 and coverage improvement across all three settings. Similar to GESTGEEEVAAL models struggle to add coverage to an existing test suite, performing best at first generation, where there is no coverage (or minimal coverage from test setup) and thus it is easiest to add new coverage. For the last and extra test generation testing, the model generates nearly 0 new coverage, testing already tested functions, rather than tempting to test new functions. This represents a frontier of test generation (improving coverage of already high coverage test suite). Similar TESTGENEVAL. Codestral has higher average pass @5 5 than GPT-4o, despite GPT-4o generating tests with higher coverage.
both last @ and first @5.. Table 11 IT
Figure 8a Figure 8b Figure T8c show the number of unique generations for each model across temperatures 0.2 and 0.8. Similar to the full TESTGENEVAL, CodeL_lama generates the fewest unique solutions, while models such as Llama 3.1 8B generate large number of unique solutions.
28

---

Published as a conference paper at ICLR 2025
Table 10: Pass rates for different models in first, last, and extra completion settings. Pass I temperature=0.2, P is for temperature=0.8.
for
First
Last
Extra
Model
Pass
Pass@5
Pass@1
Pass@5
Pass 1
@
Small Models
CodeLlama 7B
6.9%
25.0%
7.5%
22.5%
6.9%
21.9%
Gemma 9B
6.9%
18.1%
18.8%
45.0%
25.0%
45.6%
Llama 3.1 8B
14.4%
41.2%
35.0%
56.2%
31.9%
51.2%
Medium Models
DeepSeekCoder 16B
18.8%
48.8%
20.6%
60.0%
13.1%
60.6%
Gemma 27B Codestral 22B
16.2%
35.6%
31.9% 51.2% %
63.7%
30.6%
63.1%
43.1 %
67.3%
75.0%
47.5%
75.0%
Large Models
CodeLlama 70B
1.2%
26.9% 48.1%
0.0% 36.2%
61.9% 59.4%
0.0% 41.2%
56.9%
Llama 3.1 70B
21.9%
64.4%
Flagship Models
GPT-4o
36.2%
66.9%
26.9
60.6%
26.2%
61.3%
Llama 3.1 405B
33.8%
60.0%
38.8%
68.15 1. %
40.6%
71.2%
Table Coverage improvements, including coverage improvements for passing tests, and average pass @5 for first, last, and extra completion settings. Models struggle to improve coverage, and in general report substantially worse average pass than p
First
Last
Extra
Model
+Cov
Cov
Avg
+Cov
+Cov@P
Avg@S
+Cov
+Cov @
Avg
Small Models
CodeLLama 7B
9.3%
37.3%
5.8%
0.0%
0.1%
5.0%
0.0%
0.0%
5,3%
Gemma
7.9%
43.7% %
6.8%
0.1%
0.2%
17.1%
0.1%
0.1%
16.1%
Llama 3.1 8B
16.3%
39.5%
10.7%
0.0%
0.053
26.4%
0.1%
0.1%
23.8%
Medium Models
Deep'Scck Coder 16B
20.6%
42.3%
17.9%
0.1%
0.1%
21.5%
0.1%
0.1%
20.1%
Gemmna 27B
15.5%
43.6%
14.5%
0.1%
0.1%:
29.1%
0.0%
0.0%
27.5%
Codestral 22B
26.4%
39.4%
32.5%
0.2%
0.3%
48.0%
0.1%
0.1%
43.8%
Large Models
CodeLlama 70B Llama 3.1 70B
8.7%
33.3%
7.0%
0.0%
0.0%
17.0%
0.0%
0.0%
15.7%
18.8%
39.6%
16.9%
0.1%
0.2%.
29.1%
0.1%
0.1%
32.5%
Flagship Models
GPT-4o
28.2%
42.2%
34.4%
0.3%
0.6%
28,
0.3%
0.5 %
28.1%
Llama 3.1 405B
20.9%
35.2%
33.4%
0.2%
0.45%
41.3%
0.1%
0.2%
43.9%
(a) First setting
(b) Last setting
(c) Extra setting
Figure 18:
Unique generations by model and temperature for first, last and extra completion settings (out of maximum of 5 generations).
29

---

Published as a conference paper at ICLR 2025
F ANALYSIS
F.I CORRELATION WITH OTHER BENCHMARKS
F.1.1 TESTGENEVAL
16
lestGentvall cover agude1 vo Nummnttvall paesegg)
Nos Nostienbvall coverapeer1 vi insiival pass
Commation Confficent 0.88
rerreiation Confiiiire:
Hunerivvll p11009
644
Tumily
Samily
llama
gomina
gomma
nagerip
TestCenEvvll coverage@::
TesiGlnEvvll coverageg3
(a) Correlation with HvalnEval
(b) Correlation with TestEval
TestGenEval coverage@1 pass@!
Comclation Confficione
other
codallama
smal
medium
large
flapohi
TestDanEval coveragegi
(c) Correlation MMLU
Figure 19: Correlation with HumanEval (a SOTA code generation benchmark), TestEval (a SOTA test generation benchmark) and MMLU (a NLP benchmark). We find that there is a weak positive correlation between HumanEval and TESTGENEVAL, and with the exception of Gemma 9B there a similar weak positive correlation between TestEval and TESTGENEVAL. MMLU correlation is weaker, with code generation performing much better on TESTGENEVAL than MMLU.
We perform correlation analysis between TESTGENEVAL in our full test generation setting and other benchmarks. We select two benchmarks for comparison: HumanEval (a widely used code generation benchmark) and TestEval (a existing test generation benchmark).
Figure 19a shows correlation between TESTGENEVAL and HumanEval. Correlation between Hu- EvnEval (a code generation task) and TESTGENEVAL is higher with a correlation coefficient of 0.86. The major outlier here Code with a very low TESTGENEVAL. and HumanEval score. Another outlier is Gemma 9B, with lower HumanEval score than expected given its TEST- GENEVAL score. The moderate correlation illustrates that there is still information added by adding TESTGENEVAL, but that it does indeed correlate with code generation performance.
Figure 19b shows correlation between TESTGENEVAL and TestEval, a smaller scale test generation benchmark. Generally all models tend to perform, well on TestEval, however Gemma 9B performs significantly worse than expected. This because the instruct model does not follow the prompt provided by TestEval authors, instead generating code snippets. Additionally, there does is minimal
30

---

Published as a conference paper at ICLR 2025
differentiation between models, with models performing highly on TestEval (Llama 3.1 70B and CodeLlama 70B), but significantly worse on TESTGENEVAL.
Figure 19c shows correlation between TESTGENEVAL and MMLU, state of the art natural lan- guage processing benchmark. Code generation models such as Codestral and DeepSeekCoder per- form significantly worse MMLU than TESTGENEVAL. Otherwise, the performance is gener- ally correlated.
F.1.2 TESTGENEVALLITE
TestSenEval caverage@1 vs HumanEval passg]
Test coverageig passgl
Humeriena po00e1
04-
Teath pangt
limm
other
other
codetiama
coce tama
small
small
medium
medium
large
large
Tagatrp
RageVpp
0.0
coverageas)
coveragees
(a) Correlation with HumanEval
(b) Correlation with TestEval
TestGennvall cover agog1 va MMLU passgi
Eorresation Conffican:: 0.78
family
loma
gonma
other
32
small
large
faachi
TestGenEval coveragegi
(c) Correlation with MMLU
Figure 20: Correlation with HumanEval (a SOTA code generation benchmark), TestEval (a SOTA test generation benchmark) and MMLU (a NLP benchmark) with TESTGENEVALLITE. Correla- tions are similar to the full split of our benchmark.
Figure 20a Figure 20c and Figure 20b display the correlations with HumanEval, MMLU, and TestEval datasets, respectively, in the TESTGENEVALLTE dataset. Similar to TESTGENEVAL, we find that HumanEval has the highest correlation of any benchmark, however there are some outliers such as GPT-4o and Llama 3.1 8B having a higher HumanEval score. MMLU correlation is less strong, with code models such as Codestral and DeepSeckCoder scoring significantly lower on MMLU than on TEsTGENEVAL. Finally, TestEval has the lowest correlation, with Gemma 9B failing to follow the TestEval prompt.
31

---

Published as a conference paper at ICLR 2025
(a) Full-first correlation
(b) Full-last correlation
(c) Full-extra correlation
Figure 21: Correlations between the full dataset and the first, last, and extra subsets in the
SWEBench dataset
F.2 CORRELATION BETWEEN SETTINGS AND MODELS
F.2.1 TESTGENEVAL
Figure 21a Figure 21b and Figure 21c show the correlations between the full dataset and the first, last, and subsets, respectively, the TESTGENEVAL. Our full test generation setting has relatively high correlation with other settings, however it is not perfect correlation, with some models performing better on one setting than others.
(a) First-last correlation
(b) Finst-extra correlation
(c) Last-extra correlation
Figure 22: Correlations between the full dataset and the first, last, and extra subsets in TEST- GENEVAL
Figure 22a Figure 22b and Figure 22c show the correlations between the test completion settings. First and last and first and extra test completion settings have a highly positive but not perfect corre- lation. Interestingly, last generation and extra generation have perfect correlation, indicating that models performing well in one setting will in the other setting.
Figure 23 shows the correlation of all models in their ability solve the TESTGENEVAL full test generation setting. Models in the same families have the highest correlation (Gemma 9B and Gemma 27B for example, and Llama 3.1 8B and Llama 3.1 70B). Gemma 27B and Codestral 22B also very correlated in their ability to solve problems in the full test suite generation setting.
Figure 24a Figure 24b and Figure 24c show the correlations of all models in their ability solve first, last, and extra test completion tasks in TESTGENEVAL. Trends from the model predictions in the full test suite generation continue here, with models the same family most correlated in their ability to solve tasks. Llama 3.1 405B and Codestral 22B also share some similarities in their first, last and extra test settings problems solved, as well.
F.2.2 TESTGENEVALLITE
Figure 25b and Figure 25c show the correlations between the full test generation and the first, last, and extra completion settings, respectively, in the TESTGENEVALLITE split. Similar to TESTGENEVAL, correlation remains high between the full test suite generation and test completion settings. However, there are still cases where a model better at one setting than others, indicating there is information gained by having all settings.
Figure 25a
32

---

Published as a conference paper at ICLR 2025
Figure 23: TESTGENEVAL full model correlation
(a) TESTGENEVAL first model correlation
(b) TESTGENEVAL last model correlation
TESTGENEVAL extra model correlation
Figure 24: Correlations for various test completion settings between all models in the TEST- GENEVAL.
(a) Full-first correlation
(b) Full-last correlation
(c) Full-extra correlation
Figure 25; Correlations between the full test generation and the first, last, and extra completion settings in the TESTGENEVALLIEE split
(a) First-last correlation
(b) First-extra correlation
(c) Full-extra correlation
Figure 26: Correlations between the full dataset and the first, last, and extra subsets in the TEST- GENEVALLITE split
33

---

Published as a conference paper at ICLR 2025
Figure 26a Figure 26b and Figure 26c show the correlations between the test completion settings. The first test completion setting is highly correlated with both the last and extra test completion set- tings. Last test completion is nearly perfectly correlated with extra test completion performance. We hypothesize this is because the tasks are very similar, with the only difference being one additional test added the context.
618
416
Figure 27: TESTGENEVALLITE full model correlation
(a) TESTGENEVALLITE first model correlation
(b) TESTGENEVALLITE model correlation
last
(c) TESTGENEVALLITE extra model correlation
Figure 28: Correlations for various test completion settings between all models in the TEST- GENEVALLITE split. We omit CodeLlama 70B in settings where it has pass@1 1 of O.
Figure 27 shows the correlation the correlation of all models in their ability solve the TEST- GENEVAL full test generation setting. Similar to TESTGENEVAL. models in the same family (Llama 3.1, Gemma) exhibit the highest correlation between different sizes, larger models solve more of problems than smaller models, but still solve similar types of tasks.
Figure 28a Figure 28b and Figure 28curee88cshe
28c show the correlation of all models in their ability to solve the TESTGENEVAAL..A first, last, and extra test completion settings. Similar to the full test generation, models the same families are highly correlated their ability to solve similar problems. Codestral is also more correlated with Llama 3.1 8B than other models for the extra test completion task.
F.3 EFFECT OF SAMPLING ON COVERAGE AND PASS@K
Figure 29a and Figure 29h show how performance the full test generation and first, and extra test completion settings changes
tests sampled. For full test suite generation, find that coverage at k gradually increases, with no noticeable plateau the first 20 generations. The curve is slightly steeper at the beginning, but still looks linear as k approaches 20. Sampling 20 generations leads to approximately a 10% increase in overall coverage (coverage 20 23.3%).
For test completion, find that pass @k is most substantial in the first tests, but slowly increases after. @ass@ 100 performance is substantially higher than pass @ and pass@5 performance, indi- cating models capable of solving test completion tasks, but with many attempts. The plot curve
34

---

Published as a conference paper at ICLR 2025
for
first, settings
11.5
2444
644
200
15
02
(a) Coverage@M vs k (full test generation)
(b) Pass @k vs (test completion)
Figure 29: Effect of sampling more tests for all settings on Llama 3.1 8B. Coverage @k seems to gradually increase. Pass k also increases more for lower k values and seems to plateau after k=20.
is steep initially, and then decays in value as more samples are added (first test completion plateaus around 80% pass @K and last and extra completion plateaus around 90% pass@k).
F.4 EFFECT OF CONTEXT ON COVERAGE AND P
We measure the effect of context length on both coverage for our full test generation setting and for pass @5 for our test completion setting.
coverage vs context length for full setting
passg5 vs context length for first, lstt and extra settings
345
Lest
Litra
22000 -2063
400.00
00003
130000
130030
2000 80000
61304
B0000
14:000
12000
(a) Coverage by context length (full test generation)
P by context length (test completion)
Figure 30: Effect of context window for both our full setting (coverage of generated test suite) and for our test completion setting (pass@5 for our first, last and extra completion settings). We find that context length generally helps test completion, however for test generation, even receiving parts of the file under test (measured by a lower context window) seems to be effective.
Figure 30a and Figure 30b show the effect of context length both our coverage (full test gener- ation) and on pass @5 5 (test completion). For full test suite generation, context does not help much; seeing part of large file with many classes enough to generate covering tests. Since tests can mock inputs in the partial test file, this partial context is sufficient to generate large test suite. How- ever, for test completion context size helps significantly more. This is because to complete next test in file one needs knowledge of both the code under test and the existing test file. Full context ensures that the mapping between existing tests and the code under test is clear, improving performance of completing additional tests.
F.5 QUANTITATIVE ERROR ANALYSIS
F.5.1 TESTGENEVAL
We bucket errors based off the logs from executing tests. Specifically for setting and generation scrape the logs and extract the Python error class. Figure3T]showe the most frequent errors made by the best performing model for test completion, Codestral 22B: specifically assertion, assertion,
35

---

Published as a conference paper at ICLR 2025
Top errors for Codestral 22B
5000
4000
Riequency
2000
2000
1000
Assertonnror
No Asserts
InderttationErrr
NamwErrr
TypeError
Figure 31: Different errors made on TESTGENEVAL by Codestral 22B (the best model on test completion). Despite high performance, Codestral struggles to reason about execution (large number of assertion and timeout errors).
timeout, indentation and name errors. As evidenced by the large number of assertion and timeout errors Codestral 22B struggles to reason about code execution. There are also cases where Codestral 22B generates tests that do not contain assert and simply invoke the method under test. We filter these out and label such cases as an error.
10200
80200
40000
20300
Atrert
Intertedrry
(a) Top emrrr across models
(b) Top errors by model
asserts by model
25000
13000
10000
Colelition
Cooktieed 700t
Gemmee 278
Gevesta
Liam ro
Golestal
oPtic
6018
Ulema
(c) No assert errors by model
Figure 32: Common errors across all models. We find that the most common enrors include not generating tests with asserts, followed by execution and hallucination related errors.
Figure 32a Figure 32b and Figure 32c show examples of the most common errors made by all models in TESTGENEVAL. We find that all models struggle the most with following the prompt, generating tests that lack assertions. After this eтror, the most common types have to do with code execution (assertion, timeout errors), model hallucination (name error) and formatting (indentation error).
36

---

Published as a conference paper at ICLR 2025
F.5.2
TESTGENEVALLITE
Top Codestral 22B
700
600
500
400
EosertionEror
No Assens
IndeentationErro
Nametrrer
TypeError
Figure 33: Different errors made on TESTGENEEALLIIEE by Codestral 22B. Results are similar to TESTGENEVAL.
Figure 33 shows the most frequent errors made by our best performing model Codestral 22B. Error counts remain the same TESTGENEVAL.
lop aerrors across all models
8020
2020
Avertention
Bysteterr
(a) Top 5 errors across all models
(b) Top 5 errors by model
No asserts by model
2000
Genydaeec HG
demmena Ail
Comitral 236
urrea
Cridd
Coeedieeee
Genoma
1.3
Laena
Mode
(c) No assert errors by model
Figure 34: Common errors across all models. Results are similar to TESTGENEVAL.
Figure 34a Figure 34b and Figure 34c show examples of the most common errors made by all models in TESTGENEVALLIrE. Similar TESTGENEVAL. find that all models struggle the most with following the prompt, generating tests that lack assertions and making both execution and hallucination errors.
37

---

Published as a conference paper at ICLR 2025
F.6 QUALITATIVE MODEL COMPARISON
We outline three cases where TESTGENEVAL discriminates between GPT-4o, CodeStral and Llama 3.1 405B (only one model suceeds in each of these examples).
F.6.1 EXAMPLE TEST SETUP (ONLY SOLVED BY GPT-40)
class QuerySet (AltersData)
def
model-None. query-None, ...);
def repr (self):
return "<$s $r>" (self. (self._class class
name
data)
Listing 4: Query set class for managing data from database.
Our first example involves QuerySet class that manages records returned from database. The class has no database dependencies. The initialization takes a model and query, which can also not be set no database is being used. This is hard test because involves either mocking or creating a complex model object.
def
(self)
queryset QuerySet (model=None) assertEqual (repr (queryset), "<QuerySet []>")
Listing 5: Cut GPT-4o generated test
GPT-4o instantiates the QuerySet with no parameters (the simplest possible version of the test). It then tests all code methods, with all of the generated tests passing on the code under test.
from django. import connection, models
def dest_queryyset_iie
(self):
model = models..m Model Model () qs QuerySet (model=model)
salf.ase self.assertEqual (qs model, model)
Listing 6: Cut Llama 3.1 405B generated test
This is the case with both Llama 3.1 405B and Codestral 22B. Both models ignore the empty case entirely and instead hallucinate invocations or imports of models in Django (instead these models should have mocked the model object and tested the null case). All models failing properly mock the class under test's 's dependencies lead to low coverage of the source file, despite GPT-4o generating passing tests all when other models fail.
F.6.2 EXAMPLE 2 INCORRECTLY MOCKING OBJECTS (ONLY SOLVED BY LLAMA 3.1 405B)
class MiggatioonAucodettttto. def [siif_ from_state, to_state, questicner=None)
silffexisting_apps (app for app, model in pooo__state.mdeell
def changes (self, graph, trim to to_apps-None, convert_apps-Nong
Listing 7: Migration detection class to automatically detect changes needed between project states,
Our second example involves a MigrationAutodetecte class that takes in two project states and mea- sures the changes between the states passed in automatically detect code that should be migrated.
38

---

Published as a conference paper at ICLR 2025
def test_changes ():
class Model (models.Model) pass
changes = migration. changes (graph= trin_to_apps-None) self f.lffas assertEqual (len (changes), class="syntax-uubber
Listing 8: Cut Llama 3.1 405B generated test
Llama 3.1 405B generates passing test for the changes method, providing an empty graph and model, meaning the only difference between the from and states is the addition of the model. The assert passes, as the method only outputs one change.
def test_init ()
from_state Magi (spec-Projectstatee)
to to_state MaqicMock (spec=Projectttttte) autodetector
(self.from_state
self. to_state, self quw.ttioeeonnn
sel f.lff.sse assse.ssst (autodetector, from_state, self, from_state)
Listing 9: Cut GPT-4o generated test
GPT-4o attempts to mock the ProjectState class (passing instance checks of the ProjectState class). However, it misses the line in initialization where the models attribute of from.state is accessed. This leads the setup to fail, and the ingrrationAutodtteeee initialization fails. Codestral 22B makes similar error, where it incorrectly mocks the ProjectState class and misses the model attribute.
F.6.3 EXAMPLE 3 HANDLING CLASS DEPENDENCIES (ONLY SOLVED BY CODESTRAL 22B)
class De
class Exact (FiielddetDDbrreeaaaaeetttiiii Builtinlookup) def process_rth (self, compiler, connection)
Listing 10: Database classes to handle lookups and comparisons in queries.
Our final example deals with lookups and comparisons that could be applied when searching database (for example exact match, greater than, less than, etc.). The file under test has many classes and subclasses such as Exact, which depends on F seldGettbbrrepvaa ueMi xin.
def test_ exact lookup ()
1hs F ( field') lookup = Exact (1hs, 'value') self assertIn ('ss', ...)
Listing 11: Cut GPT-4o generated test
GPT-4o and Llama 3.1 405B both hallucinate class invocations, rather than mocking or using the classes provided context. not a correct class to pass as lhs (missing the required output field).
def test_exact ():
field Field(()
lookup = Exact (field, 'value') self.
(lookup. lookup_ _aokup__amm,,, 'exact')
Listing 12: Cut Codestral 22B generated test
Unlike GPT-4o and Llama 3.1 405B. Codestral 22B is able to understand the file under test. Code- stral 22B both imports and instantiates valid field under test that is also imported in the file under test and correctly instantiates the Exact class that has dependency the field.
39

---

Published as a conference paper at ICLR 2025
F.7 QUALITATIVE ERROR ANALYSIS
We manually examine errors made by Codestral (the top model for our test completion task) and highlight examples for the most popular errors for TESTGENEVAL. Generally, models struggle with reasoning about execution (high amount assertion errors) and hallucination (leads to timeout and value errors).
Assertion Errors: A common type of error is the model generating tests with incorrect asserts. This challenging, models often miss nuances in code execution and methods under test themselves.
def prepare_imga sipre_imponn ippor (path): :
"""Given a filename this will try to ca calculate the python path, checks the python path exists otherwi se returns senu
# First test generation ():
def
path = test_path / "simple_sai -y"" module_name prepare_import (str(pattt) assert module_name name == "simple_app"
Listing 13: Codestral first test method completion and code method
This example shows case where the code under test requires setup (creating directories), but the model fails to add the required setup (either mocking the os methods creating the required file in test path so that the import recognized).
def test_min maxima_ratio(1
clust1 OPTICS
or
(X)
clust2 OPTICS min maxima hara:a/eee...ttoassreerreaararan
fit (X)
assert not np.array_equal (clustl labelabela __labees__.
(clustt1.aael11
clust2. labels s_)
Listing 14: Codestral last test method completion
example where Codestral asserts that two objects that have identical labels are not equal. This is case where Codestral misunderstands the code under test, which does not include min maximaaaaaaraio in the labels.
No Assert Errors: Another common type of error is assert errors. We filter out generated tests that do not contain asserts.
def test www.wolllppp.. http:iipiiiiiihaappppeeee
fname http::/www....aaaaaaaaaaaeeeaeed ext sCI', e
()
# Open test FITS file:
pet__pkg_date filename ('
h = fits fits.open open (fname)
wcs wCS (h [ext]] header, h)
Listing 15: Codestral extra test completion (no asserts)
def hagp://aaaaattttoeeeeeaaaaaaaaa.
fig, ax = subplots (stassssttttttsssssssssstt-a "u
class="oynttxxmmmmee=2222
h p.pdecllassssspptt--umbbbbe"
40

---

data Published as a conference paper at ICLR 2025
clo.elose (fig)
Listing 16: Codestral first test completion (no asserts)
Another common type of error generating tests that lead to a degenerative state (infinite loops, timeout errors).
Timeout Errors:
def rest_server _est__erver_andder_cceeee Saaaalee__ceaau__eeeeeeee (self): from djjngo core servers. servers.basetttp import ServerHandler
class MockRequestHaaddee: class MockServer:
pass
self.server MockSreerrrreer ()
self.headers = {}
pass
server Sarver_handler.o www.cceeeeaaaaaaeee ()
self. In (' Connection', server_handles headers))
Listing 17: Codestral extra test method completion
This example shows case where Codestral incorrectly mocks a request handler, by putting self.server self.MockServer() in the class MockServer rather than in a initt__ method in Mock- RequestHandler. The dreauupheadee method requires that the headers are well formed to properly close the connection, and thus never actually closes the connection.
def Best_get ceee_tracebacc_daaee data (self):
request RequestFacttory(.oo get /*)
exc_type Temp:aaceDoeaNNtEEiist
exc_value TemplatteDoesseotxxis:: ("Error message")
tb = mock mock.Mock () reporter Exceptionneporter (request, exc_type, exc_value,
tb)
self assertrrer ('
ttanaatttttelaateede_____e _xisttt
Listing 18: Codestral first test method completion
This shows similar example with mocking. Because the mock will continuously return non None object when called by get_traceback_aat in loop (tb tb.tb_next), the model needs to mock the traceback correctly and ensure that non null returned.
Value Errors: Another common type of error is generating tests that invoke functions with incorrect arguments (a value error) or missing some precondition to the function.
def pest__imezone__eee
(self):
now t wimezone.noo future now ()
self.
future timezone.localtiime (future)
self. oene (now, future), "claag="syytaxx-numbe>>>> hour")
self aelff (tt.ttmmmmmeiniuuttreet (future, now), hour")
"clas
Listing 19: Codestral last test method completion
This example shows case where Codestral misses implementation detail of localtime in the code under test. Specifically, localtime cannot be applied on a naive datetime, instead a correct test would the Jocalize method convert datetime object to object with timezone.
41

---

Published as a conference paper at ICLR 2025
def test_get__pata
patha (stl)
view = JavaScriptCatalogg ()
paths = view.get_paths. (
self aastaaaassttttuulappppapah.
(path.joll (settings. BASESSBAAAE_DDDAA
"django/conf//ooclee")
with self.as.. aelf.assertRaiises (ValueError):
www.geet_pathh. ( ('nnexxistent (oonexistent_packgee]]
Listing 20: Codestral first test method completion
This shows different example. Specifically, the input 'django.conf" not a valid package that can be passed to the get-paths method. The code under test shows way to get valid paths (invoking the apps.get app configs method, but instead of following this, the model hallucinates a non-cxistan package).
def test_get_ppaths (selff
view = JavascriptCataloge ()
paths view.get_patie ( d'django. conf'
self. auaeeaasaautuuuaaaaaaaa.a
[path. (settings. BASE DIR,
/django/con//localee//
with self.assertRaises (ValueError):
view.get_patie ( nwnexistent__pacaaeee )
Listing 21: Codestral first test method completion
This shows different example. Specifically, the input (jjango.conf be passed to the get_paths method. The code under test shows a way to get valid paths (invoking the apps.get.app.com cop.cosfigsss method, but instead of following this, the model hallucinates non-existant package).
valid package that can
F.8 FRONTIERS TESTGENEVAL
We also show some code snippets on the frontier of TESTGENEVAL. Specifically we show two examples of code snippets that model can generate tests for and code snippets where only one model can generate tests.
No Solve Cases: Below are cases where model is capable of generating tests.
class Plane (GeometryEntity)
def ___eew___ (cls, pl, a=None, b=None, **kwargs)
pl Point3D dimeclass="syntax-mme ">3) if and b:
p2 Point (a, dimeeaasss="sstttxxxnmmmbee"""
p3 Point (b, dineclass==ssyttxxxxnmbrrrr if Point3D. www.:DDarreccolleer((pp. p2, p3)
raise ( three non-coll linear
points')
p1. rel.directtonnraaiio00 (p2)
b = rel.iirection_raat// (p3)
normal vector tuple (Matrix cross (Matrix (b)
else:
kwargs. swargs.pop// hormal_vectors a)
if is_ _s__equence (a) and len (a) =
class= Ellas=="sytaxx syntax www.sayyttttbbbeeaaa number">3:
normal_vector Point3D (a) args
else:
raise ValueBrror (filldedent ('")
Either provide class= Alass="syntaa" syntax-nunber""> 3D
points or point with
42

---

Published as a conference paper at ICLR 2025
normal vector expressed sequence of length clas=="eyytaxx-mmmb""""3"""
if all (coord.is zero for coord in normal_vectti raise ValueError (o NNormlll vector cannot be zero
vector')
return GeometryEntity. new (cls, pl, normal_vector,
**xwargs)
Listing 22: No solve problem (class with many nested methods)
The above class is an example of code file that no model is capable of generating for. This is because generating tests for such a class requires understanding complex execution (we only show one of many methods for brevity, but all methods involve complex execution sequences of methods). Furthermore, generating unit test for such a class requires intricate knowledge of the code under test or complex mocking all intermediate classes.
def write write_latex( latex ( cosmology, file, *, overwrite-Palse, cls-QTable, latex_namesee names=True,, **kwargs
format = kwargs pop ("format", "latex") if fornat not in ("latex", "ascii. : raise ValueBrror (f"format must be 'latex' or Vascii.latex', not (format)")
# Set cosmology_in_met as false for now since there is no metadata being kept table = to to_table (cosmology, cls-cis, haannooooy___mmtaaeeee
cosmo_cke cls type (cosmology) for name, col in table. columns. copy param getattr (cosmo_cls, name, None)
copy () .items ()
if not isinstance (param, Parameter) param, unit in (None, u. u.one):
continue
# Get column to correct unit table [name] <<= param. unit
Convert parameter names LaTeX format if latex_names: names:
new new_names FORMAT TABLE, get (k, k) for in
cosmo par.mterrssrrs__..
table. @ename_collmee coosmology.o patamolooyy___prmeeetee parameters_..... new_names)
table.write (file, overwrite=overrricc. format="latex"
** **xwargs)
Listing 23: No solve problem (dealing with 10/Files)
The above code file is another case where models fail generate valid tests. The most common issue across models for this class is incorrectly testing the output file, as the method under test writes its output to file. Checking this output requires intricate knowledge of the content written the file, or complex mocking of 10 operations. Ideally a model generating a unit test for this class would either need increased context, proper mocking of file writing.
43

---

Published as a conference paper at ICLR 2025
One Solve Cases: Below cases where only model is capable of generating tests (frontier problems for TESTGENEVAL).
class
(BaseTterable):
#Iterable that yields a model instance for each row from raw queryset.
class Vanee otg.aaattterriaaaaaaaassttttataa (ValuesListiterablee)
Iterable returned by QuerySet values_liesseeeee (named=True)
class
(RaseTterable): a
Tterable returned by QuerySet. vaaaet.vallun veee list (flat=True)
class QuerySet (AltersData):
GPT-40 test generation
class QuerySetTests (TestCase) (TassaCC)
def (est_queryse__eetttteeeett
(self): (sel))
state self.quezyset. wwwsttttt_e___.. ()
self.assertIIn (DJANGO 1 B_____EERIIIIIIIIIIIII PICKLE_KEY,, state)
self..assertRqual ( django._
def [est_queryeet_ttereseel]]]
self. (list (iter (self.querysell.
(I)
Listing 24: One solve problem (only GPT-4o)
This example highlights a case where order to write tests for the main class (QuerySet) the model must understand large number of classes defined in the code under test. GPT-4 successfully gen- crates tests for methods that return these additional classes.
class RawModelIterable (BaseIterable):
#Iterable that yields a model instance for each row from queryset.
class Sneevvaauestisstteeraa (valuestissttterbblee: tee
Iterable returned by QuerySet values_liss
class FlatValuestisttterbbbbl (BaseIterable): (Baattterrbblaaasstttttbbbbl
Iterable returned by QuerySet. values list (flat=True)
class QuerySet (AltersData):
GPT-40 test generation
class QussySetTests (TestCase) (Testtaase)
def t (self)
state ool PICKLE KEY], cjango.__ersion
getp:ettt____ttttt_tee_eeeee state)
def test test__uuersst_iteerssllll
self aslf.assertgquall. (list (iter (self quelf..uersstt) !I)
Listing 25: One solve problem (only GPT-4o)
44

---

Published as a conference paper at ICLR 2025
This example highlights case where in order write tests for the main class (QuerySet) the model must understand a large number of classes defined in the code under test. GPT-4 successfully gen- erates tests for methods that return these additional classes.
class Modelpperation (Operation): def init_ (self, name) self name name
@cached_propertyy def (see_))
return relf.namee lower
def references references_model model (self, name, app_ app_label): return name. lower () =
lower
def reduce (self, operation, app app_ label]:
return super () .redue (operation, app app_label) or raaas.can_ediieetthrrutet operation, app_label
def (ai_redue__hrrogghhsell,, operation, app_label). return not operation references references__mddel model (self. name, app label)
class CreateModel (Model0peration) "*"Create a model's s table. ***
class DeleteModel [Mode10peration) "*"Drop a model's s table.
Codestral 22B generated test
class TestModallperationn (unittest. TestCase)
def setUp
self.app_label. 'test_app'
self.model_name = 'TestModel'
self.fields [('id',
models ptrrr -ry=u_
self.options (' ('managed'; True]
self.bases (models Model,)
self.managers = []
def test_create_model (self)
operation CreateModel (self. name, self fields,
self.options, www.:eelf.bass... melf.mamagers)
self.mode self.model_name) name)
self..assertRgual (operation. fields,
Dalf..asserttquallqpeeeeet (cperation options,, self.options)
aelf.asserrt@gual (ow.aaaioooorerraaaao.ooe.e self.bases)
self self. aslf.assert (operatioooo www.oornmmaaarrsaaaaa self self.managers)
Listing 26: One solve problem (only Codestral 22B)
45

---

Published as a conference paper at ICLR 2025
This shows another example of a one solve problem. The challenge here is the subclassing of the Create.Model class. Codestral is able to successfully invoke these subclasses and come up with valid asserts.
46

---

Published as a conference paper at ICLR 2025
G
STATISTICAL TESTS
We run pairwise comparisons for both TESTGENEVAL and TESTGENEVALLITE splits of our data.
G.1 TESTGENEVAL
Table 95% confidence interval for pass@)) along with noise related statistics for each of TEST-
GENEVAL settings.
Setting
95% % Int.
No Solve
Tau-
Sig-Noise
Test generation
3.1%
4.7%
6.4%
2.70
Extra test completion
3,5%
12.0%
11.2%
1.21
First test completion
3.7%
25.9%
7.8%
1.87
Last test completion
4.4%
10.5%
10.4%
1.27
450
(a) TESTGENEVAL acc by problem
(b) Min elo solve problems
Figure 35: Problems by average model accuracy and min clo required to solve problems for full test generation setting
Figure 35a and Figure 35b show average model accuracies by problem, along with the min elo required to solve each problem for our full test generation setting.
Oevergg
Aumerr
(a) TESTGENEVAL cov by problem
(b) TESTGENEVAL acc per cov
Figure 36: Problems by average model coverage and average accuracy per coverage
Figure 37a and Figure 37b show the average model accuracies by problem, along with the min clo required to solve each problem for extra test completion setting.
Figures Figure 38a and Figure 38b show the average model accuracies by problem, along with the min elo required to solve each problem for extra test completion setting.
Figure 39a and Figure /9b]show show the average model accuracies by problem, along with the min elo required to solve each problem for our first test completion setting.
47

---

Published as a conference paper at ICLR 2025
(a) TESTGENEVAL acc by problem
(b) Min elo solve problems
Figure 37: Problems by average model accuracy and min elo required to solve problems for extra test completion generation setting
(a) TESTGENEVAL acc by problem
(b) Min clo to solve problems
Figure 38: Problems by average model accuracy and min elo required to solve problems for last test completion setting
(a) TESTGENEVAL acc by problem
(b) Min clo to solve problems
Figure 39: Problems by average model accuracy and min elo required solve problems for first test completion setting
(a) TESTGENEVAL.LLr acc by problem
(b) Min elo to solve problems
Figure 40: Problems by average model accuracy and min elo required to solve problems for full test generation setting
48

---

Published as a conference paper at ICLR 2025
Table 13: Model performance for the full suite generation setting. Coverage refers the cover- age of generated tests.
Model
Coverage
Win Rate
Elo
GPT-4o
35.2%
79.6%
1280.6
Llama 3.1 405B
35.0%
72.8%
1221.2
Codestral 22B
33.0%
65.8%
1161.2
Llama 3.1 70B
30.6%
66.3%
1168.8
Gemma 27B
30.1%
69.8%
1196.5
CoepSeekCoder 16B
28.2%
53.4%
1086.1
Gemma 9B
20.2%
41.2%
996.7
Ilama 3.1 8B
14.1%
27.1%
899.1
CodeLlama 70B
7.0%
7.3%
641.9
CodeLlama 7B
1.3%
1.4%
347.8
Table 14: Model performance for the first test completion setting. Pass @ generated tests pass.
refers to if any of the
Model
Pass
Win Rate
Flo
Codestral 22B
38.3%
80.4%
1288.1
Llama 3.1 405B
32.1% %
73.7%
1231.1
GPT-4o
31.9%
71.8%
1220.4
Llama 3.1 70B
19.3%
52.6%
1085.9
DeepSeekCoder 16B
18.6%
51.2%
1082.6
Llama 3.1 8B
14.4%
41.7%
1025.1
Gemma 27B
12.7%
37.6%
1003.0
Gemma
8.4%
25.9%
912.0
Codel Jlama 7B
4.2%
13.9%
774.6
Codellama 70B
0.5%
1.6%
377.2
G.2 TESTGENEVALLITE
Figure 40a and Figure 40b show the average model accuracies by problem, along with the min elo required to solve each problem for our full test generation setting.
Caeeage
03
(a) TESTGENEVALLITEE cov by problem
(b) LIT. acc per cov
Figure 41: Problems by average model coverage and average accuracy per coverage
show the average model accuracies by problem, along with the min elo required solve each problem for our extra test completion setting.
Figure 42a and Figure 42b
43 and Figure 43b required to solve each problem for our extra test completion setting.
Figure
show the average model accuracies by problem, along with the min elo
A. and Figure 44b show the average model accuracies by problem, along with the min elo required to solve each problem for our first test completion setting.
Figure
49

---

Published as a conference paper at ICLR 2025
(a) LESTGENEVALLITE acc by problem
(b) Min elo to solve problems
Figure 42: Problems by average model accuracy and min elo required solve problems for extra test completion generation setting
(a) TESTGENBVALLITE acc by problem
(b) Min elo solve problems
Figure 43: Problems by average model accuracy and min elo required to solve problems for last test completion setting
02500
(a) TESTGENEEALLITE ace by problem
(b) Min elo solve problems
Figure 44: Problems by average model accuracy and min elo required to solve problems for full test generation setting
50

---

Published as a conference paper at ICLR 2025
Table 15: Model performance for the last test completion setting. generated tests pass.
Pass@ refers to if any of the
Model
Pass
Win Rate
Elo
Codestral 22B
50.4%
79.1%
1258.6
Llama 3.1 405B
42.6%
72.1%
1195.1
Llama-3.1 70B
35.0%
62.1%
1122.9
GPT-4o
32.6%
58.0%
1103.5
Gemma 27B
32.2%
57.3%
1105.4
Llama 3.1 8B
32.0%
57.6%
1096.5
Gemma 9B
21.4%
40.8%
994.0
CoepSeekCoder 16B
17.0%
33.4%
944.7
Cada 7B
6.9%
13.9%
758.6
Cade 70B
0.9%
2.2%
420.7
Table 16: Model performance for the extra test completion setting, Pass@ I refers to if any of the generated tests pass.
Model
Pass
Win Rate
Elo
Codestral 22B
48.3%
77.9%
1 1262.1
Llama 3.1 405B
42.4%
72.9%
1216.4
Llama 3.1 70B
36.4%
65.6%
1160.7
Llama 3.1 8B
31.8%
59.0%
1121.8
Gemma 27B
31.7%
57.8%
1127.3
GPT-4o
30.4%
56.5%
1111.2
Gemma 9B
18.9%
38.2%
998.5
DeepSeek 16B
15.5%
32.2%
958.2
CodeLlama 7B
5.2%
11.6%
744.5
CodeLlama 70B
0.5%
1.0%
299.2
Table 17: 95% confidence interval along with noise related statistics for each of TESTGENEVAL- LITE settings.
Setting
95 % Int.
No Solve
Tau- Noise
Test generation
5.6%
0.6%
6.2%
1.16
Extra test completion
6.9%
11.2%
11.9%
0.31
First test completion
5.6%
22.5%
6.9%
0.87
Last test completion
7.5%
8.8%
10.0%
0.13
Table Model performance for the full test suite generation setting. Coverage refers the cover- age of generated tests.
Model
Coverage
Win Rate
Elo
Llama 3.1 405B
35.3%
77.3%
1245.2
Codestral 22B
34.9%
69.1%
1176.7
GPT-4o
32.9%
74.6%
1223.3
Llama 3.1 70B
31.3%
66.8%
1171.7
Gemma 27B
29.4%
64.8%
1148.5
DeepSeek Coder 16B
27.6%
48.6%
1045.2
Gemma 9B
20.2%
38.1%
968.0
Llama 3.1 8B
12.4%
17.1%
783.9
CodeLlama 70B
8.0%
10.9%
708.8
CodeLlama 7B
2.1%
4.1%
528.6
51

---

Published as a conference paper at ICLR 2025
Table 19: Model performance for the first test completion setting. generated pass.
Pass refers to if any of the
Model
Pass@)
Win Rate
Elo
Codestral 22B
43.1%
81.7%
1283.1
GPT-4o
36.2%
72.7%
1212.3
Llama 3.1 405B
33.8%
71.7%
1195.2
Llama 3.1 70B
21.9%
53.6%
1080.2
DeepSeekCoder 16B
19.4%
48.9%
1053.7
Gemma 27B
16.2%
42.7%
1012.9
Llama 3.1 8B
14.4%
38.3%
987.5
CodeLlama 7B
6.9%
20.2%
829.0
Gemma 9B
6.9%
19.3%
822.1
Code 70B
1.2%
4.0%
524.1
Table 20: Model performance for the last test completion setting. Pass refers to if any of the generated tests pass. We omit CodeLlama 70B because not able to complete any tests on first generation.
Model
Pass@1 1
Win Rate
Elo
Codestral 22B
51.2%
79.2%
1575.4
Llama 3.1 405B
40.0%
68.0%
1478.3
Llama 3.1 70B
37.5%
64.1%
1453.6
Llama 3.1 8B
36.9%
63.9%
1453.5
Gemma 27B
33.1% 1
57.6%
1424.4
GPT-4o
28.1%
51.0%
1373.3
DeepSeekCoder 16B
20.6%
39.9%
1310.2
Gemma 9B
19.4%
36.5%
1277.5
Codel 7B
7.5%
16.0%
1103.5
CodeLlama 70B
0.0%
0.0%
-2449.8
Table 21: Model performance for the extra test completion setting. Pass@ 1 refers to if any of the generated tests pass. We omit Cada 70B because not able to complete any tests on first generation.
Model
Pass@1 1
Win Rate
Elo
Codestral 22B
48.1%
78.1% 78.1%
1552.9
Llama 3.1 405B
42.5%
72.6%
1501.1
Llama 3.1 70B
41.9%
71.4%
1494.1
Llama 3.1 8B
33.8%
60.3%
1419.8
Gemma 27B
31.9%
56.4%
1411.4
GPT-4o
26.2%
48.8%
1350.0
Gemma 9B
25.6%
47.8%
1349.1
DeepSeekCoder 16B
13.8%
28.6%
1221.7
CodeLlama 7B
6.9%
13.5%
1058.2
CodeLlama 70B
0.0%
0.0%
-2358.4
52

---

Published as a conference paper at ICLR 2025
H LIMITATIONS
Variation due to prompt and temperature Model performance is highly dependant on the prompt and temperature used Gao et al. (2024): Wei et al. 2023). A potential limitation of our work that we primarily focused on 0-shot performance, asking each model to generate the entire test suite or to complete a test given the code under test. To mitigate this we provide our prompt in Appendix and also provide the model with the correct imports to the code under test enable prompted models to generate the test suite successfully.
Overfitting to SWEBench repositories: Another potential limitation is that our benchmark is adapted from SWEBench and as a result risks models overfitting to this specific dataset. Currently, this does not seem to be a major issue, as model performance low across the board. Even once models can achieve high coverage, a significantly harder task of achieving high mutation score (ac- tually catching synthetic bugs introduced into the code under test). These multiple levels of difficulty and numerous tasks help mitigate the risk of a model overfitting to any one task specifically.
There also risk of data contamination in the pretraining data of models. To further understand data contamination, we measure perplexity of 10 randomly selected tests in TEsTGENEVAL for Llama 3.1 8B and common frequent, and non recent code from GitHub with similar lengths. We find that the perplexity of this common GitHub code is lower than the 10 tests from TESTGENEVAL (1.6 vs 2.0), indicating that data unlikely to be contaminated. This is further supported by the low performance of all models TESTGENEVAL across the board.
Data contamination:
File level context: TESTGENEVAL. currently works at the file level. This means that the context given to the model is the source file, imports and for the test completion setting partial test file. While in many cases, this context is sufficient sometimes there are cross file dependencies that are not captured, making the context provided to the model limited. However, believe this is not major limitation, as models can mock objects which they lack context for. Furthermore, expanding to repository level context is out of context window for most state-of-ther models.
Additional baselines: One other limitation is the lack of agentic baselines. This might be a promis- ing future work direction in overcoming some of the limitations of the context windows of current models, and could potentially perform better than baselines showed. However, the benchmark is still highly valuable to the community, as the first file level test generation benchmark.
One other limitation is the compute cost of computing mutation score. Each synthetic bug we introduce the code under test, requires an additional test suite execution. However, our results show that coverage and mutation score are highly correlated. Setting timeout of one hour per mutation testing run, we only timeout on 20% of files, and get an average uncertainty of 1.1% %. We provide an option to run TESTGENEVAL. without mutation, enabling those who lack compute to still benefit from TESTGENEVAL.
Compute cost of mutation score:
Quantitative metrics such as code coverage and mutation score can approximate the quality of generated tests, however they do not perfectly measure the quality of generated tests. For example, test could be very hard to read, while still achieving high coverage and mutation score. Additionally, all test generation benchmarks assume the code under test correct. Generated tests may fail on the code under test, while exposing bugs in the code under test (a phenomenon known as the oracle problem). Despite this, current test generation approaches are still useful in catching regressions bugs introduced in future versions of the code under test.
Mise Limitations:
53


Reviews:
[{"review_id": "Ink4zyU3nR", "reviewer": "Reviewer_g3uj", "summary": "This paper introduces a new LLM for test generation benchmark, TestGenEval. It builds on the same repositories as SWEBench, but focuses on extracting the test files and code-test pairs. TestGenEval studies two tasks: test suite generation (generating an entire test suite given a codebase) and test completion (generating test methods to a partial test suite). The generated tests are not only compared against human-written tests based on similarity, but also based on code coverage and mutation score, which are more important metrics for test quality. This paper benchmarked several recent LLMs across different sizes on TestGenEval, and found that generating and completing test suites are challenging tasks for current LLMs.", "questions": "- What is the key difference between the newly created benchmark and the evaluation sets used in prior work on ML for test generation/completion (e.g., Rao et al. 2023, Nie et al. 2023, Dinella et al. 2022, Tufano et al. 2020)? Namely, Rao et al. 2023 had a pretty large dataset for pre-training, and evaluated on the task of generating first/last/additional tests. Both Rao et al. 2023 and Nie et al. 2023 also used runtime metrics (compile and pass).\n\n- line 142, \"we next extract code test file pairs from the code and test files run in the PR\": do you only consider the tests that appeared in PRs in SWEBench? Or all the tests in those repositories?\n\n- line 189, \"This setup is in line with current test completion work Rao et al. (2023); Nie et al. (2023).\": this may not be true. The \"test completion\" in Rao et al. 2023 and Nie et al. 2023 was statement-level completion, but yours is method-level completion, which sounds more like the \"test method generation\" task in Rao et al. 2023.\n\n- Which mutation tool and mutants did you use? And how many mutants do you generate for each example?\n\n- For test suite generation task, consider renaming \"Pass @ 1\" to \"Any Pass @ 1\" to reduce confusion.\n\n- line 361, \"the most common error is not generating tests with asserts\": but will that lead to a runtime error, and if not, how do you detect them? Also, do you count the generated tests without asserts as pass or not pass?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Included the measurement of code coverage and mutation score as part of the evaluation metrics, which are important metrics for measuring the quality of generated tests.\n\n+ Conducted a comprehensive set of experiments using LLMs of different sizes and different configurations.\n\n+ Performed an interesting analysis on the current LLMs' performance on TestGenEval, including the correlation with existing benchmarks and common error types.\n\n+ Paper is well-written and easy to follow.", "weaknesses": "- Not sure if it is the best to include only the tests appeared in the PRs used by SWEBench; this may miss the other tests that did not appear in those PRs (i.e., don't have recent bug-fixing changes), making the data collection process biased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jD1s3IX7h2", "reviewer": "Reviewer_Yezt", "summary": "This paper constructs TestGenEval, a comprehensive and proper benchmark for testing unit test generation abilities of existing LLMs. The benchmark unprecedentedly incorporates mutation scores into their evaluation metric. It also considers multiple generation scenarios including full, first, last, and extra test generation. 10 models including a massive Llama 405B is evaluated on the benchmark and detailed results are given in the main paper and the appendix. \nThis paper is leaning towards the rejection side since 1. the author did not make effort to evaluate if the generated test matches the definition of a unit test and merely add in mutation score to evaluate further the test effectiveness, and 2. the model selected for evaluation is questionable.", "questions": "- Why is the massive Llama 405B selected in the evaluation rather than other better-performing models like Claude and Gemini?\n- What effort does the benchmark made to ensure the test matches the conventional definition of a unit test?\n- Why isn't any agents/tools evaluated in the evaluation?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The introduction of mutation score improves on previous benchmarks and ensures a more proper benchmark\n- The work introduces the most comprehensive benchmark dedicated to test generation so far\n- The work evaluated on a large set of models, even on a very expensive 405B Llama\n- Extensive quantitative/correlation analysis, contamination check, and also detailed qualitative analysis, which shows the good quality of the dataset", "weaknesses": "### Major\n- The evaluation runs the 405B Llama model, which is quite impressive. However, it is unclear why is Llama model evaluated but not other closed-source ones (which are much easier to run comparing to Llama 405B) ones like Claude and Gemini.\n- The paper acknowledges that test generated by the LLM can be very different from what human might write. However, there is no procedure in the paper to guarantee that the model generates actually “unit tests” as deemed by human developers (except for the prompt). This means that the model can just generate one big complicated test case for all of the generation scenarios considered (full, first, last, extra), which is against the purpose of unit tests in the first place.\n- The task of test generation is generally more solvable by agents or by providing example usages. However, the paper only evaluates on vanilla models and didn’t evaluate even the simple case of adding dependencies into the prompt related to the task.\n\n### Misc\n- Citations are not wrapped in brackets (Like Jain et al. when discussing R2E), which is mildly annoying", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1mCqbj1aY", "reviewer": "Reviewer_FhSJ", "summary": "The theme of this paper is the introduction and evaluation of a large-scale benchmark named TESTGENEVAL, which is designed to measure the performance of test generation and test completion. \n\nTESTGENEVAL is constructed based on the SWEBench dataset and includes 68,647 test cases from 1,210 code and test file pairs across 11 well-maintained Python repositories. \n\nThis benchmark aims to fill the gap in existing code generation language model (LLM) benchmarks that focus less on software testing, despite the strong correlation between well-tested software and effective bug detection. \n\nTESTGENEVAL covers tasks such as initial test authoring, test suite completion, and code coverage improvement, aiming to comprehensively assess test generation performance. \n\nThe paper also evaluates several popular models with parameter sizes ranging from 7B to 405B and provides a detailed analysis of TESTGENEVAL's contribution to the evaluation of test generation performance.", "questions": "1. I would like to know what is the cost involved in executing TestGenEval? Including length of prompt, required GPU memory, running time and more cost analysis?\n\n2. How do the authors ensure the quality of the dataset, including but not limited to possible leakage of data and semantic correctness of the given code?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Continuous Improvement of Test Suites: TESTGENEVAL supports not only the generation of test suites from scratch but also the completion of existing test suites, which is particularly important for test optimization in continuous integration and continuous deployment (CI/CD) processes.\n\n2. Evaluating and Comparing Different Models: TESTGENEVAL provides a standardized environment to evaluate and compare various code generation models, including both open-source and proprietary models, helping researchers and practitioners understand the performance of different models in real-world software testing tasks.\n\n3. Test Generation for Real-World Use Cases: TESTGENEVAL is built based on real-world projects, meaning it is closer to actual development testing needs rather than being limited to academic research or toy problems.", "weaknesses": "1. Dependence on Prompts and Temperature Settings:\nModel performance is highly dependent on the prompts and temperature parameters used. The paper focuses mainly on 0-shot performance, asking each model to generate an entire test suite or complete a test given the code under test, which may limit the model's performance.\n\n2. Risk of Data Contamination:\nThere is a risk of data contamination in the pre-training data of models. Although the paper suggests that data contamination is unlikely by comparing the perplexity of tests in TESTGENEVAL with common GitHub code of similar length, it remains a potential issue.\nIn addition, since TESTGENEVAL is adapted from the SWEBench dataset, there is a risk of models overfitting to this specific dataset. \n\n3. Computational Cost:\nThe computational cost of calculating mutation scores is high. Each synthetic bug introduced to the code under test requires an additional test suite execution.\n\n4. Assumptions of Test Generation:\nAll test generation benchmarks assume that the code under test is correct. Generated tests may fail on the code under test while exposing bugs in the code under test (a phenomenon known as the oracle problem).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JsbNFbsKNH", "reviewer": "Reviewer_4d8B", "summary": "This paper introduces TestGenEval, \na novel benchmark designed to evaluate neural test generation models on real-world Python projects. \nTestGenEval is built upon 11 popular Python repositories selected from the patch generation benchmark SWE-Bench.\nIt assesses neural test generation from two key perspectives: test generation from scratch and test completion based on some existing test functions. \nBoth aspects, particularly test completion, are valuable for real-world software engineering applications, such as IDE plugins. \nThe authors evaluate several popular LLMs on TestGenEval, \npointing out that even the best-performing model GPT-4o struggles with achieving high code coverage.", "questions": "Pleased address the concerns in **Weakness**.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* A standard, real-world project-based benchmark for test generation is crucial and useful for advancing software testing research.\n* TestGenEval is the first benchmark to incorporate mutation score as a metric, a commonly used measure in software engineering to assess the robustness of test cases.\n* It offers Docker images instrumented with coverage and mutation score, ensuring the reusability of the artifacts.\n* TestGenEval is also the first benchmark to evaluate file-level test completion, which aligns well with the real-world software development workflows.\n* The authors perform a comprehensive quantitative analysis of the results, demonstrating the correlations and effects of various components within the benchmark.", "weaknesses": "* This paper lacks details on mutation score calculation.\n  The authors do not provide specifics on how mutation score is measured in the benchmark. \n  In traditional software engineering, mutation testing is creating mutants of the code under test to evaluate the robustness of the test cases in detecting these modifications. \n  However, the paper does not clarify the process used to construct program mutants for test generation, \n  which is a non-trivial task that requires further clarification.\n\n\n* The authors claim that real-world unit testing involves reasoning over complex *files*, \n  generating tests for a given *class under test*. \n  The authors also point out that the other related benchmark only measure the individual test method rather than an entire test suite is a weakness. \n  This claim is not well supported with reference nor evidence, \n  and is not commonly adopted by the software engineering community. \n  Unit testing aims to validate the correctness of the smallest building block (and hence, an \"unit\") in a complex software project.\n  In modern software engineering, the \"units\" of software are often functions in the codebase.\n  If these functions are tested/verified to be correct, then the composition should be correct without further testing.\n  Therefore, the best practices often suggest to write functions that are independent and de-coupled.\n  The more recent and popular programming languages, for example Rust, often abandon class and favor function composition\n  over inheritance.\n  Are these languages not related to \"real-world unit testing\"?\n\n* The authors claim that real-world unit testing requires reasoning over complex *files* and generating tests for a given *class under test*. \n  They argue that other benchmarks' focus on individual test methods,\n  rather than entire test suites, is a weakness. \n  However, this claim is not well-supported by references or evidence and is not commonly adopted by the software engineering community.\n\n  Unit testing aims to validate the smallest building blocks of software, or so-called \"units,\" \n  which are often functions rather than entire classes. \n  In modern software engineering, if individual functions are tested and verified to be correct, \n  the composition of them is expected to be correct without additional testing. \n  Consequently, best practices emphasize writing independent and de-coupled functions.\n\n  Furthermore, more recent programming languages, such as Rust, \n  favor function composition over inheritance, reflecting a shift away from class-based design. \n  This raises the question of whether these languages fall outside the scope of \"real-world unit testing\" as claimed by the authors, \n  despite their increasing relevance in software development.\n\n* The lack of quantitative analysis results in the main text is a significant weakness. \n  While the authors conduct five different quantitative analyses, all results are placed in the Appendix. \n  If an analysis is mentioned in the main text, it would be more effective to present the corresponding results alongside the discussion. \n  For space consideration, the authors could include only the most important analyses and results in the main text, \n  while moving less critical details to the Appendix. \n  This approach should also be applied to qualitative analyses for better readability and impact.\n\n\n* Minor writing issue: the authors cite all references without using parentheses, regardless of context. \n  In some cases, using `\\citep` instead of `\\cite` would enhance the reading experience by making the citations more contextually appropriate. \n  Adjusting this would improve the flow and clarity of the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)