Learning to Generate Unit Test via Adversarial Reinforcement Learning
Authors: Dongjun Lee
Changho Hwang
Kimin Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Code generation
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Unit testing is a core practice in programming, enabling systematic evaluation of programs produced by human developers or large language models (LLMs). Given the challenges in writing comprehensive unit tests, LLMs have been employed to automate unit test generation, yet methods for training LLMs to produce high-quality unit tests remain underexplored. In this work, we propose UTRL, a novel reinforcement learning (RL) framework that trains an LLM to generate high-quality unit test given a programming instruction. Our key idea is to iteratively train two LLMs, the unit test generator and the code generator, in an adversarial manner via RL: (1) the unit test generator is trained to maximize a discrimination reward, encouraging it to produce tests that reveal faults in the code generator’s solutions; and (2) the code generator is trained to maximize a code reward, encouraging it to produce solutions that pass the unit tests generated by the unit test generator. In our experiment, we demonstrate that unit tests generated by Qwen3-4B trained via UTRL show higher quality compared to unit tests generated by the same model trained via supervised fine-tuning on ground-truth unit tests, yielding code evaluations that more closely align with those induced by the ground-truth tests. Moreover, Qwen3-4B trained with UTRL outperforms frontier models like GPT-4.1 and GPT-4o in generating high-quality unit tests, highlighting the effectiveness of UTRL in training LLMs for the unit test generation.

Published as a conference paper at ICLR 2026
LEARNING TO GENERATE UNIT TEST VIA ADVERSAR- IAL REINFORCEMENT LEARNING
Dongjun Lee' Changho Hwang", Kimin Lee' dgjun320kai st. ac.kr
KAIST Microsoft Research
ABSTRACT
Unit testing is a core practice in programming, enabling systematic evaluation of programs produced by human developers or large language models (LLMs)). Given the challenges in writing comprehensive unit tests, LLMs have been em- ployed to automate unit test generation, yet methods for training LLMs to pro- duce high-quality unit tests remain underexplored. In this work, we propose UTRL, a novel reinforcement learning (RL) framework that trains an LLM to generate high-quality unit test given a programming instruction. Our key idea is to iteratively train two LLMss, the unit test generator and the code generator, in adversarial manner via RL: (1) the unit test generator is trained to maxi- mize a discrimination reward, encouraging it to produce tests that reveal faults in the code generator's solutions; and (2) the code generator trained to maxi- mize a code reward, encouraging it to produce solutions that pass the unit tests generated by the unit test generator. In our experiment, we demonstrate that unit tests generated by Qwen3-4B trained via UTRL show higher quality compared to unit tests generated by the same model trained via supervised fine-tuning on ground-truth unit tests, yielding code evaluations that more closely align with those induced by the ground-trus sounddtruth tests. Moreover, Qwen3-4B trained with UTRL outperforms frontier models like GPT-4.1 and GPT-4o in generating high-quality unit tests, highlighting the effectiveness of UTRL in training LLMs for the unit test generation. Code and relevant materials available at project page: https: //gp::://///nnnn/ github. wwt.ti.i.otuu......eeeooooo.ooee io/UTRL.
INTRODUCTION
Unit test is a critical component in program- ming, as it enables evaluation and verifi- cation on the functional correctness of the program produced by human developers or large language models (LLMs). Recently, unit tests are widely used as verifiable re- ward functions in reinforcement learning (RL) or test-time scaling for LLM--driee code generation, where the unit tests pro- vide task-specific feedback signals for guid- ing the generation of functionally correct code (Le et al., 2022: Guo et al., 2024).
Programming Instruction
Unit test
Generator
Rdisc
Discrimination reward
Tegur Argences
Aurin cananiing
Code reward
Reode
lng Aesmitir
Code
Generator
Figure 1: Overview of UTRL. The unit test genera- tor is trained to generate unit test that detect fault in code generated by the code generator, and the code generator is trained to produce code that passes the generated unit test.
However, implementing unit tests for each programming task is labor-intensive and challenging. since (1) unit test should con- tain functionally valid test cases, and (2) each test case in the unit test should cover challenging edge cases, capable of discriminating subtly faulty code implementations. These re- quire significant level code reasoning capability and understanding of the programming task.
Work done during internship Microsoft Research.

---

Published as a conference paper at ICLR 2026
As LLMs have shown strong code understanding and generation capabilities, recent works have explored training them to generate high-quality unit tests given programming task. A common ap- proach is to collect instruction unit test pairs, followed by supervised fine-tuning (SFT). To enable scalable annotation of high-quality unit tests, these works propose strategies such as using capable teacher models (Ma et al., 2025); applying code perturbation techniques (Prasad et al., 2025); or adopting generator-validator frameworks (Wang et al., 2025b). While promising, SFT-based meth- ods are difficult scale across diverse programming domains as they fundamentally require unit test labels, which are costly to obtain, for every training example. In contrast, RL removes the need for explicit unit test labels by directly optimizing models against reward signals, offering a more scalable paradigm for training LLMs in unit test generation. The key challenge, however, lies in de- signing reward function that can reliably assess the quality of generated unit tests without relying on ground-truth annotations.
In this work, we present UTRL, an adversarial RL framework iterating over training a unit test generator LLM and code generator LLM in an adversarial manner, each LLM defining a reward signal for its counterpart. Our main contribution lies in designing the reward for training unit test generation (i.e.. discrimination reward), which rewards the unit test generator LLM when the gen- erated unit tests successfully discriminate the code solution produced by the code generator LLM from ground-truth code solution. As a result, UTRL eliminates the need for ground-truth unit test annotations, since the reward is defined by ground truth code solution and code generated by the code generator LLM. Specifically, as illustrated in Figure 1, the unit test generator is optimized to maximize the discrimination reward, which encourages producing unit tests that can detect code generated by the code generator L.LM from the ground-truth code, while the code generator is opti- mized to generate code that passes all test cases in the generated unit tests. Through this adversarial training, the code generator LLM progressively learns to generate code solutions that are harder to distinguish from the ground-truth code solutions, and the unit test generator LLM, in turn, learns to generate highly discriminative test cases that can detect subtle faults in the near-correct code solutions, thereby becoming capable of covering challenging edge cases.
In our experiments, we evaluate the quality of generated unit tests on the TACO evaluation set (Li et al., 2023), which contains competitive programming tasks collected from multiple online judge platforms. We first show that unit tests generated by Qwen3-4B (Yang et al., 2025) trained with UTRL. when used as a code reward function for best-of-N sampling, induce 3.1x x higher code accu- racy gain, compared the unit tests generated by the base Qwen3-4B. Additionally, we demonstrate that Qwen3-4B trained via UTRL achieves unit test quality higher than the same model trained via SFT, despite not relying on unit test annotations from humans or more capable models, and even surpasses frontier models such as GPT-4.1 (OpenAI, 2025). Finally, the code generator adversar- ially trained with the unit test generator achieves code generation performance comparable to the same model trained to maximize the pass rate over ground-truth unit tests.
2 RELATED WORK
Unit test generation with LLM Unit testing has been extensively explored for reliable software development (Fraser & Arcuri, 2011; Alagarsamy et al., 2024). As LLMs show strong capability in programming, recent works have explored the automation of unit test generation using large language models (LLMs), with focus on evaluating the test generation capability of LLMs (Jain et al., 2024a; Wang et al., 2024), iteratively refining the unit test by leveraging coverage analysis or mutation testing as a contextual feedback (Alshahwan al., 2024; Altmayer Pizzorno & Berger, 2025; Dakhel et al., 2024; Foster et al., 2025), or utilizing co-evolutionary algorithm (Li et al., 2025). Another line of work investigates training LLMs for unit test generation via supervised learning, with a focus on constructing high-quality unit test annotations at scale. CodeRM (Ma et al., 2025) leverages stronger teacher models to generate unit tests automatically, while UTGEN (Prasad al., 2025) proposes a data collection strategy that perturbs code solutions to create faulty variants, and then collects high-quality test cases that discriminate between the correct and faulty code. The work most closely related to ours is CURE (Wang et al., 2025a), which explores the co-evolution of code generation and test case generation capabilities via reinforcement learning. However, our approach differs from CURE in two key aspects: (1) Unlike CURE, which assumes access dataset annotated with ground-truth unit tests, UTRL requires only instruction pair dataset, which is readily available at large scale (Li et al., 2023; Lambert et al., 2024; Prasad et al., 2025),
2

---

Published as a conference paper at ICLR 2026
and (2) UTRL is framework aimed at training LLMs to generate an optimal set of test cases, rather than a single test case.
Reinforcement learning for LLM Early works on training LLMs with reinforcement learning (RL) utilized reward models learned from human preference feedback and trained LLM with the Proximal policy optimization algorithm (PPO) (Schulman et al., 2017) for aligning LLMs with hu- man values, establishing reinforcement learning with human feedback (RLHF) as standard frame- work for instruction tuning and alignment in LLMs (Stiennon et al., 2020; Ouyang et al., 2022). To address the complexity and instability of online RL, subsequent methods proposed offline prefer- ence learning algorithms that bypass the need for reward models while maintaining strong alignment performance (Rafailov al., 2023; Ethayarajh al., 2024; Hong et al., 2024). More recently, RL al- gorithms with improved efficiency have been proposed as a variant of PPO, and such RL algorithms have been actively applied to improve the reasoning capabilities of LLMs, particularly in domains such as math and programming where verifiable rewards can be precisely defined (Shao et al., 2024; Guo et al., 2024; Yu al., 2025). However, different from code generation or math, defining the verifiable rewards for unit test generation is non-trivial and remains underexplored.
Adversarial learning and self-play Adversarial learning provides general framework in which models improve by competing against an adaptive opponent, even enabling model to learn complex data distribution (Goodfellow et al., 2014). Building on this idea, self-play has emerged as a power- ful paradigm where the opponent derived from the model itself. line of works has demonstrated that symmetric self-play can yield superhuman performance in strategic decision decision-making (Silver et al., 2017), while another line of work explores asymmetric formulations in which agents gen- erate and solve tasks to construct an automatic curriculum (OpenAl et al., 2021). With the rapid progress of LLMs across diverse domains, self-play has recently been extended to LLM-based sys- tems, enabling improvement with minimal even zero additional data. AZR introduce a zero-data self-play framework in which a task generator and a task solver co-evolve to improve the model's reasoning ability (Zhao et al., 2025). In alignment, SPIN applies self-pla to alignment by itera- tively contrasting LLM-generated responses with human responses, enabling improvement without costly preference data (Chen et al., 2024). In code generation, Sol-Ver utilizes self-play between solution generator and a test generator to jointly improve code and unit tests (Lin et al., 2025). In theorem proving. STP leverage self-play between a conjecture generator and a prover enhance theorem-proving capability of LLM (Dong & Ma, 2025).
METHOD
In this section, we present UTRL., a framework for adversarially training a unit test generator LLM and a code generator LLM via RL. Section 3.1 describes an overview of the UTRL, and the core components of UTRL are described in Section 3.2 and Section 3.3.
3.1 OVERVIEW
Our goal is train unit test generator LLM MIr, which takes programming instruction / as input and outputs a unit test T, as shown in Figure 2. A unit test T consists of multiple test cases, i.e., T {T]}. where each T; denotes single test case. Each test case Tद checks the partial correctness of a code implementation C² for instruction I, by verifying that C' produces the expected output for a given input. straightforward approach to training Mur is collect a dataset of instruction-uni test pairs and fine-tune LLMSs via supervised learning. However, curating such dataset is inherently challenging because implementing unit tests is an open-ended task with no unique solution, and is often difficult to objectively evaluate their quality.
To address these challenges, we propose a framework for training a unit test generator LLM My without relying instruction-unit test pairs. Since instruction-code solution pairs are widely avail- able, we train the Myr via RL to produce unit tests that can distinguish ground-truth code from imperfect code generated by code generator LLM, Moode- We further extend this idea into an adversarial framework: Mir improves by learning to generate unit test identifying weaknesses in code generated by Mcade, while Meods in turn improves by learning to produce code that passes increasingly challenging unit tests.
Specifically, we iterate over the following two key steps (see Algorithm 1):
3

---

Published as a conference paper at ICLR 2026
Programming instruction
Unit Test
Yeu had oamos matre houce pamiingtie
remiiing [Resseing Se tnt 1] streeiiing sassct (CA,)) (1,111)
uuter manarqueeee: grodiam canti delladd susiiaa prodacte you adiioa ofter dusatt uscedd air tite ancver danathesae
"neeeing" (Raaaiing fe tol stonaiing sasset
23, 4331
Generator
MUT
cremcsing (Ronwcing for ose chonseing sasectil, 2, 4, [C), 4), (3, 4311
Canauiaa
$1-215 1XS
SUSESS 10113415
reesceigg (Reewoinng fo use NJ chesooinng sonnetit, So 108311
2.. 16.11
Figure 2: Illustration of input and desired output of unit test generator LLM. Given programming instruction specifying input arguments and functionality of a code, the unit test generator LLM generates a set of N test cases comprehensively covering the various edge cases based on reasoning.
Algorithm UTRL
Require: Dataset instruction-code solution pairs D = [(T,, C2)).
Initial code generator LLM Moode, Initial unit test generator LLM Mor. Weight for the discrimination reward A,
Desired minimum number test cases per unit test r
1: while converged do
Training unit test generator Mer (sce Section 3.2)
3:
for (I.C") E do
4:
T T~ (-1 C~ Moatl
Generate unit test T using Mor Generate of code solutions C using Mende Roaana r), Mer == : Rudd=t- rur)
5:
6:
Pur Pur Rami
Compute reward for (sce Equation 1, 2) b Update the unit test generator Mor
7:
8:
end for
9:
I/ Training code generator Moose (see Section 3.3)
10:
for (I,C*) E D do
ON Meode (-12) Muru
b Generate code C' uuinn Meode Generate unit test T using Mor D Compute reward for C' (see Equation 4) Update code generator Monit
12:
13:
Toode Haode C,
14:
Monde RI www-uuuatteeMMM... Tends)
15:
end for
16:
end while
Step 1. Training the unit test generator Mor: For given programming instruction I, we first sample multiple code solutions using the code generator LLM MUT. The unit test generator LLM Mør is then trained to produce unit tests that (1) discriminate many sampled code solutions from the ground-truth solution as possible, and (2) contain functionally valid test cases (sce Section 3.2).
Step 2. Training code generator Mcode:
Given the programming instruction I, the code generator Moode is trained to generate solutions that maximize the pass rate over T, which generated by Mur (see Section 3.3).
By repeating these steps, the code generator progressively learns to produce more functionally cor- rect solutions, while the unit test generator learns generate increasingly discriminative and high- quality unit tests.
3.2 TRAINING UNIT TEST GENERATOR
To train the unit test generator LLM MoT to produce both effective and functionally valid unit tests, we introduce two reward components. The discrimination reward Ruise serves as the primary objective, encouraging Myr generate unit tests that can detect faults across diverse incorrect code solutions. To simultaneously ensure the functional validity of each test case, we introduce the validity reward Realid, which measures the proportion of functionally valid test cases among the entire test cases in the generated unit test. During training, the unit test generator Mer is optimized with the weighted sum of Raise and Rvaid via RL.
Discrimination reward The discrimination reward evaluates how effectively the test cases T discriminate I.L M-geerrrreedd code solutions from ground-truth code solutions. Given unit test T
4

---

Published as a conference paper at ICLR 2026
Task I
Mcode
MUT
Sample ( 6) eade solutions
C1
C5
C
Unit Test T
Ti
Pass
Fail
Pass
Pass
Puss
T2
Fail
Pass
Pass
Fail
Fail
Pass
Pass
73
Pass
Fail
Pass
Puss
Fail
Pass
T42
Fail
Jexcial
Pass
Discrimination reward: (= # detected code M)
Filter functionally valid test cases using ground-truth code solution (*
Compute discrimination reward
Figure 3: Overview of the process of computing discrimination reward with respect unit test T. First, among the test cases the unit test T, test cases that pass under the ground-truth code C* (i.e., T1, T3, Tτ) are filtered, forming set of functionally valid test cases. Second, the discrimination reward is defined as a ratio of sampled code solutions that are detected by least one valid test case. In this figure, among 6 sampled code solutions, 4 code solutions (C1, C3, C4, C3)) do not pass at least one valid test case, resulting in the discrimination reward of 0.667 (= e).
generated regarding programming instruction I, the discrimination reward corresponding to T is formally defined as follows:
1 II -
Raise (T,C,
Σ
(L= Pasa TET
(I)
C CEC
where the C* ground-truth code solution, C is a set of code solutions generated by the code generator Mcode, and (a,s((.T) T) is an indicator function that returns if the code C passes the test case T, and 0 otherwise." To compute the discrimination reward regarding unit test T, as illustrated in Figure 3, we first filter out functionally invalid test cases from the generated unit test T, ensuring that invalid test cases do not affect the discrimination reward. We then leverage a set of code solutions C generated by Mesde (i.c., C ~ Meedee I)), where the discrimination reward is computed as a fraction of the code solutions that do not pass the filtered unit test.
Validity reward The validity reward evaluates whether each test case in the generated unit test T is functionally valid (i.e., correctly maps the input and expected output of the code solution), and is formally defined as follows:
Reatia (T, TT ΣTEr Pass (C*,T)
max([T],77)
where is hyperparameter to adjust the desired minimum number of test cases in single generated unit test. We apply clipping to the denominator in Equation 2 as max T). in order to prevent unit tests with only small number of trivial test cases from receiving an inappropriately high validity reward. To be specific, if the Realid is simply defined as a fraction of valid test cases among total test cases (i.e., (T, = ΣTET Pa T)) unit tests with an extremely small number of test cases (c.g., unit tests with a single trivial test case) acquire high validity reward, which is undesirable. To mitigate this problem, we clip the denominator as max([T], 1, T). This ensures that unit tests with fewer than test cases receive validity reward proportional to the absolute number of valid test cases, thereby preventing the unit tests with a small number of trivial test cases from receiving high validity reward.
Update unit test generator LLM test comprising test cases that are both (1) highly discriminative and (2) functionally valid, we define a training reward as a weighted sum of the discrimination reward Raise and the validity reward R
In order to guide the unit test generator LLM to produce unit
TUT ABasse (T,C, C.C,C C*) + (1 A) (T, T,C C* C²,T),
where the À hyperparameter that weights the discrimination reward.
See Appendix A.3 for details about the implementation of Pass function.
5

---

Published as a conference paper at ICLR 2026
3.3 TRAINING CODE GENERATOR
After training the unit test generator LLM Mur, we train the code generator LLM Mcode via RL. Regarding code C' generated by Monaa the training reward formally defined as follows:
Redde (lade(C,,, (C, Erer Pass Passs(()
(4)
Σrer Pass(C*,
where C* is ground-trutt truth code solution. In detail, first filter functionally invalid test cases (i.e., test cases that do not pass under the ground-truth code solution C*) from T. because such invalid test cases lead to faulty evaluation of the code. We then measure the proportion of the functionally valid test cases that are passed by the generated code C. Based on the reward design, we optimize Meode produce code solutions that pass unit tests generated by Mur.
4 EXPERIMENTS
We design our experiments to investigate the following questions:
RQ1. Is UTRL effective at training LLMs to generate high-quality unit tests? (see Section 4.2)
RQ2. Is UTRL more effective than supervised learning-basee basee approaches? (sec Section 4.3)
RQ3. Is UTRL effective at improving code generation? (see Section 4.4)
RQ4. Does iterative training of UTRL enable continuous improvement of unit test generation capability? (see Section 4.5)
4.1
EXPERIMENTAL SETUP
In this section, we describe training details, baselines, and evaluation protocols for our experiments.
Training details As the base model for both the code generator and unit test generator, we use Qwen3-4B (Yang al., 2025). For RL training, adopt Grouped Relative Policy Optimization (GRPO; Shao et al. 2024 (see Appendix A.1 for details) and use 15,249 programming instruction- code solution pairs from the TACO dataset (Li al., 2023). Further details about training and prompts are described in Appendix A.3 and A.5.
Baselines We compare our method to various baselines for unit test generation. Implementation details for the baselines are provided in Appendix A.3.
LLM baselines without fine-tuning: We use 4 open-source LLMs (Qwen3-4B. Qwen3-8B. Qwen3-14B, Qwen3-32B, Deepseck-Coder Coaeeece-CCoer---eee Vodi-----ter----iee Qwen et al. 2025; Zhu et al. 2024) and closed LLMs (GPT-40, GPT-4.1: Hurst et al. 2024; OpenAl 2025) baselines!.
Supervised fine-tuning (SFT): We fine-tune Qwen3-4B with instruction unit test pairs through supervised learning. For this purpose, we use high-quality unit tests from the TACO training dataset Dur- To investigate whether reasoning signals can improve unit test generation, we construct a reasoning-augmentee dataset DreusmmUT containing unit tests paired with their cor- responding reasoning. Specifically, first generate reasoning and unit tests for each training task using Gemini-2.5- 2.5-flash (Google Deepmind, 2025).² We then filter functionally valid rea- soning-ted case pairs and use them target labels for SFT training.
We also consider CURE (Wang et al., 2025a), an RL algorithm that fine-tunes an LLM to perform both test case generation and code generation using a dataset of instruction-ume test pairs. Specifically, we evaluate unit tests generated by ReasonFlux-Codde Cokk--CodrBBBBB (BWag et al., 2025a), which is based on Qwen2.5-7B-Insttee
CURE:
(Yang et al., 2025) fine-tuned with CURE.
Ground-truth (GT) unit tests: We also compare against the unit tests provided in the ben- chamrk datasets (Li et al., 2023; Jain et al., 2024b), which we consider as ground truth (GT) and regard an upper bound since they are rigorously verified and consist of large number of high-quality test cases covering diverse edge cases.
note that open-source L.LMs used as baselines are instruction -inetttoon---euueedddd models. Instead of attaching reasoning the original unit tests in TACO, which led to frequent hallucinations and inconsistencies, we regenerate both reasoning and unit tests together.

---

Published as a conference paper at ICLR 2026
Unit test generated by
Code LLM: Qwen3-8B (32 shot)
Code LLM: Qwen3- (32 shot)
Score
AScore
Acc (%)
AAce (%pp)
Score
AScore
Acc (%)
Ace (%p)
Qwen3-4B
0,430
+0.084
9.8
+1.8
0,459
+0,057
13.4
+2.7
Qwen3-8B
0,435
+0.089
9.7
+1.7
0.470
+0.070
12.6
+1.9
Qwen3-14B
0.460
+0.114
9.8
+1.8
0.483
+.0883
11.6
+0.9
Qwen3-32B
0.484
+0.138
11.7
+3.7
0.514
+0.114
13.7
+3.0
Deepseek-Coder-V2-liin
0.433
+0.087
9.5
+1.5
0.452
+0.050
11.9
+1.2
GPT-4o
0.474
+0.128
10.9
+2.9
0.502
+0.100
11.5
+0.8
GPT-4.1
0,537
+0.191
13.3
+5.3
0.572
+0.170
15.1
+4.4
Qwen3-4B + SFT w/ Dur
0.458
+0.112
11.7
+3.7
0.491
+0.091
14.0
+3.3
Qwen3-4B + UTRL (ours)
0.578
+0.232
14.9
+6.9
0.610
+0,208
17.3
+6.6
Owen3-14B- UTRL (ours)
0.599
+0.253
15.0
+7.0
0.627
+0.225
17.7
+7.0
GT (upperbound)
0.668
+0.304
20.3
+13.1
0.699
+0,278
24.2
+13.7
Best-of-N improvement achieved by Qwen3-4B and Qwen3-14B trained via UTRL,, com- pared against baselines. We report code score (Score) and code accuracy (Acc) of the best-of-N selected code solution, and also report the increment of each metric compared code generated without the best-of-N sampling (i.e.. AScore and AAcc) This result demonstrates effectiveness of UTRL training LLMs produce unit tests with highly discriminative test cases.
Table 1:
Evaluation protocols For evaluation, we utilize rigorously verified 945 competitive programming tasks provided in the TACO evaluation dataset (Li et al., 2023) and 511 programming tasks provided in the LiveCodeBench--2 (Jain et al., 2024b). To measure the quality of the generated unit tests, we propose the following evaluation schemes:
Best-of-N improvement: To examine whether the generated unit tests include high-quality test cases, we perform best-of-N sampling for code generation, using the generated unit tests as evaluators. Specifically, we generate N candidate solutions from code LLMs and select the one that passes the largest number of generated unit tests. To evaluate the quality of the selected code, we then measure two metries: (1) code score, defined as the fraction of test cases in GT unit test (i.e., high-quality unit tests from evaluation dataset) that the code solution passes, and (2) code accuracy, which indicates whether the solution passes test cases in the GT unit test.
Unit test fidelity: To quantify how closely generated unit test approximates the evaluation induced by the GT unit test, introduce metric called unit test fidelity. For each task, first sample multiple code solutions from LLMss. We then evaluate these solutions twice: once using the generated unit test and once using the GT unit test. This yields two code score vectors, and we compute Spearman's 's correlation between them. A high correlation indicates that the generated unit test closely replicates the evaluation induced by the comprehensive GT unit test.
We provide more details about the evaluation metries in Appendix A.4.
4.2 QUALITY OF UNIT TESTS
In this section, demonstrate the effectiveness of UTRL in training LLMs to generate high-quality unit tests, compared with strong baselines. We first assess the quality of the generated unit tests using Best-of-N improvement and unit test fidelity. We then compare UTRL against recent RL-based baseline, CURE.
Best-of-N improvement Table reports the best-of-N improvement achieved by Qwen3-4B and Qwen3-14BB trained with UTRL, compared against several baselines. When used as code evaluators for best-of-32 sampling with Qwen3-8B and Qwen3-14B code generators, unit tests produced by Qwen3-4B trained with UTRL yield code accuracies of 14.9% and 17.3%, respectively. For com- parison, Qwen3-4B trained with SFT achieves only 11.7% and 14.0%. Moreover, Qwen3-4B trained with UTRL outperforms frontier proprietary models such GPT-4.1 and GPT-4o, underscoring the effectiveness of UTRL in training LLMs to generate high-quality unit tests. With increased model capacity, Owen3-14B trained with UTRL attains stronger performance, reaching code accuracies of 15.0% and 17.7%. Notably, UTRL is effective even when paired with source code generators such GPT-4o (see Table 8 in Appendix A.2 for supporting results).
7

---

Published as a conference paper at ICLR 2026
Unit test fidelity
Qwen3-14B UTRL
0.827
Qwen3-4B UTRL
0.794
Qwen3-4B SFT w/ DuT
0.566
GPT-4.1
0,800
GPT-4o
0.726
DeepSock-Coder-e Coder-V2-Lite
0.551
Qwen3-14B
0.641
Qwenn3-885
0.532
Qwen3-4B
0.528
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Spearman's correlation (-1 to 1)
Figure 4: Fidelity of unit tests generated by Qwen3-4B and Qwen3- 14B trained with UTRL, com- pared against baselines. Both model trained via UTRL achieves the unit test fidelity higher than baselines, demonstrating the effectiveness of UTRL in training LLMs to generate unit tests that closely approximates the code evaluation induced by the GT unit tests.
Unit test
Code LLM: Qwen3-8B (32 shot)
Code LLM: Qwen3- 14B (32 shot)
generated by
Score
AScore (%)
Ace (%)
AAce (%p)
Score
AScore
Ace (%)
AAcc (%p)
CURE UTRL (ours)
0.446
+0.100
9.7
+1.7
0,483
+0.083
12.7
+2.0
0,521
+0.175
14.1
+6,1
0.562
+0.162
15.9
+5.2
GT (upperbound)
0.650
+0.304
21.0
+13.1
0.680
+0.278
24.4
+13.7
Table 2: Best-of-N improvement induced by UTRL and CURE. For UTRL, we train Qwen2.5- IB-Instrect using 4.5K programming tasks in CodeContest dataset (Li et al., 2022) following the training setup of CURE, which we denote UTRL
Unit test fidelity Figure 4 presents the fidelity of unit tests generated by Qwen3-4B trained with UTRL. compared against several baselines. Qwen3-14B and Qwen3-4B trained with UTRL achieves Spearman's correlation of 0.827 and 0.794, where Qwen3-14B trained via UTRL out- performs GPT-4.1, demonstrating the effectiveness of UTRL in training LLMs to generate unit tests that induce code evaluations consistent with the GT unit tests. Interestingly, while SFT with Dur provides substantial gains in best-of-N improvement relative to the base Qwen3-4B it yields only marginal improvements in unit test fidelity. We conjecture that this is because SFT with Dur trains the LLM to generate unit tests without step-by-step reasoning, making it difficult for the model to produce logically structured test suites (i.e., ranging from basic test cases highly discriminative edge cases), which are essential for achieving high unit test fidelity.
Comparison to CURE Table 2 and Figure 6 re- port the best-of-N improvement and unit test fidelity achieved by Qwen2.5-7B-Instruu trained with UTRL versus the same model trained with CURE, an RL instruction-unit test pairs. When used as a code evaluator for best-of-32 sampling with Qwen3-8B and Qwen3-14B code generators, UTRL achieves code accuracies of 14.1% and 15.9%. which 4.4% and 3.2% higher than those obtained with CURE. Moreover, UTRL achieves a unit test fidelity of 0.593, surpassing CURE. These results show that UTRL achieves superior performance despite relying only on instruction-code pairs, in contrast to CURE requiring instruction-uni test pairs.
Unit test generation methods
Qwen2.5-7B-lnsttoes CURB Qwen2.5-7B-Instee UTRL
baseline based
0.593
0.576
0.45
0.50
0.55
0.60
Spearman's is correlation
Figure 6: Unit test fidelity of UTRL com- pared against CURE. Qwen2.55-BB-Inttmue trained via UTRL achieves unit test fi- delity higher than CURE, demonstrating superiority of UTRL over CURE.
4.3 SUPERIORITY UTRL OVER SUPERVISED LEARNING APPROACHES
To investigate whether training LLMs with reasoning-annotater annotated unit tests via SFT can improve unit test generation, also evaluate SFT with D Dre and compare with UTRL. Figure 5 presents
8

---

Published as a conference paper at ICLR 2026
Unit test generation methods
Qwen3-4B SFT Du
Qwin3-4B SFT DrasessU"
Qwen3-4B UTRL (Ouns)
Best-of-N improvement
Unit test fidelity
Code Acc. (%)
15
10-
0.648
Qwen3
Qwen3-81
Qwen3-14B 14B
GPT-4a
0.0
0.2
0.4
0.6
0.8
Code generation model
Spcarman's correlation
Figure 5: Best-of-N improvmenet (left) and unit test fidelity (right) achieved by UTRL, compared against SFT baselines (SFT w/ DUT. SFT w/ DrensonAUT).
best-of-N improvement (left) and unit test fidelity (right) for Qwen3-4B trained with SFT baselines versus the same model trained with UTRL. While SFT + DrensonUU achieves an improved unit test fidelity score of 0.604, higher than the SFT + DLT. falls behind 0.741 achieved by UTRL. Additionally, in best-of-N improvement, SFT + Deasom=I/ shows performance even inferior to SFT Dur. We conjecture that this is because the training dataset DreasomLLT comprises synthetic test cases generated by Gemini-2.5-Fiiat rather than highly discriminative ground-truth unit tests that constitute DUT Hence, the model trained with Preasoo-UT generates relatively less discriminative unit tests compared to the same model trained with Dur, which results in inferior best-of-N improvement, since Best-of-N improvement largely determined by the presence of highly discriminative test cases. Although SFT baselines directly optimize LLMs using unit test labels annotated by humans or more capable teacher models, they show limited generalization to evaluation tasks compared with UTRL. This observation is consistent with prior findings that SFT tends memorize the training distribution. whereas RL promotes better generalization in reasoning intensive tasks (Chu et al., 2025). Moreover, since collecting high-quality unit annotations for SFT is costly, UTRL offers more practical approach for training LLMs for unit test generation.
4.4 EFFECTIVENESS OF UTRL IN TRAINING CODE GENERATOR
In UTRL, the code generator is adversarially trained against the unit test generator to produce code solutions that maximize the pass rate over the generated unit tests. To evaluate the effectiveness of UTRL for training the code generator, we measure the pass@1 code accuracy of code solutions generated by Qwen3-4B trained via UTRL., and compare against two code generator training baselines: (1) RL using rewards defined as the pass rate over unit tests generated by GPT--4o, and (2) SFT using ground-truth code solution provided in the training dataset. Moreover, we also evaluate Qwen3-4B trained via RL using rewards defined the pass rate over GT unit tests provided in the TACO dataset, which we regard as our upperbound. Figure 7 shows pass @ code accuracy averaged over 945 evaluation tasks. The code generator trained with UTRL achieves a code accuracy of 15.3%, which is remarkably higher than the two baselines, and even comparable to 15.9% achieved by the code generator trained to maximize pass rate the GT unit tests. While training the code generator to maximize the pass rate over HPT--o--enerater unit tests improved the pass @ accuracy to some extent, the code accuracy saturates under 12%, and we observe that directly fine tuning code generator with ground-truth code solutions via SFT severely degrades the performance on unseen evaluation tasks, resulting in merely 3.6% pass@1 code accuracy. These results indicate the effectiveness of UTRL in training the code generator, especially when high-quality ground-truth unit tests are infeasible.
4.5 EFFECT OF ITERATIVE TRAINING
We examine whether iterative training enables the code generator to produce near- cear-perfect code solu- tion, thereby providing increasingly challenging discrimination tasks for the unit test generator, and whether the generator improves by learning to discriminate the near-perfect code from ground-truth code. Figure 8 shows discrimination rewards averaged over evaluation tasks. In the first iteration, the unit test generator is trained against code produced by the initial code generator, and the dis- crimination reward saturates after about 50 steps. At this point, stop training and update the code generator via RL to maximize its pass rate against the current unit tests. When training resumes in iteration 2, the discrimination reward drops from 0.626 to 0.375, indicating that the updated code

---

Published as a conference paper at ICLR 2026
SPT
LTRL
Discrimination reward itcration
RL with (T aesis
-
RI with GIPT-4o lotts
Pass Accuracy (%)
15.0
0.4
12.5
50
100
130
200
250
300
10.0
Discrimination reward at itcration
7.5
0.40
0.55
50 100
150 200
250
300
Training Steps
Figure 7: @ accuracy of code genera- tors on the TACO evaluation tasks. Qwen3- 4B code generator trained with UTRL out- performs the same model trained with GPT- 4o-gen no-generated unit tests and the same model trained via SFT in terms of pass accuracy.
Figure Discrimination reward evaluated at training iterations 1 and 2. During iteration training, the discrimination reward saturated after 50 steps with only about gain of 0.02, while in iteration 2, it exhibited an improvement 0.07.
generator now produces code solutions that are harder to distinguish from ground-truth, thereby creating more challenging discrimination task for the unit test generator. As training continues in iteration 2. the unit test generator improves the reward from 0.375 to 0.447. Correspondingly. as shown in Figure 9, the unit test generator iteration produces unit tests that yield superior best-of- N performance compared to iteration 1, even outperforming GPT-4.11 Furthermore, we extend the training to third iteration, but we did not observe notable improvement in discrimination reward. While we conducted these experiments using Qwen3-4B the unit test generator, believe that employing a larger-capacity model as a unit test generator would likely allow the adversarial process to sustain meaningful improvements over more iterations.
5 CONCLUSION
In this work, present UTRL., adversarial reinforcement learning framework for training LLM as unit test generator. Our approach alternates between training the unit test generator and code generator, enabling the unit generator to produce unit tests that can distinguish between near-correct code solutions and the ground-trun -ruth code implementation. Through extensive experi- ments and analysis, we demonstrate the superiority of UTRL over recent baselines and supervised bearningbbssed methods, even outperforming frontier models like GPT-4.1. We hope this work paves way for future research on training algorithm for unit test generation.
ETHICS STATEMENT
We introduce UTRL, an RL framework for training LLMs for unit test generation. As unit tests enable systematic and interpretable verification source code written by LLMs, believe the improvement automated unit test generation will contribute to more reliable and safe LLM--bsse code generation and agentic software engineering.
REPRODUCIBILITY STATEMENT
For the reproducibility of our results, we have provided detailed description of our experimental setups and prompts in Section A.3,A.4, and A.5. Additionally, to further facilitate reproduction, we will open- the training dataset, model checkpoint, and source code.
ACKNOWLEDGEMENT
This research was conducted as part of the Sovereign AI Foundation Model Project(Data Track), organized by the Ministry of Science and ICT(MSIT) and supported by the National Information AIData-WIIOI, WI101.. This work was also supported by Institute for Information communications Technology Planning Evaluation(IITPP) grant funded
Society AgeAc, K.rer. (Grant No. 2026-AIData
10

---

Published as a conference paper at ICLR 2026
by the Korea govermeen((MSIT) IRS-2019-11190075, Artificial Intelligence Graduate School Pro- gram(KAIST)). This work was also supported by the National Research Foundation of Korea(NRF) grant funded by the Korea MSS)) (No. RS-2024-00414822),
REFERENCES
Saranya Alagarsamy, Chakkrit Tantithamthavomn, and Aldeida Aleti. A3test: Assertion-augmentted automated test case generation. Information and Software Technology, 176:107565, 2024.
Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test im- provement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pp. 185-196, 2024.
Juan Altmayer Pizzorno and Emery D Berger. Coverup: Effective high coverage test generation for python. Proceedings the ACM on Software Engineering, 2(FSE) 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335. 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study foundation model post-trate training. arXiv preprint arXiv:2501. 2025.
Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel Des- marais. Effective test generation using pre-trained large language models and mutation testing. Information and Softwrre Technology, 171:107468, 2024.
Kefan Dong and Tengyu Ma. Stp: Self-play 11m theorem provers with iterative conjecturing and proving. arXiv preprint arXiv:2502.00212, 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Christopher Foster, Abhishek Gulati, Mark Harman, Inna Harper, Ke Mao, Jillian Ritchey, Hervé Robert, and Shubho Sengupta. Mutation- -uided llm-based test generation at meta. arXiv preprint arXiv:2501.12862. 2025.
Gordon Fraser and Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented soft- ware. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pp. 416-419, 2011.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Google Deepmind. Gemini 2.5 flash. https: http:///eemmmm...gggg/ee/e/eeeee flash/, 2025. Accessed: 2025-06-17.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-codes When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401.14196 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. Testgeneval: A real world unit test generation and test completion benchmark. arXiv preprint 00074.2410.00752, 2024a.

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024b.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems,
11-1211228,,,,, 2022.
Kefan Li, Hongyue Yu, Tingyu Guo, Shijie Cao, and Yuan Yuan. Cocoevo: Co-evolution of pro- grams and test cases to enhance code generation. arXiv preprint arXiv:2502. 10802, 2025.
Rongao Li, Jie Fu, Bo-Wen Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624) 2022. doi: 10. 10.1116//ciencc..abg 158. URL httpt::t//// / www. science. org/ doi abal//ab///ii 10. 10. 1126/ science abq1158.
Zi Lin, Sheng Shen, Jingho Shang. Jason Weston, and Yixin Nie. Learning to solve and verify: A self-play framework for code and test generation. arXiv preprint 1494444,0 2025.
Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. arXiv preprint arXiv:2501.0105: 2025.
OpenAl. Introducing gpt-4.1 1 Accessed: 2025-04-14.
com http:/iddxxx/eeeee -pt-ndeepgpt------ppp-a- in the api. https:// wttp///w....omiiin////tect//
2025.
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, 1lge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D'Sa, Arthur Petron, Henrique P O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint urXiv:2101.04882. 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Archiki Prasad, Elias Stengel-Eskin, Justin Chih- Chen, Zaid Khan, and Mohit Bansal. Learning 2025.
to generate unit tests for automated debugging. arXiv preprint
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: / arxiiv. org/ abs/ / 2212.412222 15115.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 14553378--5741. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
12

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- ollf-play with general reinforcement learning algorithm. arXiv preprint 1712v11712.01815, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-300 1021, 2020.
Wenhan Wang, Chenyuan Yang. Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation. arXiv preprint wrXiv:2406.04531, 2024.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving IIm coder and unit tester via reinforcement learning. arXiv preprint 2tdi::25006031333 2025a.
Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests#: High-quality test case generation for competitive programming. arXiv preprint arXiv:5506008817 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, al. Qwen3 technical report. arXiv preprint 2rX5v:2505.09388 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint a50iv: 14476, 2025.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.0333 0335. 2025.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li. Huazuo Gao, Shirong Ma, et al. beepseek-coder-v22 Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 11911, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A. GROUPED RELATIVE POLICY OPTIMIZATION
In this section, we describe the RL algorithm we used for training the LLMs in our experiments.
Proximial Policy Optimization Proximal Policy Optimization (PPO) (Schulman et al., 2017). an actor-critic RL algorithm, is widely used for training LLMs via RL. PPO updates the LLM policy Ty by maximizing the clipped surrogate objective described follows:
lo1
To(0rl oce)
(F) = Boo (112)
E
min
A(x,
TT (o occ))
1=1
<o(osk>r, oct)
e, 2,1+e) A(t, )
clip Te (o 021)
(S)
where vy policy under the training, I denotes prompt, Occ is a completion up t-th token, is f-th token generated by the LLM Towa and is hyperparameter for clipping the importance sampling ratio. Here, the computation of advantage A(x, oz) requires learning a separate value function Vg. Usually, such value function is parameterized by LLM of size comparable to the LLM policy under training, thereby requiring significant memory and computational cost.
Grouped Relative Policy Optimization In order to bypass the cost of learning the value function, Grouped Relative Policy Optimization (GRPO) has been proposed as a variant of PPO (Shao et al., 2024). In order to compute the advantage, GRPO samples a group G outputs {ol0) li=1 from the old LLM policy * for input prompt x, and computes scalar rewards fr, = R(x, o)) These rewards then normalized by subtracting the group mean and dividing by the group standard deviation. The resulting normalized score A, (renm()) is broadcast all tokens in the output sul(r)
( such that A(x, ) A; This shared advantage then used to weight the token-level policy gradients, allowing for effective optimization without learned value function. Formally, GRPO optimizes the following objective:
To(o; Ir, 0zt) Air (3)
(0) Egoin 12 (+12)
(12)
min
TB (0
L,0 021)
(o, 2r, 220
clip
1e A1)
(6)
Tea
(o
x 221)
where A, = (x-mman(r)) stl(r)
is the normalized advantage shared across all tokens in output (
A.2 ADDITIONAL. EXPERIMENTS
In this section, we provide supplementary experiments to thoroughly validate each component of UTRL, and demonstrate further advantage of UTRL.
A.2.1 EVALUATION ON LIVECODEBENCH
To evaluate whether the unit test generator LLM trained via UTRL generalizes to arbitrary di- verse programming tasks, we additionally measure Best-of-N improvement and unit test fidelity of UTRL on 511 programming tasks in LiveCodeBench (Jain al., 2024b). As shown in the Table 3, Qwen3-4B trained via UTRL achieves best-of-N improvement higher than GPT-4.1, and Qwen2.5-7B- -IIstructt trained via UTRL outperforms the same model trained via CURE. This results underscore that the effectiveness of unit test generator trained via UTRL well generalizes to diverse programming tasks.
A.2.2 COMPARISON TO BEST-OF-N DISTILLATION
As an additional baseline for training unit test generator, we compare UTRL with best-of-N distil- lation approach. For Best-of-N distillation, we sample 8 unit tests for each training instance using Qwen3-4B, select one with maximum discrimination reward, and finetune the Qwen3-4B with the selected unit test via SFT. As shown in the Table 4, although the best-of-N distillation approach yields performance comparable to SFT w/ Dur baseline even without requiring ground-truth unit tests for training, UTRL achieves superior performance.
14

---

Published as a conference paper at ICLR 2026
Unit test generated by
Code LLM: Qwen3-4B
Code LLM: Qwen3-8B
Score
Acc (%)
Score
Acc (9)
Qwen3-4B
0.704
55.2
0.714
55.4
Qwen3-4B + UTRL
0.744
59.9
0.752
59.3
Qwen2,5-7B
0.607
44.6
0.617
44.0
Qwen2.5-BB+CURS
0.660
48.2
0.671
47.7
Qwen2.5-B+UUTRL
0.686
54.4
0.693
52.4
GPT-4.1
0.732
58.3
0.735
58.3
Table 3: Best-of-N improvement evaluation on LiveCodeBench. A unit test generator trained UTRL also generalizes to unseen tasks in LiveCodeBench, outperforming CURE. Moreover, Qwen3-4B trained via UTRI. achieves performance superior to GPT-4.1.
Unit generated by
Code LLM: Qwen3-4B
Code LLM: Qwen3-8B
Score
Acc (%)
Score
Acc (%)
Qwen3-4B + SFT w/ Dur
0.368
7.8
0.380
7.7
Qwen3-4B best-of-N distil.
0.368
7.8
0.380
7.7
Qwen3-4B UTRL.
0.475
11.6
0.494
11.9
Table 4: Comparison to best-of-N distlaation approach for training unit test generator.
A.2.3 EXPERIMENT ON DIFFERENT LLM
Although we mainly train Qwen3-4B and Qwen3-14B via UTRL, we additionally show effective- ness of UTRL in training arbitrary LLMs. To this end, we train for steps and evaluate best-of-N improvement of the unit tests generated by the resulting unit test generator. As shown in the Table 7, UTRL effectively enhances the unit test generation capability when applied to the Llama3. ----t-uttuttt,, improving achieving best-of-N improvement of 11.75% on average, which is remarkably higher than Alamma-3.1-88 ---B-Inntruee model.
via UTRL
A.2.4 ABLATION ON WEIGHTING TERM FOR DISCRIMINATION REWARD
We conduct ablation study on hyperparameter which controls the weight of the discrimination reward for training unit test generator LLM. We sweep over E (0.0, 0.85, 1.0). As shown in the Table 6 and Table setting
0.0 leads to improved validity (proportion of the functionally correct test cases among entire test cases in the unit test), while resulting in lower Best-of-N improvement, since the model is guided to generate test cases that are merely valid but indifferent discrimina- tiveness. Conversely, setting À 1.0 results in a substantial fraction of invalid test cases, which also degrades unit test quality measured by Best-of-N improvement. Setting À = 0.85 achieves the best unit test quality, balancing validity and discriminativeness of unit tests. These results imply that both discrimination reward and validity reward are essential, and should be balanced appropriately to produce high-quality unit tests.
A.2.5 ABLATION ON VALIDITY REWARD
In order demonstrate the effectiveness of the validity reward and its design choice (see Equa- tion 2), we conduct an ablation study comparing the following variants of the UTRL, (1) UTRL trained without validity reward (i.c., w/o Rvald), and (2) UTRL with validity reward but without clipped normalization in validity reward term, where the validity reward is defined as ratio of valid test cases among the generated test cases (i.e., w/o clipping). Figure 10 shows the number of test cases and the ratio of valid test cases in the generated unit test over training steps. UTRL without Ryalid results the generation of unit tests with more than 50% of invalid test cases, while UTRL results in unit tests containing less than 35% of invalid test cases. Furthermore, in the case of UTRL without clipping, we observe that the model collapses to generate an extremely small num- ber of trivial test cases in order to maximize the ratio of the valid test cases, hindering the learning progress toward generating comprehensive unit tests. These results demonstrates the effectiveness of the design choice of UTRL.
15

---

Published as a conference paper at ICLR 2026
Unit generated by
Code LLM: Qwen3-4B
Code LLM: Qwen3-8B
Score
Acc (%)
Score
Acc (%)
Llama-3 3. 3.1--B--mmm Instruct
0.368
7.8
0,380
7.7
Llama-3, 3.1-8B-Imiie 3B-IInttruct + UTRL
0.475
11.6
0.494
11.9
Table 5: Performance of 1lama-3.1-8B 1-88--Instrct trained via UTRL. UTRL effectively improve the unit test generation capability of Llama-3. 1.lma-3.1-8P 3.1- -8B-IIstruct which is measured by best-of-N improve- ment.
Code LLM: Qwen3-4B
Code LLM: Qwen3-8B
Score
Acc (%)
Score
Acc (%)
0
0.509
12.4
0.524
13.1
À 0.85
0.514
13.7
0.534
13.8
À 1.0
0.485
12.3
0.492
11.5
Table Best-of-N improvement according to weighting factor for discrimination reward X. Omit- ting validity reward (A 0) degrades the quality of the unit tests owing to the functionally incorrect test cases occupying more than half the generated unit test.
A.2.6 BEST-OF-N IMPROVEMENT MEASURED WITH ADDITIONAL CODE GENERATORS
Table presents Best-of-N improvement measured when Qwen3-4B and GPT-4o used for code generator LLM. In this evaluation, UTRL still achieves the highest code accuracy and code score, demonstrating that LLMs trained via UTRL produces high-quality unit tests that can effectively evaluate code generated by various code L.Mss.
A.3 IMPLEMENTATION DETAILS In this section, we provide implementation details of UTRL and other baselines in our experiments. Hyperparameters are described Table
Training details of UTRL For training, utilize 15,249 programming task instances provided in training split of TACO dataset, where we exclude instances without ground-truth code solution and those whose unit test annotation do not follow stdio format (e.g., functional assertions), and 1,000 programming task instances for validation. In our experiment, we train cntructiio-finetune Qwen3-4B and Qwen3-144 1-BB via UTRL. For Qwen3-4B experiment, we train the unit test generator for 50 steps in the first iteration, and train the code generator for 330 steps. and then continually train the unit test generator for 170 steps in the second iteration. For Qwen3-14B experiment, we train the unit test generator for 100 steps at the first iteration, without further iteration. We use weighted sum of discrimination reward and validity reward (see Equation 3) measured on the 1,000 validation tasks validation metric to choose training steps. For the number of LLM-generated code samples to define a discrimination reward, we set the value as 8. For T, a hyperparameter to adjust the desired minimum number of the test cases in the unit test, set the value as 12. Additionally, for X, weighting factor for the discrimination reward, we set this value as 0.85. Moreover, order to ensure fair comparison with CURE, we also implement UTRL by training Owenn2.5-7B-Instrruct via UTRL for 100 steps using the 4.5K instruction-code solution pairs in CodeContests dataset, following the experimental setup of CURE. For UTRL f, we trained the model only for a single iteration.
Implementation of Pass function We describe implementation details about the Pass function, which is core component of reward computation. Given code C and test case T, consider Pass(C, (C, T) is (i.e., code C' passes test case 7' if (1) the code C' builds successfully without any syntax error, (2) the execution of test case T regarding the code C does assertion error), and (3) the execution of test case T regarding the code C' runs under 10 seconds. Otherwise, we consider Pass(C, is (i.e., code C does not pass test case T').
return any error (e.g..
Details of baselines
SFT with Dur: As training dataset, we use the 15,249 training task instances in TACO dataset, exactly same as UTRL, and we fine-tune Qwen3-4B. To label the unit test for SFT, randomly
16

---

Published as a conference paper at ICLR 2026
validity ratio (%)
À 0
64.3
0.85
57.2
1.0
49.7
Table 7: Ratio of valid test cases according to weighting factor for discrimination reward A.
Unit test generated by
Qwen3-4B
GPT-4o
Score
AScore
Acc (%)
AAcc (%p)
Score
AScore
Ace (%)
AAcc (%p)
Qwen3-4B
0.430
+0.084
9.8
+1.8
0.459
+0.057
13.4
+2.7
Qwen3-8B
0.435
+0.089
9.7
+1.7
0.470
+0.070
12.6
+1.9
Qwen3-14B
0,461
+0.115
9.7
+1.7
0.475
+0.075
13.3
+2.6
DeepSeck- CeepSeck-Coder-Lite---
0.407
+0.061
9.0
+1.0
0.458
+0.056
12.9
+2.2
GPT-4o
0,463
+0.111
9.6
+2.6
0.511
+0.094
13.8
+1,4
GPT-4.1
0.529
+0.182
12.9
+6.2
0.547
+0.148
15.6
+4.6
Qwen3-4B 4B + SFT w/ Dur
0,457
+0.127
11.7
+4.5
0,437
+0.239
14.0
+7.5
Qwen2.5-7B + CURE
0.421
+0.092
10.2
+3.0
0.459
+0.261
13.8
+7.3
Qwen2.5-7B + UTRL t
0,502
+0.173
14.0
+6.8
0.526
+0.328
16.0
+9.5
Qwen3-4B + UTRL
0.555
+0.207
14.5
+7.0
0.568
+0.353
16.9
+10,2
Qwen3-14B UTRL
0.579
+0.219
15.8
+7.8
0.578
+0.359
17.0
+11.0
GT (upperhound)
0.630
+0.284
20.2
+13.0
0.599
+0.197
21.7
+11.0
Table 8: Best-of-N improvement measured on TACO evaluation set using Qwen3-4B and GPT-4o as code generator LLM. UTRL shows the highest best-of-N improvement compared to baselines. Here, UTRL denotes the variant of UTRL using 4.5K programming tasks for training, for fair comparison to CURE.
select 12 test cases from the ground-truth unit test for each task in the TACO dataset. As shown in Figure 11 (left), we format the target response by listing the 12 test cases without reasoning and use them directly SFT labels.
SFT with DT:: a training dataset, we use the 15,249 training task instances in TACO dataset, and we fine-tune Qwen3-4B. In order to label DremsosUT with reasoning-annotate unit tests, we prompt Gemini-2.5 (iaini-..5-flass with the programming instruction to generate 12 test cases with reasoning. We then filter out functionally invalid test cases (i.c., test cases that fail under the ground-truth solution), and use the remaining valid test cases annotated with reasoning as SFT labels. We remark that this is replication of CodeRM (Ma et al., 2025), which utilized Llama3-70B-Instte to label unit test with reasoning, and train the Llama3-8B- 1ar2--8B-Instuuc with the labeled dataset.
For evaluating CURE and compare it with UTRL, we use the open-source ReasonFlux- 7B-Coder, which is Dive2.5--B-Innsreee fine-tuned with the CURE algorithm using 4.5K pro- gramming tasks from a subset of the CodeContest dataset (Li et al., 2022). In order to ensure fair comparison, train Qwen2.5-7B Instruct with UTRL using the same 4.5K training tasks (de- noted as UTRL !), and compare the resulting model with ReasonFlux-7B- 7B-Coder. Additionally, following the evaluation setup of CURE (Wang et al., 2025a), for each programming instruction in evaluation set, we sample 16 test cases to form single unit test, using the prompt format provided in the CURE paper.
CURE:
A.4 EVALUATION DETAILS
Evaluation tasks Across all experiments, we use 945 competitive programming tasks in the eval- uation split of the TACO dataset. The evaluation tasks span diverse difficulty levels, and each task annotated with highly comprehensive unit test (i.e., ground unit test) which comprises an average of 202.3 test cases. The tasks were collected from Codeforces, CodeChef, HackerRank, and HackerEarth, which are widely used competitive programming platforms, where the task distribu- tion is described in Table 10.
17

---

Published as a conference paper at ICLR 2026
clipping
wio Roddd
Best-of-N Improvements
Number test cases
18
10
Ieeation
Beration
17.6
16.7
Code Acc. (%)
16-
15.6
11.9
100
150
149
Ratio valid
14.2
13.7
0.6
0.4
13
Qwan3348B
Qwen3-8R
Quen1-14B Code generation model
150
Training Steps
Figure 9: Best-of-N performance gains obtained by UTRL at the same iterations. Across code generation models, iteration 2 results in improved best-of-N improvement compared to iteration 1.
Figure 10: Averaged number of test cases in the generated unit test and ratio valid test cases across RL training steps. Ablated vari- ants of UTRL results in generation of unit test with invalid test cases, or only few trivial test cases.
SFT with DUT
SFT with D Dreason UT
<reasonings
Reasoning for test case 1. The parpose of this test varidy the most basic </reascnings
Output:
BuzeTiin 70P1arronnts
Ioput:
Output:
Beasening
examples provided
</sesaoning>
Ingut:
Ieput:
Output:
Outgut:
Figure 11: Examples of unit test annotations for datasets for SFT baselines (SFT with Dur and SFT with D Dremoo-UT). D DressonUU is annotated by ground-truth unit tests, while D Dremon+UT is annotated by the teacher model (Gemini-2.5-flass
Best-of-N improvement For measuring Best-of-N improvement, utilize LLM for sampling 32 code solutions, and select the best code solution by using the generated unit test as a code evaluator (i.e., select the code solution that passes the largest number of test cases in the generated unit test). We then evaluate code score and code accuracy regarding the selected code solution, using ground- truth unit tests provided in the TACO evaluation dataset. An intuition behind this evaluation that more discriminative unit test will identify and select higher-quality code among multiple candidate code solutions. TACO evaluation set, we evaluate the ofst-of-N improvement on multiple code generator LLMs, Qwen3-4B, 8B, 14B, and GPT-4.. Additionally, in LiveCodeBench-ve evaluation set, we evaluate the Best-of-N improvement on Qwen3-4B and Qwen3-8B code generator LLMs.
Unit test fidelity Unit test fidelity measures whether the generated unit tests induce code evalua- tion consistent with the code evaluation by the ground-truth unit tests. For each programming task, we first sample 128 code solutions using Qwen3-4B, 8B, 14B and GPT-4o (i.e., sample 32 code solutions using each LLM). We then evaluate code score of the 128 code solutions twice, once by using the ground-trun -tuth unit test, and once by using the generated unit test, which induces two score vectors length of 128 (i.e., number of code solutions under evaluation) and each ranges from
18

---

Published as a conference paper at ICLR 2026
Methods
Optimizer
LR
Batch size
warmup
kl coef
sampling temperature
UTRL (iter
Adam W
1c-6
128
lc-2
le-3
1.0
UTRL (iter
Adam
5c-6
128
lc-2
le-3
CURE
Adam
le-6
128
le-2
le-2
1.0
SFT w/ DUT
Adam W
le-5
128
le-2
SET w/ Dresso+UT
Adam W
le-5
128
1e-2
Table 9: Training hyperparameters of UTRL and baselines.
CodeForces
CodeChef
HackerRank
HackerEarth
of tasks
540
245
20
40
Table 10: Distribution of TACO evaluation tasks over multiple competitive programming platforms.
to 1. Finally, we compute Spearman's 's correlation between these two vectors to quantify how closely the generated unit tests reproduce the code evaluations of the ground-truth unit test.
A.5 PROMPTS
In this section, we provide the prompts used in our experiments: (I) unit test generation, (2) code generation, and (3) rationalization, which were used for the baseline in Section 4.3.
Unit test generation prompt
System prompt:
You are an expert Python programmer capable of generating test cases for Python programming tasks.
Given a programming task, generate several independent test cases with corresponding reasoning.
Each test case should be independent of the others and sharply target distinct
cases that arbitrary faulty code
solutions can detected.
Before writing each test case, think deeply
about the input arguments that expese extreme or subtle edge cases, and reason about the expected output.
After completing the reasoning process, generate the test case stdio format.
Specifically, your output should follow the format below:
<reasoning>
Reasoning ng test case
First, reason about the input arguments that can discriminate an incorrect code sclution (e.g., edge cases).
Ensure the input arguments test aspects independent of previous test cases.
Then, derive the expected output from. the problem description. </reasoning>
Input:
stdio format input
Output:
stdio format output 1
<reasoning>>
Reasoning for test case 2
First, reason about the input arguments that can discriminate an incorrect code sclution (e.g., edge cases) Ensure the input arguments test aspects independent previous test cases.
19

---

Published as a conference paper at ICLR 2026
Then, derive the expected output from the problem description. </reasoning>
Input:
stdio format input 2
Output:
stdio format output 2
<reasoning>
Reasoning for test case 12
First, reascn about the input arguments that can discriminate an incorrect code sclution (e.g., edge cases) Ensure the input arguments previous test cases.
aspects independent
Then, derive the expected output from the problem description. </reasoning>
Input:
stdio format input 12
Output
stdio format output 12
...
Ensure the follewing:
1. Do not include solution code in your response; generate
exactly 12 test cases.
2. For each test case, provide a detailed rationale.
3. Each test case must be independent; avoid duplicates.
User prompt:
Here is the problem description:
[problem_query query!
Based on comprehensive reasoning, generate comprehensive unit involving several test cases for the given problem. The test cases should cover various edge cases, corner cases, and normal cases, at the same time, functionally correct.
Code generation prompt
System prompt:
You are an expert Python programmer.
Based the problem description, solve the coding problem efficiently.
Think step by step, then write a Python solution that solves the problem.
Follow the format below:
<reasoning>
Write your reascning here. </reasoning>
"'python
Write your Pythen solution here. It should run with stdio-format input.
User prompt:
Here is the problem description:
pproblem_query!
20

---

Published as a conference paper at ICLR 2026
A.6 LIMITATIONS & FUTURE DIRECTIONS
We outline the limitations of UTRL and promising directions for future research.
Performance gap with ground-truth unit tests Although we show that UTRL promising direction for improving unit test generation capabilities of LLMs, performance gap remains be- tween UTRL-generated unit tests and ground-truth unit tests. As UTRL can be combined with arbitrary online RL algorithms, believe that investigation RL algorithm enabling better ploration, and combination of UTRL with the better RL algorithm will further improve the unit test generation performance.
Extension to broad software engineering domains While our work conducts experiments on the competitive programming task domain, UTRL can be applied to broader software / program- ming domains. Scaling UTRL with large-scale instruction-code datasets covering a wider range of programming scenarios might be promising direction for future work.
Considerations for variable length unit tests UTRL trains the LLM to generate fixed number of test cases per unit test. Future work could explore framework that enables adaptive generation of variable-length unit tests depending on the complexity of the given programming task.
21


Reviews:
[{"review_id": "Z4ucCbvFDt", "reviewer": "Reviewer_R3AP", "summary": "This paper introduces UTRL, an adversarial reinforcement learning framework for training a Large Language Model (LLM) to generate high-quality unit tests. The framework co-trains two models: a unit test generator ($M_{\\text{UT}}$) and a code generator ($M_{\\text{code}}$). The core idea is that $M_{\\text{UT}}$ is rewarded for creating tests that can distinguish between code generated by $M_{\\text{code}}$ and a ground-truth solution ($C^*$), while $M_{\\text{code}}$ is rewarded for generating code that passes the tests from $M_{\\text{UT}}$. The main contribution is a method to train a unit test generator without requiring ground-truth unit tests, relying only on instruction-code pairs. Experiments on the TACO dataset show that the resulting unit test generator outperforms supervised fine-tuning (SFT) and proprietary models in producing high-quality tests.", "questions": "1.  The framework's dependency on a ground-truth executable solution appears to be its greatest limitation. Could the authors elaborate on the practical applicability of this method outside of curated competitive programming datasets where canonical solutions are available? How could this framework be adapted for a more realistic scenario where $C^*$ does not exist?\n\n2.  Figure 7 indicates that the UTRL-trained code generator does not surpass the performance of an agent trained with ground-truth unit tests. Does this imply that the primary contribution is the creation of a standalone, high-quality unit test generator, rather than a superior end-to-end training pipeline for code generation? What is the practical advantage of the adversarial loop if the resulting code agent's performance does not improve beyond the oracle baseline?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental setup is rigorous. The authors compare their method against a good set of baselines, including various LLMs, SFT, and the recent CURE framework. The introduction of \"best-of-N improvement\" and \"unit test fidelity\" as evaluation metrics is well-reasoned and provides a quantitative assessment of the quality of the generated tests.\n2. The paper is clearly written and well-structured. The UTRL framework is explained effectively with figures, an algorithm description, and formal reward definitions, making the methodology easy to understand.\n3.  The work addresses the important and challenging problem of automated unit test generation. Improving the quality of generated tests could provide a more reliable and scalable feedback mechanism for training and verifying code-generating LLMs.", "weaknesses": "The central claims of the paper rest on a fundamental assumption that severely limits its practical applicability and scalability.\n\n1.  Unrealistic Requirement of Ground-Truth Code: The entire training process is critically dependent on having access to a ground-truth, executable correct solution ($C^*$) for every training instance. This assumption is unrealistic for the vast majority of programming tasks. The primary value of automated code generation lies in solving problems for which a solution is not already known. If a correct, executable `C*` is available, the need to train a new code generator is greatly diminished. This requirement makes it impossible to scale the method using large, readily available datasets of code from sources like GitHub, confining it to niche, curated datasets.\n\n2. No Improvement in Downstream Code Generation: The ultimate goal of generating better unit tests is often to improve the performance of code generators. However, the paper's own results in Figure 7 demonstrate that this goal is not achieved. The code generator trained adversarially with UTRL's generated tests performs comparably to, but slightly *worse than*, a model trained with rewards from the ground-truth unit tests. This suggests that the complex adversarial framework is, at best, a proxy for an oracle that it cannot outperform. While it successfully creates a high-quality unit test generator, it fails to show that these generated tests provide a superior training signal that pushes the code generator beyond existing capabilities. The main loop does not seem to create a net performance gain for the overall system.\n\n3.  Overstated Claims on Scalability: Given the reliance on $C^*$ and the lack of improvement for the code generator, the claim that this method can enable better scaling of LLM training beyond current SOTA methods is not well-supported. The need for verified, executable solutions for every training problem is a far more significant bottleneck than the need for unit tests, fundamentally hindering scalability.\n\n4. The proposed adversarial framework was proposed before in other similar scenario where an external interpreter is available (e.g. for Lean with STP [1])\n\n[1] Dong, Kefan, and Tengyu Ma. \"Stp: Self-play llm theorem provers with iterative conjecturing and proving.\" arXiv preprint arXiv:2502.00212 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VNsAxEkKMT", "reviewer": "Reviewer_qbAT", "summary": "This paper presents UTRL, an adversarial RL loop training the language model to be code generator and a unit test generator. These components are trained in an adversarial manner with the UT generator acting as the discriminator -- distinguishing ground truth code solution from incorrect codes generated by the code generator while generating valid unit tests. The code generator on the other hand is optimized to have a higher pass rate over the generated unit tests. The paper makes use of the TACO dataset for code generation and verifies that their system leads to better quality unit tests and higher code generation performance when compared to different SFT and RL baselines (like CURE).", "questions": "See above in addition to the questions below:\n1. What is a tangible takeaway from Fig 7, how does the training trend contrast with other baselines?\n2. Isn't the training online for the main results reported in Table 1, or is it iterative as well -- if so how is the number of iterations set? Can Fig 8 be extended to more iterations?\n3. The paper repeatedly makes the claim that UTRL removes the need for ground-truth UT annotations. However, the training uses the regular code-generation recipe of instruction, gold solution. When these are available, ground-truth UTs can be readily collected by sampling multiple inputs from an LLM, rejecting invalid outputs. The authors should clarify what other kinds of annotations if any they are referring to.\n4. As mentioned in Sec 3.1, you generate a set of UTs for each problem. Are the number of UTs roughly similar for various baselines?\n5. Why not show the pass@1 accuracy of the code generator before and after training? Does that not improve and the improvement only come from best of n sampling? If not, could the performance be further improved by doing best-of-N sampling over the trained code-generator model or is it not effective due to the nature of adversarial training. An ablation separating the contribution of better code generation and UT generation would strengthen this work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The adversarial setting in the method intuitive and appears to be effective compared to the baselines (although the latter can be improved, see below)\n- The paper is well-written, well-structured, and easy to follow. The problem or topic is well-motivated, and joint improvements in unit test generation and code generation could be of interest to the community.", "weaknesses": "- Missing references to related work, I think the scope of related works discussed is very myopic, and I would broadly discuss the following line of work:\n    -  Self-play setups for LLMs: https://arxiv.org/abs/2407.19594, https://arxiv.org/abs/2401.01335, https://arxiv.org/abs/2502.14948\n    - In my opinion, it is worth emphasizing the adversarial training setup in general which this method builds on, e.g. GANs (https://arxiv.org/abs/1406.2661), https://arxiv.org/abs/1607.02533, etc.\n\n- Strength of SFT baseline: \n    - In table 1 and 2, why not present the UT + Reason baseline? No discussion why the code-generation performance of UT + Reason baseline is lesser in Fig 5 (a) despite better code fidelity.\n    - Alternate design, Prasad et al show that SFT supervision by training unit tests to be discriminative (they call attacking) and valid (with correct) outputs is effective. Sec 3.2 suggests such data is already collected during training, set of UTs which are both discriminative and valid. What happens when you train an SFT baseline with or without reasoning on this data? Would this remove the need for joint RL training?\n- Lack of OOD generalization: The paper only tests on an in-domain test set of TACO dataset. Can the author show improvements in code-generation and UT fidelity transferring to CodeContests, LeetCode, MBPP, LiveCodeBench and other mainstream code-generation datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLCHtnYQm2", "reviewer": "Reviewer_eXix", "summary": "This paper introduces an RL method for training model to generate unit tests. Their method trains UT generator and code generator together in an adversarial loop: the UT generator is supposed to find faults, while the code generator is supposed to improve to pass more unit tests.\n\nThe method uses iterative RL training of 2 LLMs, with each model training off each other's signal, with the main contribution being a reward for UT generation, which informs the reward for code generation. The models are trained on examples from the TACO dataset.\n\nThe method is evaluated on TACO, with results showing improved UG generation as compared to base model, and as compared to an SFT model. The method is compared to multiple baselines:\n- untrained models (closed and open-source)\n- SFT model, trained on data filtered from Gemini 2.5 \n- compare to CURE UTs generated by a Qwen 2.5 7B model finetuned by CURE authors \n- Ground-truth unit tests (ceiling)\n\nMethods are evaluated according to best-of-N reranking with different models as generators and UT fidelity, which is the Spearman correlations between the ranking of several solutions according to a generated and ground truth unit tests. \nThe results indicate that UTs generated by the proposed method improve both Qwen-3-4B and Qwen-3-14B in best-of-32 and achieve higher fidelity. They also indicate that in an apples-to-apples comparison on Qwen-2.5-Instruct, the method outperforms CURE.", "questions": "- L156-157: under (1), your goal is to discriminate sampled code from gold code. Doesn't this assume that the sampled code is incorrect? If so, does the current method still work when the generator is a stronger coding model?\n- Qwen-3-4B: is this the instruct or base model? if it's the base model, do you still get improvements over instruct model? \n- I would suggest adding experiments showing\n\t- more on pass@1\n\t- cross-dataset transfer\n\t- comparison to larger models \n- a bit of a nit: since CURE came out in June, it seems odd to call it \"concurrent work\" (L097) and this doesn't seem necessary since the authors compare against it.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Paper is clearly written and well-organized, and explanation of results is clear\n- The method appears to work well as a verifier, with improvements even over proprietary models. The adversarial nature of the method is well-motivated. \n- Unit test fidelity as a metric seems to be a novel contribution. \n- Compares to recent baseline (CURE)\n- Problem importance: unit test generation is an important problem", "weaknesses": "- no pass@1 results. From the way that the adversarial setup is described in the intro, I would have expected to see improvements to pass@1 for the adversarial generator. It seems like the evaluation (reranking@32) is geared towards the discriminator and test-time re-ranking, but does the pass@1 of the generator also improve via training? This is shown a bit in Fig 7 but should be expanded (since code score also is not quite the same thing as pass@1). Effectively, section 4.4 could be scaled up. \n- OOD generalization: both train/test are TACO, so it's not clear that the method will generalize to other datasets. This might be something that is covered in the CodeContest experiment with Qwen-2.5 compared to CURE, but this should be made more formal/explicit. Moreover, it's not clear why the paper doesn't evaluate on the same data as CURE (which they use as a baseline) or UTGEN (which they cite).\n- missing related work: https://arxiv.org/abs/2502.14948 also iteratively improves LLMs for solving and testing using RL with DPO. \n- for CURE comparison: it would be nice to also see CURE applied to Qwen-3-4B \n- limited model size: does this still result in improvements with stronger/larger models? It would also be nice to see a larger Qwen-3 model as a baseline for UT generation in Table 1 (since we don't know what size 4o and 4.1 are.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urkEioPTiu", "reviewer": "Reviewer_NJwC", "summary": "The authors proposed using reinforcement learning to jointly train a unit test generator and a code generator. The unit test generator is optimized for generating test cases that can discriminate gold code and model-generated code, while the code generator is optimized for pass rate. This eliminates the need for collecting expensive gold unit test and experimental results show that the proposed method has superior performance than SFT baselines and prompting frontier models to generate test cases.", "questions": "1. In figure 3, the caption says “Second, the discrimination reward is defined as a ratio of sampled code solutions that do not pass at least one valid test case. In this figure, among 6 sampled code solutions, 4 code solutions (C1, C3, C4, C5) do not pass at least one valid test case, resulting in the discrimination reward of 0.667”, but the from the figure it seems like C1, C3, C4, C5 pass at least one valid test case. Do the authors mean “solutions that have at least one failure case”?\n2. I wonder what the performance would be if we train the unit test generator without the adversarial setup, i.e., only train it using RL and the gold code solution to generate valid unit tests, then use it at test time to measure how much it could improve code generation.\n3. In section 3.1, the author mentioned that the two key steps are repeated iteratively. How many iterations does it need to converge, and how is the performance changing over iterations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of modeling code generation and unit test generation as an adversarial setup is sound.\n2. This setup removes the need for gold unit test data, which is particularly hard to collect.\n3. Qwen3-4B trained with UTRL outperforms GPT4.1 and GPT4o in terms of unit test generation, which is a strong result.\n4. The paper is well presented and is easy to follow.", "weaknesses": "1. The experiments are mainly conducted with Qwen3-4B, lacking analysis on the effectiveness across different models such as Llama, Gemma, or more code-specific ones like Qwen Code. I also wonder if the trained unit test generator is able to improve frontier models, when they are served as the code generator.\n2. There is a hyperparameter, lambda, that balances the discrimination reward and the validity reward, which might need tuning. However, the analysis of this hyperparameter is missing.\n3. There is little discussion on the iterative training part, where the authors only mention that the discrimination reward saturates at iter 1 and keeps improving at iter 2. But how many iterations are needed, and could we expect even more improvement after iteration 2 is not mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)