Title: Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Code LLM

Reinforcement Learning

Unit Test Generation
Abstract: Mathematical reasoning in large language models has been successfully incentivized through reinforcement learning with verifiable rewards, leading to improved one-shot precision. In this work, we turn our focus to the coding domain. Beyond one-shot precision, we highlight unit test generation as another key factor for enhancing coding ability, since accurate unit tests are essential for enabling self-checking and self-correction during inference.
Traditional approaches for fine-tuning LLMs on unit test generation rely heavily on ground-truth code solutions in the training data. We propose CURE, a novel reinforcement learning framework with a dedicated reward design that co-evolves coding and unit test generation capabilities based on their interaction outcomes—without any ground-truth code as supervision. This approach enables flexible and scalable training and allows the unit tester to learn directly from the coder’s mistakes.
Through extensive evaluations, we demonstrate that our CURE models, derived from base models of varying sizes, excel in both code generation and unit test generation. They naturally extend to downstream tasks such as test-time scaling—achieving a 6.2\% improvement over the base model—and agentic unit test generation, with a 25.1\% improvement. Our 4B model consistently outperforms Qwen3-4B while achieving 64.8\% inference efficiency in unit test generation. Notably, we also find that the CURE model can serve as an effective reward model for reinforcement learning on base models, even in the absence of any labeled supervision.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCURE is a framework for post-training and test-time scaling improvements in language models for code generation. The key idea is to have language models generate programs and tests simultaneously, optimizing the quality of both artifacts using a reinforcement learning approach and a clever reward function. The authors perform a thorough evaluation of CURE, finding that it generates better code and tests and complements existing approaches well.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The approach is sensible and well-motivated. The reward (4) may even be deeper than the authors let on. In the software engineering literature, there is a notion of “mutation testing” (see, e.g., https://ieeexplore.ieee.org/document/5487526) that is used to evaluate the quality of tests. The reward function is related to mutation testing but with the addition of agreement with ground truth tests. Very cool. Mutation testing has been used for LLM code generation before (see, e.g., the “completeness measure” of https://arxiv.org/pdf/2406.09757 that is very closely related to one half of the reward function), and the authors may enjoy exploring that space.\n- The empirical evaluation is solid.\n- The related work is honest, and the comparisons are apt.\n\n## Weaknesses\n- “without access to ground-truth code solutions” and “without any ground-truth code as supervision” are technically true but slightly misleading. The approach does require ground truth tests.\n- The distinctions with related work are subtle. One might argue that the approach is slightly incremental. There is, of course, plenty of work on using LMs to generate code, tests, and combinations thereof.\nThe authors do not outline the scope of their work. See Question 1.\n\nQUESTIONS:\n- What kind of programs and tests can this approach handle? Can it be extended to stateful programs, for example? What about flaky tests? What is the scope of this work, and can it eventually be extended beyond that?\n\nLIMITATIONS:\nNo. See question 1.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CURE, a novel co-evolutionary reinforcement learning framework that enhances code generation and unit test generation capabilities in large language models (LLMs). The main contribution of this work is its ability to co-evolve code generation and unit test generation without relying on ground-truth code solutions, which significantly improves scalability and flexibility. The proposed model is evaluated on five benchmarks, demonstrating a 6.2% improvement in code generation accuracy, a 25.1% improvement in test generation accuracy, and a 35% reduction in inference time for long-chain reasoning models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Originality**: The idea of using co-evolutionary reinforcement learning to simultaneously optimize code generation and unit test generation is novel and addresses the significant challenge of training models without ground-truth code.\n- **Impact**: The model demonstrates considerable improvements in both code generation and unit test generation across multiple benchmarks, which is highly relevant for practical software development and automation.\n- **Scalability**: The removal of the reliance on labeled data opens the potential for scaling this approach to a wider range of real-world applications.\n\n### Weaknesses\n- **Theoretical Flaws**: Proposition A.1 contains a critical error. It incorrectly assumes that the quality of a unit test can be directly tied to the quality of the code. This leads to an issue where the evaluation of a test quality is overly dependent on the code's accuracy, ignoring that a perfect test might still be penalized if the code fails in certain ways.\n\n  **Example**: A perfect unit test (with $p_u$ = 1) could be incorrectly evaluated as  $\\hat{p_u}$ = 0  if the code implementation is flawed, even though the test itself is correct. This misjudgment undermines the reliability of the reward system.\n\n- **Reward Function Issues**: The reward function may penalize high-quality unit tests, like boundary cases, while rewarding simpler, less effective tests. This could lead to suboptimal performance.\n- **Experimental Reliability**: The reward system might lead to flawed test generation. Additional experiments are needed to ensure the model can avoid producing degenerate tests (e.g., tests that pass all code but lack discriminative power).\n\nQUESTIONS:\n1. Do you agree with the theoretical flaw in Proposition A.1? If not, please provide a detailed explanation. The theoretical issue is fundamental to the reward system, and we will reassess the score based on your input.\n\n2. Does the proposed reward function adequately differentiate between high-quality and low-quality unit tests? Could it inadvertently lead to the generation of less useful tests, such as overly permissive ones that fail to provide sufficient differentiation?\n\n3. What are the experimental results under the correct reward?\n\nIf the authors can provide more details to address my comments, I would be inclined to raise my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reinforcement learning framework that jointly improves an LLM coder and a unit-test generator without ground-truth code solutions. In every training round, the policy samples $n$ candidate programs and $m$ candidate tests for the same problem, forms an $n\\times m$  execution matrix, and assigns:\n- coder reward: the number of public (ground‑truth) tests passed;\n- unit-test generator reward: a theory‑derived metric $\\mu = p_{u} (1 - p_{01}) - (1 - p_{u}) p_{00}$ that favors unit tests which pass correct code solutions and reject wrong code solutions.\n\nExperiments on five benchmarks demonstrate the effectiveness of the proposed approach. Moreover, the unit-test generator can serve as a reward model in further reinforcement learning loops.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.   Insightful theorem analysis. The paper gives a concise and rigorous derivation of the necessary and sufficient condition for the pairwise reward ordering to become almost surely correct\n2.   Facilitates Label-free RL. The proposed approach leverages mutual supervision between code solutions and unit tests to improve both the code generator and unit-test generator, representing a significant step toward truly label-free RL.\n3. Comprehensive evaluation. Results span five public benchmarks, multiple BoN settings, agentic pipelines, and API models, demonstrating consistent gains and meaningful inference‑time speed‑ups for long‑CoT testers. \n\n**Weaknesses:**\n\n1.   Discrepancy between Theorem 3.1 and the actual method. Theorem 3.1 implies that no ground-truth code solutions or unit tests are needed, yet in practice, the framework still heavily relies on ground-truth tests to estimate $p_u$, $p_{00}$, and $p_{01}$. Exploring approaches that eliminate the need for ground-truth tests (e.g., CodeT) would likely yield more convincing results.\n\n2.   Weak co-evolution coupling.  Although the paper advertises a co-evolving coder and tester, in practice the coder’s reward still comes directly from ground-truth unit tests, and the tester’s reward assumes that any code passing those same tests is correct to estimate $p_u$, $p_{00}$, and $p_{01}$. This reliance on the same public test suite for both roles means the “co-evolution” is more superficial than synergistic.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes CURE, which is an RL framework, to jointly optimize a single model for code generation and writing unit tests. The framework only depends on ground-truth test cases and does not require any code solutions. The authors evaluate their approach for a variety of models across 5 public datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
- The considered setting where test-cases are more widely available over solutions is practically relevant for coding.
- The proposed reward is well motivated and properly justified.
- The paper provides rigorous evaluation, showing significant improvements across multiple datasets and tasks.

Weaknesses and Suggestions:
- Ablation on how many ground truth test-cases are needed and what happens when test-cases tend to zero is missing. 
- Clarify in the abstract that the approach relies on ground-truth test-cases.
- Error bars are only available for a subset of the results. 
- A qualitative analysis on the types of unit-tests generated by CURE would be useful to show that they go beyond trivial cases, and what they are still missing.

QUESTIONS:
- Can you comment on what happens during reward estimation if the ground-truth test-cases are either not sound or not complete? Specially in cases where the initial test-cases are minimal and incomplete, the tester seems to be penalized for catching buggy code. 

- In Figure 5, is CURE originally trained on the same unit-tests that are also used in the ‘labeled RL’ baseline?

LIMITATIONS:
Additional discussion on how the current evaluations are limited to algorithmic coding tasks in Python is needed.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03