Rethinking Verification for LLM Code Generation: From Generation to Testing
Authors: Zihan Ma
Taolin Zhang
Maosongcao
Junnan Liu
Wenwei Zhang
Minnan Luo
Songyang Zhang
Kai Chen
Venue: NeurIPS 2025 poster ()
Keywords: Test Case Generation
LLM Code Evaluation
Human-LLM Collaboration
Primary area: evaluation
TL;DR: Current LLM code evaluation is flawed by weak test cases; we propose SAGA, a novel method using human expertise to generate superior verifiers, demonstrated by our new CodeComPass benchmark and TCGCoder-7B model, for more reliable assessment.

Abstract: Large language models (LLMs) have recently achieved notable success in code‑generation benchmarks such as HumanEval and LiveCodeBench. However, a detailed examination reveals that these evaluation suites often comprise only a limited number of homogeneous test cases, resulting in subtle faults going undetected. This not only artificially inflates measured performance but also compromises accurate reward estimation in reinforcement learning frameworks utilizing verifiable rewards (RLVR). To address these critical shortcomings, we systematically investigate the test-case generation (TCG) task by proposing multi-dimensional metrics designed to rigorously quantify test-suite thoroughness. Furthermore, we introduce a human-LLM collaborative method (SAGA), leveraging human programming expertise with LLM reasoning capability, aimed at significantly enhancing both the coverage and the quality of generated test cases. In addition, we develop a TCGBench to facilitate the study of the TCG task. Experiments show that SAGA achieves a detection rate of 90.62\% and a verifier accuracy of 32.58\% on TCGBench. The Verifier Accuracy (Verifier Acc) of the code generation evaluation benchmark synthesized by SAGA is 10.78\% higher than that of LiveCodeBench-v6. These results demonstrate the effectiveness of our proposed method. We hope this work contributes to building a scalable foundation for reliable LLM code evaluation, further advancing RLVR in code generation, and paving the way for automated adversarial test synthesis and adaptive benchmark integration.

Rethinking Verification for LLM Code Generation: From Generation to Testing
Zihan Ma Taolin Zhang Maosong Cao', Junnan Liu', Wenwei Zhang',
Minnan Luo Songyang Zhang Kai Chen
Shannhaii Al Laboratory
School of Computer Science and Technology, Xi'an Jiaotong University, China
MOO KLINNS Lab, Xi' an Jiaotong University, China
{mazihan880) xjtu. edu.
{zhangtaolin, shan {minnluo) @ux tu edu.cn cn
org.
Abstract
Large language models (LLMs) have recently achieved notable success code-generation benchmarks such HumanEval and LiveCodeBench. However, detailed examination reveals that these evaluation suites often comprise only lim- ited number of homogeneous test cases, resulting subtle faults going undetected. This not only artificially inflates measured performance but also compromises ac- curate reward estimation in reinforcement learning frameworks utilizing verifiable rewards (RLVR). To address these critical shortcomings, we systematically investi- gate the test-case generation (TCG) task by proposing muult-ddimenionna metrics designed to rigorously quantify test-suite thoroughness. Furthermore, we introduce human- LLM collaborative method (SAGA), leveraging human programming expertise with LLM reasoning capability, aimed at significantly enhancing both the coverage and the quality of generated test cases. In addition, we develop a TCGBench to facilitate the study of the TCG task. Experiments show that SAGA achieves a detection rate of 90.62% and a verifier accuracy of 32.58% on TCG- Bench. The Verifier Accuracy (Verifier Acc) of the code generation evaluation benchmark synthesized by SAGA is 10.78% higher than that of LiveCodeBench--vb These results demonstrate the effectiveness of proposed method. We hope this work contributes to building a scalable foundation for reliable LLM code evaluation, further advancing RLVR in code generation, and paving the way for automated adversarial test synthesis and adaptive benchmark integration.
Introduction
Large Language Models (LLMs) have triggered a paradigm shift in automatic code generation, demonstrating capabilities on par with or even exceeding human programmers on numerous bench- mark tasks. As LLMs become increasingly integrated into software development workflows, ensuring quality and reliability of the code they produce is paramount. This necessitates reliable evaluation methodologies, where code perffiers-typical raises a crucial question: Are the test cases of current benchmarks for evaluating models' code capabilities robust enough?
powered by test suites- play critical role. This
Initial analyses of benchmarks like HumanEval [6] (avg. 7.7 tests/problem), MBPP [3] tests/prob- lem), and EvalPlus [28, 29] (which saw 15% pass rate drop with 80x more tests) indicate the
means corresponding authors, means project lead, x means authors contributed equally. The data demo and prompts be accessed via https: / /ttptttttttt////////////tg/...tt com/ open- compass/SAGA
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fragiliity of current evaluation setups due to sparse test coverage. While methods like TestEval [43] tailor tests to specific solutions, they are inefficient for large-scale evaluation and impractical for dynamic integration into RI. training loops, thereby hindering the development of models robust against diverse failures. LiveCodeBench [20] employs L.MSs to generate numerous tests from golden solutions and synthetic inputs, aiming to enhance robustness.
However, a fundamental concern is that such methods may inadvertently create tests biased towards typical, often homogenized, LLM er- ror patterns, which starkly contrast with diverse human reasoning errors. Conversely, compet- itive programming platforms (Online Judges) possess extensive, rigorously curated test suites for assessing code robustness, but these are often private and inaccessible. This underscores the urgent need for accessible and robust code ver- ifiers to enable reliable performance evaluation and reward estimation.
(a)
Pailure Roso%%)
Rogo
LLM
Number
Figure 1: (a) Verifiers synthesized primarily from -..M-generated data exhibit high failure rate when testing human- written bugs. (b) PCA analy- sis reveals that LLM-induced errors are highly clus- tered, indicating systematic weaknesses, whereas human errors are diverse and dispersed, posing greater challenges to existing verifiers.
In this work, we first identify limitations in current benchmarks through preliminary exper- iments. Our investigation reveals critical weak- nesses in existing verifier suites. Specifically, when we took L...-generttd solutions that had passed LiveCodeBench''s private tests and re- evaluated them on LeetCode's online judge, we found that for significant portion of these solu- tions-20% for medium and 40% for hard problems, respectively- LiveCodeBench's ss verifier had missed. This demonstrates that LiveCodeBench's 's verifiers can be flawed, failing to achieve comprehensive error detection and thereby overestimating the true quality of the LLM-generated solutions. Furthermore, these LLM-centric verifiers themselves exhibit a high failure rate (i.e.. they fail to detect existing bugs) when evaluating human-written faulty code, a rate much higher than any apparent failure rate on LLM- -edeerateeddd errors (Figure (a)). PCA analysis of error patterns (Figure (bb) reveals that LLM errors cluster tightly, indicating shared systematic biases, while human errors are widely distributed across complex error landscape.
identified errors that
These findings underscore the fundamental inadequacy of constructing verifiers solely from L.M-- generated data, as such approaches misassess LLMs' coding capabilities and provide flawed training feedback, manifesting two critical challenges: (1) Test Case Homogenization and LLM-Centric Bias: LLM-based TCG methods produce test suites that mirror the generating models' error patterns and cognitive biases, creating "homogenization trap" where tests focus on LLM-like failures while neglecting diverse human programming errors (e.g.. logical flaws, integer overflows). (2) Verifier Ineffectiveness and Persistent Blind Spots: Verifiers built on such test suites exhibit blind spots for human-like like errоrs, failing to rigorously evaluate code due to challenges in generating tests for complex boundary conditions and interaction scenarios, compounded by LL-M-centric test design. These challenges impede reinforcement learning frameworks (e.g., DeepSeek-R1 [8], O1-101 (10) from leveraging verifiable rewards, leading to optimization misdirection via reward hacking".
To address these challenges, we develop a comprehensive measurement framework for code verifiers, introducing multi-dimeesional evaluation metrics (detection rate, verifier accuracy) and identifying test case diversity and per-case strength as critical quality factors. We derive upper bound for detection rate and validate it empirically across 1,500+ coding problems. Using this framework, we uncover systemic test quality issues in leading benchmarks, such as CodeForce-Con CoTT [39], where 50% of problems had tests failing to detect known errors and 84% of verifiers were flawed, further underscoring these limitations.
To systematically improve the quality of code verifiers, we formally define the Test Case Generation (TCG) task, comprising three core components: task formulation, benchmark construction, and method exploration. We introduce TCGBench, a benchmark curated by aggregating representative problems from three leading competitive programming platforms -atoorrs-Atcoder, Codeforces, and Now- coder. TCGBench curates human-verified adversarial examples spanning diverse error patterns (e.g..
Models exploit verifier weaknesses instead of achieving genuine correctness, as they are insufficiently penalized for diverse errors.

---

logical flaws, edge cases) and supports comprehensive evaluation of TCG methods. We further investigate current TCG methods and propose SAGA (Strategic Adversarial & Constraint-differentia GenerAtive workflow), novel human-LLM collaborative framework. SAGA is designed to system- atically generate high-coverage, highly discriminative test cases by leveraging both human-derived constraints from correct solutions and insights from failure modes in incorrect solutions. This dual- pronged analytical approach allows SAGA to achieve improvements over current state-of-the- TCG methods, with the Detection Rate increasing by 9.55% and Verifier Accuracy by 12.14%.
Additionally, we leverage SAGA to enhance the quality of the popular code generation benchmark LiveCodeBench-vó (subset)', and develop CodeCompass, a new high-quality code benchmark. We believe SAGA can further be employed scale datasets, enabling the production of training data with robust and accurate reward estimation for coding tasks. Our main contributions are as follows:
We construct TCGBench, comprehensive dataset from competitive programming platform, to analyze existing Test Case Generation practices. On it, we formalize multi-dimensional metries for rigorous test case quality evaluation.
We propose and validate SAGA, a novel human-LLM collaborative TCG framework (Fig. 5). By integrating insights from both correct and incorrect human solutions, SAGA generates significantly more effective test suites, improving Verifier Accuracy by 15.86% over existing TCG methods.
Using SAGA, we develop CodeComPass, a challenging benchmark with human-verified adversarial examples for robust code generation evaluation, and introduce TCGCoder-7B. SAGA-distille specialist model for capable TCG.
2 Related Work
Reliable evaluation is paramount for advanced code- sedde-eeneaaingg LLMs [37, 13], especially in Rein- forcement Learning from Verifiable Rewards (RLVR) contexts [8, 31]. However, existing benchmarks often exhibit limitations such LLMM-cenrric biases and insufficient test diversity [28, 20]. This underscores the need for more effective Test Case Generation (TCG). Current LLM-based TCG methods, primarily Direct Generation [23] and input-Interpreter [20], still struggle to produce com- prehensive and challenging test suites that cover subtle corner cases. Our work, SAGA, addresses this gap by introducing novel human -LMM collaborative framework designed to systematically generate higher-quality tests by leveraging deep human insights. A comprehensive review of related works is provided in Appendix A.
3 Evaluating Verifier Quality: Metries and TCG Paradigms
The reliable evaluation of LLLM-ggeerrtte code is critically constrained by the quality and accessibility of verifiers. Standard bench- marks like HumanEval [6] often employ lim- ited test suites, while the extensive private ver- ifiers of Online Judges remain largely inac- cessible for broader research. This scarcity of robust, accessible verifiers impedes accu- rate LLM evaluation and the advancement of RLVR. To surmount this challenge, we focus on leveraging L.MSs themselves for Test Case Generation- the systematic synthesis of test suites. Figure 2 illustrates the pivotal role of a Code Verifierin the LLM code evalua- tion pipeline. It also highlights distinct TCG paradigms:
Direct Generation
Prefdiee
Imput-hased Test Caw Generation
Mathod
Piiss
Figure 2: The code evaluation pipeline and different TCG paradigms.
"The subset used in our study comprises 101 problems from AtCoder, specifically from contests ABC387 through ABC400 and ARC190 through ARC196 (problems A to F). This selection facilitates a consistent evaluation scope when comparing with benchmarks like our proposed TCGBench, which also incorporates these recent algorithmic challenges.
3

---

Afcoder
Nowcoder
CodeForce
Aloader Bawooder
Relenken
63
Notli Assrtiy
Performenc Rate (6))
Soluta Model
Qnex-728-
661
24
669
A11
447
4.
337
64
41
Coder
B6.13
Orea Coder
offidd
D5.33
Qene-728
DS-33
Qrm--22
Owna Coder
128 V/3
Qwen 728
Qwen
TwennCoder Coder
FCG Model
TCG Medel
TCG Model
Figure 3: Direct generation issues: (a) Low quality of LLM --M--neertted tests. (b) High self seff-pass rates suggest model blind spots.
Direct Generation: An LLM directly produces complete test cases (inputs and outputs). Representa- tive works include TestChain [23], AceCoder [48], and CodeRM [33].
Input-Interpreter An L.MM generates test inputs; ground-truth interpreter (or reference solution) then computes the corresponding outputs. This paradigm, exemplified by LiveCodeBench [20] and Codeforce-COn [39], often employs random input sampling- strategy proven effecttive by Live- CodeBench for generating numerous diverse test cases and thus adopted in our baseline evaluations. EvalPlus [28], which mutates seed inputs for execution, also aligns with this approach. A comparative analysis with EvalPlus is detailed in Section 4.2.1.
Human Priors (Our Approach): LLM-driven TCG is guided by structured human expertise, strategy central to our proposed SAGA framework (detailed in Section 4).
3.1 Problem Definition
Formally, for a programming problem P E P with description D, input space Xp, and ground-truth solution fP= Xp + Yp, TCG method aims produce case set T [[T, 0:) where each input 1, E Xp and its corresponding output O = fP()). To quantify the quality of such generated test suites, we employ two key metries:
Definition 1 (Detection Rate (DR)). A solution-level metric that answers the question: "For specific buggy program, can our test suite find at least one bug?" Mathematically, DR measures test suite's 's ability detect an error in a single incorrect solution S # fp. is the probability that at least one test case in T causes S to fail. Let E, be the event that S fails on test case i. The detection rate is the probability of the union of these events: (s(T) = P(U E.). The final reported DR for TCG method is the average of these binary outcomes (1 if a bug is found, 0 otherwise) across all known incorrect solutions for given problem set.
Definition (Verifier Accuracy (VAce)). A stricter, problem-level metric that answers the question: "For a given problem, can our test suite find bugs in all known incorrect solutions?" VAcc evaluates the systematic completeness of the verifier. is a binary indicator that equals 1 if and only if the test suite T successfully detects failure in every incorrect solution within the set Swnmgg (P). Formally: VAce(T) = (AS Swonng! (P), (T) > 0), where B(.)) is the indicator function. VAcc(T) signifies that the verifier is robust enough to reject all known faulty programs for that problem.
3.2 Investigating Current TCG Paradigms and Their Limitations
To ground our exploration of TCG effectiveness, we leverage TCGBench, a dataset we curated comprising 1840 recent programming problems from Atcoder, Codeforces, and Nowcoder, along with an average of 36.66 incorrect user submissions per problem. This rich resource (detailed in Appendix D) facilitates rigorous evaluation of TCG methodologies. Our analysis employ open- source LLMs: DeepSeek-V3-0324, V3-0324, Qwen2.5-72B IIn--uutt and Qwen2.5-Coder- Cooder-32B-Instruee with the greedy decoding strategy.
For paradigm 1: Is Direct LLM-based Test Case Generation Effective?
Directly prompting an LLM to generate complete test cases (inputs ILLMM, outputs OLLM) from problem description P, depends heavily on the LLM's deep comprehension, particularly of edge cases. Our experiments (Figure 3), involving LLM generation of 50 diverse test cases per problem,

---

(a).
Atcoder
CodeForce
NowCoder
0.850
0.875
0.82
0.825
0.80
0.800
Detection Rate
0.825
0.775
0.78
Detection
0.750
Datection
0.76
0.775
0.725
0.74
0.700
DuspSenk
0.72
DampSenie
0.750
DerpStok-VV
0.675
Owend2 5 720
Queend S-Creder
0.70
Quen2,5-Code
Qwea22.5 720
Oween2 5 723
Owent Sebneer
0.650
0.700
6 Test Case Sizee
(b)
Test Case Size
Test Case Size
0,850
0.82
0.875
0.825
0.850
0.800
Detection Rate
0.825
0.70
Delection
0.70
0.775
0.725
0.74
0.700
DespSinne-V
0.72
DeepSerk
0.750
DergSeok
0.679
Qwen2.5 5 728
Qownl Seladdrr
0.70
Qwea2.5 Coder
0wn22.5-721
0.725
Owen2.5 Coder
Dwim2.5-7215
0.650
4p of &.80
A
0,700
@
Test
Test Case Size
Test Case Size
Figure 4: Experimental validation of Intt--nntrrrreter (random sampling) limitations on TCGBench. (a) Detection rate number of test cases (linear scale), showing clear saturation below 100%. (b) Detection rate vs. log of the number of test cases (semi-log scale), illustrating diminishing returns consistent with the theoretical upper bound - (1 Muff,
noff, validating the impact of correlation p.
reveal the issues. The retention rate (proportion of valid tests post-verification against ground truth) very low, indicating unreliable quality. This results in poor overall DR (often <60%) and VAce (<10%). Moreover, L.M -e--eeeretted solutions easily pass these self-generated tests. Notably, on AtCoder problems with historical official tests", LLM solutions performed substantially better on their own generated tests, suggesting such tests fail to challenge the model's cognitive biases.
For paradigm 2: Can Large Number of Inputs from an Input-Interpreter Approach Compen- sate for Low Quality?
The Input-Interpreet paradigm, where LLM generates random inputs I.Rand X'p for ground- truth interpreter, is employed by benchmarks like LiveCodeBench [20]. While generating many tests feasible, merely increasing the quantity does not fundamentally improve the detection rate. This limitation arises from inherent correlations between test cases. We propose here a corollary for the upper bound of the detection rate":
Corollary (Asymptotic Saturation of Detection Rate). As the number of generated test cases n. -> oc, if p (the average probability of single test detecting an error, 0 < p < 1) and Pett (the effective average positive correlation between detection events, Peft > 0) are stable characteristics, the approximate upper bound on the detection rate (s(T) converges to: <1
limn-+o0 (T) x 1-
This corollary implies that due to inter-test correlation Pett, simply increasing the number of random tests cannot guarantee 100% error detection; the detection rate will saturate. The underlying reasoning involves the concept of an "effective sample size" noff R3 n [11+ (n 1) ) [21], which quantifies the diminishing utility of additional correlated tests. Our experiments on TCGBench (Figure 4) confirm this: detection rates plateau (Fig. 4(a)), and marginal gains diminish rapidly. Notably, plotting the detection rate against the logarithm of the number of test cases (Fig. 4(b)) reveals a trend consistent with our derived theoretical upper bound, further validating the impact of correlation p.
Beyond DR and Acc, to more deeply quantify the intrinsic quality and efficiency of TCG strategies like SAGA (introduced in Section 4), we employ two advanced metrics. These provide observable insights into test suite characteristics related p (average test potency) and Pett (inter-test correlation):
'Early AtCoder problems provided official test cases; however, from December 2024 onwards, test cases are no longer publicly available for new contests.
'The full derivation is provided in Appendix C.
5

---

Distinct Error Pattern Coverage (DEPC): For a test suite T and Np problems. let v(tx) be the
error pattern vector for test 4.- DEPC is {o(lz) 4. E 7 and [|((4)]] 2 11 The Diversity
Ratio is DEPC(T)/n Higher DEPC suggests lower Pott, indicating broader unique error detection. Normalized Area Under the Accuracy-Number of test cases Curve (AUC-AceN): For Verifier
Accuracy (k)(k) with & tests up to N,
N
Aco(k; + Acc((k+1) (ki+1 k2)-
AUC@N R
N Kmin i= immmm
2
Higher AUC@N N indicates superior average verifier accuracy, reflecting potent (high p) and efficiently diverse tests.
Detailed explanations and derivations for these metrics are in Appendix B.
4 SAGA: A Human-LLM Collaborative Framework for Advanced TCG
The preceding analysis in Section 3.2 revealed critical limitations in prevalent Test Case Generation (TCG) paradigms. As demonstrated, Direct Generation suffers from low test quality (Figure 3), while the Input-Interpreter approach constrained by test case homogeneity and saturating performance returns (Figure 4). To address these shortcomings, & Constraint-differentall GenerAtive workflow), a novel human-LLM collaborative framework (Figure 5). SAGA systematically generates high-quality, diverse, and discriminative by maximizing test potency (7) and diversity (lowering correlation p). Additionally, we trained TCGCoder-7B, SAGA-distilled 7B specialist model from 15,000 problems (details in Appendix J), as strong TCG baseline and reference.
introduce SAGA (Strategic Adversarial
suites
4.1 The SAGA Framework: Integrating Human Expertise
Recognizing the limitations of naive TCG, SAGA explores the integration of human ex- pertise. Intuitively, leveraging human problem- solving insights should enhance TCG. Table shows that incorporating simple human priors (e.g., boundary values from Shuman) into basic TCG paradigms on TCGBench-Full yields some improvement. However, these gains are often marginal, failing to fully exploit LLM potential for detecting complex flaws. While methods like EvalPlus [28] use human solutions, it's often for mutating seeds or formatting inputs, not deeply guiding large-scale challenging test generation.
Table 1: Initial Impact of Incorporating Simple Human Priors (Smmma) into Basic TCG Paradigms
on AtCoder Results (Accuracy @ 50).
Method
Accuracy @55
Direct Gen. (Paradigm 1)
11.30%
Direct Gen. Simple Priors
15.06% (+3.76%)
Input-Interpreter (Paradigm 2)
23.36%
Input-Interpreter Simple Priors
27.95% (+4.59%%
SAGA advances beyond such superficial integration. As depicted in Figure it deeply incorporates multifaceted human programming insights-from both correct solutions (GroundTruthh, Spuman) and incorrect submissions (Human Bugs, analytical strategy. An LLM is fed the problem description /' and insights gleaned from human solutions through customized prompting module. This process generates Python Case Scripts to produce test inputs, accompanied by Math Explanations and Self- Validation code to ensure correctness and relevance. The generated inputs are then processed by an Interpreter (ground-trey -trtthhhhhhhhh solution) to yield test outputs, forming the final test cases. SAGA's core analytical dimensions are:
with LLM reasoning via a structured, dual-pronged
Multidimensional Analysis (Leveraging Snuman): This dimension extracts profound insights from correct solutions to engineer challenging tests. It involves:
1. Constraint Handling Differences: Discrepancies in how Swrong and S manage problem-
specific constraints.
2. Defense Pattern Deconstruction: This is where the Math Explanation component (Figure 5) plays a key role. transforms the implicit programming logic from human code into explicit, structured test instructions. Diverse defensive logic and problem-solving strategies within Shuman are decomposed into formal mathematical logical constraints (e.g., "equivalence class: player pairs", "boundary_ value: [(1,2), (N."- This process allows SAGA to purposefully

---

<>
Multi-Dim Analysis
Python Scripts
sirateg""
Inputs Daput (731 Sapur" "a P Caenne
GroundTruth
Anddey,aia"" Muaal (iaad)) "SerEE valdd
For 2: nepett 10
an. bay (eetlin) aloers set/rappe Mel)
Math Explanation
Test Inputs
Iry:
Interpreter
Problems
f rado
setchu sgaret)"1s) 641)
issert co cr sent, 1.!
Outputs "autpu" oaar outpal
Differential Analysis
recure Tha
reture P'DCa 3al (e0ytatees)
mine Falorr stroe
Human Bugs
Case Scripts
Self Validation
Test Outputs
Figure 5: Overview of the SAGA framework. SAGA leverages both GroundTruth (correct human solutions) and Human Bugs (incorrect submissions) alongside the Problem description. An LLM performs Multi- Salti-Dimmensonall Analysis and Differential Analysis to generate Python Case Scripts for test input synthesis. These scripts are accompanied by Math Explanations (capturing testing strategies and constraints) and Self- Validation code. The generated Test Inputs are then passed to an Interpreter (ground-truth solution) to produce Test Outputs, forming the final test cases.
larget the core mathematical and logical properties of the puublem-such as singularities, extremal values, or specific structural properties-guide guiding the generation of meaningful edge and adversarial test cases that simple random generation would miss.
3. Targeted Test Generation: Using these constraints and pitfalls to guide the LLM in constructing challenging test inputs I via the Case Scripts.
This analytical approach aims to generate test cases that not only cover a wide spectrum of valid scenarios, including complex boundary conditions and intricate interactions, but also enhance test diversity (lowering p) and individual test potency (increasing p).
Differential Analysis (Leveraging Swrong): This addresses error types missed by analyzing only correct solutions. It compares failed submissions (Swrong) with their corrected versions (S% to find inputs Lait where Swongg [ditt) + Seurrea (Aaiit) revealing common error patterns. This targets: 1. Constraint Handling Differences: Discrepancies in how Swrmn and S manage problem- specific constraints.
2. Lack of Defensive Completeness: Deficiencies in Swning related to handling edge cases or boundary inputs, as revealed by comparison with Soomea-
3. Failure Pattern Analysis: Generating specific inputs that trigger failures in Swing but are correctly handled by S'orrect
The incorporation of these differentially identified inputs lan into the test suite creates a more rigorous and challenging evaluation framework, as it specifically targets known failure modes, thereby substantially increasing the discriminative power of the resulting verifier.
The Self- Validation scripts (Figure 5) ensure that generated test inputs adhere to problem constraints and the intended testing strategy before execution. For Multidimensional Analysis. SAGA leverages insights from 10 distinct, correct user solutions per problem to broaden perspective. In Differential Analysis, meticulously pairs a user's correct solution with their most recent preceding incorrect submission. This focus closely related yet differing attempts enhances SAGA's ability to identify subtle error patterns and generate challenging corner cases. By synergistically combining these refined analytical dimensions, SAGA produces comprehensive and highly effective test suites.
is important to note that SAGA is designed for efficiency. The main computational cost lies in one-time analysis phase per problem, where the LLM performs a single, powerful analysis of human solutions to generate Python Case Script. Crucially, is this generated script- not the LLM that then programmatically generates a large volume of test inputs. This design avoids the high cost and latency of repeatedly invoking the LLM for every single test case, making the framework scalable.
4.2 Experimental Validation of SAGA
SAGA's effectiveness was initially validated the TCGBench. Figure 6 visually summarizes SAGA's multifaceted superiority in this context, comparing against the random Input- Interpeetrr baseline and its core analytical components (Multidinensional Analysis and Differential Analysis).
Key Findings from Comparison on TCGBench (Fig. 6): SAGA markedly improves solution vetting capabilities: its DR surpasses 93.81% (vs. baseline's 's 82.85% at 100) and VAce reaches
7

---

(a)
53.015
Bencian
Dill
(d)
Diter SAGA
Detection Rate 1%
BLAN
w MahiDin
SM0s
Verifler Accuraac
its
Diverdry Ratio
eg.
Digg 100:
DEPC
13.
wi MuRIDio titter
-8-
10
0
10
Test. Case Sizee
20
do
Test Case Size
33
Case
Test Sizee
Figure 6: SAGA outperforms the Baseline (Random Input-Interpreted and its individual analytical components (Multidimensional Analysis leveraging Shuman: Differential Analysis leveraging Swons) on the AtCoder subset of the full TCGBench across: (a) Detection Rate, (b) Verifier Accuracy (with (c) Distinct Error Pattern Coverage (DEPC), and (d) Diversity Ratio. Dotted lines in (a) & (b) show baseline performance at n 100. See Appendix F for SAGA's performance on other platforms within the full TCGBench.
AUC 50 values),
41.33% (vs. baseline's 21.89% at n 100) with only 50 tests. SAGA's AUC@50 50 (0.5445) more than doubles the baseline's (0.2586). showcasing superior efficiency. SAGA also generates test suites of superior intrinsic quality: it achieves the highest DEPC (broader error coverage, lower p) and Diversity Ratio (more efficient error discovery per test). Both Multidimensional and Differential analysis components individually outperform the baseline, but their synergy in SAGA yields optimal results. This robustly validates SAGA's systematic leveraging of human insights on a large scale.
4.2.1 Main Results and Analysis on TCGBench-Lite
For focused main comparisons and ablation studies, we curated TCGBench-Lite, a challenging subset of 270 problems from AtCoder, Codeforces, and Nowcoder contests since June 2024. This ensures contemporary relevance and minimizes potential data leakage. To further guarantee the impartiality of our results, particularly for the evaluation of our SAGA-distilled model TCGCoder- 7B, we employed a strict chronological split: the training data for TCGCoder-7B was sourced entirely from problems published before 2023. This temporal separation ensures the model is evaluated on genuinely unseen problems. TCGBench-Lite includes an average of 41.41 incorrect submissions (Swoong) per problem. Its difficulty distribution (Easy: 27.04%. Medium: 32.59% Hard: 40.37%) was determined by platform tags and contest characteristics (details in Appendix E). This curated set allows for rigorous yet manageable evaluation. Unless specified, all LLM M-diveenn methods use DeepSeek-
DeepSeek-V3-0324 V3-0324 as the backbone for fair comparison. We evaluate against the Input-Interprett (LiveCodeBench-stie random sampling), TestChain [23], EvalPlus [28], and our SAGA-distilled TCGCoder-7BB Results are in Table 2.
Analysis of Main Comparison (Table 2): SAGA's structured integration of human insights yields substantial gains over other TCG methods on TCGBench-Lite. Notably, SAGA achieves an AUC 50 of 0.2228, surpassing the Inpur-enterpreter (0. 234), which is limited by random sampling's 's homo- geneity. This highlights SAGA's ability to produce more consistently effective tests. While EvalPlus achieves high raw Diversity Ratio through mutation, its lower AUC@ 50 (0. 1278) suggests that SAGA's deep analysis of human solutions is more critical for overall verifier quality than mere test variety. TestChain, lacking rich human priors, performs weakest. Crucially, our SAGA-distilles TCGCoder-7B (AUC@ 50: O. 1890) outperforms all these established baselines, even when they utilize a much larger backbone model (DeepSeek- (eepSSee--V3-0324) V3-0324). This demonstrates SAGA's potential to distill effective TCG strategies into smaller, specialized, and highly capable models, thereby making advanced TCG more accessible.
Analysis of Ablation Studies (Table 2): Value of Structured Human Insights: Isolating SAGA's Multidimensional Analysis (leveraging Stuman) and Differential Analysis (leveraging Swrong) reveals that both significantly outperform simpler prompting (SimpleCOT) and the Input-Interpreed baseline in terms of AUC 50. This underscores that structured analysis of human solutions, whether correct or incorrect, is fundamental for higher-quality verifiers. The full SAGA framework, synergistically combining both analytical dimensions, achieves the optimal overall quality (highest AUC@50), confirming their complementary benefits. Importance of Prompi Design: The substantial performance degradation when replacing SAGA's detailed, drsight-driven prompts with generic CoT (SimpleCOT) highlights that the methodology of integrating human insights is as critical as the insights themselves
8

---

Table 2: Experimental Results: Main Comparison of SAGA with Baselines and Ablation Studies
on TCGBench-Lite. Metries are @ VAco@ k. AUC@ 50, and DivRatio @ 50. All methods use DeepSeek- backbone unless specified in ablation.
Method / /Confiuuratinn
DR@20 DR@50
VAcc@20
VAcc@
AUC@50
DivRatio@50
Main Comparison with Baseline TCG Methods
TestChain [23]
65.91%
68.31%
8.12%
11.88%
0.0841
50.09%
cnttr-otterrreter [LiveCodeBeench-tty. [20])
77.84%
81.07%
12.36%
16.72%
0.1234
79.42%
EvalPlus [28]
67.52%
71.12%
11.56%
15.15%
0.1139
79.27%
TCGCoder-
model)
85.14%
89.44%
17.93%
29.11%
0.1890
94.43%
SAGA (DeepSeek-V3 Backbone)
85.66%
90.62%
22.40%
32.58%
0.2228
94.06%
SAGA Ablation Studies
Analytical Component Ablation
SAGA w/ Multidim. Analysis only
84.51
88 000%
20, 20.70%
26.05%
0.1923
95.81%
SAGA w/ Differential Analysis only
84.31
88.16%
19.85%
26.67%
0.1926
94.41%
Prompt Design Ablation
SimpleCOT Prompt SAGA
83.36%
84.54%
15.61%
19.11%
0.1424
96.23%
Random Input w/ GT for SAGA
82.31%
36.64%
16.44%
22.70%
0.1616
85.38
EvalPlus w/ GT for SAGA
76.72%
79.56%
11.67%
20.44%
0.1278
89,49%
Base L.M Ablation for SAGA
SAGA w/ Qwen2.5 Co--er------nnstue
78. 88%
79.78%
19.70%
22, 26%
0.1810
96.80%
SAGA Qwen2.5-7225 2-enn.5-72--l11 Instruct
82.77%
85.08%
20.30%
26.46%
0.1943
94.92%
SAGA w/ S-reen..-Coder-22B---
86.25
90.54%
20.74%
32.73%
0.2139
94.72%
for effective TCG. Robustness Across LLMss: SAGA demonstrates commendable robustness when paired with different LLM backbones. While peak performance on specific metries can vary with yields the highest VAcc@ 50), the SAGA framework with its default DeepSeek-V3 backbone consistently provides the best overall AUC@ 50. This indicates SAGA's comprehensive design is more impactful for holistic verifier quality than relying on specific LLM capabilities or optimizing for isolated metrics like Diversity Ratio. Notably, SAGA also shows strong performance with Qwen2.5-Cove -onder-- IBIInstr (TCGCoder-7B's base model), further validating SAGA's efficacy on smaller, specialized coder models.
the LLM (e.g., SAGA with (0ven2.5-Codee=322--niis
5 Towards Advanced Applications of SAGA
SAGA's proven capability in generating high-quality, diverse, and discriminative test suites (Section 4) enables advancements in LLM code evaluation and training. This section primarily explores the development of superior code generation benchmark, CodeComPass,, while also noting the potential for SAGA-enhanced verifiers to contribute to more reliable Reinforcement Learning from Verifiable Rewards (RLVR).
CodeCompass: A LAGA-Ennancee Benchmark for Code Generation Evaluation
To address the need for more challenging and discerning evaluation of LLM code generation models, we introduce CodeCompass. The verifiers within CodeCompass are synthesized using SAGA for the 270 contemporary problems that also constitute TCGBench-Lite (detailed in Appendix E), ensuring relevance and minimizing data leakage risks. Each problem in CodeCompass features a rich test suite, averaging 50.54 SAGA-generated test cases, meticulously curated ensure comprehensive coverage (with additional manual curation applied initial generation yielded fewer than a target threshold).
For comparative analysis of verifier quality and impact model evaluation, we focus shared subset of 101 AtCoder problems that overlap between our CodeCompass verifiers and the test suites used by LiveCodeBench--vó This allows for a direct comparison on common ground.
Superior Verifier Quality CodeCompass. We first establish the intrinsic quality of CodeCompass verifiers using core TCG metrics against those of LiveCodeBench-v6 on the shared AtCoder subset (Table 3). CodeCompass demonstrates markedly higher efficacy identifying faulty solutions, greater efficiency in discovering unique error patterns (Diversity Ratio @40: +43. 133%). and faster convergence to quality. This confirms that SAGA produces verifiers that are intrinsically more diverse, discriminative, and effective.

---

60
394T
Drop Ratio (N)
Table 3: Verifier Quality: CodeCompass LCB-v6 (Shared Subset, @40 tests).
55
55.45
13.6
Metric
LiveCodeBench-v6
CodeCompass
52.67
51
DR@40
78.85 78.85%
93.44%
VAce@-40
19.61%
30.39%
AUC@40
DivRatio@:40
53.56%
0.1388
96.69%
0.1980
4.5
45..44
pass
Models
Average Pass@] (%6)
90.8%
86,256
BincCoodBamch CodeeC
38.81
Qwea2.5-128
G774
35
DespSndeV1
UWg--za2
33.8%
30,69
Qwesk
(1615)
15,8%
11.7%
26
25
LiveCodeBench CodeCompass
Easy
Medium
Hard
Figure 7: Avg. Pass @ by Difficulty.
Figure Model Ranking Changes.
Enhanced Discriminative Power for Code Generation Model Evaluation. This superior verifier quality translates directly to challenging and discerning evaluation for LLM code generation models. As shown in Figure 7, CodeComPass consistently elicits lower average Pass @ 1 rates across all problem difficulties compared to LiveCodeBench--6 the shared subset, indicating a more rigorous test. Critically, this increased stringency enhances discriminative power (Figure 8); when evaluated CodeComPass, the average Pass@) for various models drops by relative 9.56% compared to their performance on LiveCodeBench-vol This relative decrease leads to re-ranking of models (e.g., Qwen2.5-72B and Swve22.--Coder-222 switch relative positions). This ability to expose nuanced differences in model capabilities confirms that CodeComPass offers more robust and insightful assessment of true code generation proficiency.
Meanwhile, the superior SAGA-generated verifiers, exemplified by CodeComPass, promise to enhance Reinforcement Learning from Verifiable Rewards (RLVR) frameworks [8, 10] by delivering more accurate reward signals and mitigating reward hacking, thereby fostering more robust and capable code generation models.
Conclusion
This paper critically re-evaluates LLM-based Test Case Generation (TCG), highlighting current verifier limitations and formalizing key quality metrics alongside TCGBench, a foundational TCG research dataset. We introduce SAGA, novel human-LLM collaborative framework that integrates human programming insights with LLM reasoning to demonstrably generate superior test suites. Leveraging SAGA, we developed CodeComPass, an enhanced verifier suite for robust code gen- eration evaluation, and distilled TCGCoder-75 specialized, efficient TCG model. Collectively, these contributions significantly advance reliable LLM code evaluation and pave the way for future automated test synthesis and effective adversarial testing.
Acknowledgment
This work was supported by the National Nature Science Foundation of China (No. 62192781, No. 62272374), the Natural Science Foundation of Shaanxi Province (2024JC-JCQN-62) the State Key Laboratory of Communication Content Cognition under Grant No. A202502, the Key Research and Development Project in Shaanxi Province (No. 2023GXLHH-024), Shanghai Oriental Talents Project BJZH2024070. We also thank the OpenCompass team Shanghai Al Laboratory and our colleagues for their support and discussions.
10

---

References
[1] Saswat Anand, Edmund K. Burke, Tsong Yuch Chen, John A. Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, and Phil McMinn. An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw., 86(8):1978-22 -000,, 2013. doi: 10.1016/JJJSS 2013.02.06..... https:////d/o org/10 1016/j 061.
jss 2013.001100 02.
[2] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonogiou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Ka- reem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 1800, 2023. doi: 10.48550/ARXXI.222 11805. URL https: //doi. http://dtooooo/////////. 10. 48550/ aativ///rxivvv//a 2312. 11805.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108 077732, 2021. URL httth::////attt org/ htt//bs/21000 07732.
[4] S. Beyleda and V. Gruhn. Bintest search-based test generation. In Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003, pages 28 28-33, 2003. doi: 10.1 0T09/CMMPAAC.223312455533
[5] Cristian Cadar and Koushik Sen. Symbolic execution for software testing: three decades later. Commun. ACM, 56(2): 90, 2013. doi: 10. 10.1145/208777.2400755. URL https: //doi.org/10. /111/114444471 2408776 2408795.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel fierbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.033777 2021. URL https: : / aarxiviivvvv... . org/ abs 2217 03374.
[7] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-dehug. In The Twelfth International Conference on Learning Representations, 2024. URL
[8] DeepSeek-AN AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URI.
wtw.:org//bbbba/eeeeeee 12948.
[9] Zhiyu Duan, Yujia Li, Pubo Ma, Xiaodong Gou, and Shunkun Yang. A multi-layer fault triggering framework based on evolutionary strategy guided symbolic execution for automated test case generation. In 22nd IEEE international Conference on Software Quality, Reliabilit, and Security, QRS 2022 Companion, Guangzhou, China, December 5-9, 2022, pages 255-262. IEEF, 2022. doi: 10.1 109//RS--555511 0S0DRSS-557188.222200002 URL https: //tp///oii.ooo www.i.oor///a. 1ttgg/10.100 1109/ QRS-C57518 2022. 00045.
[10] Ahmed Kl-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera Gilaberte, Jakub Pachocki, Jerry Tworek, Lorenz
11

---

Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive programming with large reasoning models. CoRR, abs/2502.06607,, 2025. doi: 10445500/ARXXV.2500006600 Utt.https://doi. wwtp//1...05///221....ovoo 2502. 06807.
[11] Andrea Fioraldi, Alessandro Mantovani, Dominik Christian Maier, and Davide Balzarotti. Dissecting american fuzzy lop: A fuzzbench evaluation. ACM Trans. Softw. Eng. Methodol., 32 (2):52:1-52:26, 2023. doi: 10. 10.1145/3580596 URL httph::ttp/oo org/ 10. 1145/3580558
[12] Garrett M Fitzmaurice, Nan M Laird, and James H Ware. Applied longitudinal analysis. John Wiley & Sons, 2012.
[13] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. CoRR, 1412440..4119,, 2024. doi: 1048550ARXIV.2401..444 14196. URL https: http:::/ddoi.rrgee org/10. 18550/arXiv 2401 14196.
[14] Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45(1), December 2012. ISSN 0360-0300. doi: 10.1 (0.1145/237977722379781 https///s org/ 10. 1145/2379776. 2379787.
[15] Dong Huang, Qingwen Bu, Yuhao Qing, and Heming Cui. Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024. URL https: :tttpps://azee http://wwwww../aaaaa/ao arrg/abs/2308 088./
08784.
[16] Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agentt-aant based code generation with iterative testing and optimisation, 2024.
URL https: /wtp/v/aaivvvvvaa org/ abs/2312.... 13010.
[17] Muye Huang, Lingling Zhang, Jie Ma, Han Lai, Fangzhi Xu, Yifei Li, Wenjun Wu, Yaqiang Wu, and Jun Liu. Chartsketcher: Reasoning with multimodal feedback and reflection for chart understanding, 2025. URL https: http///axvvvv.ooo
abs/22555 19076.
[18] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
[19] Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. Testgeneval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference Learning Representations, 2025. URI. pttttps://ppeneweee
net. htl/fformmidiee 7d=
[20] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
[21] Leslie Kish. Survey sampling. york: John wesley & sons. Am Polit Sci Rev, 19(5 1965.
[22] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: / /wwppsapppppsseaaa nips cc/ cc/paper_fiitceccccceee fwwes/paper//// hash /tta////661191aaaa1a2222
Abstract-
.tmll.
[23] Kefan Li and Yuan Yuan. Large language models as test case generators: Performance evaluation and enhancement. CoRR, abs/2404. 13340, 2024. doi: 10.485500/ARXIV.2004.11 URL https:/ /doii org/10. 48550/ arxiv 220.4. 2404. 13340.
12

---

[24] Raymond Li. Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii. Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel I Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy V, Jason T. Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding. Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! Trans. Mach. Learn. Res., 2023, 2023. URL https: : /ttp:///pppeeeeeeeee/ee net/ http://fooummddooo04411....
[25] Rongao Li, Jie Fu, Bo- Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint 2xXi:2332211852: 2023.
[26] Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' A'Attmme, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition--evee code generation with alphacode. CoRR, abs/2203.07814 2022. doi: 10.48550/ARXIV.2203.. 07814. URL https: /doii org/ 10. 48550/ar55ii 2203. 078144.
[27] Guanghan Liu and Kung-Yer Yee Liang. Sample size calculations for studies with correlated observations. Biometrics, pages 937-947, 1997.
[28] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang. and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In sevethh Conference on Neural Information Processing Systems, 2023. URL https: / //openrevie.
[29] Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang, Evaluating language models for efficient code generation. In First Conference on Language Modeling, 2024. URL https: httpe:///pppevveeeeeee wet.riiwwwweeeeeeeeeeee..ee.eeee -e
[30] Kaibo Liu, Yudong Han, Yiyang Liu, Jie M. Zhang, Zhenpeng Chen, Federica Sarro, Gang Huang, and Yun Ma. Trickybugs: A dataset of corner-case bugs in plausible programs. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR), pages 113-117, 2024.
[31] Shudong Liu, Hongwei Liu, Junnan Liu, Linchen Xiao, Songyang Gao, Chengqi Lyu, Yuzhe Gu, Wenwei Zhang. Derek Wong, Songyang Zhang, and Kai Chen. Compassverifier: A unified and robust verifier for Ilms evaluation and outcome reward, 2025. URL https:
// //arxiv. org/ abs/2508. 03686.
[32] Sharon Lohr. Sampling: design and analysis. Chapman and Hall/CRC, 2021.
[33] Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. Dynamic scaling of unit tests for code reward modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers). pages 6917 1917-6935, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 779-8-89176-2577 18653/v acon URL https: httpp::://acaaanhoooggy/ wwg//020 201
doi: 10.
[34] Valentin J.M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. The art, science, and engineering of fuzzing: A survey. IEEE Trunsactions on Software Engineering, 47(11):31222331, 2021. doi: 10.1109/TSE.201. 2946563.
13

---

[35] Phil McMinn. Search-based software test data generation: survey, Softw, Test. Verification Reliab., 14(2):105-156, 2004. doi: 10.1002/STVR.294. URL https://doi org/ 10.//00.11 1002/ stvr. 294.
[36]
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen- Yih, Sida I. Wang, and Xi Victoria Lin. LEVER: learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honoluiu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 26106-26128, PMLR, 2023. URL http p/tpoeedp//rrooeeinggg mmr. wttpppesee///ni22//333 html.
[37] OpenAL. GPT-4 technicall CoRR, abx/2303.08774, 2023. doi: 10.48550/ARXIV233003 08774. URL https: org/ 10. 48550/ 2303, 20..2333377777 08774.
[38] Carlos Pacheco and Michael D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion, OOPSLA '07, page 815-816, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781 1795138356777 doi: 10. 10.1145/129786..15 29T.1199996..9199990 1297902.
URL httph:tt////oooooo org/ 10. 1145/ 1297846. 1297902.
[39] Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beech- ing, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. codeforces-cota. 2025.
co/datassets/ope-rllll
[40] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs/2308. 12950, 2023. doi: 10.48550/ARXIV223008 12950 URL https://doi. org/10. 18550/arXiv.. 2308. 12950.
[41] Kostya Serebryany and the LLVM Team. libFuzzer - library for coverage-guide guidedd fuzz testing. html, 2016. Accessed:
[42] Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-basee code generation using deep reinforcement learning. arXiv preprint arXiv:2301.1388 2023.
[43] Wenhan Wang, Chenyuan Yang. Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Ling- ming Zhang, An Ran Chen, and Lei Ma. TESTEVAL: benchmarking large language models for test case generation. CoRR, abs/2406.04531, 2024. doi: 10448500/ARXIV22406.005 https://doi. org/10..//// 4055008555iii 2406. 04531.
URL
[44] Yue Wang, Weishi Wang. Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-awart unified pre-trained encoder-decoder models for code understanding and generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8709. Association for Computational Linguistics, 2021. doi: 10.18653/V1/0021 EMM.P-MAAN.
URL
https: //ops:///oi.. org/10 10. 18653/v1/20222 emnlp- main. 685.
[45] Yue Wang, Hung Le, Akhilesh Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. EMNLP 2023, Singapore, December 6-10, 2023, pages 1069-1088. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. EMNLP-MAIN.66 68. URL nttpp:///oii. /ttp://ooo..o//./////// 10. 18653/v1. v1/////1//22..////// ewnlp-mainl happmmmin.6. 68.
[46] An Yang, Baosong Yang, Binyuan Hui, Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong
14

---

Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 2407.1667, 2024.
[47] Kohsuke Yatoh, Kazunori Sakamoto, Fuyuki Ishikawa, and Shinichi Honiden. Feedback- controlled random test generation. In Michal Young and Tao Xie, editors, Proceedings of the 2015 International Symposium Software Testing and Analysis, ISSTA 2015, Baltimore, MD, USA, July 12-17, 2015, pages 316-326, ACM, 2015. doi: 10.1 00.11452711888..778865 URL https://don org.oo..oogo// /145/2771783.22 2771805
[48] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv, 2502.01718, 2025.
[49] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong. Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 5673-5684, New York, NY, USA. 2023, Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580000 3599799. URL https://dof org/10 1145/ 3580305 3599790.
[50] Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. The Thirteenth International Conference on Learning Representations, 2025. URL https: / /oppneeeeeerereeeee net www.fooodmiiaaaoo VdrrriddyVyy
15

---

Appendix
A Related Works
A.1 Advancements and Challenges in LLM-based Code Generation and Evaluation
Large Language Models (LLMs) have revolutionized automated code generation, with models like Al- phaCode [26], GPT-4 [37], Gemini [2], and specialized code models such as CodeT5 [44, 45], StarCoder [24]. DeepSeek Coder [13], CodeLlama [40], and Qwen2.5-Coder [46, 18] demon-
strating remarkable capabilities. These models, whether trained via Supervised Fine-Tuning or
enhanced through Reinforcement Learning (RL) with execution feedback (e.g.. AlphaCode [26], PPOCoder [42], CodeRL [22]), increasingly match or exceed human performance on diverse pro- gramming benchmarks.
The reliable evaluation these sophisticated models is paramount and hinges on the quality of Code Verifiers Verifiers-ttpically test suites-thi ascertain the functional correctness of generated code. This is especially critical for RL frameworks employing verifiable rewards (RLVR) [8, 10, 36, 17], where test case quality directly impacts reward accuracy and training efficacy. However, as highlighted in Section 1. the comprchensiveness and robustness verifiers often present a significant bottleneck. underscoring the critical need for effective Test Case Generation.
To this end, the research community has developed numerous benchmarks. Early benchmarks like Hu- manEval [6] and MBPP [3] provided foundational testbeds. Subsequent efforts like EvalPlus [28, 29] and MBPP-Plus aimed to improve robustness by expanding existing test sets. More comprehen- sive benchmarks such as TACO [25], CodeContests [26] (focused on algorithmic problems), Big- CodeBench [50] (complex multi-library tasks), LiveCodeBench [20] (simulating online judge envi- ronments), and HumanEvalPacc [49] (multilingual program synthesis) have broadened the evaluative scope. Concurrently, dedicated TCG benchmarks like TESTEVAL [43] and TestGenEval [19] have emerged. Despite these advancements, persistent issues in test case quality, coverage, and potential LLM-centric biases necessitate continuous efforts towards developing high-quality, diverse, and unbiased Code Verifiers.
A.2 Methodologies for Test Case Generation (TCG)
TCG is fundamental to validating code correctness and providing feedback for both evaluation and training of code generation models. While formally defined as a distinct task in this paper, TCG principles are integral to benchmark creation (e.g.. LiveCodeBench [20], TESTEVAL [43]) and RL data pipelines (e.g., ACECODER [48], CodeRL [22]). However, suboptimal test quality in existing datasets (e.g.. TACO [25], CodeForces-CoTT (39]) can lead to overestimated model performance and reward hacking, particularly in rule-based RL systems like AlphaCode [26) and DeepSeek Coder [13], thus highlighting the urgent need for effective TCG.
Traditional TCG Techniques: Software testing has rich history of TCG. Search-Basee Software Testing (SBST) [1, 14] uses metaheuristics like genetic algorithms [4, 35] to optimize for coverage criteria, though it can be computationally intensive. Symbolic Execution [9, 5] explores program paths systematically but faces path explosion and challenges with complex code. Fuzzing [34, 11], especially coverage-guii -uidded variants like AFL [11] and libFuzzer [41], excels finding crashes and vulnerabilities by mutating inputs, though it may lack semantic depth for logical tests. Feedback- directed random testing [47, 38] also discovers defects can sometimes lack diversity.
LLaebased TCG Paradigms: With the rise of L.LMs, new TCG methods have emerged, leveraging semantic understanding for potentially greater coverage and diversity. As discussed in Section these generally follow two paradigms:
Direct Generation:
LLMs produce complete test cases (inputs and outputs). This includes iusertion-focuseee methods (e.g., CodeCoT [15], AgentCoder [16]) and direct input-output syn- thesis (e.g.. TestChain [23], CodeRM [33], AceCoder [48]), aiming for logical coverage and boundary conditions.
imput-Interpreter L.LS generate test inputs, which are then executed by ground-truth solution to derive outputs. This is seen in LiveCodeBench Co0oecodeoor [39] and related to LLM-
16

---

guided fuzzing. EvalPlus [28, 29] also aligns by mutating seed inputs for execution. TestChain [23] also showed improvements by decoupling input and output generation.
Advanced and Specialized TCG Approaches: In RL contexts, dynamic TCG is crucial. CodeRM- 8B [33] adjusts test quantity by problem difficulty. LEVER [36] uses learned verifiers, and other works focus on seeutionnfeedbace for reward model optimization, all highlighting the impact of test quality on RL outcomes. Differential testing, as in AID [30], compares program versions to expose bugs, showing strong results on datasets like TrickyBugs [30] and EvalPlus [28]. Recent efforts also explore generating tests targeting prior failures [7] or using LLMs for test suite refinement.
Despite these diverse approaches, achieving comprehensive logical coverage, generating truly diverse and challenging corner-case tests, and maintaining computational efficiency remain significant hurdles. Furthermore, many LLM-ccntric TCG methods may perpetuate biases inherent in the L.MSs themselves (Section 1). This paper's 's SAGA framework addresses these limitations by proposing a novel human-LLM collaborative paradigm that systematically integrates deep human insights from both correct and incorrect solutions. By focusing on a dynamic, adaptive, and efficient TCG process, SAGA aims to enhance the reliability of LLM code evaluation and training, paving new path for TCG research.
B Formulation and Interpretation of Advanced Evaluation Metrics
This section provides detailed explanations and interpretations for the advanced metrics introduced in Section 3.2 to evaluate the intrinsic quality of test suites.
B.1 Distinct Error Pattern Coverage (DEPC) and Diversity Ratio
Formulation (from main text): For test suite T and Np problems, let e(t2) be the Np-dimensional binary error pattern vector for test b (where v(t2), if t, reveals an error for problem j).
DEPC(T) = [v((x)) tt and [|o(t))] 2 13
The Diversity Ratio is DEPC(T)/n, where [7].
Interpretation: DEPC measures the breadth error coverage by counting the number of unique ways (pattems) in which the test suite detect failures across set of problems. A higher DEPC signifies that the test suite is capable of identifying a wider variety of distinct error types or combinations of errors. This directly relates to the concept of inter-test case correlation (pett) discussed in theoretical model (Appendix C): test suite with high DEPC is likely composed of tests that are less correlated in terms of the errors they detect, thus having lower Peft- The Diversity Ratio normalizes DEPC by the number of test cases, indicating the average efficiency of each test in contributing a new, distinct pattern. high Diversity Ratio suggests that the test suite is not only diverse but also concise, with less redundancy among its test cases.
B.2 Normalized Area Under the Accuracy-Number of test cases Curve (AUC-AceN)
Formulation (from main text): For Verifier Accuracy Acc(k) achieved with test suite of size k (up to maximum N, starting from kmin), the AUC-AceN is approximated by the trapezoidal rule:
1
N-1
Acc(ki) + Acc(ki+1) (K k;))
AUC@N
N Kemin
1= Rmnnn
Interpretation: AUC-AceN quantifies the average Verifier Accuracy the test suite size grows, providing single scalar value to compare the overall effectiveness and efficiency of different TCG strategies. A higher AUC@N N (ranging from 0 to 1) indicates that a TCG strategy consistently generates test suites that achieve higher verifier accuracy across various sizes up to N. This metric composite reflection of both the average potency of individual test cases (p) and the effective diversity of the test suite (related to Pet).
Impact of p (Test Potency): higher average p means individual tests are more likely detect errors. This leads to steeper initial rise in the Ace(k) curve and a higher overall level of accuracy
17

---

achieved, both of which contribute to a larger AUC@N N. Thus, AUC-AceN is particularly sensitive p.
Impact of Diversity (low Pent): Greater diversity (lower Peft), empirically reflected by high DEPC, allows the Acc(k) curve to sustain its rise or plateau at higher level for larger number of test cases before saturation effects become dominant. This sustained high accuracy also contributes to a larger AUC@N N.
Therefore, a high AUC@ N signifies TCG strategy that excels generating tests that individually powerful (high p) and collectively non-redundant (low Pett ) leading to efficient and robust verifier construction within the specified test suite size limit. It reflects better overall quality of the generated test cases achieving high average performance.
C Theoretical Analysis of Detection Rate
We analyze the detection rate (T) P(X 2 1) for test suite T = (ti....,tyy where X Ei=1 1x, and p; P (E)) is the probability that test t, detects an error.
C.1 Modeling Correlated Heterogeneous Bernoulli Trials
The expectation of X is [ = Ep. = np, where p = E.P. The variance is Var(X) Ep.11 pc) + Eizj Cov(1E, 1
To model the effect of correlation in a tractable manner that allows for insights similar to the homogeneous
(uniform p, uniform p), we consider approximating model. We define effective average pairwise covariance, Cell. and an average individual variance, ob = n Σ p1 p.). We then model the variance as if it arose from system with these averaged second-order characteristics:
Varapprox (X) = -x) + n(n 1)Cef-
If we further posit that these average characteristies can be related through an effective average correlation Pett such that Če# ~ Petl /avg(p: -1 pt))² more simply for conceptual linkage, Cen RJ Perp(1 - p) (assuming pt are not excessively dispersed, making oz x p(1 - p)), then:
Varapprox X) np(1 [1 (n 1)fef))
This equation models the variance of X as it were a sum n Bernoulli trials with a common success probability and common pairwise correlation Pett- The validity of this approximation depends on the actual distribution of p; and the structure of covariances. However, it serves as a useful model to understand the qualitative impact of average correlation.
Definition 3 (Model-Bu --add Effective Sample Size nag)- Within this approximating model (Eq. 1), and by analogy to Kish's 's design effect [21] (where deff = 1 (n- 1)ftt), the model-based effective sample size is:
(n-
This nht represents the number hypothetical independent Bernoulli(p) trials that would exhibit the variance given by Eq. This is consistent with adjustments for correlated data [32, 12, 27].
C.2 Upper Bound and Saturation within the Model
Using n and 5 from our model, we analyze the detection rate (T' = -P( = 0).
Theorem 1 (Model-Based Approximate Upper Bound on Detection Rate). Within the described approximating model, the detection rate (T') is approximately upper bounded by:
(s(T) (1 p) "aa (1 Io Dipe
Proof Sketch. Sketch. The probability P(X = 0) approximated by that of nea independent Bernoulli trials, which is (1 - Car
This theorem suggests that even when individual p; vary, if there's an effective positive average correlation Patt > the system behaves as if it has fewer independent tests, limiting the detection rate.
18

---

C.3 Interpretation and Implications for Performance Metrics
The derivation above, employing an approximating model based on average parameters (P, Pett). robustly indicates that persistent positive effective correlation among test case detection events leads to saturation of the overall detection rate. The core insight-that redundancy limits the marginal gain from additional tests- remains. This theoretical observation is crucial for understanding the performance of TCG strategies and their impact the empirical metrics used in this paper:
Verifier Accuracy (Ace(T)) and its relation to p:: The Verifier Accuracy any given test suite size n, Aco(Tn., is fundamentally driven by the test suite's ability to expose errors, which is heavily influenced by the average potency of its constituent test cases. higher average error detection probability, p. means that individual tests are, on average, more "powerful" or "incisive." " Consequently, a TCG strategy yielding a higher p will lead to a verifier that more readily and correctly identifies faulty solutions, directly boosting Ace(T,). While high correlation (Pett) can limit the ultimate achievable accuracy by causing early saturation
distinct error discovery, a strong p is essential for the accuracy curve to reach high level in the first place.
AUC-AceN as a reflection of sustained high p and managed Petr: The Area Under the Accuracy- Number of test cases Curve (AUC-AceN), which quantifies the average Verifier Accuracy test suite size increases, a composite reflection of both p and Pett. A high ş ensures that the Acc(k) curve rises steeply and achieves significant altitude. This initial rapid ascent and the overall height of the curve contribute substantially to a larger AUC-AceN. Concurrently, a lower effective correlation Pott (i.e., greater diversity, reflected empirically by DEPC) allows the accuracy to be sustained or to continue growing across larger number of test cases before significant saturation, thereby expanding the area under the curve. Therefore, strategies achieving high AUC-AccN those that likely generate test cases with a consistently high average error detection probability (p) and effectively manage redundancy (lower Pett). The magnitude of p is particularly critical for the "value" captured by AUC-AceN, as it dictates the average level of accuracy being integrated. DEPC and its relation Patt: DEPC empirically captures the diversity of error patterns. A TCG strategy that yields high DEPC is effectively generating tests with low effective average correlation Pett. thus mitigating the saturation effect on discovering new types of errors and allowing for a more sustained increase in overall detection capability.
Therefore, the pursuit of TCG methods like SAGA, which aim enhance both individual test strength (targeting a higher p) and inter-test case diversity (targeting a lower Pott), is theoretically well-founded for optimizing these key verifier performance metrics.
D TCGBench: Foundational Dataset for TCG Research
As introduced in Section 3.2, TCGBench the comprehensive dataset curated for our Test Case Generation (TCG) research. It aggregates 1840 recent programming problems sourced from three leading competitive programming platforms: AtCoder (https://atcod)
jj),, Codeforces (https:/ /ttps://cooeeooces. com), and Nowcoder (https: /www. nowcoder com). These platforms are recognized for their diverse algorithmic challenges. For each problem TCGBench, we also collected average of 36.66 incorrect human submissions, specifically those resulting in "Wrong Answer" (WA) or "Time Limit Exceeded" (TLE) verdicts. This large-scale collection of problems, along with their corresponding WA/TLE submissions, provides rich empirical foundation for studying human error patterns and rigorously developing and evaluating TCG methodologies. The problems are sourced from recent contests ensure currency and minimize data leakage risks when evaluating contemporary LLMs.
E TCGBench-Lite and CodeCompass: Curated Set for Evaluation
For the main experimental comparisons and ablation studies presented in Section 4.2.1, and for constructing the CodeCompass verifiers (Section 5), we curated TCGBench-Lite. This a focused subset of 270 problems sourced from AtCoder, Codeforces, and Nowcoder contests held since June 2024, ensuring high contemporary relevance and minimizing potential data leakage for evaluating newer models. TCGBench- features average of 41.41 incorrect submissions per problem.
19

---

The difficulty distribution for TCGBench- and CodeCompass (Easy: 27.04% Medium: 32.59% Hard: 40.37%) was determined by multi-faceted approach. This involved considering platform- provided difficulty tags, the type of contest round (e.g., AtCoder Beginner Contest vs. Regular Contest; Codeforces Div.4/3 vs. Div. typical problem-solving patterns associated with specific problem slots within these contests, and general community perception of difficulty for similar problems. For instance, early problems in beginner-focused contests were generally classified as 'Easy', while later problems in advanced contests those requiring complex algorithms/data structures were classified as 'Hard'. This classification aims to provide a balanced yet challenging set for rigorous evaluation. The verifiers in CodeCompass, used for code generation evaluation, consist an average of 50.54 characteristics are summarized in Table 4.
test cases per problem for these 270 problems. The
Table 4: Overview of TCGBench-Liee (Core Dataset for CodeCompass Verifiers).
Aspect
Details for CodeCompass Evaluation
Core Dataset Source
Atcoder, Codeforces, Nowcoder (June 2024 -Pesent)
Total Problems
270
Difficulty Distribution
Easy (27.04%). Medium (32.59%)) Hard (40.37%)
Scale for CG Evaluation
Avg. Test Cases/Problem: 50.54
Avg. P (for SAGA's TCG): 41.41
Primary CG Metric
Pass@k
CodeForce
(a)
95-
(b)
Bescioo
Dillee
(d)
Verifier Accuracy (%)
MabilDin
SMGA
Delection Rate
Diversitt Ratto
80
DRPC
n0
Difter
Mutcktaa
9
SAGA
15
Ruadinn togtt0
30
Size
Test Case Size
20
40
10 50 30
Test Case Size
Test Case Sizee
NowCoder
54.4%% (b)
(d)
Detection Rate %
25
Veriffer Accuracy (%)
Bus tes nit Caoe JMRC
Sandine
SANDA
Ditee
20-
Diversity Ratio
R5
to.
DEPC
Haffline 100: 83..5%
80-
w/ Mullite
25
vel Ditker
100
130
3.75
Test Case Size
50
Test Cuse Size
20
40
Test Case Size
30
Test Case Size
Figure 9: SAGA performance Codeforces (CF) and Nowcoder (NC) problems from the full TCGBench dataset: (a) Detection Rate, (b) Verifier Accuracy (with AUC@50), (c) DEPC, and (d) Diversity Ratio, compared to Baseline (Input-Interpreter) and SAGA's analytical components (Multidimensional Analysis and Differential Analysis). Dotted lines in (a) & (b) show respective baseline performance at n 100.
F SAGA Performance on Full TCGBench
To demonstrate SAGA's broader applicability beyond the AtCoder subset shown in Figure 6 (which used the full TCGBench for that visualization), Figure presents SAGA's performance on the Codeforces and Nowcoder portions of the complete TCGBench dataset. SAGA consistently replicates its superior performance, exhibiting enhanced efficacy (DR, Acc) and superior test suite quality (DEPC, Diversity Ratio) compared to the baseline and its individual analytical components across these platforms as well. This consistent pattern of improvement underscores the fundamental benefits of SAGA's structured, insight-driven approach to TCG.
20

---

To further demonstrate SAGA's broad applicability and robustness, Figure 10 presents its Detection Rate (DR) and Verifier Accuracy (VAcc) when paired with different LLM backbones (Qwen2.5- Codsr-7B-Instrucct Dween.5-72BBInntrrue and SeepSeek-V3-0322. on the Codeforces and Nowcoder portions of the full TCGBench dataset, compared against the Input-Interpreter baseline (using -33)) Across both platforms and all LLM backbones, SAGA consistently and significantly outperforms the baseline in DR and VAce at various test case sizes. Notably, even SAGA with the smaller Owen2.5-Coder--77 often surpasses the baseline that utilizes the larger DeepSeek-V3, highlighting SAGA's ability to effectively guide diverse LLMs. While larger SAGA backbones generally yield higher absolute performance, the consistent uplift provided by the SAGA framework across different models and problem sources underscores its fundamental benefits and general applicability for advanced TCG.
Atcoder
CodeForce
Nowcoder
DR (%)
40%
90.0%
50%
55.0%
90.0%
20%
25%
90.0%
-00%
Vicc
X0.0%
100
20
50
100
50
100
Test Case Size
Test Case Size
Test Case Size
SAGLA-Qecn-Cola (DR)
SAGIA-DapSec-VV (DR)
SAGA-Qwan-huler (VAss)
SAGA--DepSea (VAse)
SAGLA-Qwan7BB (DR)
Basclinc (DR)
SAGA-Qwes728 (VAse)
Bascline (VAes)
Figure 10: SAGA performance with different LLM backbones (Qwen-Coder, Qwen-72B, DeepSeek- V3) compared to the Baseline Inpuu--nterpreter with DeepSeek- V3) Codeforces and Nowcoder portions of the full TCGBench dataset. Metrics: Detection Rate (DR) and Verifier Accuracy (VAcc) at varying test case sizes. Dashed lines indicate baseline performance.
G Detailed Performance Analysis on TCGBench-Lite by Difficulty
To provide more granular understanding of how different Test Case Generation (TCG) methods perform across varying levels of problem complexity, Figure 11 illustrates the Verifier Accuracy (VAce (5) and Detection Rate (DR @50) of SAGA, its analytical components (Multidimensional Analysis only, denoted "w/ MultiDim"; Differential Analysis only, denoted "w/ Differ"), and baseline TCG methods (TestChain, EvalPlus, and Random input-Interpreter on the Easy, Medium, and Hard problem subsets within TCGeench-Lite.
Easy
Medium
Hard
4856
10.0%
25.15%
33.2%
50.0%
VAecge50
47%
50.%
31.8%
20.6%
36.7%
21.5%%
37,8%%
1600%
10.0%-
11.0%
10.05
3.5%%
330%
100%
48%
15.5%
57.5%
55.5%
DR
45%
R0%
7856
78%
706
43.7%
98%
68%
60%
58%
Basckest
EnalPett
Randun
Bltor
50%
BatChate
Eauteer
Rendup
Ex64
Dilter
500
TraeCiste
Eoatfer
Reddup
B464
TeoiChsin
MaliDim
Figure 11: Performance comparison (VAce@50 and DR@50) of SAGA, its components, and baseline TCG methods across Easy, Medium, and Hard problem subsets of TCGBench-Lite SAGA consistently outperforms baselines across all difficulties, and its full framework generally surpasses its individual analytical components, especially on harder problems.
21

---

Key Observations from Figure 11:
Consistent Superiority of SAGA: Across all difficulty tiers (Easy, Medium, Hard) and for both VAcc@ and DR 50, the full SAGA framework consistently outperforms all baseline methods (TestChain, EvalPlus, Random). This underscores SAGA's robustness and its ability to generate more effective test suites regardless of problem complexity. For instance, on Hard problems, SAGA achieves VAcc@ 50 of 25.06%, substantially higher than the Random baseline (18.47%) and other methods. A similar trend is observed for DR 50, where SAGA reaches 87.0% on Hard problems. Impact of Problem Difficulty on Baselines: The performance of baseline methods, particularly TestChain and EvalPlus, degrades more noticeably problem difficulty increases. For example, EvalPlus's ' VAce 50 drops from 38. 10% on Easy problems to mere 5.56% on Hard problems. This suggests that these methods may struggle to generate effective test cases for more complex scenarios or subtle bugs prevalent in harder problems. The Random Inpeut-eteerreter shows more resilience than TestChain and EvalPlus on harder problems but still falls short of SAGA.
Synergy of SAGA's Analytical Components: While both Multidimensional Analysis ("w/ Mul- tiDim") and Differential Analysis ("w/ Differ") components of SAGA individually outperform the baselines, the full SAGA framework generally achieves the best performance or is highly competitive.
On Easy problems, SAGA's VAce @5 (55.59%) is notably higher than "w/ MultiDim" (50.00%) and "w/ Differ" (49.15%), indicating that the combination of insights from both correct and incorrect solutions is beneficial even for simpler problems.
For Medium problems, SAGA (VAce @ 50: 33.24%) again leads, showing a clear advantage over relying on only one type of human prior.
On Hard problems, the synergy is particularly evident. SAGA's VAce 50 (25.06%) is superior to "w/ MultiDim" (21.92%) and "w/ Differ" (21.25%) This suggests that for complex problems with elusive bugs, leveraging diverse insights from both correct solution structures and patterns of common errors crucial for generating highly discriminative test suites.
similar synergistic effect generally observed for DR @50,0, where the full SAGA framework often provides the highest or near-highest detection rates.
Interestingly, the "w/ Differ" component (Differential Analysis leveraging Swieng) shows relatively strong performance on Hard problems compared to its performance on Easy problems, particularly for VAce @50. This might imply that analyzing patterns from incorrect submissions is especially valuable for uncovering the types of subtle or complex errors that characterize more difficult problems.
Effectiveness of Differential Analysis on Harder Problems:
Limitations of Simpler Priors: The performance of "Random" (nnut-Interpreter) and even "EvalPlus" (which uses human solutions for mutation) on Medium and Hard problems highlights that merely having access to human solutions or employing random generation is insufficient. SAGA's structured approach to analyzing and strategically leveraging these human priors is what drives its superior performance, especially as complexity increases.
H Supplementary Experimental Analyses
To further explore mechanisms for robust test suite construction and the value of diverse generation strategies, we present two additional studies: an analysis of mixing random test cases from different LLMs, and an ablation study on SAGA's knowledge sources.
H.1 Efficacy of Mixing Random Test Cases from Different Language Models
Our theoretical framework highlights that test suite quality is influenced by individual test potency (p) and inter-test case correlation (Pea). We investigated managing Pet by mixing random test cases (akin to the Input-Interpreter paradigm from Section 3.2) sourced from different LLMs (V3: DeepSeek-V? V3- 0324, 72B: 2-t.m2.5-7BBInntttue
--nn..--o -n-Itruruct.. The hypothesis is that tests from different models may exhibit lower inter-souree correlation, leading to improved combined suite characteristics. Figure 12 shows the AUC@ 50 when mixing random tests, with diagonal elements representing single-I.LMM suites.
Coder:
Observations
22

---

0.314
34:94
11.57
56-53
0.1985
Ovter
12B
Colar
V
CadeeForse
CoduFersc/AAlCiee
V2
2
B.2044
738
65.34
Code
0.3412
82444
0.2022
$4.57
50.52
56.2
8.347
0.1985
0.17N4
128
Cooke
12B
Color
V3
128
Cookrr
NowCCade/VAniiSe
NowCedee Ratiog
Newt NuwCooer/AAiiuugga
26.22
75.26
82134
77,99
02843
0.1965
Coder
V5
Codeer
130
Cider
V/3
Figure 12: Heatmap illustrating AUC 50 performance of mixed random test suites. Diagonal elements: suites from single model (V3, 72B, Coder). Off-diagonal (i.j): mixing random tests from model and model
Benefit of Model-Source Diversity: Mixing random tests from two different LLMs frequently yields superior AUC@50 compared to using tests from only one, across AtCoder, Codeforces, and Nowcoder datasets. For instance, on Codeforces, mixing V3 (AUC@ 50: 0.2014) with 72B (6.1691) results in mixed AUC@ 50 of 0.2629, surpassing both. This suggests complementary biases even in random generation.
Reduced Effective Correlation: The improvements imply that combining tests from different models likely lowers Perr compared single-source suites. Enhanced diversity allows the combined suite to cover a broader error spectrum, increasing AUC@50.
Surpassing Stronger Components: Often, the mixed suite outperforms even the stronger individ- ual model in the pair (e.g.. V3+72B on Codeforces). This robustly shows different LLMSs contribute unique, valuable random tests, highlighting complementary strengths.
Diversifying the source of even "naive" random tests reduces correlation and improves test suite quality, supporting our theoretical framework. While SAGA achieves this more directly via structured analysis, this experiment underscores the general impact of minimizing Pott-
Take-away:
H.2 Ablation Study: Impact of Human Knowledge Source Volume in SAGA
To dissect SAGA's knowledge source contributions, we conducted an ablation study on the Codeforces portion of TCGBench. We modified SAGA by removing its Differential Analysis component (insights from incorrect solutions, Swrang) entirely. Instead, we doubled the volume of correct human solutions (Shuman) fed into SAGA's Multidimensional Analysis component, creating a Saattidim-Enhaance version. The aim was to see if increasing one type of human insight could compensate for omitting another. Table compares the original SAGA with this Multidim-Enhaneee configuration.
Analysis of Ablation Results (Table 5):
Degradation in Core Effectiveness: Despite doubling Snuman input, the "Multidim-Enhaneee" Enhanced" version shows a clear drop in V5ce@50 (from 47.08% to 38.73%) and AUC@ 50 (from 0.3195 to 0. 0.2744) compared to the full SAGA. This suggests insights from incorrect solutions via Differential Analysis are crucial for verifier quality and cannot be fully compensated by merely increasing the volume of correct solutions for Multidimensional Analysis.
23

---

Table Ablation Study on Codeforces (TCGBench): SAGA vs. Multidim- (Double Snaman for Multidimensional Analysis, No Differential Analysis).
Configuration
DR@50
VAcc@50
AUC@50
DivRatio@50
SAGA (Multidim. + Differ.)
90.89%
47.08% 38.73%
0.3195
0.697
88.09%
0.2744
0.701
Diversity vs. Effectiveness: While the Diversity Ratio is comparable or slightly higher for "Multiiii-- this raw diversity doesn't translate to better overall verifier quality (AUC@50) This reinforces that the rype of diversity and the nature of uncovered error patterns (targeted by Differential Analysis) are critical, not just diversity as a raw count.
Implications for SAGA's Design: This strongly supports SAGA's dual-pronged approach of integrating insights from both correct and incorrect human solutions. The unique error patterns revealed by Differential Analysis appear vital for building high-quality, discriminative verifiers, and their contribution is not simply replicable by scaling up the input to Multidimensional Analysis alone.
I Model Performance on CodeCompass
To further illustrate the utility of CodeCompass as a challenging benchmark for evaluating LLM code generation capabilities, Table 6 summarizes the Pass@ performance of several contemporary LLMs. The evaluation was conducted separately for C++ and Python problem instances within CodeCompass, utilizing its SAGA-enhanced verifier suite.
The results demonstrate clear differentiation among models in both languages, underscoring and assess current code generation models. The challenging
CodeCompass' 's ability to effectively rank nature of the SAGA-goot -AGe-ggnerated test cases in CodeCompass provides a rigorous testbed for future model development and comparison.
Table 6: Performance of Various LLMs on CodeCompass (Pass@ 1) for C++ and Python.
Model
Pass@1 1 CodeCompass (C+)) Pass@1 on CodeCompass (Python)
Owen22,5Cooler322BBBmm
15.93%
12.59%
Qwen2..-77B-IInttrccc
17.04%
14.81%
GPT-4o (2024-11-20)
20.74%
14.44%
DeepSeek-V3
24.81 %
23.33%
QWQ-32B
31.85%
26.30%
DeepSeek-Chat Chat-RI
35%15%
34.07%
Qwen3-235B-A22B
43.70%
36.30%
From Table 6, it is evident that CodeCompass effectively differentiates LLM performance across both C++ and Python. While relative model rankings show some consistency, language-specinn performance variations are also notable, highlighting the benchmark's 's capacity to reveal nuanced capabilities. The generally moderate Pass ] rates underscore the challenging nature of CodeCompass due to its SAGA-enhanced test suites, making it a valuable resource for rigorous multi-lingual code generation assessment.
J TCGCoder-7B Training Details
TCGCoder-7B, our specialist 7-billion-parameee parameter model for Test Case Generation (TCG), was fine- tuned from Qwwen2.5-Coder-71--nnsss Instruct [46, 18]. The training dataset, distinct from our evaluation sets to prevent data leakage, comprised 15,000 early-stage programming problems from Codeforces and NowCoder, processed by our SAGA framework (Section 4) to generate structured outputs (Python Case Scripts, Math Explanations, Veliiaaliittoonn code, per Figure 5). The fine-tuning aimed to distill SAGA's TCG reasoning into TCGCoder-77 Key training configurations included 3 epochs, a global batch size of 16, an initial learning rate of 5e-6 (minimum 3e-7), max sequence length of 61.335

---

tokens, and the qwen2 chat template. Training utilized Fully Sharded Data Parallel (FSDP) across 2 nodes, each with GPUs.
K
Case Study: SAGA Framework and Metrics in Action
To provide concrete illustration of our methodology, this section presents study demonstrating both the SAGA framework in action and the precise calculation of our proposed evaluation metrics,
K.1 Illustrative Example of the SAGA Framework
We demonstrate SAGA's dual-pronged analytical approach using the "Christmas Trees" competitive programming problem. The task is to count the number of trees planted at intervals of M starting from coordinate A, that fall within the range R).
SAGA systematically leverages both correct and incorrect human solutions to generate comprehen- sive verifier.
1. Multidimensional Analysis (Leveraging Correct Solutions, Snuman) First, by examining correct human solution, the LLM deconstructs its defensive logic. It identifies the core mathematical condition (A + k M [L, R)), analyzes explicit parameter ranges (e.g.. - < A < 10**), and transforms implicit strategies into formal testing directives. This includes defining boundary values (e.g.. testing when targeted Python Case Scripts, such as gen_TC2() () to test extreme boundary values and gen_TC3 () for the L = R edge case.
A + k- MM) and stress strategies. Guided by these directives. SAGA generates
2. Differential Analysis (Leveraging Incorrect Solutions, Swrony) Complementing this, SAGA performs Differential Analysis by comparing correct solution with a known incorrect to pinpoint specific failure modes. For the same problem, the LLM identifies a critical_go gap corresponding to an "Off-by-one error in boundary calculation." " It deduces that this failure occurs specifically when the lower bound L is exactly on a tree's position. To exploit this, SAGA generates a targeted test case, gen_ TC1 ("O 5 10 20"), where the incorrect solution would fail by count. Other generated scripts target different identified gaps, such as handling negative ranges or potential overflows with extreme values.
Synthesis and Verification SAGA systematically applies both analytical dimensions across large number of human-provide solutions (both correct and incorrect) for each problem. This generates a comprehensive set of Python Case Scripts. These scripts are then executed to produce a large volume of test inputs, which undergo velf-vaaii ensure adherence to problem constraints. After deduplication, the resulting test suite is verified against ground-truth human solution to form the final, robust verifier. This structured process ensures the test suite is diverse, targeted, and systematically covers critical logic and subtle failure modes far more effectively than random generation.
K.2 Illustrative Example of Metric Calculation
To clarify the computation of our core metrics, we use simple, reproducible example. Consider a problem P1 for which have N, 3 known incorrect solutions, each with specific bug:
SI: Fails on negative numbers.
S2: Fails due integer overflow.
S3: Also fails on negative numbers.
Now, suppose TCG method generates a test suite T' with 7 test cases, producing the following outcomes:
1. Error Pattern Vector, DEPC, and Diversity Ratio: The error pattern vector v(tx) for each test case is a binary vector dimension N, = 3, where the j-th element 1 if the test case causes the j-th incorrect solution to fail.
DEPC (Distinct Error Pattern Coverage) the number of unique, non-zero error pattern vectors. Here, the set of unique non-zero vectors is {*[1, 0, 1]', '00, 1, 0]') Therefore, DEPC(T) = 2.
25

---

Table 7: Test case outcomes for problem PI.
Test Case
Input
Fails S1?
Fails S2?
Fails S3?
-5
Yes
No
Yes
12
999999999
No
Yes
No
-10
Yes
No
Yes
t4
0
No
No
No
-1
Yes
No
Yes
16
1
No
No
No
-20
Yes
No
Yes
Table 8: Error Pattern Vectors for the test suite T.
Test Case
Error Pattern Vector v(tx)
Test Case Error Pattern Vector v(ta)
tl
'[1, 0, 1]' '[0, 1, 0]*
t5
'[1. 0, 1J'
12
'[0. 0. 0J '[1, , 1]'
13
'[1, 0, 1]'
t7
14
'[0, 0, O]"
The Diversity Ratio is DEPC(T) /n, where n is the total number of test cases. For this suite, the Diversity Ratio is 2/7 0.286. This indicates that, on average, it takes 3.5 test cases to discover one unique bug pattern, highlighting redundancy tests 13, 15, t7 and the ineffectiveness of 14, t6.
2. Detection Rate (DR) and Verifier Accuracy (VAcc): Detection Rate (DR) is solution-level metric. For each incorrect solution, we determine if the test suite finds the bug (a binary outcome).
For S1: Does T detect the bug? Yes (e.g., tl works). Outcome = 1.
For S2: Does T detect the bug? Yes (t2 works). Outcome 1.
For 83: Does I' detect the bug? Yes (c.g., works). Outcome 1.
The final reported DR is the average of these outcomes: DR + (1+1+1)/3 = 1.0 or 100%.
Verifier Accuracy (VAce) is stricter, problem-level metric. It if and only if the test suite can detect a failure in every known incorrect solution (i.e., if the average DR is 100%). In this example, since the average DR 100%, VAcc(T)
L Observation: Analysis with State-of-the-Arr Reasoning Model
To explore the impact of advanced reasoning capabilities, we integrated SAGA with a powerful reasoning model, Qwen3-235B-A222 Counter-intuitively, this led a degradation overall perfor- mance. A subsequent manual analysis revealed this was not due to failure in reasoning, but rather practical engineering constraint. For many complex problems, the model's extensive chain-of-thought process exceeded our deployment's maximum sequence length (64k tokens), resulting in truncated and unusable script outputs. This observation suggests that while powerful, such large-scale models require significant computational resources to fully leverage their reasoning potential in complex generation tasks. Our choice of smaller default backbone thus reflects pragmatic balance between performance and accessibility for reproducible research.
M Limitations and Future Work
Our primary limitation SAGA's current reliance on a "golden" ground-truth solution as an interpreter. While effective in domains like competitive programming where oracles are available, this dependency restricts its direct application in general software engineering. Echoing insightful reviewer feedback, key direction for future work to adapt SAGA for oracle-less settings. We propose exploring pscudo-labeling approach where an initial set of diverse test cases -whose quality can be assessed using metrics like DEPC introduced herein- act as "voters" to identify high-confidence "pseudo- gold" solutions from pool of candidates. These can then seed the SAGA framework, removing the need for vre-verified oracle and significantly broadening applicability.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately state the paper's contributions: identi- fying limitations in current TCG, proposing the SAGA framework, and developing TCG- Bench, CodeComPass, and TCGCoder-77B -B. These claims are supported by theoretical analysis and extensive experimental results presented throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A dedicated "L "Limitations and Future Work" section is provided in Appendix ??, discussing aspects such the scope of human priors, generalization of TCGCoder-7B, com- putational cost of SAGA, the dynamic nature of LLMs, subjectivity in difficulty classification, and the current focus functional correctness.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical analysis concerning detection rate saturation (Corollary 1 in Section 3.2) is presented with its underlying assumptions. 'he complete derivation, including the model for correlated Bernoulli trials and the proof for the associated theorem, is provided in Appendix C. All theorems, corollaries, and definitions are numbered and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed descriptions of the SAGA framework (Section 4.1), the datasets TCGBench and TCGBench-Lite (Appendix E), the CodeComPass verifiers (Section 5), evaluation metrics (Section 3 and Appendix B), and TCGCoder-79 training (Appendix J). The LLM backbones used specified. Code and data will be released post-review (see Q5).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
28

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We commit to releasing the SAGA implementation code, the curated TCG- Bench and TCGBench-L datasets, the CodeComPass verifiers, and the TCGCoder-7B model weights upon acceptance of the paper, after the blind review period. All released assets will be accompanied by detailed instructions for use and will be licensed for aca- demic research purposes, with appropriate measures to protect any residual privacy concerns from the original public data sources. Anonymous repository links for other code, prompt templates, and the data demo are now on the first page.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/guides/ Abolic/guides/Codeeubbiisss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//ipss.. wnips.cc/publii.. wwtppppcc//ccccccccccccccccccccc guides /wweee/eeeeemieeeeeoeeoeeeeeeeee commisssionns ssisnnolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides details dataset construction (TCGBench, TCGBench- Lite, CodeComPas in Appendix E, D and Section 5), L.MMs used for experiments (Sec- tion 3.2, Section 4.2.1), 1), and key training parameters for TCGCoder-7B (Appendix J). The experimental setup for SAGA and baselines is described, cnabling an understanding of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error regions are shown in Figure 4(b) for the experimental validation of the theoretical upper bound on detection rate, illustrating the consistency of the observed trend. For the main experimental results (Acc, DR) presented in Table 2, error bars repre- senting standard deviation across multiple runs or problem subsets will be included in the Appendix ?? to provide information on result stability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Appendix J specifies the GPU resources (2 nodes, 8 GPUs each, CUDA 12.2.2 environment) for TCGCoder-7B training. LLM inference for SAGA and baseline experiments was conducted on similar high-end GPU hardware (e.g., NVIDIA A A100s). While precise execution times per problem vary, the scale of LLMs used implies standard computational requirements for models of these sizes.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
30

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / neurips...... https///eerriss..ppp..i...ecee httpppuubbttccaeeeeeeeeeeeeeeeee /tthi esGuidelinest
Answer: [Yes]
Justification: The research adheres the NeurlPS Code of Ethics. It focuses on improving evaluation methodologies for AI- Al-generated code. Data is sourced from public programming platforms, and care will be taken in any release to ensure user privacy from these original sources is respected (c.g.. by anonymizing any potentially identifying information if present, though the focus is on code logic).
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper primarily focuses on the positíve societal impact of enabling more reliable and robust Al-assisted software development through improved evaluation (Conclusion, Section 6). Potential negative impacts (e.g., accelerating automation) are briefly acknowledged in Appendix ??. The work aims to improve Al safety and reliability in coding.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
31

---

Justification: The datasets (TCGBench, TCGBench-Lite, CodeComPass) are derived from publicly available programming problems. Upon release, any data derived from user submissions (e.g. for Swrmyy will be processed to remove or anonymize any potentially identifying information to protect user privacy, ensuring they are used solely for academic research. The TCGCoder-7B model is specialized for TCG and does not pose the same general misuse risks as large foundation models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites original sources for all baseline methods (e.g.. LiveCodeBench [20], EvalPlus [28], TestChain [23]) and L.LMs used (e.g. Qwen series [46, 18), DeepSeek models [8, 13]). The programming problems for TCGBench are from public contest platforms (AtCoder, Codeforces, Nowcoder), whose terms generally permit research use. The open-source LLMs used typically have licenses like Apache 2.0, which are respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The SAGA framework, TCGBench dataset, TCGBench- Lite subset, CodeCom- Pass verifiers, and TCGCoder-7B model are introduced. Their construction, characteristics, and methodologies are documented (Sections 4.1, 1,, 5, Appendices D, J, E). Upon acceptance and release, comprehensive documentation will accompany all assets.
Guidelines:
32

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing or direct experiments with human subjects. The "human solutions" referred to are publicly available code submissions from programming contest platforms, used in an aggregated and anonymized fashion for analysis.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The research does not involve direct human subject participation that would require IRB approval. The data used (code submissions) is from public sources where users have agreed to platform terms; development, with privacy considerations for any release as noted in Q11.
Justification:
is for aggregated analysis and TCG methodology
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
33

---

Answer: [Yes]
Justification: LLMs are central to this research. The SAGA framework (Section 4.1) is human-LLM collaborative method for TCG. LLMs also used in the baseline TCG methods we evaluate (Section 3.2) and are the subjects of the improved evaluation method- ologies proposed. Specifie I.LMs (e.g.. SeepSeek-V3--0222 Qwen series) mentioned as backbones for these processes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / //nnenrrpsss.. http///ips.c//ooofeeeeeeeeeeeeee www.c///eennneeee/e/eccccceccae/ /LLM) for what should or should not be described.
34


Reviews:
[{"review_id": "CUT91COQlr", "reviewer": "Reviewer_UMHU", "summary": "This paper presents a systematic study of LLM test case generation. First, the paper demonstrates that existing popular code generation benchmarks, such as LiveCodeBench, can mistakenly consider wrong LLM-generated code as correct solutions. In particular, the verifiers of these code generation benchmarks fail to identify bugs in human-written wrong solutions. The paper also shows that human-written bugs are more diverse than bugs in LLM-written code, which are not effectively covered by LLM-generated test cases. Based on this analysis, the authors propose a series of metrics to measure the quality of LLM-generated test suites, and they propose SAGA to improve these metrics via utilizing human-written correct and wrong solutions. They construct TCGBench to verify that SAGA outperforms LLM-based test case generation baselines. With SAGA-generated test cases, they further construct CodeComPass, a code generation benchmark with more rigorous test cases.", "questions": "1. It would be helpful if the authors can present a running example to demonstrate how SAGA works. From Figure 5, it is hard to understand the concrete workflow. To my understanding, besides human-written solutions, all of case scripts, math explanation and self validation are generated by LLMs. Is this correct? From Section 3, it is unclear how the math explanation is used in the pipeline.\n\n2. How do you combine multi-dim analysis and differential analysis? My understanding is that each component produces its own case script, is it correct?\n\n3. Do you run the case script to generate multiple test cases? Do you have any filtering of the generated test inputs, and how does the filtering work?\n\n4. In Table 1, please give more details and examples of simple human priors used in this experiment.\n\n5. Is there any human analysis on the quality of LLM-generated math explanation and script? How often are they correct?\n\n6. In Figure 4, if you mix test cases from different LLMs, how does the detection rate look like?\n\n7. Have you evaluated recent thinking models on CodeCompass? Would be curious to see if their drop ratios are smaller.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper makes solid contributions to the topic of LLM-based test case generation, sharing good insights validated by comprehensive experiments. First, the comparison of assessment of LLM-generated code based on verifiers from benchmarks and Online Judges is very informative, clearly showing the quality issues of current popular code generation benchmarks.\n\n2. The analysis of human-written wrong solutions and LLM-generated wrong code is also interesting, clearly showing that LLM-generated code contains more homogeneous error patterns.\n\n3. The metric definition to quantify the quality of LLM-generated test suites is well-motivated and rigorous.\n\n4. TCGBench and CodeCompass presented in this work are valuable contributions to the community.\n\n5. Comprehensive experiments demonstrate that the proposed SAGA approach is effective.\n\nWeaknesses:\n\nThe main weakness of this work is the lack of clarity in describing the SAGA approach, probably due to the dense content. Below are my questions:\n\n1. It would be helpful if the authors can present a running example to demonstrate how SAGA works. From Figure 5, it is hard to understand the concrete workflow. To my understanding, besides human-written solutions, all of case scripts, math explanation and self validation are generated by LLMs. Is this correct? From Section 3, it is unclear how the math explanation is used in the pipeline.\n\n2. How do you combine multi-dim analysis and differential analysis? My understanding is that each component produces its own case script, is it correct?\n\n3. Do you run the case script to generate multiple test cases? Do you have any filtering of the generated test inputs, and how does the filtering work?\n\n4. In Table 1, please give more details and examples of simple human priors used in this experiment.\n\n5. Is there any human analysis on the quality of LLM-generated math explanation and script? How often are they correct?\n\n6. In Figure 4, if you mix test cases from different LLMs, how does the detection rate look like?\n\n7. Have you evaluated recent thinking models on CodeCompass? Would be curious to see if their drop ratios are smaller.", "was_revised": true, "final_justification": "The author response addressed my questions. Thus, I keep my review score.", "rebuttal": "Dear Reviewer UMHU,\n\nWe sincerely thank you for your positive review of our paper. We are especially grateful that you recognized the **solid contributions, deep insights, and rigorous methodology** of our work, as well as the value that TCGBench and CodeComPass bring to the community. Your constructive questions are crucial for us to further refine and clarify our work.\n\nBelow, we provide detailed answers to your specific questions:\n\n#### 1. Running Example and Workflow of SAGA:\n\nThank you for this question. Figure 5 provides a high-level overview, and we understand that a more detailed explanation of the concrete workflow is needed. Your understanding is correct: apart from the human-written solutions, other components such as the test case scripts, math explanations, and self-validation are indeed generated by the LLM.\n\nRegarding the role of the ***Math Explanation*** that you specifically mentioned, the LLM transforms the *implicit programming logic* from human code into *explicit, structured test instructions*. This process ensures that we are not just randomly generating inputs but are purposefully targeting the core mathematical and logical properties of the problem.\n\nFor example, in the classic \"Convex Hull\" problem, SAGA would identify from human code that the algorithm's core relies on using the cross-product to determine the geometric orientation of points. Based on this mathematical principle, SAGA would infer two key testing directions: 1) **Singularity Testing**: When the cross-product is zero, the points are collinear. SAGA would intentionally generate a test case with a \"trap\" point lying exactly on an edge of the true convex hull to test the algorithm's precision. 2) **Extremum/Degenerate Case Testing**: When all points are collinear, SAGA would generate a test case where all points lie on a single line to test if the algorithm can correctly handle this degenerate input. This process allows SAGA to systematically discover and test *logical singularities* and *structural extrema* determined by the problem's intrinsic mathematical structure, a depth that is unattainable by simple random testing.\n\nWe will add a **detailed case study to the appendix** of the final version to walk readers through this entire process, showing how these prompts are used to transform inputs into final, rigorous test cases. We will also elaborate more on the role of the math explanation in the main text.\n\n#### 2. How to Combine Multidimensional and Differential Analysis:\n\nThis is an excellent question. These two analyses are performed **in parallel**. Multidimensional Analysis utilizes correct solutions (*$S_{human}$*) to explore the solution space and defensive logic, while Differential Analysis uses pairs of correct and incorrect solutions (*$S_{wrong}$*) to target known failure modes. Each generates its own `Case Scripts`. These scripts are then **pooled and deduplicated** to form a comprehensive and diverse set of test inputs, thereby maximizing both *coverage* (from Multidimensional Analysis) and *discriminative power* (from Differential Analysis).\n\n#### 3. Test Case Generation and Filtering:\n\nYes, we run the generated Python `Case Scripts` to produce multiple test inputs. The filtering is done through the `Self-Validation` script, which is also generated by the LLM as part of the SAGA process (as shown in Figure 5). This script programmatically checks if the generated inputs conform to the problem's constraints (e.g., input format, value ranges). Any input that fails this validation is discarded, ensuring the quality and relevance of the final test suite. After obtaining the test case outputs, we also **use multiple correct solutions to verify the test cases**; only those that are passed by all correct solutions are retained.\n\n#### 4. Details on Simple Human Priors in Table 1:\n\nThe human priors here refer to adding prompts such as *\"Refer to the correct human solution and analyze the boundary conditions\"* into the prompt.\n\n#### 5. Human Analysis on LLM-generated Math Explanations and Scripts:\n\nThis is an important aspect. While we have not conducted a formal, large-scale human analysis on the correctness of every explanation and script, their quality is **indirectly validated by the effectiveness of the test cases they produce**. The high performance of SAGA (e.g., high DR and VAcc) indicates that these LLM-generated artifacts are, on the whole, correct and effective at guiding the generation of useful tests. Our *self-validation retention rate is typically above 60%*.\n\n#### 6. Detection Rate with Mixed LLM Test Cases:\n\nWe were also very interested in this point and conducted this experiment, with the results presented in **Appendix F.1 and Figure 12**. The findings demonstrate that mixing test cases from different LLMs (which have different cognitive biases) can reduce the inter-test correlation *ρ* and significantly improve the overall quality of the test suite (higher AUC@50). This **confirms the theoretical foundation of our work**. We will add a sentence to the main text to highlight this finding and point to the appendix.\n\n#### 7. Evaluating the Latest Thinking Models on CodeComPass:\n\nWe evaluated **Qwen3-235B-A22B-thinking-2507**, and the experimental results are shown in the table below. This again validates the effectiveness of CodeComPass as a high-difficulty benchmark and reveals that even top-tier \"thinking\" models still have room for improvement in generating truly robust code, highlighting the value of our work.\n\n| Model                   | Pass@1 on LCBv6 | Pass@1 on CodeCompass |\n| :---------------------- | :-------------- | :-------------------- |\n| **Qwen3-235B-thinking** | 64.36%          | 63.37%                |\n\nThank you once again for your thoughtful and constructive review. Your feedback has been crucial in helping us improve the clarity and completeness of our paper.\n\nBest regards,\n\nAuthors of Submission 17344"}, {"review_id": "t5dJ2yzzoU", "reviewer": "Reviewer_p61z", "summary": "For the task of code generation, learning verifiers that can generate a unit test suite is crucial. However, it is challenging as the verifier needs to generate tests to cover all the edge cases making it a hard problem for current LLMs. This paper first studies the existing methods– 1) That generate test cases (input and output) using LLMs; 2) Generate inputs only and use the oracle solution to get the output. With proposed metrics to test quality of generated tests, the paper shows that current methods does not generate good unit test suites. The paper then proposes SAGA, which uses incorrect solutions from humans to generate the input-interpretor, followed with explanation and self-validation code. Experiments show that SAGA performs better than the baselines. Lastly, on selected 101 problems in LCB-v6, they show that SAGA generated unit tests are more accurate than test suites of LCB-v6.", "questions": "1. In Introduction (line 38-42), the paper talks about biased generation of tests which contrasts with human reasoning errors. Are there some examples that show this?\n\n2. The Definition 1 is confusing, $E_i$ is an event such that the solution S fails unit test $i$. So can be either {0, 1}. Now, the intersections of $\\bar{E_i}$ will be empty of will be ${0}$ or ${1}$. If it is 0, then all cases passed, else all cases did not pass. The confusing part is what does the $\\mathbb{P}$ of this set mean? Given that $\\epsilon_S(T)$ is already defined, how does the second part explain it better? Overall, I feel the two definitions can be explained more simply. \n\n3. For paradigm 1, how does the Detection rate vary with test case size? This experiment was done for paradigm 2, but it would be interesting to observe for paradigm 1 as well. \n\n4. For the DEPC metric, what does the error pattern vector mean? \n\n5. In Sec 3.1, it is unclear how the human priors were added? \n\n6. In SAGA, are all the correct and incorrect solutions passed together to get the Explanations and Self-Validation code? How are the strategies in $S_{human}$ decomposed into formal mathematical and logical constraints?\n\n7. The baselines from Paradigm 1 relaxes the constraint of having a ground truth code solution which is needed in SAGA. Is there a way the method can be extended to cases where the ground truth code solution is not available?\n \n8. How does the performance compare to good reasoning models when used in any of the paradigms? Does the Qwen models used in Table 2 use the thinking mode? Since the agent needs to come up with multiple test cases, an extended reasoning should aid in better performance.", "limitations": "The paper discusses the limitations in the Appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Existing benchmarks that use models to generate unit tests might not be completely accurate at testing code solutions. This paper studies this and show that bugs introduced by humans many times are missed by unit tests generated with the language models.\n2. The proposed metrics to test efficacy of generated unit tests is interesting. \n\n### Weaknesses\n1. Some parts of the paper are not well written and hard to follow. Some points are written in the questions below. \n2. Given the complexity of the task of generating unit tests, the paper uses non-reasoning models. Whereas, I believe reasoning should help in better performance as the agent needs to think through the corner cases and simulate the code in CoT to get outputs. It would be nice to see comparison with reasoning models.", "was_revised": true, "final_justification": "I feel access to the gold solution is a big limitation of this work. In most applications, the gold solution is not provided by the user (e.g. vibe coding), and this method cannot be applied in such scenarios.\n\nBased on the response and other reviews, I would like to keep my positive evaluation and current score.", "rebuttal": "Dear Reviewer p61z,\n\nWe sincerely thank you for your review, and we are especially grateful for your recognition of our **in-depth analysis of the deficiencies in existing benchmarks** and the **innovative metrics** we have proposed for evaluating test effectiveness.\n\n**Regarding Weaknesses and Reasoning Models (Your Weakness #2, Question #8):**\n\nThis is an excellent point. We want to emphasize that the **SAGA framework itself is fundamentally designed to guide LLMs in structured reasoning**. In our main experiments, we chose DeepSeek-V3 as the backbone model based on a comprehensive consideration of performance, efficiency, and cost, which makes our method more easily reproducible. By compelling the LLM to deconstruct human defensive code patterns and analyze specific failure modes, SAGA employs a **guided, programmatic Chain-of-Thought  process**, prompting the model to \"think through various corner cases.\" The detailed prompts we have open-sourced serve as direct evidence of this reasoning-driven process.\n\nTo directly address your concerns about reasoning models, we conducted new experiments with a state-of-the-art reasoning model. The results on **Qwen3-235B-A22B** are shown in the table below:\n\n|                     | VAcc@20 | VAcc@50 | DR@20  | DR@50  | AUC@50 | DivRatio@50 |\n| :------------------ | :------ | :------ | :----- | :----- | :----- | :---------- |\n| **Qwen3-235B-A22B** | 21.48%  | 28.78%  | 84.74% | 89.10% | 0.2119 | 94.27%      |\n\nWe observed that the performance with the reasoning model actually decreased. After manual investigation, we found this was *not because of a lack of reasoning ability*, but due to **practical engineering challenges**—the model's inference sequence length exceeded the limits of our deployed model (our Qwen3 deployment has a sequence limit of 64k), causing some difficult problems to not be parsed successfully. This also indicates that using such a large reasoning model significantly increases inference and time costs, with limited actual performance improvement. This result **highlights our rationale for choosing DeepSeek-V3**: it is more accessible and reproducible for the community. We will add this experiment as a new section in the appendix.\n\n**On Biased Test Generation (Your Question #1):**\n\nThank you for requesting specific examples. An illustration can be found in **Figure 1(b)** of our paper. The PCA analysis shows that errors caused by LLMs are **highly clustered**, indicating a systematic bias—they tend to make mistakes on similar types of problems. In stark contrast, human errors are **diverse and widely distributed**. This means that if test suites are generated without reference to human error patterns, the scores of LLM-generated code could be **artificially inflated**. This directly explains a key finding in our work: when we re-evaluated code that passed on LiveCodeBench on a real online judge platform, we found its verifier accuracy to be alarmingly low. This is precisely because the tests in LiveCodeBench, lacking insights from diverse human failure modes, missed bugs that a more robust verifier should have caught.\n\n**On the Clarity of Definitions (Your Question #2):**\n\nThank you very much for pointing out where a clearer explanation was needed.\n\nLet us re-clarify these two core metrics:\n\n- **Detection Rate, es(T):** This is a **solution-level** metric that measures, \"Can our test suite T find *at least one* bug for *this specific* buggy program?\" Mathematically, it is the **probability of the union of events**. You correctly identified *Ei* as the event that \"solution S fails on test case i.\" Therefore, *es(T)* is the probability that **at least one** of these failure events *E1, E2, ..., En* occurs, which is *P(E1 ∪ E2 ∪ ... ∪ En)*. Our use of *1 - P(∩Ei_bar)* in the paper was a way to calculate this union probability using its complement, where *Ei_bar* is the event that \"solution S *passes* on test case i.\" We acknowledge this notation may be unintuitive and will revise the final version to explain it directly with simpler language and the *P(∪Ei)* form.\n- **Verifier Accuracy, VAcc(T):** This is a stricter, **problem-level** metric that asks, \"For a given problem, can our test suite T find bugs in *all* known incorrect solutions?\" It evaluates the **systematic completeness** of the verifier. *VAcc(T)* equals 1 only if the Detection Rate is greater than 0 for every single incorrect solution for that problem, making it the **gold standard for measuring verifier robustness**.\n\nWe will revise this section with clearer language and examples in the final version to ensure the logic and distinction between these two definitions are perfectly clear.\n\n**On the Relationship Between Detection Rate and Test Scale for Paradigm 1 (Your Question #3):**\n\nWe focused the \"Detection Rate vs. Test Scale\" experiment on Paradigm 2 due to the intrinsic differences between the two paradigms. The core advantage of Paradigm 2 (Input-Interpreter) is its ability to easily generate a massive and diverse set of test inputs. In contrast, Paradigm 1 (Direct Generation) struggles to produce a large number of unique and complete test cases (including both input and output), making it less suitable for an analysis that requires varying the test scale from 1 to 100. This is also why in our main experiments comparing different methods, we report performance at a fixed scale (e.g., @20 and @50).\n\n**On the Meaning of DEPC's Error Pattern Vector (Your Question #4):**\n\nThe error pattern vector *$v(t_k)$* for a test case *$t_k$* is a binary vector where each element corresponds to one of the *$N_p$* incorrect programs in our benchmark. If *$t_k$* causes the *j-th* incorrect program to fail, the *j-th* element of the vector is 1; otherwise, it is 0. DEPC then counts only the number of **unique non-zero vectors** in the entire test suite. This allows us to see precisely which set of incorrect programs each test case exposes. If two test cases have identical error pattern vectors, it means they are **functionally redundant**.\n\n**On How Human Priors are Incorporated (Your Question #5):**\n\nThis is detailed in our SAGA framework. Human priors (*$S_{human}$*) are insights derived from correct human-written code. We feed these correct solutions, along with the problem description, into our **Multidimensional Analysis prompt** (available in our anonymous code repository). The LLM is then tasked with **reverse-engineering the defensive logic** (e.g., boundary checks, special value handling) from this human code. These machine-extracted strategies are the \"human priors\" that guide the generation of new, more challenging test cases.\n\n**On the Workflow of SAGA (Your Question #6):**\n\nFor Multidimensional Analysis, we provide correct solutions to give the LLM a broad perspective on robust coding patterns. For Differential Analysis, we provide a correct solution and its most recent preceding incorrect submission to help the LLM pinpoint specific failure modes. Decomposing strategies into formal constraints is a core task specified in our prompts; solutions are not input all at once but sequentially. We understand this part is complex, so we will **add a detailed case study in the appendix** of the final version to walk readers through the entire process with a concrete example.\n\n**On Extending SAGA to Cases Without a Ground Truth Solution (Your Question #7):**\n\nThis is a key question about the scope of SAGA's application. SAGA currently requires a ground truth solution to act as an infallible interpreter. This is a limitation we acknowledge in **Appendix J**. Extending SAGA to settings without an oracle (e.g., through majority voting or by learning a verifier model) is an important and exciting direction for future work.\n\nOnce again, we express our deepest gratitude for your valuable feedback. Your guidance has been crucial, and we are confident that with your insights, the revised paper will be much stronger and clearer.\n\nBest regards,\n\nAuthors of Submission 17344"}, {"review_id": "lyPhlhjrpE", "reviewer": "Reviewer_Y2Dv", "summary": "This paper systematically analyzes the limitations of test cases in current LLM code generation evaluation, and proposes a human-LLM collaborative test case generation framework SAGA. SAGA generates high-coverage and high-discrimination test cases by integrating correct/incorrect human programming solutions. The authors construct a TCGBench dataset for evaluation, and develop a new benchmark CodeComPass and a dedicated model TCGCoder-7B based on SAGA. Experiments show that SAGA significantly outperforms existing methods (such as LiveCodeBench) in detection rate (DR) and verifier accuracy (VAcc), and CodeComPass can more rigorously evaluate LLM code generation capabilities.", "questions": "Overall, this is a work of good quality. I have some concerns that the author can address and further enhance the paper accordingly.\n\nFirst, how efficient is SAGA? It seems SAGA requires multiple LLM calls, thus, high computational and monetary cost. Please justify it.\n\nThen, SAGA requires correct/incorrect human codes, which introduces another cost, i.e., human effort to collect such programs. It is easy to collect this kind of program from competition-level datasets such as Codeforces. However, for real-world software projects, it will be challenging. As a result, the generalization ability of SAGA is unclear, which should be discussed.\n\nAgain, the evaluations are based solely on competition-level datasets, which hinders the conclusion of the experiments.\n\nTCGBench-Lite selects datasets after June 2024 to avoid data leakage, but does not explain how to isolate the evaluation set when training TCGCoder-7B. Please clarify the training/validation/test set division of TCGCoder-7B.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n+ Target an important problem – the low quality of the current evaluation of code models\n+ A new test case generation process with theoretical support\n+ A new dataset for future evaluation of code models\n+ Comprehensive experiments\n\nWeakness\n\n- The efficiency of SAGA is unclear. SAGA needs to call LLM multiple times, which seems costly\n- SAGA relies on a large number of correct/incorrect human codes. The generalization of SAGA to other task domains (with low human codes) is unclear.\n- Codeforces and Nowcoder contain competition level programs. Lack of real-world software support (such as programs from GitHub repos).", "was_revised": true, "final_justification": "Good work. Please consider adding the discussions in the response in the final version.", "rebuttal": "Dear Reviewer Y2Dv,\n\nWe sincerely thank you for your thoughtful and constructive review. We are very encouraged that you recognize the **importance of the problem** we address, the **theoretical novelty of the SAGA framework**, the **contribution of our new dataset**, and the **comprehensiveness of our experiments**. Your insights on practical issues such as cost, generalization, and human effort are highly perceptive and align perfectly with our own considerations during this research. We greatly appreciate this opportunity to elaborate on how we have addressed these key points.\n\nBelow, we respond in detail to the weaknesses you identified:\n\n**1. On the Efficiency and Cost of SAGA:**\nThis is a critical practical concern, and thank you for raising it. We would like to clarify the operational flow of SAGA, which is designed to be **more efficient than it might initially appear**. The main cost of SAGA lies in its *analysis phase*. The LLM performs a single, powerful, and focused analysis of human code (correct and/or incorrect solutions) to generate a Python `Case Script`. This script is a standalone program that contains not only the logic for generating test inputs but also the *Math Explanations* and *Self-Validation* code.\n\nCrucially, it is this generated Python script—***not the LLM***—that then programmatically generates a large volume of test inputs. This design **avoids the high cost of repeatedly invoking the LLM for every single test case**.\n\nFurthermore, we completely agree with your concern about democratizing advanced TCG methods. This is precisely why we developed **TCGCoder-7B**. Our goal was to demonstrate that SAGA's complex reasoning capabilities can be ***distilled*** **into a smaller, more efficient model**. As our results in Table 2 show, TCGCoder-7B's performance **significantly surpasses baseline methods that use much larger models**. This proves that with specialized training, a small-sized model can provide top-tier TCG capabilities, allowing it to be easily deployed in most research environments and thus greatly lowering the computational barrier.\n\n**2. On Dataset Selection, Human Effort, and Generalization:**\nYou noted that our evaluations are based solely on competition-level datasets. This is correct, and we want to emphasize that this was a **deliberate and strategic choice, not a limitation.**\n\n-   ***High-Quality Ground Truth:*** In the task of Test Case Generation (TCG), the most critical challenge is **evaluating the quality of the generated test cases themselves.** Competitive programming platforms provide a massive source of near-perfect \"ground truth\" solutions, validated by millions of submissions. This is a level of reliability that **no other source (like GitHub) can currently match**. Without this reliable anchor, our research could not have conducted rigorous, reproducible, and quantitative evaluations.\n-   ***Rich Failure Modes:*** Our SAGA framework, especially its differential analysis, heavily relies on learning from **diverse human errors.** These platforms offer hundreds or thousands of real, labeled \"wrong submissions\" for each problem. This invaluable \"database of failures\" is an **indispensable resource** for training and evaluating our methodology, as it reveals the subtle mistakes human programmers make when facing complex logic.\n\n**On Generalization to Real-World Software:**\nWe completely agree that the ultimate goal is to generalize to real-world software like GitHub projects. However, we must recognize that this is a ***\"walk before you run\"*** **process**. The entire field is still in its very early stages when it comes to handling complex, multi-file, dependency-heavy real-world codebases (as evidenced by the struggles on benchmarks like BigCodeBench).\n\nTherefore, our strategy is: **First, master the art of generating high-quality tests in a \"sandbox\" that, while domain-specific, is extremely logically complex and has a perfect evaluation environment.** We believe that by validating SAGA's effectiveness on competitive code, we are building the **most solid methodological foundation** for tackling the larger, messier real-world problems in the future. Our work demonstrates a clear and significant advantage in this critical domain, which is a **key step toward that ultimate goal**.\n\n**3. On the Train/Test Split for TCGCoder-7B:**\nThank you for raising this critical point for clarification. We assure you that we took **meticulous measures to prevent data leakage** and ensure a fair evaluation.\n\nOur process is as follows:\n\n-   **A Strict Chronological Split:** The training data for TCGCoder-7B is sourced entirely from earlier problems on Codeforces and Nowcoder, the vast majority of which are from **before 2023**. Our evaluation set, TCGBench-Lite, on the other hand, consists exclusively of contest problems from **June 2024 onwards**. This strict temporal separation ensures that our model is evaluated on unseen problems, guaranteeing the **impartiality of our results**.\n-   **Proactive Semantic Deduplication:** We went a step further by using a **SentenceTransformer model** to deduplicate all problem descriptions, filtering out any pairs with a cosine similarity greater than 0.9 to ensure problem independence. Our recent manual checks revealed that even for these highly similar problems, their core solution logic and patterns are entirely different.\n\nWe hope these detailed clarifications fully address your concerns and demonstrate the rigor of our experimental design. Your practical and insightful questions have been invaluable in helping us to better articulate the contributions and context of our work. Thank you again for your diligent effort and feedback.\n\nBest regards,\n\nAuthors of Submission 17344"}, {"review_id": "upzI7UpulS", "reviewer": "Reviewer_2jtr", "summary": "The paper provides a critique of Test Case Generation (TCG) for measuring LLM-based code generation. The critique asserts that existing benchmarks suffer from homogeneity of test cases that, in some cases, allow subtle faults to go undetected.  The paper defined four (4) metrics for test case quality.  Test case quality was studied by assembling and measuring results from TCGBench, a benchmark aggregated from existing code generation benchmarks. The paper then introduces Strategic Adversarial & Constraint-differential GenerAtive (SAGA), a workflow that combines LLM and human insight to generate improved test suites using novel methods for multi-dimensional and differential analysis. The SAGA method is then used to develop CodeComPass (a verifier) and TCGCoder-7B (a test case generation model).", "questions": "No questions.", "limitations": "The lack of a discussion on related work is a critical defect. The paper includes a reasonable section on limitations, but as Appendix J. Limitations are not addressed or summarized in the body of the paper.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength - The paper starts with a relevant and important question - \"Are the test cases of current benchmarks for evaluating models’ code capabilities robust enough?\"\n\nWeakness - A significant defect in this work is that there is no discussion of related work in the body of the paper. This includes both work that has been done to measure/improve individual benchmarks like HumanEval and work to measure/improve the quality of code assessment generally.\n\nWeakness - Section 2 is largely a restatement of the limitations of empirical testing (in contrast to formal methods). The section introduces definitions for DR and VAcc, but then concludes by defining two (2) new additional metrics. What constitutes a distinct error pattern is defined, by not well explained in a practical setting.\n\nWeakness - Section 3 begins with the claim \"Section 2.2 revealed critical limitations in TGC paradigms, such as low test quality and homogeneity. It is not clear how this statement follows from what was presented in Section 2.2 or Figure 4.\n\nWeakness - While the idea of leveraging human effort (especially already expended human effort) is a good one, and while the method shows improved testing capability, there isn't enough detail to clearly understand (or reproduce) what was done. The descriptions of Multidimensional and Differential Analysis are brief and vague.  Since this is one of the main contributions of the paper, there should be more here.\n\nStrength - It would seem that the paper provides a path toward generating a better verifier approach for existing and to-be-constructed benchmarks that would make for a valuable paper, but the result is a bit muddled in its presentation. Weaknesses as described should be addressed.  It might be that the measurement and improvement (SAGA) portions should be separate papers.", "was_revised": true, "final_justification": "I appreciated the author's rebuttal and their addressing my feedback. I have also looked at the material provided as response to other reviewer's comments. On the basis of these clarifications and improvements I have increased my overall rating to 3.", "rebuttal": "Dear Reviewer 2jtr,\n\nWe thank you for your feedback, and especially for recognizing the **importance and relevance** of the question we address, as well as the **potential path and value** our work offers for building better verifiers. Your comments have helped us identify several areas where the presentation clarity and impact of our paper can be substantially improved. We assure you that we have taken all your points seriously and have prepared the following clarifications, which we hope will address your concerns. We sincerely thank you for the time and effort you have invested in reviewing our manuscript.\n\nWe will address the weaknesses you pointed out one by one:\n\n**Weakness 1: \"no discussion of related work\"**\n\nThank you for pointing out this issue. We would like to clarify right away: **we do have a very comprehensive related work section, located in Appendix A, and we commit to moving a condensed summary into the main body in the final version, adopting a \"main text summary + appendix details\" structure.**\n\nThe reason for this initial arrangement was that the main text space was insufficient to accommodate the full version. In order not to sacrifice the depth and completeness of the review—which we feel is critical for readers to fully understand the landscape—we made the difficult decision to place the full discussion in **Appendix A**. This section (A.1 and A.2) covers advancements in LLM-based code generation, various evaluation benchmarks (like HumanEval, EvalPlus), and both traditional (SBST, Fuzzing) and LLM-based TCG methodologies. We sincerely apologize that this information was not easily discoverable. In the final version of the paper, we will move a condensed yet thorough summary of this related work section into the main body to ensure all readers can understand how our work is situated within the broader academic context. Furthermore, we will cite the latest research in the updated version and **sincerely welcome any relevant references you feel we may have overlooked**. We will incorporate them into our final revision.\n\n**Weakness 2: Clarity of Section 2 and its metrics**\n\nWe appreciate your feedback on the clarity of Section 2. Our initial intention was to build a comprehensive measurement framework, and we now recognize that our initial explanation was too dense. The four metrics (DR, VAcc, DEPC, AUC-AccN) are designed to assess test suite quality from multiple perspectives.\n\n-   DR and VAcc measure **effectiveness** (Can it find bugs?).\n-   DEPC and AUC-AccN measure **intrinsic quality and efficiency** (How diverse and potent are the tests?).\n    The \"error pattern vector\" is a practical method for defining diversity: it is a vector that represents which incorrect programs a single test case can detect a bug in. We have provided more detailed descriptions of all these metrics, their formulas, and their interpretations in **Appendix B**. Based on your feedback, we will revise Section 2 in the main text to provide more intuitive explanations and examples, referencing the clearer descriptions in the appendix.\n-   The practical value of these metrics is demonstrated throughout the paper:\n    -   In **Figure 6**, we use them to precisely show ***why*** SAGA is superior to the baselines (as it achieves higher DEPC and AUC-AccN).\n    -   **Appendix B** provides their full mathematical derivations and practical interpretations.\n\n**Weakness 3: The logical link between Section 2.2 and Section 3**\n\nThe assertion at the beginning of Section 3 that \"Section 2.2 revealed limitations\" is directly supported by the experimental results presented in that very section, specifically **Figure 3 and Figure 4**.\n\n-   **Figure 3** shows the *low quality* of directly generated tests (low DR and VAcc).\n-   **Figure 4** shows the *homogeneity* of tests from the Input-Interpreter paradigm, evidenced by detection rate saturation, which validates the theoretical upper bound for the most commonly used paradigm today.\n    These results demonstrate what we refer to as \"low test quality and homogeneity.\" We will rewrite the opening of Section 3 to explicitly reference these figures and findings, thereby establishing a clearer motivation for SAGA for the reader.\n\n**Weakness 4: Lack of detail on the SAGA method**\n\nThank you for the emphasis. The power of SAGA lies in its structured, human-insight-driven process. The core mechanism of SAGA operates through **detailed, structured prompts, which we have open-sourced in the supplementary materials** (with full prompts for both Multidimensional and Differential Analysis). These prompts are effectively the *\"source code\"* of our method. To make it completely transparent and reproducible, we will **add a new, detailed running example to the appendix** of the final paper. This example will demonstrate:\n\n1.  How a problem description, a correct human solution, and a buggy human solution are fed into the SAGA prompts.\n2.  The LLM's structured output, including the `constraint_analysis` and the generated `Case Scripts`.\n3.  How these scripts are executed to produce the final test cases.\n    This will provide the concrete, reproducible detail that you rightly pointed out was missing.\n\n**Limitations:**\n\nThank you for your attention to this. A dedicated and detailed discussion of limitations and future work is located in **Appendix J**. We discuss the scope of human priors, generalization, computational cost, and other important factors. We recognize the importance of highlighting these limitations and will **add a summary of this section to the conclusion in the main body** to ensure this crucial context is not missed by readers.\n\nWe are confident that these clarifications and the planned revisions, directly guided by your valuable feedback, will address the issues you have raised and significantly strengthen our paper. Thank you again for your rigorous review and for helping us improve our work.\n\nBest regards,\n\nAuthors of Submission 17344"}]

Author rebuttal:

Meta-review: The paper introduces a new benchmark (called CodeComPass) for LLM code generation.  

Strengths: very timely and the problem being addressed is immediately useful for many people in the NeurIPS community.

Weaknesses: writing is at times dense and unclear, some concerns have been brought up about reproducibility (the authors posted a message with a reply).

The reason for acceptance is that the community needs better benchmarks, which is a need this paper fulfils.

The discussions centred on: (1) clarifications about what the authors do exactly (2) clarifications of definitions (3) the fact that the algorithm relies on human written solutions to coding problems.
Decision: Accept (poster)