How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective
Authors: Xianzhen Luo
JinYang Huang
Wenzhen Zheng
Qingfu Zhu
Mingzheng Xu
Yiheng Xu
YuanTao Fan
Wanxiang Che
Venue: ICLR 2026 Poster (2026)
Keywords: Code LLMs;Benchmark;Evaluation;Test Case
Primary area: datasets and benchmarks

Abstract: Evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task. 
Existing benchmarks often evaluate the exclusion ratio on large, unstructured collections of wrong codes, 
suffering from high computational costs and score inflation.
Furthermore, they inadvertently reward generators that detect common, trivial bugs, while failing to penalize their inability to identify rare yet critical faults.
In this work, we connect two fundamental questions: (1) What is the minimal set of wrong codes sufficient to represent the entire error space? and (2) What is the minimal set of test cases needed to distinguish them?
We introduce a novel framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code-test matrix, where rows represent wrong codes and columns represent test case results.
The rank of this matrix specifies the minimal number of independent error patterns (wrong codes) and provides a tight upper bound on the number of test cases required for complete fault coverage.
Our objective is to identify a basis of size equal to the matrix rank that maximizes internal diversity.
To tackle this NP-hard problem, we propose WrongSelect, an efficient approximation algorithm to select maximally diverse wrong codes.
Applying this framework to millions of competitive programming submissions, we construct TC-Bench, a compact, diverse, and inflation-resistant benchmark. Extensive experiments show that even the most advanced test case generation methods achieve only ~60\% exclusion rates on TC-Bench, exposing a significant gap in their diagnostic power and highlighting substantial room for future improvement.
Our dataset is available at: https://huggingface.co/datasets/Luoberta/TC-Bench and our code is at: https://github.com/Luowaterbi/TC-Bench.

Published as a conference paper at ICLR 2026
How MANY CODE AND TEST CASES ARE ENOUGH? EVALUATING TEST CASES GENERATION FROM A BINARY-MATRIX PERSPECTIVE
Xianzhen Luo Jinyang Huang Wenzhen Zheng' Qingfu Zhu Mingzheng Xu"
Yiheng Xu* Yuantao Fan Wanxiang Che
Harbin Institute of Technology Central South University StepFun 4 Peking University [xzluo, qfzhu, car) ir hit.edu.ccr
tss.ikecsuu...uk...... Jecs.. edu.cr cn
ABSTRACT
Evaluating test cases automatically generated by Large Language Models (LLMs) is critical yet challenging task. Existing benchmarks often evaluate the ex- clusion ratio on large, unstructured collections of wrong codes, suffering from high computational costs and score inflation. Furthermore, they inadvertently re- ward generators that detect common, trivial bugs, while failing to penalize their inability to identify rare yet critical faults. In this work, we connect two fun- damental questions: (1) What is the minimal set of wrong codes sufficient represent the entire error space? and (2) What the minimal set of test cases needed to distinguish them? We introduce novel framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code- test matrix, where rows represent wrong codes and columns represent test case results. The rank of this matrix specifies the minimal number of independent eгror patterns (wrong codes) and provides tight upper bound on the number of test cases required for complete fault coverage. Our objective is to identify a basis of size equal the matrix rank that maximizes internal diversity. To tackle this NP-hard problem, we propose WrongSelect, an efficient approxima- tion algorithm to select maximally diverse wrong codes. Applying this framework to millions of competitive programming submissions, we construct TC-Bench, a compact, diverse, and anflation-resistail benchmark. Extensive experiments show that even the most advanced test case generation methods achieve only 60% ex- clusion rates on TC-Bench, exposing a significant gap in their diagnostic power and highlighting substantial room for future improvement. Our dataset is avail- http:/ggiinggtteeoootttttt/s///a
able at: http:///gthhbb.cmmmmoouwwtttt/C Bench.
and our code is at:
INTRODUCTION
The capability of Large Language Models (LLMs) in solving algorithmic coding problems is key measurement of their intelligence (OpenAl et al. 2024 2025 Jain et al.) 2024]. The evaluation of code solutions relies heavily on test cases. Golden Test cases (GTs), created by problem authors and continually refined and expanded by experts, are considered the correct solution. A solution is deemed correct only if it passes GT's. Current Code Reinforcement Learning with Verifiable Rewards (RLVR) methods similarly rely on test cases to compute rewards, placing substantial demands on the comprehensiveness of test cases (Le al.) 2022 Guo et al.) 2025 Team et al.) 2025) Zeng et al.) 2025a). As shown in Figure [] (a), the GT of a graph theory problem should encompass various graph sizes and structures, such as chain, tree, and star. Failure to cover all scenarios will compromise the reliability and lead to the false positive problem.
boundary-conditiin set equivalent to
GTs consist of few simple public test cases intended to clarify the problem and larger set of private test cases used to assess correctness. However, these critical private test cases are scarce
"Corresponding author. 'Equal contribution.

---

Published as a conference paper at ICLR 2026
Golden Test case
Augment Test case
WrongSelect
Basis
Rank=2 0
V
Method
Method
WC
AC= 0 WA=
WC
Easy
Tree
Star
T
T2
T2
WC
011
Diverse
WC
Wrong Code Result
T
T
T
Transformation
WC
AC
AC
WA
WC²
AC
WA
AC
Evaluation
Overload
Valid Score
T T2
Ti
T
O(In d|)
WC;
AC
WA
60
WC,
AC
WA
WA
95 92
O(12 21)
30
Inflated
Efficient
WC,
WA
WA
WA
Score
(o) Row Data
(b) ALL Wrong Codes
(c) Ours
A comparison of two evaluation frameworks for Augment Test cases (ATs). Both frame- works start from the same raw data (a), which consists of many wrong codes (WCs) and their exe- cution results on Golden Test cases (GTs). (b) The naive evaluation utilizes the full set of WCs and an unprincipled number of ATs, suffers from prohibitive computational costs, and leads inflated scores. (c) In contrast, our proposed framework first processes this data with WrongSelect to select a compact yet representative diagnostic basis (TC-Bench). Evaluation using this basis not only highly efficient but also yields more valid scores.
Figure I:
and expensive to create. To address this challenge, existing methods either manually construct test cases (Khan al. 2023) or automatically augment test cases (ATs) using LLMs (Cao et al., 2025 Ma et al. 2025b) Yang et al. 2025 Wang et al.) (025c). The emergent methods introduce the need evaluate their quality. The evaluation includes ensuring that their ATs valid (passing correct codes) and useful (excluding wrong codes (WCs) ). Since many methods are seeded with correct codes, their ATs are naturally valid. Thus, the core challenge shifts to assessing their usefulness. The straightforward approach is to collect as many wrong codes as possible and evaluate all ATs to determine how many WCs they can exclude. However, this incurs immense computational costs and suffers from inflated scores shown in Figure [ (b). This cost, product of the number of ATs and WCs, can be prohibitively high. Furthermore, one WC doesn't equal one kind of error. Indeed, the population of WCs is dominated by numerous trivial repetitive errors, with only few representing core, terdett- faults (Figure T (a) ). A mediocre method that only identifies common
can thus achieve the small number of critical faults gets statistically overwhelmed. Consequently, this diminishes the benchmark's 's discriminative power. Conversely, some heuristic methods selecting small subset of hard-to- oiltit eиrors yield overly sparse evaluations (Cao et al., 2025), unable to continuously reflect model capabilities.
similar to superior method that finds rare corner cases, as
These limitations raise fundamental questions: What constitutes an efficient and informative collec- tion of WCs for evaluating ATs? What principles should govern its size and member selection? The dual relationship between test cases and code also leads to another critical question: How many test cases are necessary to comprehensively define the solution space for a given problem?
We propose that an ideal WCs set should neither be heuristically nor randomly selected, but should be compact and diverse set of WCs that acts as a diagnostic basis, effectively spanning all unique error patterns of the problem. We propose to interpret the execution outcomes of WCs across GTs as a mapping from abstract reasoning errors to observable behavioral patterns. this binary representation, the accepted (AC) is denoted as 0 and wrong answer (WA) as 1. Each WC is thus represented as a binary vector, and the entire collection forms a Code- Test binary matrix. The matrix rank quantifies the maximum number of distinct error patterns present among WCs. More- over, provides an upper bound on the minimal number of test cases required to distinguish these error patterns. However, matrix produce multiple possible bases. An optimal diagnostic basis
2

---

Published as a conference paper at ICLR 2026
should consist of WCs representing minimally overlapping error patterns to maximize diagnostic breadth and information efficiency. Bases containing many similar WCs with highly overlapping error patterns suffer from redundancy. thus reducing discriminative power. As finding the most di- verse basis is NP-hard, we design WrongSelect, a greedy- efficient approximation algorithm that iteratively selects WCs that maximize diversity at each step, yielding the final basis.
To construct our high-quality benchmark, we collect numerous problems with their GTs and user submissions from prestigious algorithm competitions like USACO, NOI, and ICPC. We rigorously filter submissions, retaining only those with complete execution results on GTs. Next, we trans- form the codes for each problem into a binary matrix and calculate its rank to characterize the error pattern complexity. Then, we employ WrongSelect to efficiently select a maximally diverse set of WCs, constructing a structured diagnostic basis (Figure (c) ). Last, we meticulously review, stan- dardize, and translate all problem descriptions into English to ensure consistency and quality. The resulting benchmark, named TC-Bench, contains 877 problems with a total of 9347 WCs. The final set of WCs constitutes less than 2% of the original submissions. This reduction, combined with the principled number of the necessary test cases, can lead to a near-quadratic decrease in evaluation cost, dramatically improving efficiency. To validate TC-Bench,
reproduce and evaluate 5 com-
mon test-case generation methods (Jain et al. 2024] Zeng et al. 2022b Zhang et al., 22233 He et al.)
2025
Gu et al. 2024) 13 leading L.MMs (DeepSeek-Ale al. 2024 Int Hui et al. (2024). Experi- mental results show that even the state-of-the
method Claude4-1 Wlaude4-Thinking with LCB achieve only
approximately 60% performance. By eliminating redundant error patterns and surfacing critical cor- ner cases, TC-Bench ensures that method's 's ability to handle these challenges is directly reflected in its score. This directly prevents the score inflation that plagues less-curated benchmarks.
Our contributions can be summarized as follows:
We propose novel framework based matrix rank that, for the first time, unifies two fundamental questions: the minimal number of wrong codes needed for evaluation and the minimal number of test cases needed for coverage. This framework provides a principled method for constructing a structured diagnostic basis.
We construct and release TC-Bench, a compact and diverse benchmark built on our the- ory. By design, TC-Bench has a high signal-to-noise ratio, enabling efficient, reliable, and inflation-resistant evaluation of test case generation methods.
Through extensive empirical experiments, we uncover significant deficiencies in current mainstream test-case generation methods and LLMs when dealing with complex error pat- terns, providing clear guidance for future research.
METHODOLOGY
This section details our principled approach to constructing TC-Bench. We first formalize the prob- lem finding maximally diverse basis within a binary Code-Test matrix (Section 2.1.. Recogniz- ing this problem as NP-hard, we then propose WrongSelect, a greedy approximation algorithm for this task (Section 2.2 Finally, we detail the data processing pipeline used to apply this framework in practice to build TC-Bench (Section 2.3
2.1 PROBLEM FORMULATION
Identifying diverse underlying errors in a vast collection of WCs would require immense manual effort from algorithm experts, which is clearly infeasible. Therefore, the key challenge is to finding a formal transformation that can equivalently represent the diversity of underlying errors.
Our inspiration comes from how codes are evaluated. A code is considered correct if and only if it passes GTs, which are assumed to cover all problem requirements and boundary conditions, thereby defining the solution space. For any code, we can get its result on GTs. For example, the result ["AC", "WA", "WA"] represents a code that passes the first case but fails the other two. Such a result sequence be regarded as a behavioral mapping erroneous reasoning of a code into a concrete pattern within the solution space. Collecting all such signatures across codes allows construct empirical space of failure modes for problem.
failure signature, translating the abstract
3

---

Published as a conference paper at ICLR 2026
However, this raw space is highly redundant: it contains identical signatures, and some patterns may simply be combinations of other ones. To extract a compact and informative benchmark from this landscape, structured analytical tool is required.
Binary Matrix Representation We formalize this space of failures as binary matrix M of size where n is the number of WCs and is the number of GTs. Each entry My is defined as:
{1 if the i-th WC fails on the j-th test case, Mij- 0
if the i-th WC passes the j-th test case.
Each row vector of M thus represents the failure signature of the i-th WC. For instance, signature ["AC", "WA", "WA"] becomes the binary vector [0, 1, 1].
Optimization Objective With this binary Code- Test matrix in place, task reduces to selec- tion problem: how to choose from the m failure signatures a representative and compact subset to serve benchmark. An ideal subset I must satisfy the following two requirements. Com- pleteness and Irredundancy. The selected set should capture the full complexity of M without redundancy. In linear algebra, this corresponds precisely basis, i.e., the row vectors in I are linearly independent and their number [Z] equals the rank of M. This constraint guarantees that the number of selected WCs neither too many nor too few, but exactly sufficient to span all distinct error modes. Notably. since the row rank equals the column rank, this same value | also provides another important insight: constitutes a theoretical upper bound on the minimum number of test cases required to distinguish all independent error modes. Diversity. Multiple bases may satisfy the rank condition. Ideally, a perfect basis would consist of mutually orthogonal failure signatures, meaning each error mode is completely independent and contributes unique dimension. However, in real-world error data, this kind orthogonal basis rarely exists. Our practical goal is therefore find a basis that approximates orthogonality by max- imizing the diversity among its members (i.e., minimizing their overlap). To measure the overlap between signatures, adopt the Jaccard similarity, which quantifies the ratio of jointly failed test cases to the total failed cases across both signatures. A lower Jaccard score indicates lower similarity. Formally:
basis. Concretely, must be row
(T.,T,))
|r I 1-
where r counts the jointly failed test cases (intersection) and ||r|] is the total number of failed tests for a signature (size the set).
Beyond pairwise similarity, we must assess the diversity of the entire basis I. We therefore define our global objective as minimizing the average pairwise Jaccard similarity among all members I:
min F(I)= I
Σ
Jr,,
(2)
In summary. our problem is formalized as follows: given a binary matrix M. find a row basis I that minimizes the average pairwise Jaccard similarity F(I). This is combinatorial optimization problem known to be NP-hard. In the next section, we present a greedy algorithm, WrongSelect, designed to efficiently approximate this solution.
2.2
WRONGSELECT
2.2.1
PRINCIPLED PRE-FILTERING
The quality of the final basis critically depends on the quality of the candidate pool. In practice, raw data often contains noise, such as problems lacking sufficient WCs or WCs failing on all test cases. To address this, pre-filtering is designed to systematically remove these noise at both the problem level and the code level.
Problem-Level Filtering via Column Analysis In practice, observe that some M contain columns filled entirely with "I" as shown in Figure 2 This indicates that all WCs fail in one case. The analysis subset shows that this phenomenon arises from three main causes: (1)
4

---

Published as a conference paper at ICLR 2026
Web crawler
Step1: PRE- SRE-FILTEE FIRTIINNG
Step2: RANDOM-RESTARE LOCAL SEARCH
110
F
FOn fot NO
Column
010
R
011
Easy To Hack
Teurreut
J I Tic
0011 1
M
001
M
o 1 Tont
Traet nolghhoe
001
Laurrent
[011
foc.daaunttiin
Row
010
010
Fin
010
0 o Foul 011
011
011
BackfrackingSrrnnggnaa
111
Meaningless Noise
Fin
Searching Data Structures
Gready loundry
Recursion
Tree Divide And Conquer
Number Theory Sorlinggaah
Greedy
Problem Description
Inappropriate Question
Clean & Format
Correct Code
Dynamic Programming
Translation
Time Top 20%
An overview of the TC-Bench construction pipeline. It begins with raw data collection, followed by a two-step WrongSelect working on the transformed binary matrix M. Step pre-filters the problems with an all- "1" column and removes codes whose rows have too many "1"s. Step 2 samples numerous initial bases Teurrent from the filtered M' and iteratively minimizes the diversity score by swapping internal and external rows. The best local optimum is chosen to approximate the global optimum. Concurrently, problem descriptions are standardized and correct codes are sampled from the top 20% performers, ensuring the overall quality of TC-Bench.
Figure 2:
GT exhibits incremental difficulty (e.g., gradually stricter constraints on time or space complexity); (2) the number of WCs for the problem is insufficient; or (3) the problem or GT is overly simple, involving only single extreme scenario. Although the first case is reasonable, it is relatively rare, and manually distinguishing it is prohibitively costly. More importantly, all-ones columns open the door to hack scores. Therefore, ensure the diagnostic value of each problem, we exclude all problems containing all-ones columns from our dataset. This excludes about 5% of raw problems.
Code-Level Filtering via Row Analysis Another observation is that some WCs fail on exces- sively high proportion of GTs. Such WCs typically pass only the public test cases while failing almost all private ones. They act strong background noise: any mediocre test can easily elim- inate them, leading to inflated evaluation scores and severely diminishing the discriminative power of the benchmark. To mitigate this, we compute the failure rate of each row, which is defined as the proportion of I's relative to d. Accordingly, we set the filtering threshold т 80%. A WC exceeding highly likely to fail all private test cases. Such WCs generally correspond triv- ial or common error patterns, and removing them helps benchmark to diagnose more nuanced and complex failure modes. This removes 13% of raw WCs and M turns into M'.
As the final quality control step, we exclude all M' with rank less than 5 (Rt' < 5). A low rank indicates insufficient diversity in error patterns and is not suitable to be used in a benchmark. Only matrices M' that pass all these filtering stages are considered qualified candidates and proceed to the subsequent basis selection process.
2.2.2 RANDOM-RESTARE LOCAL SEARCH
On the filtered matrix M', our objective is to select a basis I that achieves the lowest possible F(T). We adopt a local search optimization strategy to approximate the optimal basis.
Starting from complete but randomly chosen initial basis, the algorithm iteratively improves the basis by performing local modifications. Specifically, it explores the neighborhood of the current basis, defined as all new bases that can be obtained by a single swap operation (exchanging one member inside the basis with one outside). If there exists a neighbor that achieves better diversity (lower F(I)), the basis is replaced by the best neighbor, and the process repeats. This iterative improvement continues until no better neighbor exists, i.c., the current basis converges to a local optimum. To mitigate the risk of being trapped in poor local optima due initialization, we employ
5

---

Published as a conference paper at ICLR 2026
random-restale mechanism. The local search process is repeated multiple times from different random starting points, and finally, the best solution among all local optima is selected as the output.
Take "Step2" in Figure 2 as an Example. M' has rank of R² 2. Assume the initial random basis is Zeurrent = [ 0, I, [0, 1, 1]]I with a diversity score of (Z = 0.5. The only external candidate is the vector M' Zeurrent
[0, 1, OJ. The algorithm then explores the neighborhood of Tearrent- It first considers swapping the internal vector Your = [0. 0. 1] with the external vector 0j. The resulting set, [[0, 1,0,1,0,, [0. 1, 1]]. valid basis, but its score /' 0.5 provides Fin [0, 1,
no improvement. Next, consider swapping Fout [0, 1, with rin - [0, 1, 0j. This produces a better basis Zemp [[o, 0, 1], [0. 1, O... Its diversity score F(Znep)
O. After evaluating all neighbors, since a better neighbor is found, the algorithm updates its state: Teurrent + [[o, 0, 1], [0, 1, OJl. A new search iteration begins from this basis. As this basis is now perfectly diverse (F 0), no further swaps can improve the score, so the algorithm has converged to local optimum. This result is saved, and the random-restart mechanism initiates a new search from another random starting point. Algorithm in Appendix A.2 illustrates the pseudo code with detailed explanation.
Although the nested structure suggests high theoretical complexity, in practice the algorithm con- verges rapidly in both the inner and outer loops. Moreover, several parts of the procedure can be parallelized easily, making the overall runtime highly efficient.
2.3 BENCHMARK CONSTRUCTION
Evaluating test cases requires not only wrong code, but also first generating them from problem descriptions and validating them against correct code. This section details the full pipeline of data collection, filtering, and cleaning used to construct our benchmark.
Raw Data The raw data comes from top-tier programming contests and high-quality training sets, including USACO, IOI, and ICPC. In total, it initially contains 3,321 problems and 2,230,009 sub- missions. We retain only problems for which the full execution results of WCs on GTs are available. After this step, we obtain 1,763 problems, containing 15,457 correct codes and 554,056 WCs.
Problem Description To ensure fair and consistent problem comprehensions, we apply rigorous standardization to problem descriptions. We first remove problems that heavily rely on images, cannot be automatically evaluated (e.g., interactive problems, multi-solution tasks), or require highly constrained runtime environments. We then clean the statements by removing source tags, URLs, and HTML, as well as rewriting non-standard mathematical formulas. Finally, we employ GPT-4o to translate non-English problems and manually proofread to ensure semantic consistency.
Wrong Code To ensure consistency of the evaluation environment and avoid noise introduced by environment-speciff factors, retain only C++ submissions labeled as WA, including 1,698 prob- lems and 282,458 WCs. Next, our principled pre-filtering leaves 1,133 problems with 33,846 WCs. For each problem, perform random-restart local search with both outer and inner loops set to 1000 iterations. Figure 9 shows that loops converge rapidly, demonstrating the efficiency of our method. Ultimately, 13,400 wrong codes constitute the maximally diverse basis for all problems. Figure 10 illustraaes the distribution of WCs per problem before and after WrongSelect.
Correct Code Since correct codes consistent with GT, their primary differences lie in runtime and memory consumption. In Section 4 we show that overly loose or overly strict sets of cor- rect codes can bias evaluation results. Therefore, for each problem, we randomly select correct submissions from the top 20% after min-mat max normalization of runtime.
Through this principled pipeline, we ultimately construct TC-Bench, a high-quality diagnostic benchmark with 877 standardized problems, 9347 core WCs, and 7016 correct submissions. More details regarding the construction process are available in Appendix [..] Furthermore, we present a case study in Appendix C to empirically validate the practical effectiveness of WrongSelect.
3 EXPERIMENT
After constructing TC-Bench, this section presents the experimental design and evaluation results of different test case generation methods.

---

Published as a conference paper at ICLR 2026
3.1 EVALUATION SETUP
3.1.1 MODELS & METHODS
Models We evaluate SOTA LLMs via API, including GPT-4o, Claude-Sonnet-4, Claude-Sonnet- 4- 4-Thinking, DeepSeek- V3, Owen-Coder-Paie and Qwen3-235B-A225 We also evaluate Qwen-2.5 and Qwen-2.5-Coder families of varying sizes (7B, 14B, 32B). Due to space constraints, the results the 7B and 14B LLMs are presented in Appendix AA.3 We note that DeepSeek-RI struggles to reliably generate test cases. Therefore, its results are excluded from the main experiments but discussed in Appendix A.5 In total, we evaluate 13 LLMs.
Methods Based on whether correct code is available during generation, methods be categorized into two classes. The first class does not rely on correct code. CRUX (Gu et al. 2024) directly generates inputs and outputs. PSEUDO (liao et al. (024) generates both inputs and candidate solutions, then obtains outputs by ex- ecuting the solutions and taking the majority-voted output as the result. Going further, ALGO (Zhang et al.] 2023) prompts the LLM produce imput generators (execute to obtain inputs) and brute-force oracle solu- tion (lower the difficulty).
Execute
Generated by Model
CRUX
PSEUDO
input 1,1,2
output
input
solutions main(y)
600 AAB
Yes
X10
output
Degrade
ALGO
Input Generator
Oracle main()
output
generator()
LCB
Random Input Generator
input
Edge-case Generator
HT
Random Input Generator
input validator
input
input
Edge-case Generator
def
ssvalid(): bool
True
When the correct code available, output correctness be guaranteed by executing the inputs on it. Live- CodeBench (LCB) (Jain et al.) 2024) requires LLM to generate both multi- ple random and edge-case input gen- erators. It should be noted that we select one representative implemen-
Figure 3: CRUX, PRESUDO, and ALGO construct the out- put, while LCB and HT depend on the correct code to gen- erate the output.
tation for each category, and the other variants are in Appendix A.1
3.1.2 PIPELINE & METRICS
Test Case Generation For each problem in TC-Bench, ATs are first generated by the evaluated methods. For methods that do not rely on correct code, only cases accepted by all correct code are considered valid. We define PassRate as the proportion of valid cases among all generated cases. Formally, for set of problems Q: PassRate Ty. is AT's for problem g., and (tlldd)t,,) is if test ty is valid, and 0 otherwise.
प En.ee (TTh Ener, IsValid(t,)). where
Wrong Code Execution To measure the effectiveness of the valid ATs, we define HackRate. A WC from TC-Bench is considered excluded if it fails (c.g.. WA, Time Limit Exceeded (TLE), RE (Runtime Error)) are counted as successful exclu- sion. The HackRate represents the proportion of WCs that successfully excluded. Formally: Enre [w.,] Ewew, IsExcluded(w)) (w)), where Wg. is WCs for problem 9: and IsExcluded (w) is WC w is eliminated, and 0 otherwise.
least one valid AT. All failure types
HackRate =
Q
3.2 RESULTS
Table presents the results for various model and method combinations TC-Bench.
TC-Bench Reveals a Significant Performance Ceiling for Current Technologies. Even the best- performing combination, Claude-4 + HT, achieves less than 63%. This result strongly validates

---

Published as a conference paper at ICLR 2026
Table 1:
Performance comparison for all evaluated model-method combinations. PR denotes Pass- Rate and HR denotes HackRate. AC represents the percentage of non-excluded wrong codes. WA, RE, and TLE are all considered exclusions and contribute to HR. PSEUDO of Qwen3 is anoma-
lous due to API frequently returning empty or low-quality responses.
LLM
Method
PR
AC
WA
RE
TLE
HR
Open Source
CRUX
26.71
84.57
13.51
0.89
1.03
15.43
PSEUDO
35.04
79.52
18.59
1.02
0.78
20.38
Qwen2.5-328
ALGO
20.48
78.04
20.29
1.33
0.33
21.96
LCB
57.62
48.39
48,46
2.07
1.08
51.61
HT
65.46
69.27
29.17
1.22
0.34
30.73
CRUX
22.68
81.27
16,31
0.91
1.51
18.73
PSEUDO
37.72
79.23
18.72
0.98
1.07
20.77
Qwen2.5- 201255--oddr3321
ALGO
21.33
81.41
17.27
0.85
0.46
18.59
LCB
59.65
41.90
54.90
2.21
0.98
58.10
HT
66.53
56.24
40.98
1.98
43.76
CRUX
37.90
83.01
15.54
0.85
0.60
16.99
PSEUDO
19.58
88.32
10.97
0.37
11.68
Deepseck-
ALGO
28.22
70.78
27.53
1.24
0.44
29.22
LCB
46.58
41.17
55.68
2.06
1.08
58.83
HT
63.51
50.58
46.34
2.05
1.03
49.42
CRUX
26.30
69.10
27.14
1.76
2.00
30.90
PSEUDO
9.85
97.54
2.15
0.19
0.12
2.46
Qwen3-235B-A22B
ALGO
25.90
70.23
27.84
1.28
0.65
29.77
LCB
70.40
54.03
41.25
2.40
2.32
45.97
HT
55.35
69.20
28.50
1.61
0.69
30.80
CRUX
29.26
67.65
28.79
1.71
1.85
32.35
PSEUDO
40.15
67.11
29.57
1.45
1.87
32.89
Qwen-Coder-Plux
ALGO
30.43
67.04
31.15
1.31
0.50
32.96
LCB
77.73
38.54
57.98
2.28
61.46
IIT
67.28
46.93
50.06
2.09
53.07
Closed Source
CRUX
42.43
70.77
26.25
1.46
1.52
29.23
PSEUDO
50.90
73.33
24.01
1.03
GPT-4o
ALGO
24.43
75.51
22.87
0.97
0.65
24.49
LCB
68.51
42.45
52.68
2.66
2.21
57.55
HT
47.68
49.45
47.48
2.16
0.92
50.55
CRUX
32.93
76.31
21.11
1.14
1.44
23.69
PSEUDO
64.72
63.97
32.35
1.23
2.45
36.03
Claude4
ALGO
32.12
69.17
29.01
1.20
0.62
30.83
LCB
55.49
37.92
58.29
2.63
1.15
62,08
HT
71.56
37.04
58.58
2.86
1.53
62.96
CRUX
30.47
66.26
31.14
1.44
1.16
33.74
PSEUDO
23.56
85.98
12.84
0.51
0.67
14.02
Claude4-Thinkae
ALGO
32.41
64.54
33.68
1.22
0.56
35.46
LCB
75.79
37.65
59.65
1.93
0.78
62.35
HT
71.24
39.69
57.26
2.08
0.97
60.31
that WrongSelect indeed selects diverse and challenging error basis, revealing a performance gap that would otherwise be masked in unfiltered benchmarks. This suggests that there is substantial room for improvement in handling complex and diverse errors, and TC-Bench serves as a reliable yardstick to measure this progress.
High PassRate does not Equate High Hackrate. A high PassRate score can hacked by generating large number of casy test cases. For instance, Qwen2.5-32B and Deepscck-V2 V3,, CRUX's PassRate significantly higher than ALGO's, s, yet its Hackrate score substantially lower.
The Impact of Methodology Far Outweighs That of the Base Model. The results consistently show that the choice of method has a much greater impact on final performance than the scale or even the source (open-source vs. closed-source) of the base model. For instance, while Qwen2.5-Coder. 32B has fewer parameters than the activated parameters of Deepseek- V3, their HackRate scores with the LCB method differ by only 1%. In contrast, Qwen2.5-Coode--228. LCB's HackRate is
8

---

Published as a conference paper at ICLR 2026
TCG Testoosn-Evel
TCS
Au (c-Berch(aan))
80
Testonee-Ew Evol
Besch(Uun)
60
Percent
20
CRUX
PRESUDO
ALGO
LGB
HT
Rank
20
25
30
(a) Evaluation Performance
(b) Rank Distribution
Filter Top 20%
60
LaB Posslate
Hadelltte
GRUX PussRate
(c) Normalized Time Distribution
(d) Top 20% Stability Test
Figure 4: (a) Rank distribution of filtered WCs across methods. (b) Comparison of evaluation results against TC-BENCH (Ours). (c) Normalized execution times of correct solutions, primarily distributed below 0.2. (d) Sensitivity analysis showing stable PassRate and HackRate when filtering with random subsets of & correct solutions.
nearly 40% higher than CRUX. Furthermore, we observe that top-performing open-source models (e.g., Qwen-Coder-Pluse ries) across various methods. We hypothesize that this is because test case generation is specialized task that is underrepresented in existing large-scale code pre-training corpora, thus limiting the per- formance gains through model scaling or different training corpus. Further experimental analyses, study on Time Scaling, and a summary of common error patterns, are detailed Appendix A.3
competitive with leading closed-source models (e.g., the Claude4
4 DISCUSSION
Unfiltered and Heuristic Code Sets Lead to Biased Evaluation. To validate the impact of code subset of 100 problems on Claude-4-
selection strategies, we conduct rigorous comparison Thinking. We compare TC-Bench against three baselines: TCGBench Ma et al. (2025b) (denoted as All WC), which uses the full set of wrong codes; TesiCase-Eval Cao et al. (2025). which randomly samples 20 codes; and TCG Yang et al. 2025), which selects wrong codes from those passing at least 60% of test cases. Unfiltered sets lead to score inflation. As shown in Figure 4 (a), the full set leads to severe score inflation. For instance, LCB exhibits near-perfect performance ( 100%) on AIL_WC, whereas its score on TC-Bench drops to just over 50%. This inflation masks the method's incompetence on core, difficult error patterns. Crucially, TestCase-Eval exhibits scores and trends highly similar AlL_WC across all methods. This indicates that naive random sampling, while potentially reducing dataset size, fails to exclude redundant error patterns and thus cannot resolve the issue of score inflation. Heuristic Selection results in under-representation. Conversely, TCG yields significantly lower scores. While this might seem rigorous, our rank analysis reveals it stems from insufficient coverage. Figure the rank of the error space varies significantly per problem. While most are below 20, some approach 30. TCG's 's rigid limit of 5 codes forces drastic represetprrnnnntaiion for high-complexity problems. While this lowers the performance scores of current methods, it makes future methods prone score inflation: they would only need to cover a maximum subset five patterns rather than the complete emror space to achieve perfect scores. In summary, TC-Bench strikes the optimal balance. By selecting basis defined by the problem's

---

Published as a conference paper at ICLR 2026
intrinsic rank, it avoids both the inflation of coverage- based methods and the under-reprresentatin of heuristic constraint methods, serving as a stable and fair test suite.
Rank serves as the Upper Bound for the Necessary Number of Test Cases. The row rank, which represents the number of independent error patterns, equals the column rank, which represents the number of independent diagnostic dimensions. In an enror space defined by rank R, there are only R linearly independent diagnostic dimensions. Any additional test case is merely a linear combination of these basis dimensions and does not provide new information for distinguishing existing error patterns. Therefore, R test cases are sufficient to distinguish all error patterns, serving as a compact upper bound. Consider a concrete example matrix with R
1
12
4
W1
0
0
W2
1
0
W3
0
1 0
0
W4
1
Here, columns l and l/ are linearly independent, but ly linear combination (133 4 + (2). Any wrong code failing on t or t2 implies predictable behavior on fz. Thus, 23 offers no new diagnostic dimension. The set {f1, f2, {1} is sufficient to distinguish all unique error patterns. This framework addresses a critical flaw in previous evaluations where the number of test cases was arbi- trary. Consequently, problems with small diagnostic dimensions were often "over-tested," inflating scores, while complex problems were "under-tested." Using Rank as the budget ensures fairness: allows simple problems to reveal performance gaps while ensuring complex problems are tested with sufficient depth.
Correct Code Selection Influence Results. Unlike WCs, which have failure signatures, correct codes all behave identically on GT, differing only in runtime and memory usage. This makes their se- lection more subtle. Using only single correct solution as a validator is insufficient. Certain invalid input may still have an output under specific code. Our initial exploration shows that as the number of correct codes increases (as shown in Figure more ATs are filtered, leading to higher HackRates. However, not all filtering is beneficial. Many complex but valid ATs are wrongly discarded due to timeouts by slow correct codes. Worse, such now-perfornance correct codes show inconsistency across environments (different OJ platforms). Performance profiling reveals a highly skewed distri- bution: most correct codes cluster in the top 20% after applying min min-max normalization to runtimes (Figure (c)). These segh-performance codes are stable across platforms. Consequently, we adopt biased random sampling strategy: for each problem, we retain only correct codes within the top 20% normalized runtime and randomly sample 8 from this set. Repeated experiments confirm that this strategy yields highly stable evaluation outcomes (Figure (d)).
AT Uncover Latent Bugs Beyond GT. An interesting phenomenon emerged during evaluation: some wrong codes labeled as Wrong Answer under GT's produce Runtime Error Time Limit Error when executed on ATs. To verify whether this is due to server overload, we conduct controlled experiment. We sample 350 WCs that exhibited RE/TLE and combined them with about 2.6k ran- dom WCs. Running these a 128-core machine, we gradually reduce concurrency from 128 to 88 tasks. The RE/TLE frequency remains nearly constant regardless system load. This strongly suggests that advanced methods are indeed capable of producing stricter and more challenging ATs than official GTs, revealing hidden bugs related to performance and robustness.
5 CONCLUSION
Existing evaluation practices suffer from inflated scores and unclear principles regarding how many codes and test cases necessary. We addressed this gap by formalizing benchmark construction as a binary-matris rank problem, which jointly determines the minimal code basis and the upper bound on test cases. To approximate its NP-hard solution, we introduced SrongSelect and applied it to large-scale competitive programming data, resulting in TC-Bench, a compact and diverse diagnos- tic benchmark. Experiments show that TC-Bench reveals substantial gaps in current methods and provides a faithful foundation for advancing research on test case generation.
10

---

Published as a conference paper at ICLR 2026
ETHICAL STATEMENT
The data for the proposed methods is drawn solely from publicly accessible project resources on reputable websites, ensuring that no sensitive information is included. Moreover, all datasets and baseline models used in our experiments are also available to the public. We have taken care to acknowledge the original authors by properly citing their work.
ACKNOWLEDGE
We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004 and 62476073.

---

Published as a conference paper at ICLR 2026
REFERENCES
Introducing Claude 4. https//www.attrrrpi..co//owwwww
Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?, 2024. URL https:
7/arxiv. www.wii.rrrr..ooo / abs 2412. 02883
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards effective code-integrated reasoning, 2025. URL
https://arae wwg.xvv..orraaeooo wwtp///////...ooo 250.5.2252555555
Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, Yixuan Tan, Jiafu Tang. Yilin Tang, Junlei Wu, Qianyu Xiao, Can Zheng, Shouchen Zhou, Yuxiang Zhu, Yiming Huang. Tian Xie, and Tianxing He. Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems, 2025. URL httpla://aaxxiii/ org. aww.///.......//// 06821
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guam Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. net /wtp/ppd22 peddfiiddkt... wtw.kkktttttmmmtt
URL https::/
Mouxiang Chen, Zhongxin Liu, He Tao, Yusu Hong. David Lo, Xin Xia, and Jianling Sun. B4: Towards optimal assessment of plausible code solutions with plausible tests. In Proceedings of the 39th IEFF/ACM International Conference on Automated Software Engineering, pp. 1693- 1705, 2024.
Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang, Yang Zhang, and Chuchu Fan. RI RI-code-interpreter:: Training Ilms to reason with code via supervised and reinforcement learning, 2025. URL Intp:::://///. /taptps://axxi..... org. abs/25055555 21668
Jonathan Cook, Silvia Sapora, Arash Ahmadian, Akbir Khan, Tim Rocktaschel, Jakob Foerster, and Laura Ruis. Programming by backprop: Llms acquire reusable algorithmic abstractions during
code training, 2025. URL https: / /ttp///avvvv///////////ee////avr httaaaawwvvvrrrreee/////
wtwggb/bbbb.e. abs///aa//// 2506. 18777
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent- rlvr: Training software engineering agents via guidance and environment rewards, 2025, URL
https://arxivi http://///axii..o//aa/////////// wttp:bo//go//////eo 11425
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang. Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu. Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei. Tianyu Sun, W. L. Xiao, Wangding Zeng. Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie. Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu. Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu,
12

---

Published as a conference paper at ICLR 2026
Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek-V2 V3 Technical Report, 2024, URL attps::/arxxe org/ abs 2412 19437
Mchdi Fatemi, Banafsheh Rafice, Mingjee Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning, 2025. WtR.https:/// /tt///aaxxxia v. wttp://wwww.///////....0 2etp//550400........
Jia Fu, Xinyu Yang. Hongzhi Zhang. Yahui Liu, Jingyuan Zhang, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Klear-codetest: Scalable test case generation for code reinforcement learning, 2025a.
05710
Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu and Yu Cheng. Scaling reasoning. losing control: Evaluating instruction following in large reasoning models, 2025b. URL https::/arxx:
org/ /ae///a///// 2505. 14810
Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htttps://openzeett Ffpg52swvC
net forttfformm?dd
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808) (118808))6633-666 ):6336638 2025.
Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. HardTests: Synthesizing High- Quality Test Cases for LLM Coding, 2025. URLpttps: //tp:vv./aiivva//a http://///aaa..oooc////////522.2
Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent- code generation with iterative testing and optimisation, 2024. URL https:
//arxiv. wwt./oogg/bbbbaooooooo
13010
Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-Coder Technical Report, 2024. URL https:// srxiv.org/ea
2409. 12186
Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark, 2025. URL https: /ttp//ixvvv.oro/.///. org abs 2lbs/2410.000 00752
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Lolar-Lezama, Koushik Sen, and Ion Stoica. Bench: Holistic and Contamination Free Evaluation of Large Language Models for Code, 2024. URL. httt:sss//rrsii..rr///tttt
2403. 07974
Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, and Furu Wei. Preference Optimization for Reasoning with Pseudo Feedback. In The Thirteenth International Conference on Learning Representations, 2024.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
0360.0770
Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. XCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval, 2023. URL https::/aaxxtt //arxiia.....//////////.aaa/ariv wtg....ig/g.//...........
abs /2303 03004
13

---

Published as a conference paper at ICLR 2026
Hung Le. Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Mastering code generation through pretrained models and deep reinforcement learning. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Informa- fion Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December www.. cc/paper files/paalee http:/ppppp/////apeeeep Abstract
Coderl:
9. 2022, 2022. URL
3e34119ee..a3$ -t wow.aattttooooeeeeeee.mmm html
Bin Lei, Yuchen Li, and Qiuwu Chen. Autocoder: Enhancing code large language model with AIEV-INSTRUCT, 2024. URL
14906
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, and Dayiheng Liu. Cort: Code-integrated reasoning within thinking, 2025. /tth::::: //rxxva..vvvv//////// org/ httgg/bb//2500.0 09820
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger. Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023. URL htRtttts::///tt
org/ abs 2305 06161
Zihan Ma, Taolin Zhang, Maosong Cao, Junnan Liu, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking verification for UIm code generation: From generation to testing, 2025a. URL httts::////attttt wrg. /500 06920
Zihan Ma, Taolin Zhang, Maosong Cao, Junnan Liu, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking Verification for LLM Code Generation: From Generation to Testing, 2025b. URL httts:ttts::::/xiii.t.t
/ts//////00 06920
Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents, 2025. URL htttt::://rxii../rrrrrr/// a 2406. 12952
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li. Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Prochl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter. Evan Mays. Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan
14

---

Published as a conference paper at ICLR 2026
Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk; Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi. Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. OpenAl ol System Card, 2024. htt.htt::: http::::///axxv...rre.. org. www.g/gg////ooooooero.o..e..eeee 2412 16720
OpenAI, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, 03 con- tributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive Programming with Large Reasoning Models, 2025. URL hwtphhttt/ss/ttttt org/
3bcs/2502.065 06807
Patomporn Payoungkhamdee, Pume Tuchinda, Jinheon Baek, Samuel Cahyawijaya, Can Udom- charoenchaikit, Potsawee Manakul, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, and Sarana Nutanong. Towards better understanding of program-of-thougghe reasoning in cross-lingual and multilingual environments, 2025. URL
abs 2502. 17956
ByteDance Seed, :, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xi- aochen Zuo, Chi Zhang. Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng Chen, Hongmin Chen, Riwei Chen, Liangqiang Chen, Zixin Chen, Jinsong Chen, Siyan Chen, Kaiyuan Chen, Zhi Chen, Jin Chen, Jiecao Chen, Jinxin Chi, Weinan Dai, Ning Dai, Jiahui Dai, Shihan Dou, Yantao Du, Zhengyin Du, Jianhui Duan, Chen Dun, Ting-Han Fan, Ji- azhan Feng, Junda Feng. Ziyuan Feng, Yuwei Fu, Wenqi Fu, Hanjie Fu, Hao Ge, Hongyi Guo, Mingji Han, Li Han, Wenhao Hao, Xintong Hao, Qianyu He, Jerry He, Feng Hc, Wen Heng, Zehua Hong, Qi Hou, Liang Hu, Shengding Hu, Nan Hu, Kai Hua, Qi Huang. Ziyue Huang, Hongzhi Huang, Zihao Huang, Ting Huang. Wenhao Huang, Wei Jia, Bin Jia, Xiaoying Jia, Yuhua Jiang, Haobin Jiang, Ziheng Jiang, Kaihua Jiang, Chengquan Jiang, Jianpeng Jiao, Xiao- ran Jin, Xing Jin, Xunhao Lai, Zheng Li, Xiang Li, Liyi Li, Hongkai Li, Zheng Li, Shengxian Wan, Ya Wang, Yunshui Li, Chenggang Li, Niuniu Li, Siyu Li, Xi Li, Xiao Li, Aoyan Li, Yuntao Li, Nianning Liang, Xinnian Liang, Haibin Lin, Weijian Lin, Ye Lin, Zhicheng Liu, Guanlin Liu, Guanlin Liu, Chenxiao Liu. Yan Liu, Gaohong Liu. Juncai Liu, Chundian Liu, Deyi Liu, Kaibo Liu, Siyao Liu, Qi Liu, Yongfei Liu, Kang Liu, Gan Liu, Boyi Liu, Rui Long, Weiqiang Lou, Chenwei Lou, Xiang Luo, Yao Luo, Caiping L.v, Heyang Lv, Bole Ma, Qianli Ma, Hongzhi Ma, Yiyuan Ma, Jin Ma, Wenchang Ma, Tingting Ma, Chen Mao, Qiyang Min, Zhe Nan, Guanghan Ning, Jinxiang Ou, Haojie Pan, Renming Pang. Yanghua Peng. Tao Peng, Lihua Qian, Lihua Qian, Mu Qiao, Meng Qu, Cheng Ren, Hongbin Ren, Yong Shan, Wei Shen, Ke Shen, Kai Shen, Guangming Sheng, Jinlong Shi, Wenlei Shi, Guang Shi, Shuai Shuai Cao, Yuxin Song. Zuquan Song, Jing Su, Yifan Sun. Tao Sun, Zewei Sun, Borui Wan, Zihan Wang, Xiaohui Wang. Xi Wang, Shuguang Wang, Jun Wang. Qinlong Wang, Chenyuan Wang, Shuai Wang, Zihan Wang. Chang- bao Wang, Jiaqiang Wang, Shihang Wang, Xuwu Wang, Zaiyuan Wang, Yuxuan Wang, Wenqi
15

---

Published as a conference paper at ICLR 2026
Wang, Taiqing Wang, Chengzhi Wei, Houmin Wei, Ziyun Wei, Shufa Wei, Zheng Wu, Yonghui Wu, Yangjun Wu, Bohong Wu, Shuang Wu, Jingqiao Wu, Ning Wu, Shuangzhi Wu, Jianmin Wu, Chenguang Xi, Fan Xia, Yuqiao Xian, Liang Xiang, Boren Xiang, Bowen Xiao, Zhen Xiao, Xia Xiao, Yongsheng Xiao, Chao Xin, Shulin Xin, Yuwen Xiong, Jingjing Xu, Ziwen Xu, Chenyin Xu, Jiayi Xu, Yifan Xu, Wei Xu, Yufei Xu, Shikun Xu, Shipeng Yan, Shen Yan, Qingping Yang, Xi Yang, Tianhao Yang. Yuchang Yang, Yuan Yang, Ximing Yang, Zeyu Yang, Guang Yang. Yifan Yang, Xuesong Yao, Bairen Yi, Fan Yin, Jianian Yin, Ziqiang Ying, Xiangyu Yu, Hongli Yu, Song Yu, Menghan Yu, Huan Yu, Siyu Yuan, Jun Yuan, Yutao Zeng, Tianyang Zhan, Zheng Zhang, Yun Zhang, Mofan Zhang, Wang Zhang, Ru Zhang, Zhi Zhang, Tianqi Zhang, Xinyi Zhang, Zhexi Zhang, Sijun Zhang, Wenqiang Zhang, Xiangxiang Zhang, Yongtao Zhang, Yuyu Zhang, Ge Zhang, He Zhang, Yue Zhang, Renjie Zheng. Ningxin Zheng, Zhuolin Zheng, Yaowei Zheng, Chen Zheng, Xiaoyun Zhi, Wanjun Zhong, Cheng Zhong, Zheng Zhong, Bao- quan Zhong, Xun Zhou, Na Zhou, Huan Zhou, Hang Zhu, Defa Zhu, Wenjia Zhu, and Lei Zuo. Seedl.5-thinking: Advancing superb reasoning models with reinforcement learning, 2025. URL https:://axiis http:/axxvvvrrraaaaa/ abs/ 20000///55444 2504 13914
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement fine- tuning via adaptive curriculum learning, 2025. URL httpps//aaxxiv.or////a/////////
05520
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.205.. 2025.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving Ilm coder and unit tester via reinforcement learning, 2025a. URL
03136
Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests+: High-uuu test case generation for competitive programming, 2025b. URL https::/arxiv.org//
2587.05817
Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. CodeContests4: High Quality Test Case Generation for Competitive Programming, 2025c. URL https://arxiv..t http:////xxxvvorgaaa.. org/abs/
2506. 05817
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms, 2025. URL /attl::p//////// / arxiv. org/ abs /2504. 14655
Jiacheng Xu, Bo Pang, Jin Qu, Hiroaki Hayashi, Caiming Xiong, and Yingbo Zhou Clover: A test case generation benchmark with coverage, long-context, and verification, 2025a. URL https:
//aaxiivorg/aaaaaaaee. 2502. 08806
Zhangchen Xu, Yang Liu, Yueqin Yin. Mingyuan Zhou, and Radha Poovendran. Kodcode: A di- verse, challenging, and verifiable synthetic dataset for coding, 2025b. URL https://arxiv.
org/abs/ 2503.5033 02951
Zheyuan Yang, Zexi Kuang, Xue Xia, and Yilun Zhao. Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase- Eval: A Systematic Evaluation of Fault Coverage and Exposure, 2025.
www.aawwwww....raaaooooooooooere abs / /aloos//556...0 12278
Yufan Ye, Ting Zhang, Wenbin Jiang, and Hua Huang. Process-supervised reinforcement learning for code generation, 2025,
01761
Jiahao Yu, Zelei Cheng, Xian Wu, and Xinyu Xing. Building coding agents via entropy-enhanced multi-turn preference optimization, 2025a. URL http://ss//xxx..or///aaaa/s////
12434
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangji Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025b. URL https: 7/arxiv http://wwww//b//....//.o 14476
16

---

Published as a conference paper at ICLR 2026
Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao- Zhang. Z1: Efficient test-time scaling with code, 2025c. URL
abs./504..000 00810
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.064 06471, 2025a.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACE- CODER: Acing Coder RL via Automated Test-Case Synthesis, 2025b. bttps://////
25g/aba/2502... 01718
Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. Codedpo: Aligning code models with self generated and verified source code, 2025. URL
/tttps://axxi...o abs///bs//aa aww.//21100 2010.4100.0000 05605
Kexun Zhang. Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. ALGO: syn- thesizing algorithmic programs with generated oracle verifiers. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:: .npresniiiii -ttp::/4444cc-atttttt---ooooo Lie.eeeccco4200eaaaaaaaeceeee--e
Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. ol ol-coder: ol replication for coding, 2024. URL http::///aaaiii..rrg///////////
001
Sining Zhoubian, Dan Zhang, and Jie Tang. Rest-rl: Achieving accurate code reasoning of llms with optimized self-training and decoding. 2025. URL http::ttt////t //axivv.. abs abs//508.
19576
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
RELATED WORK
Test Case Generation As private ground-truth test cases are scarce. researchers have turned to
LLMs for automatic test case generation. (Cook et al. 2025| Chen et al. 2025 Shi et al. 2025 Seed
al.
2025
Fatemi et al, 2025: [Ahmed al.) 2024 Yu et al. 2025b] Zhoubian al., 2025 Lei
et al.
2024
Early work had models directly produce complete test cases, i.c. input-output pairs. Gu
ct al.
2024
Chen al. 2023 Zeng et al., 2025b Xu et al., 2025b Payoungkhamdee et al. 2025
However, because such outputs often unreliable, Jiao et al. (002); Li al. (2023) let the model
generate both an imput and a candidate solution, then execute the solution to derive the output. Other methods introduced input generators to replace raw inputs (Jain et al. 2024 Cao al. [025 Xia
et al. 2025), or validators to enforce format and range constraints hefore execution (He et al. 2025
Fu et al. (2025a). Some methods enhance the model's ability to generate test cases through training. such as via SFT (Supervised Fine-nuning), RL(Reinforcement Learning)and other techniques. (Li
al. 2025: Bai et al. 2025 Zhang al. 2024; Wang et al. 2025a) Most recently, multi-round
generation and execution feedback has led test case generation to agent workflows (Wang et al.,
2025c Da al. 2025 Ye et al. 2025 Zhang et 2025
Yu al. 2025a Huang et al. 2024).
Test Case Evaluation Evaluation originally followed traditional software testing, emphasizing coverage and distinguishing between buggy and fixed code. (Xu et al. 2025a Yu et al. 2025c)
SWT-Bench (Mündler al., 2025) and TestGenEval (Jain et al., (025) transform from SWE- Bench (Jimenez et al. 2024), providing buggy implementations and their corresponding fixes. Oth- extend beyond single languages update recent codebases. For algorithmic problems, TestB- val collected 210 problems but still relied on coverage metrics. More recent works shifted toward end-to-end evaluation with large collections of correct and wrong submissions, measuring how often generated test cases exclude incorrect code. (Ma et al. 2025aa Yang et al. 2025 Wang et al. (025b) However, these approaches either rely on ad-hoc manual selection or expand code without lection or analysis. VCC-Bench is the first to study how many codes and test cases are sufficient, and provides a principled, efficient evaluation framework.
Code-Tee Test Matrix CodeT Chen et al. (2023) and B4 Chen et al. (2024) share the concept of using exccution results on test cases (the Code- Test Matrix) as behavioral signatures. CodeT assumes correct code behaviors clustering to select a consensus set. B4 calculates the probability of observing the matrix to select the most likely correct cluster. However, these methods aim for solution selection where the correctness of code and tests is unknown, utilizing the matrix primarily for signature matching or probabilistic modeling. In their context, the algebraic rank and basis are the primary interpretative tools. In contrast, TC-Bench operates ground truth with guaranteed correct tests and wrong codes. We view the matrix complete Error Space and apply linear algebra operations calculate the Rank and Basis, representing this error space most efficiently.
consistent while incorrect results are diverse, utilizing signatures for
18

---

Published as a conference paper at ICLR 2026
A.2 WRONGSELECT
Algorithm WrongSelect
Input: Raw matrix M. filter threshold т, restart count E, local search step K
2: Output: The optimal basis I'
3:
b Phase 1: Principled Pre-filtering
4: M' & Filter(M,,
5:
R' rank(M))
6:
I + 0
7:
Fmin +- 00
8:
Phase 2: Random- Restarr Local Search
9: for = 1 to E do
10:
Teunrent RandomBasis(M R!)
Generate random initial basis
11:
Feurrent f- F F[eerrem)
for 1 to do neitthe + Teurrent Recsnnigghtor F eurrent
13:
15:
for each rin M' Teurrent and each Your E Teurrent do
Trmmp + (Ieurrent (Fout)) {rin}
Traverse each neighbor
17:
if raak (Temp) = R' then
if (Jump) besstneighhor then
19:
Inisttncighhor f- Tromp Fhestanighhhor < F (Ieep)
20:
21:
end if end end for
22:
23:
24:
Feest neighbor < Feurrent then
D Move to the best neighbor
25:
Tgurrent neighbor neightee
26:
Feurrent f
F neighbor
else
28:
break
D Local optimum reached, exit inner loop
end if
30:
end for
31:
if F
Coureett
< Fmin then Pmin F Faurent
32:
33:
I" + Ieuneut
34:
end if
35:
end for
36:
37:
return I"
Phase 2 in Algorithm [ illustrates the pseudo code. The algorithm consists of two nested loops: the outer loop explores multiple random starting points to ensure global search breadth, while the inner loop refines each starting point local optimum, ensuring local search depth. Given the pre-filtered matrix M', each outer iteration begins by generating random initial basis Zcurrent. The inner loop then iteratively improves this basis. In each iteration, the algorithm systematically explores the neighborhood of the current basis: neighbor basis is obtained by swapping one mem- ber inside the basis with one outside, while maintaining the same rank. We compute the average Jaccard similarity F(Itemp) for each neighbor. If the best neighbor Thest neightor is superior to the current solution, Zeurrent is updated accordingly, and the process continues. Otherwise, when no better neighbor exists, the algorithm concludes that local optimum has been reached and the inner loop terminates. After each outer iteration, the current basis is compared with the best basis found far, and the best is updated if necessary. The outer loop repeats this procedure from multiple ran- dom initializations, and finally, the best basis across all runs is returned as the approximate global optimum.
19

---

Published as a conference paper at ICLR 2026
Table 2: Model Performance Comparison
LLM
Method
PR
AC
WA
RE
TLE
HR
Crux
26.86
81.44
16.14
0.89
1.53
18.56
PSEUDO
9.52
98.86
1.06
0.05
0.04
Qwen2.5-7B
AL.GO
12.37
89.61
9.25
0.67
10.39
LCB
42.38
52.46
43.69
2.29
1.56
47.54
HT
58.51
68.78
28.66
1.53
1.03
31.22
Crux
29.12
81.22
16.36
1.00
1.42
18.78
PSEUDO
14.97
93.92
5,30
0.32
0.46
6.08
Qwen2.5-14B
ALGO
19.82
86.91
11.82
0.71
0.56
13.09
LCB
49.71
49.65
46.63
2.17
1.55
50.35
HT
70.79
64.23
33.83
1.29
0.64
35.77
Crux
33.13
80.53
17.15
1.15
1.18
19.47
PSEUDO
16.49
88,65
10.31
0.55
0.50
11.35
Qwen2.5-CCdder-B
ALGO
14.27
92.80
6.62
0.40
0.17
7.20
LCB
41.94
57.83
39.02
1.92
1.23
42.17
HT
71.02
78.08
20.47
0.97
0.48
21.92
Crux
26.18
81.27
16.44
1.05
1.24
18.73
PSEUDO
10.32
95.46
4.16
0.28
0.10
4.54
Qwen2.5-Coom (onn2r--Coder--ee 14B
ALGO
27.92
95.45
4.20
0.22
4.55
LCB
51.87
46.05
49.95
2.34
1.66
53.95
HT
73.07
68.45
29.75
1.21
0.58
31.55
A.3 MAIN RESULTS
The Usage of Correct Code is Performance Watershed. Across nearly all models, methods that rely on correct code (LCB, HT) significantly outperform those that do not (CRUX, PSEUDO, ALGO) on HackRate. Although methods like PSEUDO and ALGO attempt to ensure correctness by having the LLM generate its own solution (or even a simpler brute-force one), the success of this process is constrained by the LLMs' an incorrect solution, it not only fails to generate complex test cases, but even simple ones are filtered due incorrect outputs. All this leads to low PassRate, which in turn severely impacts the Hackrate. Their performance is sometimes even worse than the simplest CRUX method.
sooblem--oovinggg capabilities. When the model generates
Performance Gains Primarily Come From WA. Through fine-grained analysis of exclusion reasons, we find that the primary performance gain from advanced methods with specific edge case generators, such LCB and HT, comes from significantly improved WA exclusion rate. For error types like RE and TLE, scores do not show a significant gap compared to simpler methods like CRUX. This suggests that the core advantage of current SOTA methods lies generating ingenious test cases that probe for algorithmic logic flaws. Crafting test cases that effectively trigger robustness failures may be a different, and perhaps a more difficult challenge.
Implementation Details Significantly Impact Final Performance. Although the five methods are conceptually progressive, specific implementation details, such as prompts and pipelines, can cause substantial performance variations. The concepts of ALGO and PSEUDO are similar. but ALGO simplifies task by asking model to generate a simpler brute-force solution. However, PSEUDO often outperforms ALGO because it generates 10 solutions and uses majority vote, whereas ALGO generates only one. Similarly, although HT adds an input validator over LCB, underperforms most models. We attribute this to implementation choices, such as allowing the edge case generator to return empty and providing simpler few-shot examples, which may lead the model "get lazy" and produce less complex test cases.
A.4 TEST TIME SCALING
To investigate the quantitative impact of increasing the number of test cases, conduct scaling experiment. For each problem, we used its rank R' as the base number of test cases (Ix) and proportionally scaled this number up to 5x, observing the trend in HackRate.
20

---

Published as a conference paper at ICLR 2026
Qwen2 5-32B
Quend 5-Cader-328
DcepSook-V0
Owen3-2358-4228
600
607
035
60%
408
23%
23%
Coder
4p
Cloudet
Cloudet Thining
80%
607
607
60%
50%
20k
20%
0%
Dack
Figure 5: Results of test case scaling for each model and method. The x-axis represents the number of test cases, scaled as multiples of the problem's 's rank from 1x to 5x.
The addition of test cases exhibits significant diminishing returns. As shown in Figure 5 the gain from scaling from 1x to 2x is the most significant across all combinations. As the number increases from 3x to 5x, the performance curves generally begin flatten, or even saturate. This suggests that blindly and massively increasing the number of test cases is an inefficient strategy. After covering the regular error patterns, additional test cases are likely just re-validating known failures rather than uncovering new, deeper defects.
The relative performance ranking among methods remains highly stable across all scales. Crucially, this experiment validates the effectiveness of setting the base number of test case as the problem's rank R'. While increasing the number of test cases does improve HackRate, the perfor- mance curves for each method almost never intersect. For instance, for Deepseek-V3 and GPT-4o, the five methods are well-separated. This stability demonstrates that TC-Bench, is already ef- ficient and reliable benchmark for differentiating the performance of various test case generation methods. It successfully captures the core discriminative power of different methods without incur- ring the high computational cost of scaling.
The conclusions from main experiments demonstrate good scale-invariance. Finally, this scaling experiment further reinforces the core findings from our main experiments. For example, the performance gap between methods that rely on correct code (LCB, HT) and those that do remains significant at all test case scales. Similarly, the impact of methodology continues to outweigh that of the base model.
A.5 COMMON FAILURE
To better understand the causes behind low scores, we conducted qualitative analysis of failed generations and identified three major systematic shortcomings.
Task Confusion and InstructionFFolloviig Failures When prompted to generate test cases, many LLMs instead output complete solutions the problem. This issue is particularly common when both test cases and solutions are requested together. We hypothesize that this stems from the infrequency of test-case generation tasks training data and weakened
prructton-followinn abil- ity after long-co cot training (Fu et al.] (025b). DeepSeek-RI exhibited this issue most severely. As shown in Figure within CRUX and PSEUDO, 74% and 60% of its outputs, respectively, are direct solution code rather than valid test cases. Among the remaining outputs, many are unusable due to formatting errors, such as embedding executable code inside JSON. Because the extractable test cases are too few, RI is excluded from the main experiments. This finding highlights that successful test-case generation requires not only strong reasoning ability, but also precise task comprehension and robust formatting control.
21

---

Published as a conference paper at ICLR 2026
C
json
To solve this problem, we need to determine the number of players observed by each observer placed different nodes in tree. Each player starts their respective starting node and moves along the shortest path to their destination node at speed of one edge per sccond. observer node j \) observes players who arrive exactly at W [_ seconds. The solution involves efficiently counting the players who pass through each node (( at exactly W_j !) seconds. fiа Solution Code
"test case 0": "7nll 1 1 6:n5'n3 6'nl 7 4'ml 6'nl 3 ,
Standard CRUX Response
"test case_ 9"; "300000'nn + "-joinstt(mmax(1, i-10)) for range(1, 300000)) "I300000''n" "'n", "'nnjoin(f"' [i] [i+2] ffor in range(2, 300002)) 300002))
"python
import sys
sys.strecunssonlimiit$000000000
arr =[]] 9n val arr "join(map(sir, arr)
X
def main(:
data sys sys.stdin wew.drrrrddppiiiiiti
op_ lines -0
not data:
forj inrange(n val);
op lines.append(f" (n val) (i
test cases("test case 5"]
=f" (n val) in (arr str) In""""" join(op lines)
Not TestCase but Solution Code
b Generate Testcase-Code
Direct Solution Code
Figure 6: The figure shows the errors that occur in the direct generation Testcases by models like RI, using the CRUX and PRESUDO algorithms. Subfigure (a) shows the normal output, subfigure (b) demonstrates the insertion of generated code or direct responses in the form of Testcase code within a string, and subfigure (c) shows the model not following the Testcase generation instructions and instead directly providing the solution.
Lack of Resource-Aware Generation Many problems require test cases at large boundary condi- tions. As shown in Figure a, we observe that numerous methods attempt to construct overly large inputs (e-g., huge graph structures), leading to out-of-memory crashes or timeouts during execution. This reveals a deeper limitation: while LLMs are proficient in generating algorithmic logic, they lack awareness of physical execution constraints such as memory and runtime. A robust test-case generation pipeline must therefore incorporate mechanisms like input partitioning streaming to adapt to limited system resources.
m. values =[1, 10, 100, 50000)
max courses values =[1, 10, 100, 1000) max credits values -f1, 2,
1
2
max offort valucs -[1, 50, 100, 200]
for in m. values:
for max courses in max courses values: for mas credits mas credits values: for max effort in max effort valux:: construct inputs()
3
4
4
(a) Example of Memory Explosion
(b) Valid Testcase
(c) Invalid Generted by LLM
Figure 7: Subfigure shows that the memory explosion is caused by the model constructing excessively large number functions during case generation. Subfigure b, e presents a failed case where the model fails construct connected graph as required. The task specifies that all node instances must be able to reach node n. but the constructed graph does not satisfy this connectivity condition.
22

---

Published as a conference paper at ICLR 2026
Failure to Construct Required Complex Data Structures Some problems in our benchmark admit only test cases with highly constrained structures. As shown in Figure (b. c, in one problem, every valid imput must be specific type of connected graph. However, none of the tested methods successfully produced even a single valid input. As a result, these problems ended up with zero usable test cases. This underscores that generating high-difficulty test cases be challeng- ing as solving an algorithmic problem, requiring deep understanding of both data structures and algorithms.
A.6 RESULTS OF SUPPLEMENTARY EXPERIMENTS
6000
75
0.8
4000
Number of bins
Cumulative probability
0.6
Solutions:
2000
Filter Top 20%
45
0,4
0,2
9
17
25
Figure 8: The left subplot shows the interval count statistics and cumulative probability curve of the time-norme normalizdd correct answers. In the right subplot, the Hackrate continuously increases as the number of correct answers increases.
30
Problems Modified
950
Avg Improvement
29
850
Convergence Point
26
120
0,08
80
0.06
60
0.04
0.02
20
5
11
19
25
40
61
176
364
1007
Figure 9: The heuristic algorithm designed converges the eighth turn.
23

---

Published as a conference paper at ICLR 2026
300
Before op litizaation optimization
250
Cumulative Probebilityy
04
Froquency
06
100
04
100
aptiniiatiio
00
ophimiaation
10 20 30
90
60
70
Count/Rank
Count
Figure 10: Distribution of the number of WCs per problem before and after the WrongSelect. The histogram (left) compares the initial count of WCs against the rank (i.e., the final count of WCs). The cumulative distribution function (CDF) on the right further illustrates this shift. The results demonstrate dramatic reduction in the number of required codes, highlighting the compactness and efficiency of our resulting benchmark.
A.7 THE USE OF LARGE LANGUAGE MODELS (LLMs)
This article utilizes large language models (LLM) solely for writing refinement and graphic en- hancement, with other applications purposes involved.
24

---

Published as a conference paper at ICLR 2026
B APPENDIX B
B.1
BENCHMARK CONSTRUCTION
This section will detail the process involved in constructing the dataset, including the repairing of Wrong Codes, operations related to the clarity of problem statements, and statistical data.
B.I.1 WRONG CODE
Code Cleaning After processing the wrong codes in Section 2.3 for all retained wrong codes, we used public test for testing. For all execution results such CE, TLE, MLE, EXE, as well as codes that resulted in WA but with empty outputs, manual fixes and reviews were performed. As shown in Figure TT] (a) illustrates a piece of unusable file operation code, which the script does not include the definitions of Fin and Fout. For this type of code, we removed the corresponding file operations. The error in (b) arises because the unistd.h library already defines function named link _arra, which confictts with the array link_array defined in the code. (c) presents an example of incomplete code that requires manual supplementation. and the corrected code, after making modifications we tested the code using private test cases, with the requirement that the test results remain consistent with the crawled results.
ensure consistency between original
il void FILEIOO
includessuiitd..b>
#ifdef intLSY
Fin("in.in");
int ch[maxn<<<][[6] link array[maxn<ee]]
Hendif
extern intlink array(intokdddd const char *oldpath, intnewfd, const char "newpath, intflags);
il void FILEIO string pname } #ifndef intLSY
fin((paame=".in")...o c:tr))
(b YStandard library name conflict
[ttp:p/ammet..uttt____""" #cndif
int main()
il void FILEIO OICONTEST( string pname {
n qr-rread0.mmpppp
(iin((nmmme"..c iin"nnn) .in").csstt))::
for =1; <=m; i++)
#ifndef intL.SY
{
Fout(ppaamct...org .out"") str();;
int ent;
#endif
ent-greadO::
Undeclared function: Fin, Fout
Manually complete the code
Unavailable file operation
Incomplete code
Figure 11: (a), (b), and (c) respectively present three examples that we encountered when repairing Wrong Code.
25

---

Published as a conference paper at ICLR 2026
B.1.2 PROBLEM DESCRIPTION
Regarding the problem statement processing in Section2.3] this subsection provides detailed exam- ples and explanations for problems that heavily rely images and Special Judge problems.
For problem statements that rely on image-based basee understanding, such as Stars (see image in ref- erence F2g the problem includes an image that is necessary for understanding in order to generate test cases or solve the problem. We manually reviewed this type of problem statement, fil- tered out the problems where images affected the understanding of the question, and deleted them. In this step, we deleted total of problems.
Star
There are some stars in the sky, each with different position, and each star has coordinate. If a star has k stars in its lower-left (including directly left and directly below), we say that this star is level k.
For example, in the image below, star is level (because stars 1, 2, and are its lower-left), and stars and 4 are of level In the example image, there 1 star of level 0, stars of level 1, of level and star of level 3.
Given the positions of the stars, output the count of stars at each level.
Figure 12: This is an example of problem that can only be solved with image understanding.
The problems with Special Judges involve multiple outputs, answer ranges, and interactive prob- lems. In total, we removed 42 Special Judge problems. Ball Moving Game is an example with multiple solutions, as shown clearly in Figure 13 which illustrates the existence of multiple an- swers. Problems like Idea also explain that as long as the answer satisfies a certain range, it acceptable.
Ball Moving Game:
Little C' is stuck, but he believes you can solve it. Please provide operational plan to achieve Little C's goal. There may be multiple valid solutions, and you only need to provide one. The problem guarantees that there is always least valid solution.
Idea:
For each output file, if more than 95% of the lines have answer with error of no more than 25% compared to the correct answer, you will receive score. The error considered be within 25% if, for correct answer X, your answer lies within the closed interval [0.8X, 1.25X].
Figure 13: The image presents two examples of problems with multiple solutions. In the Ball Moving Game, the same input can lead to various outputs, while in the Idea problem, the output simply needs to fall within a given range.
26

---

Published as a conference paper at ICLR 2026
We also selected all interactive problems, such the shown in the reference, The Adventure of Lord | where the problem statement clearly states "This an interactive problem." " This type of problem requires complex interactions and support, making it unfriendly for test case generation.
The Adventure of Lord I:
This an interactive problem.
During the evaluation, the interactive library will call the 'explore' function exactly once.
It is guaranteed that the graph used in this problem fully determined before the interaction begins and will not be dynamically constructed based on the interactions with your program. Therefore, the interactive operations the problem are deterministic, and you do not need to worry about the specific implementation of these operations in the interactive library.
The data guarantees that the time required for the interactive library to run will not exceed second under the given call limits. The memory used by the interactive library is fixed and does not exceed 128MB.
Figure 14: The image illustrates an example an interactive problem, which necessitates specific and intricate interaction checks during evaluation. To streamline the evaluation process, have removed this part of the problem.
27

---

Published as a conference paper at ICLR 2026
Example of problem statement cleaning As shown in Figure the following is example of problem statement cleaning. For demonstration purposes, we have created a sample problem to illustrate the main cleaning tasks. In this process. irrelevant background information is removed, image links and other URLs are discarded, and the phrasing is made smoother. 'hee data range is kept the most general case. These tasks typically do not follow universal pattern and require manual inspection. After cleaning the problem statement, we used GPT-4o for translation. In this step, organized each data entry and deleted 15 problems that were difficult to handle. Then each translation was semantically proofread and certain inappropriate expressions were adjusted for accuracy. The final processed problem statement can be found next page
Tour de Bytcotia
Markdown Format
# Tour de Byteotia
Background:
In the depths of distant universe, there exists a kingdom surrounded by stars and brillianee "The Kingdom of Stars." This kingdom is home countlless whe www.eee-pysseerriiieeeee mysterious stellar trajectories and intertwined fates. One One-day, schelars discovered an aneient prephecy foretelling that a breken constellation would bring about abeut the of the world Only by guthering five lest stellar gems can this disaster be prevented.
schelars
you, you, chesen here, bear the
suspensibillityy----annggiii fate.
2 Remove Irrelevant Message
Problem Description:
Given an undirected graph with vertices and m edges, determine the minimum number of edges to remove so that vertices numbered less than or equal not appear on any simple cycle.
Input Format:
The first line contains three integers n, m, and k, representing n nodes, m edges, and as described in the problem statement.
Output Format:
The first line contains one integer representing the minimum number of edges to removed;
The following lines each output two positive integers a.b representing the removal of the edge between a and b. Output the vertex with the smaller number first, then the vertex with the larger number.
Sample:
3 ADD Sample
Input:
4 Remove Fig, URL HTML
Output:
Constraints and Hints:
5 General Description
ofr-40%% 4ag449996ssflleeaaaaxcc010000000
For all data, 15n51, |5cc://0000000055522222000000000
Figure 15: To facilitate demonstration, we constructed an example of problem statemmett cleaning, which the common cleaning procedures are integrated.
28

---

Published as a conference paper at ICLR 2026
Final problem statement:
Tour de Bytectia
## Problem Description
Given an undirected graph with $nş vertices and edges,
+ determine the minimum number of edges that need to be removed
+ so that all vertices with indices less than or equal to sks are not part of any simple cycle.
## Input Format
The first line contains three integers $n$, Sms, and skě, representing ing the number of vertices, the number of edges, and the significance $K$ as described the problem statement.
The next Smş lines each contain two integers $u$ and Svš, indicating bidirectional edge between $u$ and $vş. There is at most one edge between any pair of vertices.
# Output Format
The first line contains an integer 5kě, representing the minimum
4 number of edges to be removed.
The next Skş lines each contain two positive integers $a$ and $bş, indicating the removal of edge between $a$ and $bs. Output the vertex with the smaller index tirst, followed by the 4 vertex with the larger index.
# Examples
#*# Input:
11 13 5
1 2
3
5
2 8
4 11
9
2 3
9
10
### Output:
9
3 5
Data Range and Hints
For all data, $1 \le n Ale 1,000 000s, $0 Ale m \ie 2, 000, 000s, $1 4 \ee k Ale ns, $1 Ale u v Ale ns.
29

---

Published as a conference paper at ICLR 2026
C CASE STUDY
To demonstrate the practical effectiveness of our method, we conduct a case study on "Sliding Window", classic problem requiring the Monotonic Queue algorithm. The problem involves an integer array of length N(S 10%) and a window of size K 10%). The window slides from the leftmost the rightmost of the array, moving one position at a time. The goal is determine the maximum and minimum values within the window at each step. The output requires two lines: the sequence minimums followed by the sequence of maximums.
The optimal solution emplays a Monotonic Queue to achieve time complexity of O(N). Specif- ically, to calculate the maximums, we maintain monotonically decreasing queue that stores array indices. As we iterate through each element in the array, we first pop the elements at the back of the queue their corresponding values are less than or equal to the current element. This because these smaller and older elements can never serve as the maximum for future windows, Next, the current index is pushed to the back. Then, the front of the queue is popped if its index is out of the current window scope. Finally, the value corresponding to the index at the front of the queue represents the maximum of the current window. The minimums are calculated analogously by main- taining a monotonically increasing queue. The standard solution is shown below.
Standard Solution
#includectitts/stdcc++ using namespace std;
int a[1000005] , k
deque int main() {
ein >> n >> k
for (int cin >> a a[i] ;
1 ; i <= <= ++)
for (int i - i <- inn; i ++) { while empty () (m)pet())))) & a[i] alq.back ()))
qacko_bbcc() () ;
back (i)
(f. front ()
k + 1)
supop_front. ;
(f >= k) cout << alq.front front ()) << " ";;
cout << endl q.clear()
for (int () () & a[i] > a[q.back ()))
; i <= n ++) {
While(!q, empty
q.pop_back ;
q. p.push_bar back (i) ;
if. front() () i - k 1)
f.pop_front(( () ;
if (i >= k) cout << alq. front ()) << "
30

---

Published as a conference paper at ICLR 2026
Initially, this problem involves 96 Wrong Codes (WCs). After applying WrongSelect, only 8 basic WCs are retained. Their failure signatures are presented below:
0
0
0
0
0
0
1
07
1
0
0
0
0
0
0
0
0
0
1
I
0
0
1
0
0
1
0
0
1
0
0
0
0
1
0
C.I BASIC WRONG CODES
We meticulously analyze the retained Basic WCs and characterize their underlying error patterns.
Basic WC1 fails due to insufficient memory allocation for the queue array, where the size is set to 510,000 instead of the required 1,000,010.
Basic WC1 (0000000110)
Ficllueecsstdio>
#inel ude<cstring> using namespace std; struct node int x,
1istl (510000), (10100000001111111110110000
Should expand 510000 to 1010000
int a [510000), ,, int main &n, &m)
scanf i=1;i<en;i+t) ("ad",
for (int int head-1,tail-1; [istl(1).x=a[1];
1isti(1).p=1;
for (int 1=2;i<en;i++) {
while (headk=tall@i-list) [head] p>>=m) head++; while (haad<=aal11@1llstl [tail] (>)a
[1st list [tail].p=i. ((iii=)) ",iist [head]. :x);
printf("\n");
head=i,tail-
list2[1).
list2(1).p=1; for (int i=2;is=m;it*) {
while ( (head] pp>=m) head++; . while [beaaccttaaileellstt(ttaaalllll (itatl)aa><(()tallla>- list2 [ist2[++aaall [tail] pFii; (1>=#m))rintff " 1,st2 [head] x);
31

---

Published as a conference paper at ICLR 2026
Basic WC2 exhibits incorrect order of operations where the answer is retrieved before updating the tail with the current element, causing the current element to be ignored in every window.
Basic WC2 (1111001111)
sinclude-biis//ttdeee
#define 11 long long #define inf 2139062143
#define MAXN 1001000 using namespace std; inline int read() {
int k=0,f=11 char ch=getchar ();
(Lisiigt (ch)) {
(ch==( f=-1;
ch=getchar ();
while (isdigit (ch) {
x=x+10+ch-100 0', ch=getchar ();
return x*f;
int n,m, q [MAXN] [2], [d(, [MAXN], ans [MAXN] [2]
int main
n-read () m-reed ();
[1]=];
for (int i=1;
{
[i]=
while (hd[0) <=t] \ \&q[[d [0]] [0]<=i-m) hd[0]++>
// Swap the order of yellow and red.
ano[[111000=aaa[h111011111111111
while [ai.eeehhel....e1000eeaa01101010 [0]]]]]]{{ t1 [1] --;
g[++El[0]] [0]=i;
while [ <-imm-))))) [d[1]
// Swap the order of yellow and red.
[as
while oit.ecchhebl.eeelllleeggggglllld ti[1)-- --;
[[++c][1]] [1]=i;
for (int i=m;i<n;1+H) printf(" a, [1]n[[i][] [1]
printf ("&d\n", ans [n] (0]);
anss"""
for (int L=m:isn;ite printf ans [i] [1]);
printf ("Sd", ans [n] [1]);
32

---

Published as a conference paper at ICLR 2026
Distinct from the queue error in Basic WC1, Basic WC3 allocates insufficient memory for the input array using a size of 10 rather than the required 10
Basic WC3 (0000001000)
finclude<birs/stdeet
using namespace std;
int n, k; int tail, front; struct node { int pos, val;
g[1000000101;
int a [1000110;; // 100010 -> 1000010
int main() {
scanf (canf(""dee ("$d8á", &n, ;
for (int i-1;is-n;i+H) {
scanf (alil;
front=1;
tail=1;
v[1],va==a[]);
q[l].pos=1;
for (int i-2;i<=n;i+*) {
while (tail>=front & vttail).val>=a[tt [i]) tail--;
-vatail).al==((1;;
[atail]ppos=i
while wow.aiil.oopgg_ffreeee postt>>k) fronttt;
if (i>=k) cout<<q(front]] val<<" ";
cout<<endl;;
front=1;
tail=1;
q[1] [[]]
q[l].pos-1;
for (int i-2;icen;i+*)
while (tail>=front & valk=a)) tail--;
[[+ttail].wal=tt .va==a[i;;
while (q[tail] -pos-q(front] pos+1>k) fronttt;
is (i>-k) cout (oroot](front] val<<" ";
33

---

Published as a conference paper at ICLR 2026
Basic WC4 contains subtle logic error in queue maintenance by performing unnecessary and erroneous comparison with the head element while updating the tail. This additional operation pre- vents current elements from entering the queue, causing the queue to potentially become empty during the sliding process. In this state, accessing q top () triggers undefined behavior, retriev- ing residual garbage data from the underlying memory address.
Basic WC4 (0001001011)
#include <bits/stdct+.h
using namespace std; struct node int x, bh; friend bool operator < (node x, node (return x.x>y.x;)
a [1000001]; struct nodel int b,; friend bool operator (nodel x, nodel y) (return x.x<y.x;) a2 [1000001]; meloritty_queeetndder ql;
priority__qeueehodell q2;
int n, inline int read() (...}
inline void write (int x) (...)
int main
int i;
n=read ();
k=read(); ();
int tail=2;
int head=kt1;
for (i=1;i<-n/i+t) a[i] xx=a2 [[]]] .bh=a2 [i] .b-=
i; for (1= qi qu.push (a q2 push (a2 [i]);
write (ri ().)) printf(" ");
for (shead<-n;tail+: htttnrttttttttttt) head++)
while (q1.top() .bhstail & !ql. emlpy ()) pl.pop
// remove
() x [head] .x)
ql gl.push (a [head))
write (q1 top :x);
printf(" ");
printf("An");
tail=2;
head=kt1;
write (q2. (x.))
printf(" "); for (zhead<=n;tail+: h
head++)
while (qp. .bhstail && eq2.empty ()) q2. ()
// remove
lop () x<=a [head] x)
qu.push (a2 [head])
write (q2 top :x);
printf(" ");
34

---

Published as a conference paper at ICLR 2026
Basic WC5 represents scope error where the head and tail pointers of the queue incorrectly re-initialized inside the loop.
Basic WC5 (0001000010)
#include <stdio.h>
#include <stdlib.h>
#define 2 1000001 int main int ri=1;
int m, (at (int) +2);
int (aa== (in)) (1n
int (ftne((innt))mllloccsssofoo()))) int (Id=)))mmmlllllccsieee (int) *Z); scanf(" (" for (i-1;i<-m;itt) {
scanf ("Sd", (da
for (i=1;i<=m;itt) {
while [i] <min [ri])
ri--;
ri++; [in]di]=da
id[ri]=i;
[Eddlle)
lett;
is>in)) { printf mimi)]
printf ("\n");
for (i=1/i<<m/it+) {
1e=0; // Move outside the loop
ri=1;
ri--;
ritt;
max[ri]=da[i]]
id[ri]=i;
[Edi[[le)tn<ii) {
lett;
(f>=)= {
printf ( ("$d ", max [le]);
return 0;
35

---

Published as a conference paper at ICLR 2026
Basic WC6 attempts Sparse Table (ST) optimization but fails due to implementation error where the allocated table size is too small for the problem constraints.
Basic WC6 (0000010101)
#incuudetbttt/sttde++... using namespace std;
const int
1e6;
int (N] [18] [N], p[N]; // 18 -> 20
int maxx (N), [N]n[[N];
int n, k, le, ri;
void init
for (int j 1; j< 18; j++)
for (int st st[i]]] [il = max (st[i]li - 1],st + (1<$) - 1]][} 11);;
i<=n&& (1 << j) 1<=n; +ti)
void initl()
for (int j 1; j< 18; j++)
for (int = i<=n&& ( << 3) I<=; ++1)
st [il = min sl,t + (1<<j - 1)]li
4 11);;
int rmq 1, int z) {
int d = 1 1; return max (st [1]
(1<<p[d)) (p(d)))
int rmql (int i,int r) {
int d = 1 1; return min (st [1] [pld]], (1<<p[d)) (p(d))))
int main() {
scanf ( ("&daa", &n, sk);
for (int i = 1; i<=n; i++)
scanf ("8d", st]i][0] = alil;
initl();
for (int = = 1; i<=n; itt)
p[i] = p[i-1]];
if -- [<<p[i] + 1)
++pli];
for (int 1; i<=n kt 1; i+t)
minn [i] rmql (i, + k -1);
for (int i 1; i<=n k 1; itt) cout<sminn "; printf("An")
36

---

Published as a conference paper at ICLR 2026
Basic WC7 attempts to fix the boundary error seen in Basic WC6 by incrementing the Sparse Table size by 1, yet it remains insufficient for the maximum constraint.
Basic WC7 (0000000100)
#include<<iirssstccc++... using namespace std;
const int
1e6;
int (N] [19] [N], p[N]; // 19 -> 20
int maxx [N), miin[[N];
int n, k, le, ri;
void init for (int j 1; j< 19; j++)
for (int st st[il]] [i] = max (st[i] - 1],st + (1<$) - 1]][} 11);;
i<=n&& (1 << j) 1<=n; +ti)
void initl()
for (int j 1; j< 19; j++)
for (int = i<=n&& (1 << 3) I<=; ++1)
st [i] = min sl, + (1<<j - 1)](i
4 11);
int rmq 1, int z) {
int d = 1 1; return max (st [1] [ (1<<p[d)) (p(d)))
int rmql (int i,i r) {
int d = 1 + 1; return min (st [1] [p[d]],
(1<<p[d)) (p(d))))
int main() {
scanf ( ("&dto", &n, sk);
for (int i = 1; i<=n; i++)
scanf ("ad", st]i][0] = alil;
initl();
for (int = = 1; i<=n; itt)
p[i] = p[i-1]];
if -- [<<p[i] + 1)
++pli];
for (int 1; i<=n kt 1; i+t)
minn [i] = rmql (i, + k -1);
for (int i 1; i<en k 1; itt) cout<sminn
";
printf("An");
init();
for (int = 1; i<=n; itt)
st[ alill
for (int = 1; i<=n - kk 1; i++)
maxx[i] rmq
1);
for (int i = 1; i<=n - k 1; i++)
(out<<maxxx
return 0;
37

---

Published as a conference paper at ICLR 2026
Basic WC8 incorrectly updates the head pointer instead of the tail pointer during the first element's insertion. Additionally, it omits the insertion of the first element when initializing the second queue.
Basic WC8 (1100000000)
Ancluddebbits/stddctt
using namespace std;
const int N=1e6+3;
int n,k;
int a [N];
int h=0,t=-1; t=-1;
int q[N];
int main() { <in>>n>>k; for (int i=1;1<=;;it> {
cin>>a(il;
q[+th]=1; // q[+tt]=1
for (int i=2pi<en; i++) {
while (b=ttaalt) --t;
g[+tt]=i; (f couk<a/g)cut ";
cout<<endl;
h=0,t=-1;
// add q[+-t]=1
for (int 1-2;1<-n;i+* {
while (i-t1 while (uhh)))
q(+t]==i; (E>=)) cout<<a ";
return
The eight Basic WCs effectively map to the specific requirements data structures and algorithms inherent to this problem. These error patterss include resource allocation for different variables, as well as the position, order, and conditions for queue initialization and maintenance.
On hand, Basic WCs cover boundary constraints across different variables and granularities. For instance, Basic WC3 represents a resource error in the input array while Basic WC1, Basic WC6, and Basic WC7 target the queue. Specifically,, the internal hierarchy among Basic WC1, Basic WC6, and Basic WC7 introduces tiered validation mechanism. A less advanced test case gener- ator, which could produce medium-scale inputs but struggles with maximum constraints, can still identify Basic WC1 and receive partial credit. This effectively avoids the "all-or-nothing" scoring trap. ensuring that the benchmark gives non-zero scores to generators that possess intermediate ca- pabilities. Conversely, only top-tier generators that hit the absolute maximum boundary can exclude all these Basic WCs achieve perfect score.
On the other hand, the basis preserves logic specificities. Basic WC5 incorrectly re- re-initializes queue pointers inside the loop, causing the state of the sliding window to be lost at every iteration. To expose this, the test case generator must produce inputs where the window's extremum is deter- mined by historical element rather than the current one, verifying the persistence the queue. Basic WC8 exhibits dual failures, specifically on the first element's insertion and the second queue's 'ss initialization. This forces the generator to produce edge cases where the first element is the strict maximum minimum for the initial windows. By retaining these basic error patterns, TC-Bench
38

---

Published as a conference paper at ICLR 2026
ensures that the evaluation reflects model's ability to cover the entire spectrum of the solution space.
C.2 EXCLUDED WRONG CODES
Following the analysis of the Basic WCs, we proceed examine the Excluded WCs verify whether their eтror patterns are effectively encapsulated by the basis. We select Excluded WCI as a representative example, whose failure signature is reconstructed by the combination of Basic WC8 and Basic WC4. Excluded WCI exhibits a classic Off-by-one boundary error. During the sliding process, the code fails timely pop the element exiting the window, causing the queue to retain invalid, expired data throughout both the initialization and maintenance phases. Crucially, this composite behavior is spanned by the basis. Basic WC8 precisely mirrors the initialization failure, it retains stale data from the first queue due missing head pointer update. Meanwhile, Basic WC4 captures the maintenance failure, where additional comparison causes the queue to become empty. In this state, accessing the queue retrieves residual garbage data from memory. Together, these underlying mechanisms fully cover the error pattern of Excluded WC1.
Basic WC8 1
1 0
0
0
0
0
Basic WC4 0
0
1
0
1
Excluded WC1 1 1
1
0
1
Excluded WC1 (1101001011)
#includeebbltssstdde+...
using namespace std; const int N=1000005; int a,b; int g [ [[N),num num [N), f1 (N] F2 (N);
int main scanf ( ("$dèd" &b)
for (int i=1;i<=a;i+4) i++) {
scanf
int head=1,tail=1;
for (int (ii;i<a;;i+))
while (num [head] <i-b &sheadk=tail) // i-b+1
head++; while
tail--;
num num++ttail]=i;
q[tail]=g[i]; fi[i]=q[head]
head=1,tail=0:
for (int i=1;i<=aritt) {
while dibubbbaaaaldd---tteeeedettaall head++; (ttp::aaaaaaaaaaaaaaaaaaaaaaaaae tail--; num[+ttail]=i;
q[tail]=g[i]; f2li)=q[head];
for (int i=byi<=a;itt) cout<<fl(i)<e" cout<<endl; for (int --b;i<=a;i+h, [ut<<f2[i]<<< cout<<endl; return 0;
39

---

Published as a conference paper at ICLR 2026
Similarly, analyze Excluded WC2, whose failure signature corresponds to the combination of Basic WC8 and Basic WC5. Excluded WC2 contains a boundary error in the monotonic queue maintenance. By using the fixed condition tą= instead of the dynamic hj=t, the tail pointer incorrectly decrement past the head pointer, violating the valid window scope and accessing invalid historical data. This error pattern also effectively spanned by the basis. Basic WC8 captures the initialization failure, where the pointers fail to correctly establish the queue's ' start (updating head instead of tail), reflecting the error's mishandling of the absolute beginning. Basic WC5 captures the scope maintenance failure, where the queue's 's dynamic state is ignored (resetting pointers inside the loop), mirroring how Excluded WC2 ignores the dynamic head boundary and corrupts the persistent state.
Basic WC5 0
0
Basic WC8 1 1 0
0
Excluded WC2
Excluded WC2 (1101000010)
iinclude<iostreem>>>
#include<cstdio> #ineludeccstrinng>
using namespace std;
const int N-1e6+5; int n, P int main() {
scanf ("Sddd",,n,n)) for (int i=1;is-n;itt) int h=1,t=0; for (int i=1;is=m;itt) {
;alil);
Mhile(q[t)>aii)44 [>=1] t--;// t>=1 h<=t
q[++t)=a[)),(tt)i;;
shile((p!hhlii---1ihhhh))) h+t;
if>i)=k) printf ">h(h);;
menset (q, 0, sizeof (q)); memset (p, sizeof (p));
cout<<endl;
h=1,t=0;
for (int i-1;i<=n:itt)
t>-1] t--;// t>=1 -> h<=t
t>=1)
([++t)alli,,ptt-ii)
Mhile(pphjKii--iideeee h+t;
iE(i>=k) printf [hh))
return 0;
40

---

Published as a conference paper at ICLR 2026
C.3 REPEATED WRONG CODES
Finally, verified whether identical failure signatures indeed correspond to semantically equivalent emror patterns. We selected a cluster of Repeated WCs sharing the binary signature 0001 10001011111111111111111111111111 01011110
We specifically examine the representative example, Repeated WC1, shown below. This code hibits a logic flaw during the queue maintenance phase: when inserting the current integer, the code compares it against the queue head rather than the queue tail. In monotonic queue, the tail ele- ments must be compared and popped to maintain monotonicity. Comparing against the head (which typically holds the window's extremum) creates irrelevant condition. Consequently, elements that should have been removed remain in the queue, corrppting the window's state.
We thoroughly inspected other WCs within this same signature cluster. While they exhibit syntactic variations in implementation, we confirm that they all share the exact same root cause: the failure to correctly remove invalid elements due to flawed comparison logic. This confirms that our signature- based grouping effectively captures semantically similar faults. Due to space constraints, only key segments of these Repeated WCs are presented below.
Repeated WC1 (0001001110)
#include<bits sstdtttttstttt.tt #define maxn 1000010 using namespace std; int pos [maxn] que [maxn]; int k, int a [maxn];
int fminn [maxn], fmaxx [maxn]
void dpmin int h = 1, for (int
0;
1;
n; ++)
while (pos [h] < -i- 1 & h <= t)
++ hi
while (que[t] > a[i] && t) t;
que(++ t] a[il, pos[t] i;
fminn[i] que[
void dpmax
int h = 1, t =0; for (int
1; <= n; itt)
while (pos [h] < - k & h <= t) ++ h;
while que a[i]] 88 == t) t;// q(h] -> qitl
quel++ t] a[i] pos[t] =;;
fmaxx[i] que [h];
int m)in() {
scanf (eanf("Ada) ("Adad" &n, 8k);
for (int i = 1; <= n; ++) (a dpmin();
dpmax() ();
for (int kj i <= ni ++) printf(" ", fminn [i]);
printf (" )
for (int k; <= n; ++) printf(" ", fmaxx [i]); printf("An")
return 0;
41

---

Published as a conference paper at ICLR 2026
Repeated WC 2
// Boundary offset error head = 1; tail 0;
for (int i 1; i <= ny ++i)
while (head rail as i-k-1 ) head++;// Should be: i-k+1 while (head <= tail & alq[tail]] <= a[il) tail--;
g[+ttail] i;
if (i b= k) cout << a[q[head]] << ;;
Repeated 3C3
// Incorrectly checks queue tail (x) for expiration for (int 1=k;i<en;1+t) {
while (maxn (r]] r-;; rtt;
maxn [x]=;; while (1<=r maxn [ < i-k+1) 1++; // Should be: maxn [1] cout<<a [maxn [1]]<<" ";
Repeated WC4
11 Incorrectly accesses value array 'a' instead of index 4 array 'b'
for (int i=1;i<=nzitt) {
if (hh<=tt&& a [b [hh]] <=i-k) { // Should be: [hh] hh++;
while (tbc= tt--;
Repeated 5
// Incorrectly compares with queue front while updating tail for (int i 1; <= n; i++)
while (Iq.empty() & al groront () < a(l))) // Should be: q.back() q. q.pop_back ();
q. q.push_baat back (i);
if (i q. front () >= m) q.rop_front ();
if (i >= m) cout << a front ()) << ";
C.4 DIAGNOSING REALISTIC SCENARIOS
To further verify the diagnostic value of our benchmark in realistic setting, we conducted an evaluation using the SOTA combination: Claude-4-Thinking with the LCB. We generated 40 test cases. The results showed that the generated test suite successfully excluded 6 out of the Basic WCs but failed to expose Basic WC6 and Basic WC7, They are resource allocation errors requiring queue capacities of approximately 3 x 10y and x 10 respectively. Triggering these specific faults requires forcing the monotonic queue to fill up to these limits. Mathematically. this demands worst- case scenario where both the window size K and the array length N approach 108, and crucially, the input array must follow specific monotonic pattern (e.g., strictly increasing decreasing)
42

---

Published as a conference paper at ICLR 2026
enough elements are pushed into the queue. We manually inspected all 40 generated test cases and confirmed that while the model generated large random arrays, failed to construct this specific, structurally extreme boundary case. This demonstrates that TC-Bench effectively points out a specific weakness in current SOTA generation methods. Ultimately, this confirms that TC-Bench significantly streamlines diagnostic analysis: by narrowing the analytical scope from the entire raw dataset to a compact set of Basic WCs, it enables researchers to pinpoint model weaknesses through just few representative examples rather than sifting through massive redundancy.
This case study provides strong empirical evidence for the practical effectiveness of our method. First, the retained Basic WCs are confirmed be different error patterns. Second, the analysis of Excluded WCs demonstrates that redundant codes are essentially composite errors. Third. the inspection of Repeated WCs confirms that identical failure signatures reliably map to semantically equívalent root causes, validating our signature-based grouping strategy. Fourth, the real-world evaluation highlights the benchmark's discriminative power. Collectively, these results affirm that TC-Bench successfully constructs compact, rigorous, and representative error space, capable of delivering fine-grained and high-sensitivity evaluations for test case generation.
43


Reviews:
[{"review_id": "R0uJrrLU56", "reviewer": "Reviewer_tm2i", "summary": "This paper addresses an important problem in LLM evaluation: how to construct an efficient, reliable, and unbiased benchmark for assessing the quality of automatic test case generation methods. The core contribution is a novel framework based on binary-matrix theory. This framework uses the matrix rank to simultaneously determine the minimal number of wrong codes required for evaluation and a theoretical upper bound on the number of test cases needed for full error pattern coverage. Based on this framework, the authors devised WrongSelect to find an approximate solution to this NP-hard problem, culminating in the construction of TC-Bench.", "questions": "See Weaknesses. I will be glad to raise my score if the authors could provide a sufficient rebuttal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles the crucial challenge of evaluating test case generation methods for code.\n2. The author processes an efficient approximation algorithm combining pre-filtering and random-restart local search, WrongSelect. This provides a reasonable and effective solution to the NP-hard problem of selecting a maximally diverse diagnostic basis from a vast collection of wrong codes.\n3. The authors release a compact and diverse benchmark, TC-Bench. By design, TC-Bench can reduce the computational cost of evaluation and is resistant to score inflation.", "weaknesses": "1. Insufficient discussion of related work. The paper could be strengthened by a more thorough discussion of related work that also leverages code-test properties. For instance, CodeT [1] or other approaches that also use binary matrices, such as B4 [2].\n\n2. Limited theoretical justification for the rank-coverage duality. A central claim of the paper is that the matrix rank provides an upper bound on the minimum number of test cases required for fault coverage. While the paper lacks a formal proof or a detailed discussion to substantiate this claim. A deeper analysis would strengthen the claim beyond intuition.\n\n3. Limitations of the chosen diversity metric. The paper employs Jaccard similarity to measure the overlap. It implicitly assumes that all GTs are of equal importance. In practice, some GTs may represent extremely rare and critical edge cases, while others are more trivial. The Jaccard metric cannot capture this weighted difference. Have the authors experimented with or considered other similarity metrics that could account for the varying significance of different test cases?\n\n[1] Chen, Bei, et al. \"CodeT: Code Generation with Generated Tests.\" The Eleventh International Conference on Learning Representations.\n\n[2] Chen, Mouxiang, et al. \"B4: Towards optimal assessment of plausible code solutions with plausible tests.\" Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PEf6dsWQWA", "reviewer": "Reviewer_jxDY", "summary": "This paper proposes a new benchmark to evaluate LLMs' ability to generate high quality test cases for coding problems. The quality is determined by the efficiency in covering the error space of the problem. To improve quality representation of the error space, the authors form an error matrix, consisting of all the submissions by their pass/fail results on all the test cases. Since the rank of this 2D matrix conveniently represents the \"coverage\" of submission patterns, they develop a greedy algorithm to maximize the diversity of wrong code submission, which in return reduces to a more compact submission set for efficient evaluation.\nThe authors then evaluate several test case generation methods on various LLMs on two major metrics – the accuracy of generated test cases (PR) and the degree of successful filtering of wrong code submissions (HR). The evaluation results show that the choice of generation method matters significantly more than the choice of the underlying LLM. At the same time, the relatively underwhelming numbers (best combo leads to 60~% in HR) points to the importance of constructing a high quality set of wrong code – the analysis on the unfiltered version of the dataset reveals inflated scores, which is inaccurate.", "questions": "* How much variance in PR / HR do you observe by running the optimization multiple times? When you say convergence, do they arrive at the same set of WCs?\n* The analysis comparing the method against the \"All WCs\", are the all WCs before or after pre-filtering?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* A meaningful contribution for a much-needed area, both in terms of efficiency and framework. The method can see practical use cases. \n* The proposed method for selecting the minimal set of wrong code submissions is principled and reasonable.\n* The outcome dataset effectively pinpoints the need for better TCG methods, citing the underwhelming numbers.", "weaknesses": "* Apple-to-apple comparison against the previous work: It would be stronger if you could replicate the other test case evaluation methods on the same problem set and show how TC-Bench more critically measures the TCG quality by LLMs. I'm aware this is done partially by comparing the method against the \"All WC\" counterpart.\n* WrongSelect's robustness: It's uncertain if the greedy algorithm yields a stable set of problems. \n* The need of translation: I assume the dataset is entirely sourced from non-English problems. The choice of data sources affect the quality and the number of (valid) wrong code submissions for problems. Have authors explored English data? Reducing the submission count to less than 2% depends on the population of wrong code submissions.\n\n---\n\nBelow are NOT the weaknesses of the method, but I think they are important to raise for a higher quality submission. I'm leaving here as I don't see fit in \"Questions\"\n* Presentation 1: Overall there is high usage of acronyms, such as \"some WCs labeled as WA under GTs produce RE or TLE when executed on ATs\". I got used to it by the end a little bit, but I often had to go back and refresh my glossary many times. I'd highly suggest fixing the inconsistency (e.g., \"wrong code\" and WC) and reducing overall abbreviations.\n* Presentation 2: Plot styles are inconsistent – mostly comic sans but some of them serif (Times?) out of a sudden. It looks like being patched together. I recommend following a single style.\n* Appendix B has no content", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IdMqiqLLtq", "reviewer": "Reviewer_tPeL", "summary": "This paper addresses how to rigorously evaluate LLM-generated test cases by reframing benchmark construction as a binary matrix problem over wrong codes (rows) and golden tests (columns). The central idea is to select a compact yet representative set of wrong codes that preserves the matrix’s diagnostic capacity while avoiding redundancy and score inflation. Concretely, the authors require the kept rows to form a row basis whose size equals the matrix rank (capturing all independent error modes), and among all such bases, they choose one that maximizes diversity by minimizing average pairwise Jaccard similarity of failure signatures.\n\nThey propose WrongSelect, an efficient approximation combining principled pre-filtering and random-restart local search. Pre-filtering drops problems with any all-ones test column (hack-prone) and removes trivial wrong codes that fail ≥80% of tests; then the local search swaps rows in/out while maintaining rank to minimize the diversity objective.", "questions": "could you provide an example for the effectiveness of your approach in a code example?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a fresh and elegant formulation that connects test selection to linear algebra. \n\nModeling wrong-code failure signatures as rows in a binary matrix and selecting a rank-preserving, diversity-maximizing row basis is a principled way to eliminate redundancy while retaining all independent error modes. This perspective clarifies what “good coverage” means and gives a theoretical upper bound on minimal tests.\n\n2. The empirical results are strong. \n\nThe proposed selection method yields a compact, diverse benchmark that meaningfully stresses current test generators, and the reported exclusion rates show clear, consistent improvements over baselines. The ablations and end-to-end evaluations convincingly demonstrate both effectiveness and practicality at scale.", "weaknesses": "1. The approach assumes access to a sufficiently rich pool of golden tests (public + private) to build informative failure profiles. \n\nIn many real-world settings, curating or synthesizing high-quality golden tests is costly and time-consuming. This reliance reduces the method’s portability and makes it difficult for third parties to reproduce or extend the benchmark to new domains or problem sets without significant investment.\n\n2. While the formulation is theoretically sound (rank preservation plus diversity), the paper provides limited qualitative analysis to illustrate practical effectiveness. \n\nConcrete case studies—e.g., representative wrong-code examples retained vs. removed, error modes uncovered by the selected basis, or developer-facing insights derived from the reduced matrix—would strengthen the claim that the method improves understanding and diagnosis beyond raw metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X8aszRMCUa", "reviewer": "Reviewer_D88H", "summary": "This work addresses the challenge of evaluating automatically generated test cases for code using LLMs. The authors propose a new framework that models the relationship between code and test cases as a binary matrix, where the matrix rank reveals the minimal number of distinct wrong codes and test cases needed for evaluation. They introduce WrongSelect, an efficient algorithm for selecting a maximally diverse and representative set of wrong codes to form a compact benchmark, TC-Bench. Built from millions of competitive programming submissions, TC-Bench offers an efficient, diverse, and inflation-resistant benchmark for assessing test-case generation. Evaluation across 13 leading LLMs show that even the best current methods achieve only around 60% fault detection, highlighting significant room for improvement in automated test generation and evaluation.", "questions": "- How practical is this technique in practice/industry?\n- Do you think your technique generalizes on other datasets, when your evaluation is only using programming contests?\n- Do you think other similarity metrics would fit better than Jaccard?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Formulating problem as code-test matrix\n- Number of evaluated LLMs\n- Approximation algorithm\n- Benchmark construction", "weaknesses": "- Source of data from programming contests\n- Theoritically ok, but what about practicality?\n- Not studying other similarity metrics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)