Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
Authors: Zhe Zhang
Runlin Liu
Aishan Liu
Xingyu Liu
Xiang Gao
Hailong Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Dynamic Benchmark Construction
LLM Evaluation
Primary area: datasets and benchmarks
TL;DR: We propose CODE2BENCH, an automated pipeline for dynamically constructing rigorous code benchmarks from recent real-world GitHub repositories to combat data contamination

Abstract: The evaluation of code-generating Large Language Models (LLMs) is fundamentally constrained by two intertwined challenges: a reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing. This paper introduces a new benchmark construction philosophy, Dual Scaling, designed to systematically address both limitations. Our approach involves continuously
scaling the source of problems from dynamic, real-world code repositories and systematically scaling the rigor of tests via automated, high-coverage Property-Based Testing (PBT). We instantiate this philosophy in CODE2BENCH, an end-to-end framework that leverages Scope Graph analysis for principled dependency classification and a 100% branch coverage quality gate to ensure test suite integrity.
Using this framework, we construct CODE2BENCH-2509, a new benchmark suite with native instances in both Python and Java. Our extensive evaluation of 10 state-of-the-art LLMs on CODE2BENCH-2509, powered by a novel "diagnostic
fingerprint" visualization, yields three key insights: (1) models exhibit a fundamental performance gap, excelling at API application (Weakly Self-Contained tasks) but struggling with algorithmic synthesis (Self-Contained tasks); (2) a model’s
performance is profoundly shaped by the target language’s ecosystem, a nuance we are the first to systematically quantify; and (3) our rigorous, scaled testing is critical in uncovering an "illusion of correctness" prevalent in simpler benchmarks. Our work presents a robust, scalable, and diagnostic paradigm for the next generation of LLM evaluation in software engineering. The code, data, and results are available at https://code2bench.github.io/.

Published as a conference paper at ICLR 2026
CODE2BENCH: SCALING SOURCE AND RIGOR FOR DYNAMIC BENCHMARK CONSTRUCTION
Zhe Zhang', Runlin Liu', Aishan Liu"; Xingyu Liu', Xiang Gao "Hailong Sun SKLCCCS, Beihang University "Hangzhou Innovation Institute of Beihang University (zhangzhe2023, runlin22, liuaishan @buaa edu. en (1xingyu, xiang_gao, sunhl) ) @buaa. edu.cm en
ABSTRACT
The evaluation of code-generating Large Language Models (LLMs) is funda- mentally constrained by two intertwined challenges: reliance on static, easily contaminated problem sources and the use of superficial, low-rigor testing. This pa- per introduces new benchmark construction philosophy, Dual Scaling, designed to systematically address both limitations. Our approach involves continuously scaling the source of problems from dynamic, real-world code repositories and sys- tematically scaling the rigor of tests via automated, high-coverage Property- -aase Testing (PBT). We instantiate this philosophy in CODE2BENCH an end-to-end framework that leverages Scope Graph analysis for principled dependency clas- sification and a 100% branch coverage quality gate to ensure test suite integrity. construct
Using this framework, with native instances in both Python and Java. Our extensive evaluation of 10 state-of- the-art LLMs on CODE22ENCH-25099 powered by novel "diagnostic fingerprint" visualization, yields three key insights: (1) models exhibit a fundamen- tal performance gap, excelling API application (Weakly Self-Contained tasks) but struggling with algorithmic synthesis performance is profoundly shaped by the target language's 's ecosystem, a nuance we are the first to systematically quantify; and (3) our rigorous, scaled testing is critical in uncovering an "illusion of correctness" prevalent in simpler benchmarks. Our work presents a robust, scalable, and diagnostic paradigm for the next generation of LLM evaluation in software engineering. The code, data, and results are available at htttts://cooeeeeetee wtw.:e/bbbncn.............. github.io/
B
benchmark suite
tasks); (2) model's 's
INTRODUCTION
As Large Language Models (LLMs) are increasingly integrated into software development work- flows Jimenez et al. )2023); Git: cur the need for accurate and realistic evaluation of their coding capabilities has become paramount. However, the current landscape of code benchmarks fundamen- tally constrained by two intertwined challenges: reliance static, easily contaminated problem sources and the use of superficial, low-rigor testing.
First, 1 the static nature of canonical benchmarks like HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) leads to an inevitable obsolescence; their problems, having existed for years, are likely part of LLM training corpora, turning evaluation into exercise in memorization rather than true generalization Carlini et al. 2022 1); /aanz et al. (2023). While dynamic "live" bench- marks Jain et al. (2024)); Li al. (2024b) have emerged, they often source problems from competitive programming, which may not reflect the complexity of real-world software engineering. Second, @ the superficial testing common to most benchmarks, often relying on a handful of example- tests, creates an illusion of correctness. As highlighted by EvalPlus Liu et al. (2023a). this insufficient test rigor fundamentally limits their ability to uncover the subtle, edge-case failures that define the gap between functional code and production-ready software. As summarized in Table [ existing methods fall short across key dimensions, highlighting the significant limitations remaining and the
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of CODE2BENCH-2509 with existing code generation benchmarks.
Deps Handled
Rigorous Test
Multi-lang Design
Benchmark
Source
Dynamic
HumanEval Chen et al. 2202
Manual
X
X
X
MBPP/Austin et al, 2021
Manual
X
x
X
EvalPlus
Lau Lau
al. 2125a
Manual
X
X
LiveCodeHeechh Jain et
20124a
Contests
RepoBench Liu
202
Project Codebases
X
HumanEvall Zheng al.
2023b
Manual
BigCodeBench Zhuo
2024
Synthetic
DevEval Li 2024c
Project Codebases
EvoCodeBench al.
2024b
Project Codebases
X
CoDE2BENcn:-2009 (Ours)
Project Codebases
V
necessity to design new benchmarks. To break this cycle of obsolescence and superficiality, we argue that a paradigm shift is needed. We propose new benchmark construction philosophy centered on two core principles: (1) Scaling the Source, by dynamically and continuously ingesting diverse array of problems from the ever-evolving landscape of real-world code repositories; and (2) Scaling the Rigor, by systematically generating comprehensive test suites with deep, verifiable coverage through Property-Baad -Based Testing (PBT) Claessen Hughes 2000).
We instantiate this philosophy in CODE2BENCH, novel, end-to-end framework that automates this dual-scaling process. To Scale the Source, CODE2BENCH first addresses the challenge of classifying diverse, real-world code. Our analysis of the existing benchmark landscape reveals an implicit bifurcation in evaluation focus, which we formalize into two primary task categories: (1) Self- Coff-Contained (SC) tasks, which require pure, dependency-fres logic, reflecting the focus of benchmarks like HumanEval Chen et al. (2021) on core algorithmic reasoning; and (2) Weakly Self-Contained (WSC) tasks, which require the correct application of common libraries, capturing the focus of benchmarks like BigCodeBench Zhuo et al. (2024) on practical API application. This principled classification, enabled by our Scope Graph- based analysis, allows us systematically generate tasks that target these distinct developer skills. @ To Scale the Rigor, CODE2BENCH then employs a powerful Property-Baue Based Testing (PBT) engine and a stringent 100% % branch coverage quality gate. This ensures that every problem in our benchmark is not only realistic but also fully-explorable logical challenge, backed by a test suite capable of deep, diagnostic validation.
Using this framework, we construct CODE2BENCH- -200,, a new, multi-facett benchmark suite with native instances in Python and Java, curated from recent, real-world repositories. Our extensive evaluation of 10 ttate-of-the-art LLMs this suite demonstrates the power of our approach. The synergy of a scaled source and scaled testing enables an unprecedentedly fine- ane-grained diagnostic, revealing: (1) performance gap between models' ability in algorithmic synthesis (SC) and API application (WSC); (2) the profound impact of language paradigms on model failure modes, a nuance are the first systematically quantify; and (3) the critical role of rigorous testing in uncovering the "illusion of correctness" prevalent in simpler benchmarks.
We summarize our contributions follows:
We propose CODE2BENCH, a novel framework that introduces and operationalizes the Dual Scaling philosophy for benchmark construction, systematically scaling the source of problems with dynamic acquisition and the rigor of tests with 100% coverage PBT quality gate.
We construct and release CODE a high-quality, contamination resistaan benchmark with native tasks in Python and Java, demonstrating significantly higher complexity and test rigor than prior work (Table
We provide deep, diagnostic analysis of state-of-the-ard LLMs, introducing novel visualizations and uncovering key insights into their strengths and weaknesses rorl-world coding scenarios.
2

---

Published as a conference paper at ICLR 2026
1. Scaling the Source
Candidate
def merge_dict (bese, update):
github
return merged
Dynamic Acquisition
Principled Classification
Semantic Filtering
Temporal Fillering
Scape Graph Analysis
LLM-as-a a--da--
def normalize(a: np op.ndarmay):
SC Pool (- 10K Candidates)
Candidates
return result
.....
WSC Pool 5K Candidates)
WSC Candidates
WSC Candidate
2. Scaling the Rigor
Leaderboard
Test Synthesia
Benchmark Instantiation
Property-Basee Baarry---Beeddd-a Testing
100% Coverage
Versioned Benchmark
Input: Candidate Function
Only -40%, of nighh-quyy candicates pass rigrroua gate
Multi-Faceted Suite Java)
Diagnostic Fingerprint
Figure 1: Overview of the CODE2BENCH Framework.
2 THE CODE2BENCH FRAMEWORK: A DUAL SCALING APPROACH
In this section, detail the architecture and technical components the CODE2 framework. Our methodology is built upon the core philosophy of Dual Scaling: continuously scaling the source of benchmark problems ensure realism and novelty, and systematically scaling the rigor our tests to enable deep, diagnostic evaluation. We organize our discussion around these two principles.
2.1 SCALING THE SOURCE: DYNAMIC ACQUISITION FROM REAL- WORLD CODE
Temporal Filtering for Contamination Resistance. The primary threat the validity of LLM evaluation data contamination, where benchmarks become obsolete as their contents are absorbed into training corpora. To combat this inevitable obsolescence, our framework's 's first principle is to dynamically source problems that are provably unseen. We achieve this through Temporal Filtering, deterministic strategy grounded in the version control timestamps of real-world code. Our method leverages simple axiom: model cannot have been trained on code that did not exist prior to its knowledge cutoff date. For each model under evaluation, we run our acquisition pipeline extract functions exclusively from GitHub commits created after its official knowledge cutoff.
Scope Graph-based Analysis for Dependency Classification. To impose meaningful structure on our diverse pool functions, we automate their classification based on dependencies. We employ a Scope Graph-based analysis Néron al. (0155) formal, language-agnnn agnostic method that precisely identifies all external dependencies, a task where simpler methods like AST traversal often fail.
Our classification algorithm is deterministic, two-step process: (1) Dependency Identification. where we use the Scope Graph to compute the set of all unresolved references (D) for each function. (2) Rule-based Classification, where apply rules based predefined allowed libraries, Lallowed'
0. it is classified as Self-Contained (SC) (pure algorithmic reasoning). If D's dependencies resolve entirely within Callowed, it is Weakly Self-Contained (WSC) (API application).
Otherwise, it is discarded (e.g., Project-Dependent!
This principled, automated process is cornerstone of our framework, enabling the targeted evaluation of distinct model capabilities.
Program Analysis for Testability and Complexity. Following dependency classification, we apply
final layer of automated program analysis to ensure candidates are both testable and non-trivial.

---

Published as a conference paper at ICLR 2026
First, to guarantee testability, use Control-Flow Graph (CFG) analysis to discard functions lacking a verifiable, input-dependent output (c.g.. no return statement). Second, to ensure a meaningful complexity, we filter functions based their Cyclomatic Complexity McCabe (1976), targeting range (e.g.. [2, 10)) that balances challenge with solvability. This dual-filtering stage is crucial for curating high-quality benchmark of tasks that are both evaluable and diagnostically valuable.
LLM-based Semantic Filtering. While prior analyses ensure structural soundness, they cannot distinguish a meaningful task from a trivial onc. To assess for semantic relevance and conceptual challenge, therefore employ an L.LM-as-a-Judae /Zheng et al. (2023a); Li et al. (2024a). This final filtering step, designed for high reliability via deterministic decoding and a structured classification prompt, ensures our benchmark contains problems of genuine substance. A rigorous validation, detailed in Appendix D confirms its near-perfect agreement with human experts (Cohen's K = 0.95).
2.2 SCALING THE RIGOR: AUTOMATED SYNTHESIS VIA TROPERTY-BASED (ESTING(PBT)
Property-Basse Testing (PBT) for Comprehensive Input Generation. Traditional example- based tests verify function against small, fixed of known inputs. In contrast, Property- -Based Testing (PBT) (PBT) Claessen & Hughes (2000) explores a much larger input space by generating hundreds or thousands of random, yet structured, imputs and asserting that general property of the code holds true for all of them. In our framework, the core property we test is functional equivalence with the ground-truth implementation. For any valid input x generated by our PBT engine, the output of an L.M--gnerrted function (L) must match the output of the original, real-world ground-truth function fg((x), The ground-truth function thus serves as a perfect test oracle.
The process of input synthesis is driven by automated strategy generation. For each function candidate, our framework analyzes its signature, including parameter types and type hints, to compose set of PBT strategies. These strategies the input domain, designed to produce a rich distribution of values including typical inputs, boundary cases (c.g., empty lists, zeros, min/max values), and complex nested structures (c.g.. variable-share shaped lists of dictionaries.. This automated process yields a comprehensive suite of hundreds of input-output pairs (x,, (x,)) for each function, forming the foundation for the rigorous validation described next. The specific PBT libraries used for each language (e.g., Hypothesis for Python, jqwik for Java) are detailed in Appendix E
not simple random generators, but intelligent explorers of
The "Great Filter"; A 100% Coverage Quality Gate. While Property-Based Testing generates high volume of diverse inputs, quantity alone does not guarantee rigor. test suite, however large, is only effective if it thoroughly exercises the internal logic of the function under test. To enforce this level of rigor systematically, we introduce the final and most stringent stage of our pipeline: the "Great Filter", a quality gate that mandates 100 % branch coverage.
The mechanism is as follows: after a PBT suite is synthesized for ground-truth function Sg-- we execute the entire suite against fgr itself and measure the resulting branch coverage using standard language-speciff tools (e.g., coverage py). A function candidate and its corresponding test suite are only accepted into the final benchmark if and only if this execution achieves 100% branch coverage. This seemingly simple requirement has profound impact on the final benchmark's 's quality and character. It acts as a powerful, dual- -urlppe filter:
It filters out inadequate tests. If a PBT suite fails to achieve full coverage, it indicates that the input generation strategy was not sophisticated enough explore all logical paths of the function. Such a test suite would be incapable of providing a truly rigorous evaluation, and is discarded.
It filters out untestable functions. More importantly, if even a well-designed PBT strategy cannot trigger all branches, it often signals that the function itself "untestable" in isolation. This typically occurs in functions with defensive code for unreachable states, complex error handling coupled to external systems, or other logic that cannot be exercised through its public API. These functions, while present in real-world code, are unsuitable for a standalone, functional correctness benchmark.
The "Great Filter" is the primary reason for the significant reduction in candidates observed in data funnel (as shown in Figure []. It is a deliberate trade-off, prioritizing uncompromising rigor over sheer volume. The result smaller, but significantly more potent, benchmark where every
4

---

Published as a conference paper at ICLR 2026
single problem is guaranteed to be non-trivial, fully-explorable logical puzzle, backed by test suite capable validating every branch of its solution.
Instruction Generation for Task Specification. To ensure fair and effective evaluation, each task is accompanied by a clear, unambiguous instruction. Our framework automates the generation of these instructions by refining a function's 's original source docstring and signature using powerful LLM (GPT-40) with deterministic decoding. To mitigate potential biases, we also employ a back-translation perturbation technique Zhuo et al. (2024); Wang al. (2022); Dhole al. (2021).
Crucially, the instruction style is systematically adapted to the task's dependency classification, ensuring the model is provided with the precise context needed for the specific challenge:
For SC Tasks(e.g. SC-Python, SC-Java): Instructions use language-native conventions-Pyython docstrings with types like 1 list and dict for SC-Python, and Javadoc with Java types like List<String> for SC-Java. Though library-free, these tasks assess a model's proficiency with each language's s core built-in features and data structures.
For WSC Tasks: The instruction made library-aware and language-native It explicitly names the required external libraries (e.g., NumPy) and uses the precise, idiomatic types of the target language's s ecosystem (e.g., numpy. ndarray). This targets the evaluation on a model's practical ability correctly apply common APIs.
Benchmark Instantiation and Runner Generation. The final step of our framework packages each curated problem into an executable benchmark instance. This instance comprises two key components: test suite and test runner. The test suite, containing hundreds of input-output pairs, is generated using a native Property-Basee Testing (PBT) engine (e.g., Hypothesis iqwik) k) to ensure high coverage and rigor. To conduct the evaluation, corresponding language-native Test Runner is automatically generated. The runner is responsible for deserializing the test suite, executing the LLM-gennrated code, and performing a rigorous deep comparison against the ground-trutt outputs. To guarantee the runner's 's correctness, we perform dry run where the LLM's Mss function is replaced by the ground-truth function, ensuring the entire test harness passes flawlessly before evaluation. This end-to-end native approach, validated by dry run, ensures our evaluation is both stringent and reliable. Further details are in Appendix
THE CODE2BENCH 2509 BENCHMARK SUITE
Table 2: Quantitative characteristics of the CODE2BENCH-2500 compared to prior benchmarks.
Metrie/ / Dimension
SC-Python
WSC-Python
SC-Java
HumanEvol
MBPP
Scale & Complexity
Tasks
217
194
974
Avg. Lines of Code (LoC)
20.6
18.3
14.1
7,3
6.5
Avg. Cyclomatic Complexity (CC)
5.3
2.6
3.6
2.3
Difficulty (E:MHH Ratio)
0.30:0.40:0.30
0. 0.28:0.41:0.31
0.27:0.4300.30
11. Testing Rigor
Avg. Test Cases per Task
-500
-500
-500)
-7.8 Variable
-3.0
Test Coverage Guarantee
100% Branch
100% Branch
100% Branch
Variable
III. Diversity & Extensibility
Source Type
Real Self-Contained (Python)
Real Self-Contaised
Real
Hand-Crafted Self-Contained
Crowd Sources Self-Contained (Python)
Dependency Scope
>3 Libraries
Language Exiensibility
(Python)
Java Native
(Python)
CoDE2BENCH-28 -2509 is a new benchmark suite, automatically curated from May to September 2025, designed to overcome the limitations of prior benchmarks by systematically expanding evaluation along three key dimensions: Testing Rigor, Dependency Level. and Framework Extensibility. Figure visually situates our benchmark within this landscape, showcasing its significant leap forward compared to predecessors like HumanEval and BigCodeBenchh. We targeted actively maintained, open-source repositories on GitHub. To minimize noise and bias, enforced the following criteria: (I) Community Validation: Repositories must have 2 500 stars; (2) Active Maintenance: Commits within the last 3 months; and (3) Domain Diversity: We employed stratified sampling across 10 diverse domains (e.g., Web, ML, System) to prevent represspretnttation of any single field. We actively filtered out homework assignments and tutorials. A detailed disclosure of the selection
5

---

Published as a conference paper at ICLR 2026
criteria, sampling strategy, and the full repository list is provided in Appendix
Husartbrl
Mare
The quantitative evidence for this advance- ment is detailed in 2 Our instances demonstrate significantly higher structural complexity (e.g., an average Cyclomatic Complexity of 5.3 for SC-Python vs. 2.8
RogeiBench
80-Pyttonn
Extensiie
nguaga
Extensiblitty
Dependency
for Hvla and an order-of-magnitude increase in testing rigor, featuring -500 test cases per task with a guaranteed 100% branch coverage. Beyond these metrics, the suite's 's high quality is rooted in the rich diversity of its tasks, soureed from 220 Python and 189 Java recent, real-world repositories. This ensures wide topical cov- erage, while the successful instantiation of a native Java suite provides concrete proof of our framework's 's extensibility. This com- bination of complexity, rigor, diversity, and extensibility provides more challenging and realistic platform for assessing the true capabilities of modern LLMs. More details can be found in Appendix
Modeca
Sealing
Limttd
(2D)
Testing Rigor
Moderate
Dependence Level
Whatty Corbann
(M5001
Geff Gatcoommm
Baaic
Figure 2: The multi-dimensional evaluation landscape.
4 EVALUATION
4.1 EXPERIMENTAL SETUP
Evaluated Models. We selected a diverse suite of 10 state-of-the-art Large Language Models, encompassing both leading closed-source APIs and prominent open-source families. A cornerstone of our evaluation integrity is the strict prevention of data contamination. The CoD = 32BBNCE NCCH -C--25C benchmark constructed exclusively from code committed after May 2025, date subsequent to the knowledge cutoff of all evaluated models.
Evaluation Protocol. Our primary metric is Pass @ Kulal et al. (019). which measures the functional correctness of the gentraedd solution and closely mirrors a developer's 's real-world experience with coding assistants Jain al. (2024a). All evaluations
conducted in zero-shot, deterministic setting, employing greedy decoding (temperature 0) as is standard practice (Zhuo et al. (2024); ; Roziere al. (2023). For each task, the model received standardized instruction containing the function signature and natural language description(See more details in Appendix
Execution Environment. The generated code for each task was executed in a sandboxed environment against the full suite of PBB--genede PBT-generated tests (500 per task). A sunguage-specifte test runner performed differential testing, comparing the output of the model-generated code against the ground- -tuthh implementation. A task is considered passed only if it correctly solves all test cases. Open-source models were served via vL..M Kwon et al. (202). while others were accessed through their official APIs.
4.2 MULTI- HULTI-DIMENSIONAL DIAGNOSTIC OF LLM CAPABILITIES
primary limitation of existing benchmarks is their inability to provide deep, diagnostic insights. We argue this stems from two fundamental constraints: narrow source of problems and superficial testing. The CODE2BENCH framework overcomes these limitations through two core principles: scaling the source from dynamic, real-world code, and scaling the rigor of evaluation via Property- Based Testing (PBT). In this section, we demonstrate how this dual-scaling approach enables an unprecedentedly fine-grained diagnostic of LLM coding capabilities.
Table 3 presents the overall Pass@1 performance, revealing clear capability tiers among models. The data indicates that performance varies significantly across our three benchmark components,

---

Published as a conference paper at ICLR 2026
Table 3: Pass@ performance (%) on the CODE2BENCH 2509 suite.
Model
SC-Python (%) [95% CI)
WSC-Python (%) [95% CT)
SC-Java [95% CT]
Closed-Source Models
Claude-4-sone
40.1 [33.6 46.5]
38.7 [32.0 45.4]
47,4 [40.9 53.4]
Gemini-2.5-Flaar
37.8 [30.9 44.2)
36.6 [29.4 43.3]
45.0 [39.0 51.0]
Open-Source Models (Ordered Scale)
DeepSeek-V2 V3
34.4 [28.4 40.4]
37,6[31.4 -4.3]
44.8 54.2]
Qwen3-235b-a22
34.6 [28.6 41.0]
43.3]
46.6
53.0]
Llamaa--scout
25.8 -
32.5(26.3-3922
44.2
Qwen3-32b
31.3 [25.8 33.9]
34.5(27.-41.2)
43.0
Mistral-small-3.1 (24B)
30.4 [24.4-36.9]
38.7 (32.5-2554
(37.444444
Qwen3-8b
25.1 [19.3 31.4]
34.0 -0077
3320 44.22
Cemmma-nn-ebbitt
22.6 [17.1 28.6]
[19. 32.5]
34.5 [28.5 40.2]
Qwen3-1.7b
14.3 - 19.4]
16.5 [11.3 -1.6]
17.7 [12.8 22.5]
0.0
Gaade-S4
0-5
Despoesiov33
Uaame-Jccut
Owerd-328
Hitraks
dwank-ik
Gribal Hibor
Gemma-2n
Bennk-1.1 1.79
Sydlastt
Austiou
(8-8P 201
Dusai Whapo ut ie0 www.a Jiovinaa
Figure 3: Fingerprints across the three evaluation trakks-S-PPtti SC-PPthon (left), WSC--yythe (mid- dle), and SC-Jave (right)-shown ridgeline plots. Each curve captures model's outcome distribution, ranging from SyntaxExs to Perfect, with key pass rates annotated.
hinting the different skills they evaluate. To move beyond these aggregate scores and understand the underlying reasons for these variations, we now turn to more fine-grained analysis.
To dissect how and why models succeed or fail, we introduce granular, multi-stage outcome spectrum, visualized as a diagnostic fingerprint for each model in Figure 3 Outcome categories are ordered reflect progression from catastrophic failure to complete success: Synt (code fails to compile/run), Runt imeErr (code crashes during execution), LogicErr (code runs but logically incorrect on all test cases), followed by a four-tier partial success range based on pass rates-from Fail (<20%) to NearPerf (>=98%) -nd culminating in Perfect solutions. Figure 3 visuulizess each model's distribution across this spectrum. The synergy between the aggregate scores the table and these distributional fingerprints reveals two profound insights into the nature of [.LM coding intelligence.
Decoupling Algorithmic Synthesis from API Application. The diagnostic fingerprints (Figure left vs. middle) show systematic shift in failure modes: on SC-Python. the dominant failure is LogicErr, indicating a core challenge in first-principle reasoning. Conversely, on WSC-Python. this peak vanishes and Runt me emerges as primary obstacle, suggesting the challenge shifts the correct application of external APIs.
Language Paradigms as Performance Scaffolding. The comparison between sc-Python and SC-Java (Figure 3 left vs. right), reveals the profound impact of language paradigms. In Java, the prominent Logi cE and Runt imeErr peaks in Python sharply suppressed, while
7

---

Published as a conference paper at ICLR 2026
performance in the Perfect category surges for all models. We hypothesize this is not because models are inherently "better" at Java, but because its static type system acts as a powerful "perfor- mance scaffolding", pruning vast space of potential emrors compile time. This demonstrates that an L.M's coding ability is not an abstract quantity but is fundamentally intertwined with the target language's ecosystem, crucial interaction framework is the first to systematically quantify.
4.3 THE EFFECTIVENESS OF PBT-GENERATED TESTS
A core tenet of the CEDE2BENCH framework is scaling test rigor via Propert-Baaeed Testing (PBT). This section provides quantitative evidence for the necessity of this approach by analyzing the prevalence of Necar-Prrfett suite but fail on a handful of subtle edge cases. These instances represent an "illusion of correctness" that would likely go undetected by conventional, less rigorous benchmarks.
that pass at least 98% of our comprehensive test
Table quantifies the frequency of these "near-miss" failures, where a solution passes the vast majority of test cases (>98%) but ultimately fails. The data reveals significant and consistent pattern: on average, 6.94% % of submissions for SC-Python tasks fall into this treacherous category.
This finding directly underscores the critical necessity of our Property- Based Testing (PBT) method- ology. Without the exhaustive, edge-case-driven verification enabled by PBT, these nearly 7% sub- missions-which are functionally almost correct-oould have been falsely classified successes by conventional, sparse test suites. This would lead to significant overestimation of model capabilities. hop-performing models not immune
this illusion of correctness; "DeepSeek-V3" and "Claude-4-sonnet,, for example, approximately 8% of their submissions fall into this category. This demonstrates that even the most capable models consistently struggle with the final frontier of logical robustness, a weakness that only a truly rigorous testing paradigm like PBT can re- liably expose.
Table 4: Prevalence of "Near-Perfed" Failures (Pass@ 29%% in CODE2BENCH.
Model
SC-Py (%)
WSC-Py (%)
SC-Java
(%)
Claude-Sonnet-4
8.76
5.15
2.41
2omini-2.55Flash
6.45
5,67
4.02
DeepSeek-V3
7.80
3.61
2.41
Qwen3-235b-a22b
6.45
3.61
2.01
Interestingly, the rate of near-perfect fail- ures is lower in WSC-Pythor. (4.57% and lowest SC-Java (2.25%). This
LLAMA-4-scout
8.29
5.15
2.01
1-ttal--mmall3.1111
6.91
5.67
2.01
Qwen3-32b
7.37
3.61
1.61
Qwen3-8b
6.76
4.64
2.41
aligns with our findings in Section 4.2 In WSC tasks, the problem often a binary choice of the correct API call, leaving less room for "almost correct" logic. In Java, the strict type system likely prevents many of these subtle logical errors at the compi- lation stage. Therefore, the high rate of peer-et failures in SC-Python highlights its unique position as the most challenging testbed for a model's 's pure, unaided logical robustness.
Gemma-3n-e44
5.53
5.67
2.81
Qwen3-1.7b
5.07
3.09
0.80
Avg. (%)
6.94
4.57
2.25
Ultimately, this analysis validates the central role of rigorous, scaled testing, By systematically uncovering these near- perfect failures, CoDa2BeNch provides more accurate measure of a model's true capabilities and offers invaluable, fine-graii finenerained feedback for identifying and rectifying their most subtle weaknesses.
4.4 THE IMPACT OF DYNAMIC SOURCING AND REAL- WORLD COMPLEXITY
To situate CODE2BENCH 2509 within the existing landscape, we conduct a direct comparison against EvalPlus, state-of-the-ar benchmark that enhances HumanEval/MMBP with more rigorous, mutation-baa based testing. While EvalPlus represents the pinnacle of evaluation on static, well-known problem sets, CODE2BENCH- -509 introduces the dimensions of dynamic sourcing and real-worid complexity. This comparison aims to answer a critical question: how does a model's performance on canonical programming puzzles translate its ability handle fresh, complex code from the wild?
8

---

Published as a conference paper at ICLR 2026
The head-to-head comparison is visualized in Figure 4 which plots the Pass@1 I scores of ten prominent LLMs on our SC-Python-250 (X-axis) against their performance on HumanEval (Y-axis). The results stark and reveal three critical insights:
Systematically More Challenging Benchmark. The most striking observation that all models located deep in the red-shaded re- gion, far above the y = I diagonal of equal performance. This demonstrates that presents systemati- cally higher level of difficulty for all mod- els, without exception. For instance, top- performing models like Claude-4-Sonnet, which achieve near-perfect score of 97% on HumanEval, see their performance plum- met to 40.1 % on our benchmark-4 drop of over 50 percentage points. This substantial performance gap suggests that high scores on legacy benchmarks may create an illusion of capability that does not hold up against the complexity of novel, real-world code.
CODE2BENCH-2555
80%
HoreenBoal> SC Si-itylian--500
Hounodtall Paugg
10%
Prym Uliea Podtemmeen
Midd
49%
39%
SC-Pyilae-550
Hunential
10%
SC-Pytleo-2500 Pawjji
Probing Generalization Over Memoriza- tion. This performance delta is not merely matter of difficulty, but of the fundamental capabilities being measured. HumanEval's problems are years old and likely part of the training corpora. The lower performance on SC-Python-2555 -2500 benchmark guaranteed to be stronll indicates that our evaluation measures model's true generalization ability on novel algorithmic challenges, rather than its capacity for pattern memorization. This underscores the critical need for dynamic, contamination-resiste benchmarks to ensure a fair and realistic assessment of an LLM's problem-solving intelligence.
Figure 4: Performance on Evalplus and CODE2BENCH-2509
RELATED WORK
Code Generation Benchmarks Evaluating Large Language Models (LLMs) on code generation
tasks is an active research area, with numerous benchmarks proposed. Early benchmark:Cassaun
ct al. (2023 Li et al. (2022) like HumanEval Chen et al. (202I). MBPP Austin et al. (2021). and
APPS Hendrycks et al. (2021) provide static collections of isolated code snippets with tests, proving
valuable for initial model development but facing limitations regarding dataset contamination Carlini
ct al. (202I); Sainz et al. (2023); Yang et al. (2023); Team et al. (2024) and the lack real-world context and dependencies. Efforts like EvalPlus Liu et al. (2023a) enhance static benchmarks with more robust tests via mutation. However, these may not fully represent real-world code complexities or provide automated construction from code repositories. Multilingual benchmarks Yan al. (2023) like HumanEval-X /Zheng et al. (2023b), Aider polyglot benchmark pool ButoCodeBennh Chou et al. (025) evaluate cross-language abilities but are at risk of data leakage. Benchmarks focusing on repository-
context or dependencies Tang et al. (2023); Li et al. (2024h); Yu et al. (2024); Wang et al. (2024) include RepoBench Liu al. (2023b), CrossCodeEvall Ding et (202)) R2E
Jain et al. ( (2024b), DevEval et al. (2024c). Big Codeeeeeec Zhuo et al. (2024). and CODEAGENT Zhang al. (2024). WebBench|Xun Xu al. (2025) introduces sequential. real-world web development tasks. While these capture aspects of real-world interaction, they often lack sufficient testing.
BENE2BENCH stands out by offering an end-to-end pipeline for dynamically generating rigorous benchmark instances from recent real-world GitHub repositories.
key limitation of static benchmarks is the risk of test set contamination, where models may have been trained on the same data used for evaluatiorCarlin) et al. (2021) Sainz et al. 2023);; Yang et al. (2023; Team et al. (2024). This has motivated the development of "live" benchmarks. DynaBench Kiela et al. (2021) identified these challenges and advocated for continuously evolving benchmarks. Chatbot Arena Chiang et al. (2024) provides a platform for dynamic evaluation based
Data Leakage and Live Benchmarks
interactions. LiveBench White et al. (2024) sources new data from specific, non-code domains like mathematics and news, while LiveCodeBench Jain et al. (2024a) collects recent problems from competitive programming platforms. Other methods leverage L.LMs

---

Published as a conference paper at ICLR 2026
for task mutation to generate problems, such EvoEval Xia al. (202). Evocodebench Li al. (2024b) introduces a periodically updated benchmark to mitigate leakage, while DOMAINEVAL Zhu et al. (2025) employs dynamic data sources with automated updates for the same purpose. Both efforts currently focus on Python and may lack rigorous test. Areta-Hard-Auto Li et al. (2024d) filters crowdsourced prompts into benchmarks. While existing initiatives have made progress in addressing contamination and enabling dynamic evaluation, they often rely on specific data sources, focus on evaluation platforms, or lack systematic, automated pipeline for constructing high-quality benchmarks from real-world code at scale. CODE2 BENCH bridges this gap by providing a novel, automated, end-to-end pipeline that dynamically extracts, filters, and constructs rigorous benchmark.
CONCLUSION & FUTURE WORK
We introduced Dual Scaling, new philosophy for benchmark construction, and presented CODE2BENCH, a framework that operationalizes it by systematically scaling the source of problems from real-world code and the rigor of tests via high-coverage PBT. Our evaluation on the result- ing CCDE2BENCH -0NCCH--5999 benchmark provided deep, diagnostic analysis of modern code LLMs, revealing consistent gap between their API application (WSC) and algorithmic synthesis (SC) capabilities, and quantifying for the first time how language paradigms shape their failure modes.
Limitations and Future Work. Our work, while establishing a robust framework for evaluating
functional correctness, opens several avenues for future expansion. We plan extend our Scaling the Source principle to repository-level, Project-Dege Direect-eepenentt (PD) tasks to assess codebase understanding. Concurrently, we will expand our Scaling the Rigor principle to incorporate non-functional properties such code efficiency and security. By evolving along these axes, CODE2BENCH will continue to provide a challenging and realistic measure true software engineering competence.
ACKNOWLEDGEMENT
This work was partly supported by Natural National Science Foundation of China under Grant No. 62472017, and partly by Guangxi Collaborative Innovation Center of Multi-source Information Integration and Intelligent Processing.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the full reproducibility of our work. The complete CODE2BENCH- 2509 benchmark suite, including all task instructions, ground-truth solutions, and PBTT-ennrated test suites scripts, is also included. The project, including all data and results, is also available at our anonymized repository: https::/codeebeneeee www.eeeebbencggithu.bbb. github.
REFERENCES
URL htt.t:/sss/aaaaaaeee...////////r
URL htthttpss:// http:psss/wwwwwuusss.oo
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language 2021.
models. arXiv preprint
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp. 2633-2650, 2021.
10

---

Published as a conference paper at ICLR 2026
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costinn Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions Soffware Engineering, 49(7):36753361, 2023.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. In Forty-first International Conference Machine Learning, 2024.
Jason Chou, Ao Liu, Yuchi Deng, Zhiying Zeng, Tao Zhang, Haotian Zhu, Jianwei Cai, Yue Mao, Chenchen Zhang, Lingyun Tan, et al. Autocodebench: Large language models are automatic code benchmark generators. arXiv preprint arXiv:2508.09100 2025.
Koen Claessen and John Hughes. Quickcheck: a lightweight tool for random testing of haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, pp. 268-279, 2000.
Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, et al. N1-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02721, 2021.
Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36:46701-46723, 2023.
Linyuan Gong, Sida Wang, Mostafa Elhoushi, and Alvin Cheung. Evaluation of llms on syntax-aware code fillidethhemmidlee tasks. arXiv preprint arXiv:2403.0483 2024.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. Xiv preprint arXiv:2105.. 09938, 2021.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXivv2403.07974 2024a.
Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and lon Stoica. R2e: Turning any github repository into programming agent environment. In Forty-first International Conference on Machine Learning. 2024b.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-hench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. arXiv preprint 1044 14337, 2021.
Sumith Kulal, Panupong Pasupat, Kartik Chandra. Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-baad based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposiun on Operating Systems Principles, 2023.
11

---

Published as a conference paper at ICLR 2026
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi. Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of alm-as-a-judge. arXiv preprint arXiv:2411.16599 2024a.
Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Evocodebench: An evolving code generation benchmark with domain-speciffe evaluations. Advances in Neural Information Processing Systems, 37:57619-57641, 2024b.
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et al. Deveval: A manually- annoottedd code generation benchmark aligned with wal-world code repositories. arXiv preprint arXiv:2405. 19856, 2024c.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024d.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):
2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 233 Red Hook, NY, USA, 2023a. Curran Associates Inc.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03009, 2023b.
Thomas J McCabe. A complexity measure. IEEE Transactions on software Engineering, (4):308-320, 1976.
Pierre Néron, Andrew Tolmach, Eelco Visser, and Guido Wachsmuth. A theory of name resolution. In Programming Languages and Systems: 24th European Symposium on Programming. ESOP 2015, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2015, London, UK, April 11-18, 2015, Proceedings 24, pp. 205-231. Springer, 2015.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Oscar Sainz, Jon Ander Campos, Iker García-Ferrene Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. arXiv preprini 1rXiv:2310.1801 2023.
Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng. Helan Hu, Kaikai An, Ruijun Huang, et al. MI-bench: Evaluating large language models and agents for machine learning tasks on repository-level code. arXiv preprint arXiv:231 1223311.09835 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055: 2024.
tree sitter. "an incremental parsing system for programming tools", 2024. URL
https
7/tree- site-sstters github.ito/ Accessed: 2024.
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang. Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212. 10264, 2022.
Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, and Dacheng Tao. Oop: Object-oriented programming evaluation benchmark for large language models. arXiv preprint arXiv:2401.06622 2024.
12

---

Published as a conference paper at ICLR 2026
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406.
2024.
Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. Top leaderboard ranking= top coding profi- ciency, always? evoeval: Evolving coding benchmarks via Ilm. arXiv preprini arXiv:2403.19) 19114, 2024.
Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. Web-bench: A Ilm code benchmark based on web standards and frameworks, 2025, /tt.tttsss///art...rr////////// wttp:/wwwwww..g./////..a////////
07473
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang. Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, et al. Codescope: An execution-based multilingual multitask multidimensional benchmark for evaluating Ilms code understanding and generation. arXiv preprint 1tt::2311.088500 2023.
Shuo Yang, Wei Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850 2023.
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li. Qianxiang Wang, and Tao Xie. Codereval: A benchmark pragmatic code generation with generative pre- trained models. In Proceedings of the 46th IEEE/ACM International Conference Software
Engineering, pp. 1-12, 2024.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as-a- llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023a.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual bench- marking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5673-5684, 2023b.
Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Shing-Chi Cheung. Domaineval: An auto-constructed benchmark for multi-domain code generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 26148-26156. 2025.
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877. 2024.
13

---

Published as a conference paper at ICLR 2026
A
PREPROCESSING AND DATA STRUCTURING
This appendix details the preprocessing and data structuring steps performed in the Function Filtering pipeline (Section 2) to transform raw source code into standardized representation suitable for subsequent analysis.
A.1 SOURCE CODE PARSING VIA ABSTRACT SYNTAX TREES (ASTS)
Following the initial identification candidate functions from the source code repositories, the raw text code of these functions and their surrounding context is processed using Tree-sitter tree sitter (2024). Tree-sitter a parser generator tool that produces concrete syntax tree parsers for various programming languages. Unlike traditional compilers that focus semantic analysis, Tree-sitter is specifically designed for source code analysis tools, providing robust, incremental parsing capabilities and generating detailed, well-structured Concrete Syntax Trees (CSTs), which closely related to Abstract Syntax Trees (ASTs).
For each identified function, the corresponding source code snippet is parsed into its AST representa- tion. The AST is a tree structure that represents the abstract syntactic structure of source code written programming language. Each node the tree denotes construct occurring in the source code (e.g., function definition, variable declaration, expression, statement). Parsing the raw code into an AST is crucial as
Standardizes the code representation, abstracting away syntactic variations (e.g.. whitespace, comments) and providing a consistent structure regardless of the original formatting.
Exposes the hierarchical and relational structure of the code, making it amenable to systematic program analysis techniques.
A.2 EXTRACTION OF RELATIONAL INFORMATION
From the generated ASTs, we extract essential relational information and metadata for each candidate function. This extracted information is crucial for the subsequent dependency analysis, program analysis, semantic filtering, and benchmark instance construction stages. Key information extracted includes:
Function Signature:
The function name, parameters, and their declared types type hints (if available). This information is directly used for generating the benchmark instruction.
Source Code Snippet: The exact lines of code corresponding to the function definition, serving as the Ground Truth implementation and the basis for program analysis.
Import Statements: Identification of modules or names imported within the function's 's scope or in its surrounding file. This is vital for understanding potential external dependencies.
Metadata: Information such as the original file path and commit hash, aiding in traceability and ensuring the recency of the code source.
This structured extraction process transforms the raw, unstructured code into a queryable and analyz- able format, laying the foundation for the automated benchmark construction pipeline.
14

---

Published as a conference paper at ICLR 2026
B SCOPE GRAPH BASED DEPENDENCY ANALYSIS
This appendix provides formalized technical explanation of the Scope Graph-based dependency analysis employed in the Function Filtering stage of the CODE2BENCH. This analysis is fundamental to identifying and controlling external dependencies of candidate functions, enabling the classification of tasks into Ceff-Contained (SC) and Weakly Self-Contained (WSC) categories crucial for rigorous and standardized evaluation.
B. SCOPE GRAPH MODEL
After parsing the source code of a project into Abstract Syntax Trees (ASTs) using Tree-sitter, we construct a Scope Graph G (V, E). The vertex set V includes:
Scope nodes V, representing hierarchical scopes like modules, classes, functions, blocks. Each scope E S represents region of code where identifiers are defined and resolved.
Definition nodes D C V, representing the points where identifiers are defined (c.g., variable declarations, function definitions). Each definition d corresponds specific identifier name id(d).
Reference nodes R C V, representing the points where identifiers are used (referenced) within the code. Each reference r R corresponds specific identifier name id(r).
The edge set E includes:
Scope hierarchy edges Escope C S X S, representing nested scopes (e.g., function scope nested within a class scope).
Definition edges Edef C S D, representing that definition d is contained within a scope s.
Reference edges Eref S S X Rt representing that a reference r occurs within a scope s.
Binding edges Evind C D, representing that reference r resolves to a definition according to the language's scoping rules. These edges are established during the resolution process.
Thus, V = SUDUR R and E = Escope U Edes U Ere, U Evind-
B.2 DEPENDENCY RESOLUTION PROCESS
For each function candidate F, we identify the set of all identifier references Rp C R occurring within its body scope sp. For each reference r Rp, the dependency resolution process attempts to find a corresponding definition d E D such that a binding edge (r, d) E Ebind can be established by traversing the Scope Graph outwards from 8F according the language's lexical scoping rules.
A reference E Ry is classified as an unresolved reference if no definition E D is found within the analysis scope of G' that can legally bind to. Let Up S RF be the of all unresolved references for function F. These unresolved references UF represent the external dependencies of function F.
The Scope Graph approach offers conceptual language-agnosticisne as the graph structure and resolution mechanism are based universal programming concepts (scopes, bindings), abstracted from specific syntax by the AST input from Tree-sitter. Language--specfff scoping rules are encoded in how the resolution traversal and binding edges Evind are determined.
B.3 SC/WSC CLASSIFICATION BASED ON DEPENDENCIES
Based on the set of unresolved references UF and the function's import statements, classify function F:
Let Latlowed be the predefined set of allowed common external libraries. For each unresolved reference r E UF, we attempt to determine its origin based on the function's 's import statements and knowledge of library APIs. A reference is considered resolved to allowed library I Lallowed if its name id(r) corresponds to an identifier provided by library I, and library is correctly imported or accessible within the scope of function F.
15

---

Published as a conference paper at ICLR 2026
Let Fllloowed C Up be the subset of unresolved references that resolve to identifiers within libraries in Lallowed-
Self-Contained (SC): Function F is classified as SC and only if its set of unresolved references is empty, i.e., Ur = 0. This means all identifiers defined locally language built-ins.
Weakly Seff-Contained (WSC): Function F is classified WSC Up of 0 and all unresolved references resolve to allowed libraries, i.c., UF = Flllooeed
Discarded: Function F discarded if Up * and U,latowed & Up. This means there are unresolved references that are not from allowed libraries.
This systematic, formalized approach, leveraging the Scope Graph representation, provides precise and robust method for controlling dependencies and classifying functions, which is essential for the reliability and scalability of the CODE2BENCH
16

---

Published as a conference paper at ICLR 2026
C PROGRAM ANALYSIS FOR TESTABILITY AND COMPLEXITY
This appendix provides further details on the program analysis techniques employed in the Function Filtering stage (Section 2 to ensure candidate functions are functionally testable and represent non-trivial coding challenges. Our analysis focuses on properties derived from the Control Flow Graph (CFG) and the structural complexity of the function implementation.
C.1 CONTROL FLOW ANALYSIS FOR TESTABILITY
To identify functions amenable to automated testing and output verification, we perform Control Flow Graph (CFG) analysis. For a given candidate function f. its CFG represents all possible execution paths through the function's 's code. Nodes in the CFG correspond to basic blocks code (sequences of instructions executed sequentially), and directed edges represent potential control transfers between these blocks (e.g., branches, loops, function calls).
We analyze the structure of the CFG filter out functions that inherently lack verifiable output. Specifically, we identify functions where:
The CFG contains no paths leading return statement. Such functions typically perform actions (e.g., printing to console, modifying global state) without providing value that can be casily captured and compared against expected output in an automated differential testing setup.
All paths leading to a rrturn urn statement return only constant values, or values derived solely from constants without any dependency input parameters complex intermediate computations. While these functions have a return value, their behavior is trivial and does not require probing with diverse inputs.
Functions matching these criteria are excluded from the candidate pool, as they are either difficult to test functionally in isolation or do not represent meaningful code generation tasks for evaluating LLM capabilities beyond simple retricval.
C.2 COMPLEXITY ASSESSMENT VIA CYCLOMATIC COMPLEXITY
To focus the benchmark tasks that require models to generate non-trivial code logic, we assess structural complexity of each candidate function using Cyclomatic Complexity (CC) Cyc Cyclomatic Complexity is quantitative measure of the number of linearly independent paths through a program's 's source code. is calculated based on the CFG the function using the formula:
= +P
where:
is the number of edges in the CFG.
N is the number of nodes in the CFG.
P is the number of connected components in the CFG (for a single function, P 1).
Therefore, for single function, CC' E' - N + 2. CO is strongly correlated with the number of decision points (e.g., if, while, for, case) in the code, providing an estimation of the code's logical complexity.
We employ CC as a filter to select functions that fall within a desirable complexity range, avoiding tasks that either too simple be challenging or excessively complex to be solvable reliably testable as isolated henchmark instances. Specifically, we define a range [COmin: CC (e.g.. [2, 10) and include only functions whose calculated CC falls within this range.
Functions with CC < CCmin (e.g., CC < 2) typically represent very simple linear code or trivial control flow structures that offer little challenge.
Functions with CC > CCmar (e.g., CC > 10) may indicate highly complex logic, potentially involving deep nesting, numerous branches, or intricate loops, which can be challenging for LLMs to generate correctly and for automated tests to cover exhaustively. Furthermore, such high complexity in real-world function might often be coupled with complex, uncontrolled dependencies.
17

---

Published as a conference paper at ICLR 2026
D SEMANTIC FILTERING AND DIFFICULTY ASSESSMENT
This appendix provides additional technical details regarding the LLM-based Semantic Filtering process used in the Candidate Filtering stage to refine the candidate function set through semantic assessment. While the main text outlines the motivation and overall structure, here we present in detail the prompts employed during this stage of semantic filtering.
To validate the robustness and objectivity of our LLM-based filtering, we conducted two inter-rater reliability studies. Inter-LLM Agreement. We measured the agreement between three diverse, state-of-the-art L.M judges (GPT-44,, Claude-4-Sonnet, and Qwen3-Max) 100 function candidates. For the binary task of semantic filtering (trivial vs. meaningful), the judges achieved Fleiss' Kappa of 0.92, indicating almost perfect agreement, Human-LLM Agreement. We further compared the judgments of our primary LLM judge (GPT-4o) against a consensus gold standard established by two human experts with extensive programming experience. On set of 50 functions, the analysis yielded a Cohen's 's Kappa of 0.95.
random sample of
Table 5: Prompt Template for Self-Contained Ground Truth Filter in CODE2BENCH
# System
## Task Description
You are an expert in the field of coding, tasked with determining whether a given Python function suitable for generating instruction (question).
The function will be analyzed based on its characteristics, functionality, and adherence to specific criteria. If the function meets the criteria, it is deemed suitable; otherwise, is not. ## Criteria for Suitability To determine whether a function is suitable for generating an instruc- tion, consider the following criteria:
### 1. Function Parameters
**Basic Types Only** The function's parameters must be basic types (e.g., 'int', 'float', 'str', 'dict', etc.)...
### 2. Function Complexity
**Meaningful Complexity The function should provide meaningful test of the model's capabilities...
### 3. Side Effects and Dependencies
"*oo Side *; The function should not have side effects (e.g., modifying global variables, writing files, etc.).
** External Imports**: The function should not import other modules or depend on external
libraries...
## Output Format If the function is **suitable** return:
"jjson
"Suitable": true,
"Reason": "The function meets all criteria for generating instruction."
the function not suitable, return:
## Examples
## Note
Ensure that your analysis thorough and considers all aspects of the function.
Provide clear and concise reasoning for your decision. Only return the Json.
# User
Please check the last result:
[aast Result]
Error response:
[Error Response]
[Function Message]
18

---

Published as a conference paper at ICLR 2026
Table 6: Prompt Template for Weakly Self-Contained Ground- Truth Filter in CODE2BENCH
# System
You are an expert in Python coding, tasked with determining whether given Python function meets the requirements below for generating benchmark. The function is weakly self-contained, meaning depends only on Python standard libraries or specific external libraries (e.g., numpy, re, pandas) and no custom modules. You will analyze the function based its characteristies, functionality, and adherence to specific criteria. the function meets the criteria, is deemed suitable; otherwise, it is not.
## Criteria for Suitability To determine whether function is suitable, consider the following criteria:
### 1. Function Parameters
Basaic and Library Types**: Parameters must be basic Python types (e.g., 'int', 'float', 'str', ) or types from standard/external libraries (e.g., numpy.ndarray', 're.Pattem"). If the parameters' type is missing, but you can infer from the code, is **suitable*** If the function relies on methods attributes of unknown objects, it is **not suitable**, But if the function uses lib types (e.g., 'numpy.ndarray'. 'pandas. DataFrame") ") it is **suitable**,
### 2. Function Complexity **Meaningful Complexity**; The function should provide a meaningful test of the model's capabilities, with clear logic and purpose.
the function overly long but trivial (e.g.. repetitive assignments), is *ono suitable**. If the function is too simple (e.g., basic getter/setter), is **not suitable**. ### 3. Domain Knowledge ** **Geneaal Applicability**: The function should not require highly specialized domain knowledge
to understand or implement..
### 4. Property-Based Testing
**onstructiblee Inputs**; The function should allow generating random inputs for property- based testing to verify its behavior.
## Output Format
the function #$suitable**, return:
return:
"'jonn
"Suitable": true.
"Reason": "The reason why the function meets all criteria for generating a benchmark."
the function not suitable, return:
"'json
"Suitable": false,
"Reason": "The reason why the function is not suitable for generating benchmark."
## Examples
## Note
Provide clear and concise reasoning for the decision.
the function meets our standards but is missing imports, it still suitable.
If the function only uses 'typing" for type hints, it is not suitable.
Only return the JSON output in the specified format.
# User
Please check the last result:
[ast Result]
Error response:
[Error Response]
[Function Message]
19

---

Published as a conference paper at ICLR 2026
Table 7: Prompt Template for Weakly Self-Contained Difficulty Assessment in CODE2BENCH
# System
## Task Description
You are an expert code analyst tasked with assessing the difficulty level of weakly self-contained Python function. A weakly self-contained function depends only on Python standard libraries or specific external libraries (e.g., NumPy, pandas, re) and custom modules. Assume the function is valid and suitable for analysis. Assign a difficulty level of "Easy", "Medium", or "Hard" based on the complexity of its logic, structure, required concepts. and cognitive load understand. ## Criteria for Difficulty Assessment
### Easy Difficulty **Logic**: Very simple, minimal or no branching. single loop or direct
parameter use.
**Structure** Short, linear, immediately clear control flow.
**Concept***; Basic Python constructs (variables, operators, lists, strings) simple library calls (e.g.. Counter from collections, basic re matching, pandas filtering). **Cognitive Load**; Minimal; purpose and execution are obvious at glance.
**Exmple***
"python
from collections import Counter
def count_ word frequencies(text: str) dict[str, int]: """Count the frequency of each word in a text string. Args:
text: Input string containing words.
Returns:
Dictionary mapping words to their frequency.
words = text. (ext.lower(.split))
return dict( (ountte((((()))))
### Medium Difficulty- Logic: Moderate complexity, with loops, conditions, or data transforma-
tions (e.g.. filtering, sorting, deduplication).
Structure: Traceable control flow, possibly nested loops or multiple steps, moderate length.
Example:
Hard Difficulty
Logic: Complex, with nested loops, intricate transformations, non-trivial algorithms (e.g., multi- dim aggregation, complex grouping), subtle edge cases. Structure: Dense or multi-step control flow, significant state management.
Example:
## Output Format Return ONLY JSON object containing the assessed difficulty level: "'json
"Difficulty": "Easy/Mcdium/Hard"
## Note
weakly self-contained function depends only Python standard libraries or specific external libraries (e.g., NumPy, pandas, re) and no custom modules.
Focus logic and structure, not the library's complexity (e.g., simple Counter usage is Easy, complex NumPy array ops are Hard).
Analyze the function's 's code, docstring, and logic thoroughly.
Only return the JSON output the specified format.
# User
[Function Message]
20

---

Published as a conference paper at ICLR 2026
Table 8: Prompt Template for Self-Contained Difficulty Assessment in CODE2BENCH
# System
## Task Description
You expert code analyst. Your task is the difficulty level of the provided Python function. Assume the function is generally valid and suitable for analysis. Assign difficulty level of "Easy". "Medium". or "Hard" based on the complexity of its logic, structure, required concepts, and cognitive load to understand.
## Criteria for Difficulty Assessment
### Easy Difficulty
Logic: Straightforward, minimal branching (simple if/else), possibly a single simple loop. Direct
of parameters.
Structure: Typically short, linear control flow. Easy to follow step- ste-bb--tep..
Concepts: Relies fundamental programming constructs (variables, basic operators, standard data types, simple function calls).
Cognitive Load: Low; the function's purpose and execution immediately apparent. Example:
"python
def parse_ message(messagee str) -> str:
message is None:
return ""
message= = message.strije stt.sttrriirrr((((
# Simple string checks and manipulations
not
"run_slov"" slo_ssloww "run slow")):
return
message = message(len("run ("run slow")
while httaggge.ttripOssstttttitttt
message = message.strip([[:]
return message
### Hard Difficulty
Logie: Moderate complexity. May involve nested loops, multiple non-trivial conditions, manipu- lation of data structures (e.g., iterating through lists/dicts with transformations), implementing common simple algorithm, or tracking state across iterations.
Structure: Control flow is more involved but still reasonably traceable. Function length might be moderate.
Example:
### Hard Difficulty
Complex logic. Might involve recursion, implementing non-trivial algorithms. Structure: Can have nested structures, complex control flow, significant state management, or rely on clever interactions between code parts. May not be long but could be dense.
Logic:
Example:
## Output Format
Return ONLY a JSON object containing the assessed difficulty level:
"Difficulty": "Easy/Medium/Han"
# User
[Function Message]
21

---

Published as a conference paper at ICLR 2026
E PROPERTY-BASED TESTING
This appendix provides additional technical details regarding the Property-Based Testing (PBT) process generate rigorous test cases for CODE2BENCH, While it describes the core principles. here we elaborate on the technical implementation.
Our PBT approach is centered around defining strategies for generating diverse. valid inputs for given function and verifying core property against the ground truth implementation.
E.1 STRATEGY BUILDING AND INPUT SYNTHESIS
The Strategy Builder component analyzes the function's 's signature, parameter types, and inferred constraints from type hints, docstrings. and static analysis of the ground truth code. then leverages a PBT library (c.g., Hypothesis for Python) to compose generation strategies for each function parameter. These strategies are designed explore a wide range of valid inputs, including typical values, edge cases (e.g., empty lists, zero, maximum/minimg input types within complex structures (e.g., lists of dictionaries, tuples of specific types). The process is constraint-aware, ensuring generated inputs adhere to inferred conditions.
values), and combinations of different
For example, for function taking list of integers 'def process(data list(int))). the strategy might generate lists of varying lengths, containing both positive and negative integers, zeros, and potentially boundary values like 'sys.maxint' For a function taking a string with specific format requirements, the strategy would be built to generate strings adhering to that format.
E.2 PROPERTY DEFINITION AND VERIFICATION
The core property we verify for generated code is functional equivalence with the ground truth implementation. For every input z; generated by the strategies, we compute the expected output V: far (2:), where for is the ground truth function. The PBT framework then requires that for any generated input 2;, the output of the generated code fLLM (2;) must equal yi- Any input x; where fLLM((x,) * y; constitutes test case failure, and the PBT framework can then attempt to "shrink" x; to find minimal failing input.
E.3 ENSURING TEST RIGOR AND COVERAGE
To ensure the generated test suites are truly rigorous, we incorporate a quality control step based on test coverage. After generating suite of (æ;, 34) pairs using PBT strategies, execute these test cases against the ground truth implementation itself. We use code coverage tools (e.g.. "coverage-py" for Python) measure the branch coverage achieved by the generated test suite the ground truth code. Only test suites that achieve high coverage threshold (c.g.. 100% average branch coverage) are accepted and included in the final benchmark instance. This filtering step is crucial: even if strategy can generate many inputs, if those inputs don't 't exercise the complex branching logic of the function, the resulting test suite is not rigorous enough effectively verify model implementations.
E.4 CODE EXAMPLE (PYTHON/HYPOTHESIS)
Below is simplified illustrative example using the Python Hypothesis library to demonstrate how strategies and properties are defined.
import hypothesis. strategies from hypothesis import given, settings, Verbosity
1. Define strategies for input generation Strategy for generating arbitrary strings string_strategy - textext ()
def ground_trutbrreeersess (s: str) -> str: return s[::-1]
22

---

Published as a conference paper at ICLR 2026
def 11m generated reverse str) -> str:
Hypothetical LLM output, might have bugs return "" join (reversed (s))
Define a property using @given decorator lgiven (s=string_strategy) $ Use the defined strategy to generate input
'g'
@settings (max examples=1000) # Example settings for testing
def cest_revers_propertt(())..
Property: Reversing twice returns the original string
assert reverse_string seevcrse_strin(se) = s This property tests the function against itself
Property: LIM output matches Ground Truth output expected_cutply
(ronn__stth_rreersssss)
actual _ociial__upuutttt 11m gem_ggeerateedeem.o reversee assert actual_outpure expected_output 8 This property tests LiM output against GT
This example illustrates the basic concept of defining strategies for input generation and asserting properties about the function's behavior for these inputs. In the CODE2BENCH pipeline, these principles are automated and scaled to generate comprehensive test suites for a wide variety of functions extracted from real-world code.
23

---

Published as a conference paper at ICLR 2026
F TESTCASE RUNNER GENERATION
This appendix provides a detailed presentation of the prompts used for generating test runners (Section 2). Since test runners are inherently language-specifie, we designed tailored prompts for each programming language. The corresponding prompts are listed in Tables 9 through TO
Table 9: Prompt Template for SC-Java Testcase Runner Generation in CODE2BENCH
# System
## Task Description
As an expert Java developer specializing in test case generation and function signature translation, your task is generate Java test file and function signature based on Python function and its test cases. The test file will load test cases from JSON file and execute tests using provided Helper. method.
### Requirements
*ava Test File**;
Generate complete, executable Java test file in the 'p0' package.
**Function Signature**:
Provide only the function signature in the 'pO.Tested' class.
3. **Special C
**yppe Safety**;
Ensure "TestCase" fields exactly match JSON keys and types (e.g., 'lines" - 'List<String>', dine_index: > 'int').
**Type Definition Rules**
Follow these rules to determine where to define types:
Usage Scenario Location Example
Used function signature 'tested.java" 'public static class TagInfo ()*
Used in both 'tested.java'
Sharre types always in implementation!
## Input Format
**Test Cases JSON**; A JSON array of cases, provided "Iesttcaeessstr"
**Pyhhon Function**: A Python function to be tested, including its signature and implementation, provided as code.
## Output Format "plaintext
<code>
[Java test file]
</code>
<signature>
[Java function signature]
</signature>
## Examples
## Note
Ensure the generated Java test file is complete and executable.
The function signature should be valid Java function signature that matches the Python function's 'ss behavior.
Only return the Java test file '<code>' and the function signature in '<signature>' tags.
# User
The previously generated runner code:
[Runner Code]
The previously generated runner code resulted in the following error during execution:
[Error Message]
[Function Message]
24

---

Published as a conference paper at ICLR 2026
Table 10: Prompt Template for WSC-Python Testcase Runner Generation in CODE2BENCH
# System
## Task Deseription
You are expert Python developer specializing in property based testing and test case execution. Your task generate * **complete and executable Python script** that loads test cases from a JSON file generated by Hypothesis-based Testcase Generator and re-runs the test logic to verify the behavior of function under test ('funcl') against ground truth function ('func0"). The functions depend only standard libraries specific external libraries (e.g., NumPy, re) and no other custom modules. The script will:
1. Load test cases from the JSON file
containing 500 test cases, each with "Imputs" ". dictionary mapping 'func0"ss argument names to serilaziil values.
2. Re-run the test logic by calling 'func0" (ground truth) and 'funel" (under test) with the loaded inputs, comparing their outputs via differential testing.
3. Compare outputs using:
For basic types and their combinations ('int', 'float', 'str', 'list', 'dict', etc.), use deee__comperd from the 'helper' module.
For third-pan party library types (e.g., 'numpy.ndarray'. 'tuple" of humpy.ndarray provided comparison functions (e.g.. 'np.allclose") or custom logic if none is provided.
use library-
4. Report test results, indicating whether each test case passes or fails, with detailed failure information (inputs, expected output, actual output).
## Input The input is Python Testcase Generator script that includes:
The ground truth function 'funcO', its dependencies (c.g.. 'numpy'. 're"), and implementation.
2. Hypothesis strategies and "@example" decorators defining input generation logic.
('tett<<function 3. test function ('test_ <function "iunction_name>") that generates and saves 500 test cases to
"test__cseeee cases.json', each containing only '"Inputs"
Provided in *<Testcase Generator>* tags.
## Output Generate complete, executable Python script that:
## Example
## Note Focus on Loading and Re-testing: Load test cases from test_cases.json and verify funel against funcO) using differential testing.
Preserve Input Format: Ensure inputs match funcO's signature, converting ISON-serialized inputs
(e.g., list mp.ndarray) as needed.
Output Comparison:
Use deep_ cespporrereeeee from helper for basic types and combinations (int, float, str, list, dict, etc.).
Use library-provided comparisons (e.g., np.allclose for numpy.ndaraa) for third-party library types, custom logic if none provided.
Executable Code: The script must be complete, self-contained, and executable.
Differential Testing: Since test cases contain only inputs, compute expected outputs by calling func0 and compare with funcl's 's outputs.
External Libraries: Include imports for funcO's 's dependencies (e.g., numpy, re) and handle their data types (e.g., np.ndarray).
# User
The previously generated runner code:
[Runner Code]
The previously generated runner code resulted in the following error during execution:
[Error Message]
[Function Message]
25

---

Published as a conference paper at ICLR 2026
G INSTRUCTION GENERATION
Table 11: Prompt Template for WSC-Python Instruction Generation in CODE2BENCH
System
You are python programming expert who is refining doestrings in existing programs. You will be given a python function in a python file with an existing (possibly underspecified) docstring with corresponding some input-output examples extracted.
Your goal is to refine the associated doestring by making it more informative, precise and complete without adding verbosity or detailed programming logic the docstring. When there a doestring, the docstring is used to evaluate the code generation capabilities of a model.
The docstring should particularly describe the format and types of the expected inputs and output as well as the behavior of the function. Do not guess outputs for functions. Finally, do not throw away existing details from the docstrings and only insert content you are sure about. Do NOT have repeated content in the docstring and ONLY describe the high-level function behavior without going into implementation details. ## Requirements
** Core Description Fidelity**: The docstring must accurately reflect the function's behavior and describes the task this function solves. ** P*y close attention to the sequence of checks, conditions, and resulting actions within the code.
** Highlight any ** special rules** that affect the model's correct understanding of the function's behavior, such as:
Recursive behavior
Merging, flattening, filtering, transformation logic
Edge cases or type-specific handling
Magic numbers or constants
**Docstring **finement**: the function already has a docstring, integrate and refine its content to meet these requirements. Do not discard existing accurate information.
**Conditional Example Handling**;
You should judge whether to include examples based on the original docstring's content:
**If the original docstring contains an 'Examples' section**;
Preserve all original examples ** **erbatim*** in the final doestring's 'Examples' section.
- Format them clearly in angguage-Agnostccce
showing input and expected output).
Do ** *o add or modify examples from the "Example Usages* data.
## Input "python
(ground teuthhtttthhhhhhhhh function_ ]
### Output Format
Return the doestring in <docstring> tags following the Google-style format. Include the function signature in <signature> tags, with TODO placeholder for the implementa- tion.
#### Example output:
## Note
Only add examples in Doestring when the function already has 'Examples' section in the docstring. Do not add examples from the 'Example Usages' data if the original docstring does not contain 'Examples.
- If the signature has type hints, import nessesary types from the standard library (e.g.. 'from typing import List, Dict') in signature.
# User
<function>
FFunction Code]
</function>
<example usages>
[Example Usage]
<lexample usages>
26

---

Published as a conference paper at ICLR 2026
Table 12: Prompt Template for SC-Python Instruction Generation in CODE2BENCH
# System
You are programming documentation architect specializing in creating precise, implementation- agnostic specifications. Generate a docstring that enables accurate reimplementation in any programming language. When there is a docstring, the docstring is used to evaluate the code generation capabilities of model.
## Requirements
** Core Description Fidelity**; The docstring must accurately reflect the function's ' behavior and describes the task this function solves. **Pay close attention to the sequence of checks, conditions, and resulting actions within the code.
2. Highlight any ** speecial rules** that affect the model's correct understanding of the function's behavior, such as:
Recursive behavior
Edge cases or type-specific handling
Magic numbers constants
Special settings that may affect the difficulty for others to correctly implement functions based doestrings. e.g., the Ground Truth function may add some special string at the end of the result, so the docstring should mention this case, otherwise, the model may not be able to implement the function correctly.
2. ** ****n*gaaae LLangugge-A - Terminology**: Use universal concepts for types and logic.
Describe parameters and return values using * "*concepuaa types** ** (e-g., "an integer", "a boolean value", "a sequence of numbers". "a text string") instead of language-specie "bool', 'list', 'str').
type hints ('int',
Describe operations conceptually (e.g., "checks if X contains Y", "iterates over the elements". "applics a function to each element") rather than Python built-ins ('s1. "ind."in""""""" 'sl 's..lellac''. 3. **Docstring Refinemen*** *# If the function already has a docstring, integrate and refine its content to meet these requirements. Do not discard existing accurate information.
4. **Conditional Example Handling**.
*f the original docstring contains an 'Examples' section**;
Preserve all original examples * **verbatim** in the final doestring's 'Examples' section...
## **nnut Structure**
"python
Iground truth pnaction_codee
### Example Usages:
[example_useer laces_usae__d___
(Note: Example Usages will be provided in format like input/output pairs.)
## **Output:
Only return the docstring content in <docstring> tags and the function signature in the <signature> tags. The docstring should be enclosed in triple double quotes "Docsttrnn goes here" """"). The function signature should be formatted in "'python' code block with the function name and TODO comment indicating where the implementation should go.
## Note:
Only add examples in Docstring when the function already has an 'Examples' section in the docstring. Do not add examples from the 'Example Usages' data if the original docstring does not contain 'Examples'.
the signature has type hints, import nessesary types from standard library (e.g.. 'from typing import List, Dict') in signature.
# User
<function>
Function Code]
</function>
<example usages>
[Example Usage]
<lexample usages>
27

---

Published as a conference paper at ICLR 2026
Table 13: Prompt Template for SC-Java Instruction Generation in CODE2BENCH
# System
You are an expert Java architect and technical writer, specializing in creating high-quality, profes- sional Javadoc documentation. Your task is to generate a precise and informative Javadoc for given Java method, enabling another senior Java developer to re-implement accurately within a complete, self-contained "Tested.java' file. ## Core Objective
The generated Javadoc and method signature must serve as perfect "specification" for the provided ground-truth method.
## Requirements
The Javadoc must accurately reflect the method's behavior, including the precise sequence of checks, conditions, and resulting actions within the code.
1. Core Description Fidelity:
2. Edge Cases: Detail how the method handles edge cases, such as null, empty, special/magic values.
3. Data Structures: Accurately describe parameters and return values using standard Java types.
4. Javadoc Refinement: If the method already has a Javadoc, your primary goal is to refine and enhance it to meet these high standards. Integrate existing accurate information with your new insights. Do not discard valuable details from the original author. Example Handling:
If the original Javadoc contains an example (e.g., in '<pre>@code .t.eccddd...pp........ ...///r>'' the original example verbatim.
block): Preserve
If not: Omit any example section entirely. ## Input Structure**
"java
(ground truth_jett java method_eooeeeeeeeee _
## Output Structure
Return single "<signature>' tag containing complete, self-contained, and runnable "Tested.java" file content. The content must be enclosed in "'java' code block and include all necessary imports, the generated Javadoc, the 'public class Tested', and the public static method signature with /// TODO: implement this method comment.
<signature>
"java
All necessary imports le.g..java.uut *) should be here.
import List:
import aava.util.Mapp
public class Tested
y**
High-quality Google-style Javadoc goes here.
*/
public static ReturnType methodNaamePPrrameerrlye parameterName)
TODO: implement this method
</signature>
## Final Instructions
Ensure the method signature in the "<signature>' tag perfectly matches the ground truth, including visibility ('public static"), return type, method name, and parameter types.
Don't 't add implementation details.
Include all necessary imports based on the types used in the signature.
# User
<function>
[Function Code]
</function>
28

---

Published as a conference paper at ICLR 2026
H BENCHMARK
H. BENCHMARK DETAILS
This appendix provides detailed breakdown of the construction process and diversity analysis for the CODE2BENCH-2555 suite. All data was sourced from public GitHub repositories with commits made between May 2025 and September 2025.
To ensure the diversity, quality, and representativeness of the source code used in CODE2BENCH, we adhered a strict, multi-stage data selection protocol. This appendix details the criteria and sampling strategies employed during the "Scaling the Source" phase.
We targeted high-quality, actively maintained, and well-tested open-source repositories hosted on GitHub. To be included in our initial candidate pool, repository must meet the following quantitative thresholds:
Community Validation: 2 500 Stars. This threshold filters out personal experiments and ensures baseline of community scrutiny and adoption.
Active Maintenance: At least one commit within the months prior to our data collection cutoff (May 2025). This ensures the code reflects modern coding practices and library versions.
Test Availability: Must contain an identifiable test suite (e.g.. presence of tests/ directory, usage of pyt pytest/junit). This is crucial for verifying our ground truth extraction.
License Permissibility: Must be under permissive licenses (e.g., MIT, Apache 2.0, BSD) to allow for redistribution and modification.
To mitigate noise and bias, we applied semantic filters to exclude repositories that do not represent real-world software engineering contexts:
Repositories with keywords like "assignment", "course", "tuto- rial", "learn-pythonnjava"" or "leetcode" in their description or README were excluded. These typically contain toy problems distinct from production -radde code.
Homework & Tutorials:
Aggregators & Forks: We excluded repositories identified as mere collections of other projects (e.g., "awesome-xxx") or direct forks without significant divergence, ensuring unique data sources.
To prevent domain bias (e.g., over-represennaation of web frameworks), employed stratified sampling strategy based on GitHub topics and repository tags. We categorized candidates into 10 primary domains covering the spectrum of software development:
1. Web Development (e.g., frameworks, clients)
2. Data Science & Machine Learning (e.g., analyties, pipelines)
3. System Utilities (e.g., cli tools, file manipulation)
4. Network & Protocol (e.g., async io, sockets)
5. Security Cryptography
6. Database & Storage
7. Text Processing NLP
8. Media & Graphies (e.g., image processing)
9. Scientific Computing
10. Development Tools (e.g., linters, parsers)
We performed random sampling within each stratum to build our final source list of Python and Java repositories.
To fully support reproducibility and auditability, the complete list of source repositories is available open-source project page: htttps://ccdeberrchc... thub.. io/
29

---

Published as a conference paper at ICLR 2026
H.2 THE DATA FUNNEL: FROM RAW FUNCTIONS TO GOLD STANDARD
The final size of our benchmark is a direct result of stringent, multi-stage filtering pipeline designed to prioritize quality, realism, and rigor. Table 14 illustrates this "Great Filter" process for the Python components, starting from over one million recently updated functions identified in 220 repositories. This rigorous process ensures that only the most suitable and high-quality candidates become part of the final benchmark.
Table 14:
The Data Funnel for CODE23ENCH-2500 Python components, illustrating the multi-stage
filtering process that prioritizes quality and rigor.
Stage
Filtering Action & Criteria
SC Candidates
WSC Candidates
Initial Pool
Functions parsed from commits
-1.17
Million
Dependency Filter
Scope Graph: Strictly SC WSC compliant
27,649
12,335
TestabilityCComllext
Testable outputs Cyclomatic Complexity [2.10]
7,102
3,278
4. Sub sab-samp
Breadth-firs sampling for diversity
901
432
5. Semantic Filter
LLM-as-a-judge removes trivial tasks
479
315
Great Filter
PBT: 100% Branch Coverage Guarantee
H.3 TASK DIVERSITY ANALYSIS
The high quality of CODE2BENCH-2500 rooted in its rich diversity of tasks and application domains.
H.3.1 SC-PYTHON: ALGORITHMIC AND REAL- WORLD LOGIC
The 217 tasks in the SC-P SC-Python component cover wide spectrum of real-world programming challenges beyond simple puzzles. Key functional categories include:
Text Processing Formatting: Ranging from LaTeX sanitization (strip_latex) complex wrapping for SVG elements (vrap_text_for_sve
Classic & Modern Algorithms: Includes fundamental algorithms like levenshtein and binary _earcch as well as logic relevant to modern development tools like parsing LCOV reports (parse_lcov)
Challenges models to parse structured data from unstructured text, such as extracting JSON from noisy strings Lee.tract_balanceee (parse_version).
Parsing& Extraction:
or parsing version numbers
AVLLLM-Specific Logic: A unique feature is the inclusion of tasks from the Al ecosystem, such as formatting LLM error messages (format 11m aerror_messaee and parsing model outputs from prom_value).
Complex Data Structure Manipulation: Requires deep understanding of nested structures (e.g.. #latten_state_diict deep_merge).
H.3.2 WSC-PYTHON: A BROAD AND REALISTIC API ECOSYSTEM
The 194 tasks in the WSC-Python component require the of over 35 distinct libraries and module ensuring faithful evaluation of a model's practical API fluency. The distribution is representative of real-world Python development, covering wide and diverse ecosystem rather than focusing on a narrow set of APIs. The required libraries span multiple key domains of software engineering:
Data Processing & Text Manipulation: significant portion of tasks involve core data han- dling using standard libraries like re, json, ast, datetime urllib, unicodedata. and base64.
This count hased unique top-level import statements, e.g. re, numpy, scipy acipy.se sp.ttiiall..
30

---

Published as a conference paper at ICLR 2026
Scientific Computing & Data Science: The benchmark probes capabilities in specialized numeri- cal and data-centric domains, requiring libraries such as numpy, scipy (including submodules like scipy. sesipy..ppaiiie ial. transform) and pandas.
Machine Learning: Uniquely, our suite includes tasks that interact with the machine learn- ing ecosystem, leveraging modules from scikit-learn like TfidfVectorize and
Advanced Standard Library Proficiency: Beyond common utilities, the tasks require deep knowledge of Python's standard library, including advanced modules such as iteroolss collections enn (e.g., Counter, def.lltitttl diffibb, bi bisect, and struct.
This broad and realistic library coverage ensures that WSC- Pytto provides holistic assessment of a model's 's ability to function as a practical coding assistant in diverse range of real-world scenarios.
H.3.3 SC-JAVA: VALIDATING EXTENSIBILITY WITH DIVERSE TASKS
The successful generation of the 249-task SC-Java suite provides concrete evidence of our frame- work's extensibility. The quality of this component is validated by its high diversity, which mirrors the real-world complexity found in its Python counterpart. The tasks span a wide array of application domains:
String Manipulation & Parsing: A large portion of tasks involve complex string operations, such as format conversion (convert (Case)) cleaning (cleanText), and escaping for different contexts (escapeJsonStringg ascapeCsvFielle
Encoding & Data Conversion: Numerous tasks focus on byte-level manipulation, primarily converting between byte arrays and hexadecimal strings (e.g., bytesToHex),, common task in systems programming and networking.
Mathematics & Algorithms: The suite includes non-trivial algorithms like checksum validation (uuhBBanCaarVVeril fy) and edit distance calculation (LevenshteinDistannce
Domain-Speciffe Logic: Crucially, the tasks are not generic puzzles but rooted in spe- cific application domains, including game development (e.g., Minecraft metadata transforma- tion, trans forMMet aDecaMMde and systems utilities (e.g., calculating CPU affinity masks, maskTocpuuffiiitty)
This demonstrates our framework's ability extract meaningful and realistic algorithmic challenges from any complex, real-world codebase, regardless of the programming language.
H.4 BENCHMARK TASK EXAMPLES
This appendix provides examples of representative benchmark instances from CODE2BENCH 2509. Each example showeases complete task, including the instruction provided to the Large Language Model (L..M), the ground truth implementation from which the task was derived, the Property-Bane Testing (PBT) script used for generating comprehensive test cases, and the test runner script for evaluating the LLM's generated code.
H.4.1 SC PYTHON EXAMPLE: MERGE JSC_URRRRRRRRVVEEEEEEE
This example demonstrates a Self-Contained (SC) task in Python, requiring the recursive merging of JSON-like objects without external dependencies beyond standard library features.
Task Instruction
def barre_jaco_recurre ivee (base, update): "**Kecursively merge two JSON- like objects.
The function merges nested structures with the following rules: If both inputs dictionaries, recursively merge them. both inputs are lists, concatonate them.
31

---

Published as a conference paper at ICLR 2026
For all other cases, the update value overwrites the base value. base object left unmodi fied; new merged object is returned.
Args:
base: Base JSON-like object fdictionary, list, or primitive value),
update: Update JSON-1 object to merge into base.
Returns:
A new JSON-like object containing merged content from base and update.
Examples:
Input: base ("a": 1), update ("a": 2)
Output: ("a": 21
Input: base [1, 2j, update = [3, 41
Output: [1, 2, 3, 4]
Input: base ("a": ("b"; 1)1, update ("a"; ("e"; 213 Output: ("a": ("b": 1, "c": 21)
TORO: implement this function pass
Testcase Generator
from hypothesis import settings, given, Verbosity, example from hypothesis import strategies as st
import json import
import atexit
import copy
Configuration
TEST WEST_CASE_DIE os pat.pahh.aaa ("tes_ah"ttst__ccasss"" os mak.dirs (TEST (TEST_CASE DIR, criis_cc--True
TEST FIIEE path. (TISTTTT_CASSE generated_cases stop_collecting False
"test_cascs. json")
Global flag to control case collection
Ground truth function
def verge_json_recursivee (base, update):
if not isinstance (base, dict) not isinstance (update, dict):
if isinstance (base, list) and isinstance (update, list);
return base update
return update
merged base.copy ()
for key, value in apdate.itemt ():
if key in merged:
merged[key] = merge__jacnreccrsivee (merged [key], value)
else:
[key] value
return merged
Strategy for JSON-like objects
json ctrn_sstreegyyyeee rt.ruuuur st.one_of
st st.integers(), st. allow_infindtyyFaass)))
32

---

Published as a conference paper at ICLR 2026
st text (st characters (whitelist (tteeeelsttctteepeeees ('L', 'N', 'P", '8", 12'))), st st.booleaas
1),
lambda children: st st.onf
st sistis 1 max_size=5),
st (N'),, max_size=5), children, max_size=5)
(st. characteesss i ist_caageooess---- (1''
, max_leaves-5
Hypothesis test configuration Esettings exanxxem=meese0000,,,,,,,, verbesity- verbose,, print_blab-True)
Bgiven beasse=json_sttateee def
update=json strategy)
(base, update)
global stop suep_collecting if step_collecting: return
base_copy copy. deepcopy (base)
upaate appy copy, deepcopy (update)
expected parge__sso__ecuurr vive (base_copy, update_copy)
if isinstance (base, (dict, list)) or isinstance (update, (dict, list)):
generated cases cases. (1
"Inputs"; ("base": base, "update": updatc), "Expected"; expected
1)
if 1en (earlggnnate_aaassee >- 500:
stop coop_collecting = True
Save test def save_test_cases/
with open (TEST _asSESST________ FILE, "x") as f: jaon. (gencrated_cascss, f, indent-2, ensurc_ascii-Falss print (f"v Saved (len (generated_cases test cases to (TEST_CASE_FILE)
atexit. www.stergiiiree..... (save dest_cases
Testcase Runner
import json
import os
from tested import merge_ison_recursive funel
from helper import deep compare
Configure save path TEST CEST_CASE_DIR os os. papppaaaapaaaaaa. ("tsst_cassss"
TEST_CASE__OE SSS__CASS__SON_PTT = os. os. path. jein (TEST_CASE_DIR, "test_cases..joo"
def foad_test_cases_ sarasss_rro_jssee from_json()) ():
if not os.path. exists exists (TEST CASE JSON PATH): print (E"JSON file not found: (TEST_CAS) (NSS__CAAK_USON_PTHH
return []
Read JSON file with open (TEET_TATT_SSSEEEE JSON__AL "r") as f: test_cases json. load (f)
return
33

---

n-gram and determines the proportion of n-grams that appear more Published as a conference paper at ICLR 2026
def run_tosts_withh_ooddddd
(test_cases):
for i, case in enumerate (test_ (teat_cases):
inputs case [ ["Inputs"]
outpt = case "Expected"]
Run function under test actual_output funcl (*+inputs)
$ Copy matrix to avoid in-place
modafication
Check if results match using deop_ compare
if not deep_compare, deep_conpare(cctuull_ouuuu (actual_output, expected_cutput, tolerance-le-6):
print (E"Test case 1) failed:")
print Inputs: (inputs)")
print Expected: (expected output)")
print (f" Actual: (actual_output)")
else:
print (f"Test case {i + 1} passed.")
if namc main
test_cases load_test cases___cces_____eeee saases_frem_jooo pun_tests with Loaded_casee cases (test_cases)
H.5 WSC PYTHINN EXAMPLE: CALCULATE_ NGRAM_REPETIIIN
This example demonstrates a Weakly Self-Contained (WSC) task in Python. requires interacting with function from the standard library DosleectionsCounter" calculate n-gram repetition in text.
Task Instruction
from collections import Counter
def caloulate__grran_ooppttttoo (text: (text: str, n: int) -> float:
Calculates the proportion of repeated n-grams in given text.
This function splits the input text into words and generates n-grams
of the speci fied size 'n'. It then computes the frequency of each
than once. If there are no n-grams (e.g., when the text is empty 'n' is larger than the number of words in the text), the
function returns
Args:
text (str): The input text to analyze, consisting of words separated by spaces.
(int]: The size of the n-grams generate (e.g., for bigrams, for trigrams).
Returns:
float:
The proportion of n-grams that are repeated in the text. Returns 0 if no n-grams be generated.
Raises:
VaIueRrror:
is less than or equal to 0.
TODO: implement this function
pass
Testcase Generator
34

---

Published as a conference paper at ICLR 2026
from hypothesis import settings, given, Verbosity, example
from hypothesis import strategics as st import json import os
import atexit
import copy
from collections import Counter
Configuration
TEST CASE_ DIR = os os pa...ahhhcmm t"testh
TEST_CASE_
os us.makcdirs (TEST_AASE_DIN, DIR, cxist_ck-Trus
TEST @eTr_AAS__FILA _ILE os. bath.joing CERRS__CAE__DIEEE
"test_cases.jjae json")
generated_cases
stop_collecting False
Global flag to control case col lection
Ground truth function def salculate_ngree repetition (text, n)
words = text split()
ngrams [tuple (words : n]) for in range (len (words) 1)]
ngram_counts Counter (ngrams)
total_ngramt len (ngrams)
repeated_ngrans sum for count in ngram s values if count
1)
return repeated_ngrans / total_ngrams if total_ngraat > else 0
Strategies for generating inputs def text strategyyt
return st.text(
Ulppabbe--tt.caaaaaet (whitelist_care mnnitelisttcategoriess- ('L', 'N', '2)),
non__odeepi, max_ cox_ccpppoin--12)) min_size=0, max_size=100
def ()strategyy);
return st. integers (min_value-1, max_value=5)
Hypothesis test configuration
@settings (max_exaaa) mma__exampees-1000 10000, verdesitty-lerroosttyy..eee
print_bleb--rrue)
egiven (tes==text_strateegg (), n=n_strategy
(example(text="")) n-1)
eexample n=1)
(teanppletext"aaaa s", n=2)
texample a b c c", n=2)
f", n=3)
[exaatleetextt"1
a", n=3)
def wtw.lallltetenggnoooeeeeeeeeeeee
global stop_collecting if ctop_ccoleettig: return
Deep copy inputs to avoid mocatination text_copy copy deepcopy (text)
n_copy copy deepoopy (n)
Call funco to verify input validity try:
expected = calculate mawullatengraam.me www.aanarraaaeaaaaaaaeaae www.eetieeetitiiiee (text_ (ope copy, b__ooy) except Exception:
return # skip inputs that cause exceptions
Store inputs only snnerated_cases.aape append (( "Inputs": "text": text_copy,
35

---

Published as a conference paper at ICLR 2026
"n": coy
})
Stop collecting after 500 cases if 1en (gnneggated_caaee >- 500:
stop_collecting = True
Save test def save_test_cases
with (TEST_CASE, "w") as f:
json.dung (generated_caaees, f, indent-2, ensure_ascii-Falss print Saved {len (generated_ test cases to [TRST_CASP_FTIB]]
(save_tstaaass ate.
atexit. reexit.registee ster see
Testcase Runner
import json
import
import copy
from collections import Counter
from helper import deep_compare
from tested import calculatee_nrran.rrpeiitt as funcl
Configure save path
TEST CASE_AAEE__RC o..path.c.... abspathbsathhcecoocc ("test_cases")
TEST_CASE_USON_PP
os. path. join (TEST_CASE "test_cases.jjoo""
Ground truth function def salculate__graa__eee rewatttteeetttinnnneeeoo (text, n)
words = text. split
ngrams [tuple (words [i + n]) for in range (len (words) 1)]
Counter (ngrams)
total_ngrams len (ngrams)
repeated_ngrams sum (1 for count in ngram_counts.. carra__coutss.vlluss () count
1)
return repeated_ngrans / total_ngrams if total_ngran > else 0
def load_test_cases, (c..asss_froo_ssoon (): if not cs os.path. exists extssssssssssss (TEST CASE JSON PATH):
print (S"JSON file not found: DOST__CASE_UOON_PATT
return [] with DeeeT_EEE_SE__OOOPPTTTT "r") as f;
test_cases json. 1oad (f)
return test_cases
def compare outputs (expected, actual): Use deep_compare for basic types (int, float, str, etc.) return deep_compare (expected, actual, tolerance=le-6)
def run run_tests_witte srathh_loaed__eee ceaaded_caseeddeeeee (test_cases) for i, case in enumerate (test_cases); inputs copy deepcopy (case (" Inpuuss" text inputs inputs("text") "text"] n inputs("n"
Run ground truth and function under test expected _otttedd = actual_output funel (text, n)
ingram_repetitionteee.o (text, n)
Compare outputs
if compare_outputs (expected_output, actual_output):
36

---

Published as a conference paper at ICLR 2026
print (E"Test
1) passed. ")
else:
print (E"Test case {i + 1) failed:")
print (f" Inputs: (inputs)")
print (f" Expected: (uspecttd_output)>"
print (f" Actual: [actual_output]""
if __mme___ -- "_main_". ::
test load test cases from isen run_tests_ aun.eeeesuutthooooeeeeeeeeeeeeee (tost_ccass))
37

---

Published as a conference paper at ICLR 2026
1 DETAILED EVALUATION
1.1 COMPUTE RESOURCES
detail the compute resources utilized for evaluating the Large Language Models the CODE2BENCH- 2509 benchmark. The evaluation was conducted on an infrastructure consisting of server-grade machines. Open-source models with fewer than 32B parameters (as listed in Table 2) were served using vL..M on a cluster equipped with NVIDIA GPUs. Specifically, these models were evaluated on machines featuring NVIDIA A100 80GB GPUs. The evaluation environment for these models was containerized to maintain isolation and consistency. For larger open-source models (>= 32B parameters) and all closed-source models, evaluation was performed by accessing their respective official APIs. The compute resources for these API-based evaluations are managed by the model providers and are not under our direct control or knowledge. Therefore, we cannot provide specific details on the underlying hardware, memory, parallelization used by these providers. The Testcase Runner execution for each task (which involves loading test cases, running the generated code and ground truth, and performing differential testing) primarily CPU-bound and ran standard server CPUs, such as Intel(R) (R) Gold 52 CPU @ 2.30GHz, featuring 64 logical cores. These machines were equipped with 125 GiB of RAM and SSD storage for the benchmark data and test cases. The total compute time required for the comprehensive evaluation of all 16 models across the 1 1163 tasks in CODE2BENCH- 2509 was substantial. While precise timing varies per model and task, we estimate the total GPU-hours consumed for the open-source model inferences to be approximately 200 GPU-hours. The total CPU-hours consumed for Testcase Runner execution across all models (including running the generated code and ground truth against approximately 500 tests per task per model) is estimated to be approximately 200 CPU-hours.
Benchmark Construction Cost. The construction of the CODE2 BENCH-2ECH2 suite is a compu- tationally intensive but one-time investment. Generating rigorous Property-Baan Based Testing (PBT) suite with guaranteed 100% branch coverage takes approximately 5 CPU-minutes per task. This process includes the iterative generation of PBT driver code by the LLM, execution of test cases, and coverage verification. Constructing the entire suite required approximately 83 CPU-hours. Crucially, this pipeline embarrassingly parallelizable, allowing the entire benchmark to be regenerated in under one hour standard high- nigh-performanee computing cluster, making continuous dynamic updates highly practical.
Lightweight Mode via Test Suite Minimization. To facilitate rapid model iteration and reduce evaluation overhead, we introduce a "Lightweight Mode." " We frame the test suite reduction as a Set Cover Problem and employ greedy algorithm to select minimal subset of test cases that maintains the original 100% branch coverage of the ground truth. This optimization typically reduces the test volume from 500 30-50 test cases per task (a ~00 reduction), significantly lowering the evaluation cost while preserving the diagnostic integrity regarding logical correctness.
1.2 EVALUATION INSTRUCTION
Table 15: Prompt Template for SC-Python Benchmark Runner in CODE2BENCH
# System
You are an expert in the field of coding, helping users write Python code. ## Input
The user provides you with an function signature and docstring, you should generate a Python function based on them.
## Output
"python The generated Python code.
## Note
Only output Python code with possible type import statements but without doestring and any additional information.
# User [Instruction]
38

---

Published as a conference paper at ICLR 2026
Table 16: Prompt Template for WSC-Python Benchmark Runner in CODE2BENCH
# System
You are a highly skilled Python programming expert tasked with implementing a function based on its specification, using the allowed libraries. Implement the Python function described below. Your implementation should strictly adhere to the behavior specified the docstring and utilize only the explicitly allowed external libraries. ## Output Format
"'python The generated Python code.
Provide ONLY the Python code for the function implementation with corrsponding libraries imported. Do not include any additional information or explanations.
# User [Instruction]
Table 17: Prompt Template for SC-Java Benchmark Runner in CODE2BENCH
# System
You are an expert in the field of coding, helping users write Java code.
## Input
The user provides you with an function signature and docstring, you should generate a Java function based on them.
## Output
"'java
The generated Java code.
## Note
Provide only Java code within a "'java"" code block. Include a complete public class named Tested with package name and necessary imports. Do not add main method repeat the docstring.
# User [Instruction]
39

---

Published as a conference paper at ICLR 2026
J CASE STUDY: UNCOVERING THE ILLUSION OF CORRECTNESS
Our diagnostic approach, combining a scaled source real-world problems with the scaled rigor of Property-Based Testing, allows to beyond simple pass/fail metrics and uncover nuanced failure modes. This case study on a Weakly Self-Contained (WSC) task from CODE2BENCH -509 illustrates how our framework reveals the critical gap between functional plausibility and engineering robustness.
J.1 THE TASK: A NUMERICALLY SENSITIVE PROBLEM
WSC Task #81 requires the implementation common operation in numerical analysis. The fround-truti implementation, sourced from a mature scientific Python library, is highlly efficient and numerically stable vectorized solution using NumPy:
first wirst_ddiviie _iviiiedd differecee function,
Ground Truth (Vectorized, Sumerically-Stablee
2 def first divided difference (d, fct, fctder, atol=le-12, rtol=le-12):
dif = np. repeat (d [None, :l, (d), axis-0)
close_ np. isclose (dif, dif. T,, atol=atol, stol=rtol) dif[close_] fetder (dif(clcse_)) dif[-clcse_] (fct (dif(-close_) fct (dt.. (dif[-close_] TIclos-close_)
7
8
return dif
J.2 THE
BUT FLAWED LLM SOLUTION
Remarkably, nearly all 10 evaluated models failed this task in the exact same way. They did not produce syntax errors or obvious logical flaws. Instead, they generated a functionally plausible solution that mimics textbook implementation using scalar Python loops:
Typical LLM-Generated Solution (Scalar, Naive)
2 def (iirst_diiideeeddiffeeer..
(d, fct, fetder, atol-le-12, rtol=le-12):
3 n (en(d)
4 fdd np. zeros ( (n, )
for i in range (n)
for j in range (n)
if isciscl (d[il, d[jl, atol-atol, rtol-rtal):
Eddli, il fetder (dlil)
else:
Eddli, il (fet(d[i)) fet (d(i))) (d[i) d(i])
return fdd
The diagnostic power of our benehmark revealed in how this seemingly correct solution failed. The code generated by DeepSeek-V3 for this task passed an astonishing 98.8% of our PBT-generated test cases (494 out of 500). It only failed on a few specific, numerically challenging inputs where the different order of floating-por point operations between the vectorized and scalar approaches led to minute rounding errors. These tiny discrepancies, while functionally insignificant in many contexts, caught by strict-tolerance deep comparison function. For example, failing test reported:
Mismatch found. Expected:
....219--8,, Actual: ...224e-08
J.3 ACTIONABLE INSIGHTS FROM "NEAR-MIS MISS" FAILURE
This single "near-miss" failure pattern, consistent across the entire model spectrum, provides several highly actionable insights that would be invisible conventional benchmarks:
For LLM Developers: This reveals that models learn to be academically correct. but not industrially robust. They successfully reproduce textbook patterns but lack essential engineering knowledge regarding idiomatic code (vectorization), performance optimization,
40

---

Published as a conference paper at ICLR 2026
and numerical stability. To close this gap, training data should be augmented to explicitly reward these non- functionaal properties. Our benchmark, with its real-world ground truths and precision-sensitivv tests, provides ideal data for such targeted fine-tuning.
For Benchmark Designers: This case powerfully validates our deep testing approach and exposes the limitations of shallow test suites. A typical benchmark, likely using only a few simple integer-based test cases, would have falsely labeled this numerically unstable solution as a success. Only through the exhaustive, edge-case-driven nature of our PBT methodology is the critical difference between "toy" solution and robust one revealed- penaldzinniinig brittle, "good-enough" outputs and rewarding true engineering rigor.
This example epitomizes the diagnostic philosophy of CODE2BENCH: to not just reveal what fails, but provide deep insights into why it fails and how future models can be improved.
41

---

Published as a conference paper at ICLR 2026
K SCALABILITY
K.1 ADVANCED EXTENSIBILITY: HIERARCHICAL. DEPENDENCY RESOLUTION
The dependency classification into SC and WSC, described in the main paper, provides a robust foundation for benchmark curation. However, a significant portion of real-worid code involves functions that call other project-internal functions. While these are classified as Project- Deeeeet-Depnnent (PD) and typically discarded, a substantial subset of them are. in fact, hierarchically testable. This observation opens a powerful new avenue for Scaling the Source even further.
K.1.1 THE CONCEPT OF LAYERED COLF-CONTINNED (LSC) TASKS
We define a Layered Self- Self-Contained (LSCC function as function that is not strictly SC WSC itself, but whose entire set of project-internal dependencies recursively resolves to a set of functions that are all either SC or WSC.
Consider function fA that calls another internal function fB-
If fB is Self-Contained (SC), then the functional behavior of fA is fully determined by its own logic and the well-defined, dependency-free logic of fB.
Similarly, if fB is Weakly Self-Contained (WSC), the behavior of fa is determined by its logic and the behavior of fB. which itself is only dependent on of allowed public libraries.
In both cases, the complete functional behavior of the top-level function fa can be fully specified and is not reliant any un-testable, opaque, proprietary internal state. Therefore, it is suitable candidate for a rigorous, standalone benchmark task.
K.1.2 METHODOLOGY FOR LSC TASK GENERATION AND VERIFICATION
Our CODE2BENCH framework can be extended identify and generate these LSC tasks through recursive dependency analysis powered by our Scope Graph:
1. Recursive Dependency Resolution: When a function fA is initially classified PD due to a call to an internal function fB, our framework does not immediately discard Instead, it recursively runs the dependency analysis on fB. This process continues until all dependencies are either resolved primitives, allowed libraries, or the dependency chain terminates.
2. Hierarchical Test Oracle Construction: To create a test oracle for an LSC function like Sa. we provide not only its own source code but also the source code of its entire dependency tree of SC/WSC functions (c.g.. fa. and any functions fa calls). This complete, self-contained bundle of functions serves as the ground-truth implementation.
3. PBT-based Verification: Property-Based Testing is then applied the top-level function /A. The PBT engine generates inputs for fa, and the complete, bundled ground-truth implementation executed to generate the expected outputs. The 100% branch coverage quality gate is applied to this entire bundle, ensuring that the tests thoroughly exercise not only the logic of fA but also the interactions with its internal dependencies.
This extension to handle LSC tasks dramatically increases the pool of high-quality, testable functions that can be extracted from real-world repositories. It allows our framework to capture more complex, multi-function interactions that are representative real-world software design, while still maintain- ing the rigorous, deterministic verifiability that is the hallmark of our approach. This represents significant future direction for scaling the realism and complexity of the CODE2BENCH suite.
K.2 GENERATING SYNTAX- AWARE CODE COMPLETION TASKS
A key design principle of the CODE2BENCH framework is its extensibility beyond single-function generation. The true assets curated by our pipeline are the large collection of high-quality, real-world ground-truth functions, each paired with comprehensive suite of high-coverage Property-
42

---

(PD) tasks that Published as a conference paper at ICLR 2026
Tests. This powerful combination of "solution" (fgi) and a rigorous "verification" mechanism (the PBT suite) provides uniquely powerful foundation for generating a wide array of challenging and realistic software engineering tasks. In this section, we detail how framework can be systematically extended to Code Completion.
Our framework automatically generate high-quality, syntax-aware completion tasks. Inspired by prior work on structured code completion [Gong et al. (2024), we can leverage our curated SC and WSC function pools to create distinct types of completion challenges:
code
Completion-SC (Algorithmic Logic Completion): For our Self- Soff-Contained (SC) tasks, which are rich in algorithmic logic. we can create completion benchmarks by masking entire logical blocks.
Completion- WSC (API Call Completion): For our Weakly Self-Contained (WSC) tasks, which are centered on library usage, we can create completion benchmarks by masking specific API calls.
K.3 RIGOROUS VERIFICATION VIA PBT
The most significant advantage of deriving completion tasks from CODE2BENCH is the automatic inheritance of our rigorous verification mechanism. Unlike many completion benchmarks that rely on simple syntactic checks (e.g.. exact match or BLEU score), can evaluate the functional correctness of the completed code.
K.4 FUNCTIONAL CORRECTNESS VS. ROBUSTNESS TRADE-OFF
As highlighted by reviewers, our stringent 100% branch coverage gate effectively filters out complex defensive logic (e.g.. unreachable error handling branches) that is difficult to trigger via random inputs. This represents a deliberate strategic choice: we prioritized establishing an unimpeachable "gold standard" for core algorithmic and logical correctness over the coverage of defensive programming constructs. While this ensures absolute verifiability, it temporarily de-emphasizes the evaluation of code robustness. However, our framework is designed to retrieve this filtered data. In future work, we plan to construct a dedicated "Robustness Benchmark" by tasking models to add comprehensive error handling (c.g., try-catch, input validation) to the verifiable "happy path" implementations curated in this work.
K.5 BEYOND FUNCTION-LEVEL ISOLATION.
The current iteration focuses on function-lever tasks to ensure unit-testable rigor. We acknowledge that real-world software engineering involves complex, project-level dependencies. Crucially, our framework methodologically ready for this expansion. Our Scope Graph analysis ($??) natively supports resolving cross-file dependencies and class hicrarchies. By combining this with Stateful Property- -Baed Testing (Stateful PBT), which can generate sequences of API calls rather than static data, we envision scaling our rigorous verification pipeline to assess multi-fure function interactions and state management.
K.6 MULTI-DIMENSIONAL EVALUATION.
Our primary metric is Pass based on functional correctness. While fundamental, this does not directly measure other code quality attributes such as efficiency, security, or readability. However, our massive suite of ground-trudt -tuth functions and generated test cases serves versatile asset. Future iterations can leverage these assets to benchmark execution time (efficiency). check against secure coding standards (security), or serve as references for style compliance (readability), providing more holistic assessment of LLM coding capabilities.
43

---

Published as a conference paper at ICLR 2026
L LIMITATIONS
Despite its strengths in generating dynamic, rigorously tested, and realistic tasks focusing on func- tional correctness, CODE2BENCI122509 like many benchmarks, has limitations in its evaluation scope. Our primary focus is on assessing the functional correctness of generated code, measured through @ 1 against comprehensive PBT-generated test suites. While functional correctness is paramount, real-world software development necessitates evaluating other crucial aspects of code quality, such as efficiency, readability, style, security, robustness to invalid inputs, and the ability to generate accompanying documentation or tests. CODE2BENCH- 25N9H currently does not directly evaluate these important dimensions.
Furthermore, the current iteration of CODE2BENCH-2555 2509 primarily focuses on code generation tasks, where models are required to generate a complete function implementation based on natural language instruction and function signature. However, the underlying structure of the benchmark, including the availability of ground truth implementations and the rigorous, diverse test cases generated via PBT, offers significant potential for evaluating LLM capabilities beyond simple generation. By leveraging the ground truth and PBT-generated test suites, the framework could be extended to support other task types crucial for software development workflows, such code completion (filling in missing parts of code), code editing/repair (modifying existing code to meet new requirements or fix bugs), and assessing code reasoning abilities through execution prediction or debugging tasks. Expanding to these diverse task types would provide a more comprehensive evaluation of LLMs' understanding and manipulation of code, moving beyond pure synthesis.
Future work could explore extending the benchmark to incorporate metrics and testing methodologies for some of these additional code quality attributes and diverse task types, providing a more holistic assessment of LLM capabilities in full software development context.
Our evaluation focuses on a diverse suite of state-of-the-ar art, cnntruction-oollvviiii code generation models. We acknowledge that our study does not include models specifically designed or fine-tuned for multi-step, algorithms Chain-of- Chain-of-Thought prompting for code). This exclusion was a deliberate choice based on two primary considerations. First, our preliminary explorations indicated that the verbose, multi- step reasoning outputs of such models often exceeded practical token limits for large-scale, automated evaluation harness, presenting significant computational and financial costs. Second, and more critically, the primary goal of C2DE:BENC is evaluate model's ability to generate direct, production-sty -tyyle code from real-world specifications, a task for which current Instruction-followee models the most direct fit. While evaluating deep reasoning capabilities is an important research direction, it represents a different evaluation paradigm that is beyond the scope of our current study.
reasoning (e.g., models employing complex search
44


Reviews:
[{"review_id": "U8Xr0XRylQ", "reviewer": "Reviewer_Esu7", "summary": "This paper introduces Dual Scaling, a benchmark construction philosophy in CODE2BENCH framework to scaling from dynamic, real-world code repository and generating rigor test with 100% coverage. Using this method, the authors further build CODE2BENCH-2509, with 411 Python instances and 249 Java instances. The paper conducted comprehensive experiments on closed source models and open source models, and the result suggests a performance gap between API application tasks and algorithm synthesis tasks.", "questions": "1. The paper mentions differences between Java and Python fingerprints (Figure 3) as LLM's coding ability intertwined with their target language's ecosystems. Could the authors clarify whether this refers to differences between interpreted and compiled languages, or to other ecosystem-level factors?\n\n2. The framework requires generating hundreds of PBT-based test cases per function and enforcing 100% coverage. Could the authors quantify the computational and time costs of this process?\n\n3. A typo: Last line in page 4: perturbation techniqueZhao -> missing a blank", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark designs rigorous and strong test cases. It not only accounts for edge cases but also ensures complete test coverage, substantially outperforming other benchmarks that rely on sparse test examples, which may lead to incorrectly judged “pass” cases.\n\n2. The paper provides a carefully designed implementation in both Python and Java, addressing not only translation between languages but also their distinct type systems and library ecosystems. This enables meaningful cross-language comparison and reveals how LLM performance depends on the target language’s constraints.\n\n3. The authors effectively decouple API-calling ability from algorithmic implementation ability. The experiment suggests that models perform better at API usage than at algorithmic reasoning. This insight offers a valuable lens for future work on improving model reasoning.\n\n4. The authors also emphasize clarity and unambiguity when generating instructions, which contributes to the benchmark’s reliability and reproducibility.", "weaknesses": "1. Although CODE2BENCH draws its source data from real repositories, the benchmark tasks remain function-level and isolated. This design simplifies testing but does not capture cross-function or module-level dependencies, which are prevalent in real-world software engineering. As such, the benchmark evaluates isolated reasoning rather than full software generation ability or collaborative code development.\n\n2. As mentioned by the authors, real-world code often includes numerous defensive branches and error-handling structures. The current test generation strategy struggles to fully cover these fragmented control flows, which are often filtered out because they fail the 100% coverage requirements. While this improves test rigor, it also excludes many defensive programming constructs that are significant in real-world software development.\n\n3. The filtering process relies on a fixed list of *allowed libraries* to define “Weakly Self-Contained” tasks. This helps maintain consistency but may also limit domain diversity, since tasks from less common libraries or specialized fields are excluded. As a result, limiting the allowed libraries may constrain the representativeness of the benchmark and may introduce unexpected bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LPsC3i604U", "reviewer": "Reviewer_dNZT", "summary": "This paper proposes CODE2BENCH, a benchmark construction framework for evaluating code-generating LLMs. The framework addresses data contamination through temporal filtering (extracting functions from GitHub commits after model knowledge cutoffs) and improves test rigor through Property-Based Testing with 100% branch coverage. Tasks are classified into Self-Contained (SC, pure algorithmic reasoning) and Weakly Self-Contained (WSC, API usage) using Scope Graph analysis. The authors construct CODE2BENCH-2509 with Python and Java tasks and evaluate 10 LLMs, reporting three insights: SC vs WSC performance gap, language ecosystem impact, and \"illusion of correctness.\"\n\nThis paper shows good engineering efforts but unclear novelty contribution. Two critical flaws need to be addressed: (1) No validation of benchmark value. The paper lacks direct comparison with existing benchmarks (HumanEval, MBPP, BigCodeBench) on the same models, making it impossible to assess whether CODE2BENCH provides unique insights or simply evaluates differently. (2) No novelty contribution in analysis. the three \"key insights\" are either expected results (SC/WSC gap), intuitive observations already explored (language ecosystem), or incremental findings without baseline comparison (illusion of correctness).", "questions": "Do you have any existing comparison data (even partial) showing how the same models perform on CODE2BENCH vs existing benchmarks? Can you clarify why this comparison was not included in the paper?\n\nCan you clarify how your three key insights differ from findings in BigCodeBench (WSC tasks), HumanEval-X (multi-language), and EvalPlus (test insufficiency)?\n\nDo you plan to validate CODE2BENCH's value through comparison with existing benchmarks? What would be your approach?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Important problem: Addresses real limitations in LLM code evaluation, which are data contamination and superficial testing.\n\n2. Solid engineering: Scope Graph analysis for dependency classification is technically sound. Property-Based Testing with 100% branch coverage demonstrates rigor. The framework automates benchmark construction.\n\n3. release code, data, and results.", "weaknesses": "1. Lack of Direct Comparison with Existing Benchmarks\nFor a benchmark paper, it is crucial to demonstrate how the new benchmark compares with existing ones when evaluating the same models. The paper only shows Table 1 comparing characteristics (Dynamic, Rigorous Test, etc.) but lacks direct performance comparison with these baselines. Without evaluating the same 10 models on existing benchmarks, it's impossible to determine whether CODE2BENCH provides unique insights, whether the lower pass rates reflect higher quality or different task distribution, or whether the three \"key insights\" could be revealed by existing benchmarks. This comparison is essential to establish the benchmark's value.\n\n2. Missing Details of Validating Benchmark Construction Method\nThe paper proposes Dual Scaling with temporal filtering, Scope Graph classification, and PBT with 100% coverage, but provides less details to validate these components. What happens with 80% coverage instead of 100%? Does temporal filtering applied to HumanEval produce similar contamination resistance?  An ablation study may be a good way to explore these points.\n\n3. LLM Analysis Lacks Technical Contribution and Novelty\nThe paper presents three \"key insights\" as major contributions: (1) SC vs WSC performance gap. it is an expected result since BigCodeBench already focuses on API tasks and it's well-known these are different skills; (2) language ecosystem impact, which is also intuitive and already explored in HumanEval-X; (3) illusion of correctness. EvalPlus already demonstrated this, and without comparison to EvalPlus, the 6.94% figure lacks context. The analysis is descriptive rather than prescriptive, providing no actionable insights for improving models or evaluation. Without showing these insights are unique to CODE2BENCH or impossible to obtain from existing benchmarks, the analysis appears to justify the benchmark circularly rather than contribute genuine discoveries.\n\n4. Limited Validation of Practical Utility\nThe paper doesn't show whether performance on CODE2BENCH correlates with real-world coding capabilities (e.g., SWE-bench).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S79H83ekKX", "reviewer": "Reviewer_9nHm", "summary": "This paper introduces a novel benchmark construction paradigm termed \"Dual Scaling\", which addresses two critical challenges in evaluating code generation LLMs: the reliance on static, easily contaminated problem sources and the use of superficial testing methods. The authors present the Code2Bench framework, which dynamically acquires problems from real-world code repositories (Scaling the Source) and integrates property-based testing (PBT) with a 100% branch coverage quality gate (Scaling the Rigor) to build the Code2Bench-2509 benchmark. Empirical evaluations demonstrate that this benchmark effectively uncovers performance gaps between models in algorithmic synthesis (SC tasks) and API application (WSC tasks), while quantifying the influence of language ecosystems on model behavior.", "questions": "- It is unclear what criteria were used to assess the testing rigor for all benchmarks presented in Figure 2. Could the authors clarify how testing rigor is defined and measured?\n- Why is the Java track limited to SC only? Where is WSC-Java?\n- The authors state that each benchmark task includes approximately 500 test cases. Given that these test cases are selected after PBT generation and a 100% branch-coverage gate, does the pipeline need to generate and evaluate hundreds or even thousands of candidate inputs per task before filtering? If so, what are the actual computational costs (time and resources) per task? Similarly, for the evaluation stage, running ~500 test cases per task can significantly increase runtime and resource usage. Could the authors provide quantitative measurements of the end-to-end evaluation time per model and per task, and discuss any mechanisms (e.g., batching, caching, reduced-cost modes, seed control) used to ensure scalability across many models and large task suites? Guidance on lighter-weight modes that preserve diagnostic value would also be useful.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The 100% stringent branch coverage gate and large PBT-generated suites substantially reduce false positives and expose \"near-perfect\" failures that many benchmarks miss.\n- The outcome spectrum and “diagnostic fingerprints” provide more granular failure analysis (SyntaxErr/RuntimeErr/LogicErr vs. partial pass bands), illuminating the algorithmic synthesis vs. API-application divide and the role of language typing in error suppression.\n- WSC-Python spans >35 libraries; SC-Java demonstrates multi-language extensibility. Tasks show higher cyclomatic complexity and test volume than legacy benchmarks.", "weaknesses": "- Some of the figures in the paper are not very clear or visually polished. For example, in Figure 2, there is noticeable overlap between text elements and between text and icons, which affects readability. Improving the clarity and layout of the figures would make the presentation more professional and easier to interpret.\n- In the Related Work section, the authors assert that existing live benchmarks rely on narrow or specific data sources. However, Code2Bench is also curated from specific GitHub repositories without disclosing the selection criteria, repository sampling strategy, or inclusion/exclusion policies. This lack of transparency makes it difficult to assess source diversity, potential sampling bias, and contamination risk. Moreover, prior benchmarks such as DomainEval also use GitHub repositories to collect domain-specific tasks. The novelty of \"Scaling the Source\" appears limited.\n- Evaluation scope focused on functional correctness: Important dimensions like performance/efficiency, readability/style, security, robustness to invalid inputs, and documentation/test generation are not directly evaluated.\n- Project-Dependent problems are mostly discarded in this pipeline. Although LSC is discussed as future work, the current evaluation does not yet include multi-function or multi-file context tasks that matter in industrial settings (e.g. I/O, resource handling, exceptions, and protocols).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)