AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
Authors: Changzhi Zhou
Ao Liu
Yuchi Deng
Zhiying Zeng
Tao Zhang
Haotian Zhu
Jianwei Cai
Yue Mao
Chenchen Zhang
Lingyun Tan
ZiyanXU
Bohui Zhai
HengyiLIu
Speed Zhu
Wiggin Zhou
Fengzong Lian
Venue: ICLR 2026 Poster (2026)
Keywords: code generation
benchmark
autocodebench
llm
Primary area: datasets and benchmarks

Abstract: Large Language Models (LLMs) have shown impressive performance across diverse domains, with code generation emerging as a particularly prominent application. However, existing benchmarks designed to evaluate code generation exhibit several critical limitations. First, most rely on manual annotations, which are time-consuming and difficult to scale across programming languages and problem complexities. Second, the majority focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and imbalanced language coverage. 
To overcome these challenges, we present AutoCodeGen, an automated framework for constructing high-difficulty, multilingual code generation datasets without manual annotations. Our approach guarantees correctness and completeness by generating test inputs with LLMs, obtaining test outputs within a multilingual sandbox, and further enhancing quality through reverse problem generation and multi-stage filtering. 
Based on this novel method, we introduce AutoCodeBench, a large-scale benchmark suite spanning 20 programming languages with balanced coverage. AutoCodeBench is designed to rigorously evaluate LLMs on diverse, challenging, and realistic multilingual programming tasks. Extensive experiments reveal that even state-of-the-art models struggle on these tasks, particularly in low-resource languages.
Besides, we release complementary training and evaluation resources, including a large-scale, verifiable multilingual instruction dataset generated via the same pipeline, as well as a multilingual sandbox with high-concurrency support. We hope these contributions will provide a solid foundation for future research and inspire the community to explore more automatic and scalable approaches to multilingual code generation, with a particular emphasis on advancing progress in low-resource languages.

Published as a conference paper at ICLR 2026
AUTOCODEBENCH: LARGE LANGUAGE MODELS ARE AUTOMATIC CODE BENCHMARK GENERATORS
Jason Chou
Ao Liu"
Yuchi Deng
Zhiying Zeng Lingyun Tan Ziyan Xu
Tao Zhang
Haotian Zhu
Jianwei Cai
Yue Mao
Chenchen Zhang
Bohui Zhai
Hengyi Liu
Speed Zhu
Wiggin Zhou" Fengzong Lian
Hunyuan Team, Tencent
https://autooodeeeeen giw.hb..iihoooooooooo
ABSTRACT
Large Language Models (LLMs) have shown impressive performance across di- verse domains, with code generation emerging as particularly prominent ap- plication. However, existing benchmarks designed evaluate code generation exhibit several critical limitations. First, most rely manual annotations, which are time-consuming and difficult to scale across programming languages and prob- lem complexities. Second, the majority focus primarily on Python, while the few multilingual benchmarks suffer from limited difficulty and imbalanced language present AutoCodeGen, automated
coverage. To overcome these challenges, framework for constructing high-difficulty, multilingual code generation datasets without manual annotations. Our approach guarantees correctness and complete- ness of test cases by generating test inputs with LLMs, obtaining test outputs within multilingual sandbox, and further enhancing quality through reverse problem generation and multi-stage filtering. Based on this novel method, we introduce Au- toCodeBench, large-scale benchmark suite spanning 20 programming languages with balanced coverage. AutoCodeBench is designed to rigorously evaluate LLMs on diverse, challenging. and realistic multilingual programming tasks. Extensive experiments reveal that even -fateoof-the-ar models struggle on these tasks. par- ticularly in low-resource languages. Besides, we release complementary training and evaluation resources, including large-scale, verifiable multilingual training set generated via the same pipeline, as well as a multilingual sandbox with high- concurrency support. We hope these contributions will provide solid foundation for future research and inspire the community to explore more automatic and scalable approaches multilingual code generation.
INTRODUCTION
Recently, Large Language Models (LLMs) have advanced rapidly, achieving strong performance
across wide range of tasks (OpenA] 2024 Gemini 2025 DeepSeek-A] [025b] Anthropic (025b).
Among these, code generation has emerged as a central indicator of both intelligence and practical utility, drawing increasing attention from academia and industry alike (Chen al. 202 Jimenez ctt al. 2024; Jiang et al., 2024). Many powerful LLMs, such as Claude Opus 4.1 (Anthropic 2025a),
already widely adopted in Al-assisted coding tools (Cursor 2025 Anthropic 2025) Through the ability to generate executable code, LLMs can substantially accelerate programming automation and reduce manual development effort.
To evaluate and advance these capabilities, a series of benchmarks have been developed (Wang et al. 2025a). Early efforts such as HumanEval (Chen et al.] 2021) and MBPP (Austin al. 2021) focused on short, algorithmic Python tasks. More recent benchmarks (Peng et al. 2024
Jimenez 2024. Jain al. 2025 Zheng al. 2025 Zhu al.) 20255 Chai 2025
Bytedance 2025
Zan et al. 2025
target more challenging and realistic programming tasks, including
Equal contributions. Project leads.

---

Published as a conference paper at ICLR 2026
Table Comparison of Code Generation Benchmarks. MLing: MultiLingual: MLogi: MultiLogical, refers to programming problems that require the model to simultaneously implement multiple core functionalities. HFree: Sluman-Freee BDist: Balanced Distribution of multiple languages. DSize:
Data Size. PLen: Problem Length. Further details are provided in the Appendix B
Benchmark
MLing
Mggi
HFree
BDist
Difficulty
Category
DSize
PLen
HumanEval
X
X
X
5
164
134.1
MBPP
X
X
X
6
378
50.5
LiveCodeRench
X
X
X
4
1 1100
469.6
FullStackBench
X
X
X
12
1687
184.3
McEval
X
X
9
2007
146.7
AutoCodeBench
V
V
14
3920
498.2
competition-lewe problems and multilingual scenarios. However, constructing such benchmarks remains costly and labor-intensive, which hinders scalability and makes it difficult to guarantee both high difficulty and broad coverage. As shown in Table [ widely used multilingual benchmarks such BullStackBench (Bytedance, 2025) and McEval (Chai et al. 2025) suffer from imbalanced language distributions and limited diversity. These issucs stem from the inherent bias manual annotation. For example, most annotators proficient in Python algorithmic tasks but lack expertise in domains such as Elixir-based communication development. Moreover, the rapid progress LLMs has rendered many of the overly simple problems in these benchmarks obsolete. These limitations naturally raise a critical question: Can we automatically construct high-quality code generation benchmarks that scale while ensuring both comprehensiveness and diversity?
In this paper, we propose AutoCodeGen, an automated workflow centered on I.LM-sandbox interac- tion, synthesize challenging multilingual code generation datasets without manual annotations. Unlike previous data synthesis approaches (Luo et al. 2024 Wei et al.) 2024b; Xu et al., 2025 Ahmad et al., 2025) that directly rely on L.LMs to generate test functions, we adopt more intuitive and reliable strategy. The LLMs first produce test inputs, and multilingual sandbox executes these inputs to obtain the corresponding outputs. This design effectively mitigates a common limitation of LLMs, namely the tendency to generate incorrect outputs when confronted with high-difficulty problems. Furthermore, we introduce a reverse reverse-generatee generation paradigm, where code solutions and test functions are synthesized first, followed by the construction of the programming problem itself. This ensures that the resulting tasks are not only sufficiently challenging but also verifiably correct.
Based on the automation workflow, introduce AutoCodeBench. large-scale, fully automated code generation benchmark, as shown in Table Compared with previous multilingual bench- marks (Cassano al., 2022) [Bytedance] 2025 Chai al. 2025). ) ours simultaneously offers high difficulty, diversity, and practicality, with a balanced distribution of problems across 20 programming languages. We intentionally include some multi-logical problems test the LLMs capacity for multitasking in single problem. The key contributions of this paper are as follows:
1. AutoCodeGen. We propose an automated workflow based on LLM-sandbox interaction, where LLMs generate test inputs and sandbox executes them to obtain outputs, create high-quality multilingual code generation tasks.
2. AutoCodeBench. We introduce AutoCodeBench, a large-scale, fully automatic code generation benchmark with 3,920 problems, evenly distributed across 20 programming languages, featuring high difficulty, practicality, and diversity. We also construct Lite and Complete versions to enable efficient and high-quality evaluation. The evaluation results show that current LLMs still struggle with complex and diverse multilingual programming tasks, especially in multi-logical and low-resource language scenarios.
3. Training and Evaluation Resources. We propose AutoCodeInstruct multilingual code generation training set constructed using the same pipeline AutoCodeBench, ensuring comparable quality. We design a two-stage GRPO training to demonstrate the potential of this dataset in enhancing code generation capabilities. Besides, release Multilingual
2

---

Published as a conference paper at ICLR 2026
Code Snippets
4P
Calcalate the Sam Twe katbers
cer toLseeaa Inn, B: tut u
Panlic Delpst
det tut cantas Int, bo 221) cnt2
Traus and, 302
9s bisplt Relumn the sam er and
Sh
GO
CUMBINT
php
Test
Priunte
Pone
motic test preattil (1,2) Dipected datport:
del Test tvt seiff Bu by rszected sait,, ancert reealt expectes, recled: (oeauit) Test case
Sampling
JS
for
SF
resilt a twe sanch, bi
Mospertesppe tresithurrs
Figure I: The overview of AutoCodeGen. It first generates code solution and the corresponding public/private test input functions based on multilingual code snippets (D). They are concatenated and executed in sandbox to obtain test outputs, which are then combined by the LLM into complete test functions (2.3.4) Based on the code solution and test function, the LLM is prompted to
generate accurate programming problems (5). Finally, three-stage data filtering is applied: multiple sampling to remove too easy problems (G), LLM-as-Critic to discard low-quality ones (7), and diversity-basee tagging ensure distributional variety (8).
Sandbox with high-concurrency support, which can be employed for both model evaluation and RL training.
AUTOCODEGEN & AUTOCODEBENCH
In this section, we first present how AutoCodeGen constructs the AutoCodeBench family of bench- marks, and then provide an overview of AutoCodeBench.
2.1 AUTOCODEGEN
Our AutoCodeGen is fully automated workflow based on LLM-sandbox interaction for constructing verifiable code generation datasets. It first generates large-scale multilingual data with guaranteed executability and correctness, then applies a three-stage filtering strategy to ensure the benchmark challenging, high-quality, and diverse. As illustrated in Figure [ the workflow includes four key stages: Code Solution Generation (D), Test Function Generation (2)3).)) Programming Problem Generation (5), and Data Filtering (6.).)).
2.1.1 CODE SOLUTION GENERATION
We begin by extracting multilingual code snippets from Stack-Edu (Alla al. 2025). a large-scale dataset educational code filtered from The Stack v2 (Lozhkov et al., 2024). as seeds. These seeds span function-level, llass-level, and file-level code, sourced from real GitHub repositories, ensuring di- versity and practicality. Using a language-specific few-shot prompt, we guide DeepSeek-V3 refine and evolve these seeds into verifiable and self-contained code solutions. During this process, the model removes non-essential logic and adds appropriate comments for clarity. We then validate the correctness of the generated solutions by multilingual sandbox.
2.1.2 TEST FUNCTION GENERATION
We enhance efficiency and edge-case coverage by first generating test inputs via LLMs and then executing them in a sandbox to obtain the corresponding outputs. Specifically, is divided into the following three steps:
Test Input Generation The test input functions (both public and private) are generated alongside the above code solution, ensuring alignment between the solution and its inputs. The public test input
3

---

Published as a conference paper at ICLR 2026
Table Programming language translation pairs.
Origin
Target
Origin
Target
Origin
Target
Origin
Target
Origin
Target
Python
R
Python
Ruby
Java
Scala
Java
C#
Shell
Perl
Python
Elixir
Python
Julia
Java
Kotlin
JavaScript
PHP
C++
Rust
Python
Swift
Python
Racket
Java
Dart
JavaScript
Typescript
function includes no more than basic cases and serves demonstration purposes; will be embedded into the final programming problem an illustrative usage. In contrast, the private test input function contains 7+ cases, including edge cases, and functions as the comprehensive test for verifying the correctness of the code solution.
Test Output Generation We concatenate the code solution with test input functions and execute them in the sandbox obtain the corresponding test outputs.
Integration We prompt DeepSeek- V3 -3-0333 -0324 with both the test input functions and output results from the sandbox generate coherent and verifiable test functions. Finally, we validate the correctness by executing the code solution together with the generated public and private test functions in the sandbox.
2.1.3 PROGRAMMING PROBLEM GENERATION
Generating high-quality programming problems is challenging, as it requires detailed and accurate problem descriptions. We find that models often omit key information when generating programming problems, such as the entry point specified in the test function. Therefore, we define set of specifications, such as explicit input/output formats, function and class names, to ensure that the generated problems are detailed and well-structured. We prompt DeepSeek -33--ee------- -V3-032--2 to generate high-quality programming problems based on the code solution with appropriate comments and the corresponding test function, while embedding the public test function as example usage.
Through these three steps, we obtain a large-scale multilingual dataset, where each instance is represented tuple <programming problem, code solution, public test function, private test function>.
2.1.4 DATA FILTERING
Finally, We apply three filtering and sampling steps to ensure the high-difficllty, high-quality, and diversity of the final benchmark.
Difficulty Control Too simple programming problems are barely meaningful for evaluating the code generation capabilities of current LLMs. To address this, we employ a moderately capable code model, Seppseek-Code--Z2 sample answers for each problem ten times using the model and validate the correctness via sandbox exccution. We discard problems that are solved in all attempts. Take Python as an example, DeepSeek-Coder-V22
to filter out too casy problems. Specifically, we
t can filter out 25.1 % of the whole problems.
Quality Control During the aforementioned problem generation stage, we define six specifications to guide the generation of detailed and accurate programming problems. To further ensure high quality, we employ DeepSeek- -aiek-RR--- RR-0528 to critique each <problem, test function pair. Only the data whose test functions completely accurate and fully aligned with the problem are retained.
We aim for our benchmark to cover as many real-world scenarios as possible. To this end, we perform diversity-based sampling on the existing data to construct the final benchmark. We use DeepSeek -V3 -V3-00 V3-0324 24 to label each problem. We then divide the problems into different pools by category and perform cyclic sampling, ensuring broad representation of programming scenarios.
Diversity Sampling
2.1.5 APPROXIMATE LANGUAGE TRANSLATION
For Python, C++, Shell, Java, JavaScript, and Go, we directly use the workflow described above. For the other 14 languages, while the proposed workflow is still applicable, choose to employ
4

---

Published as a conference paper at ICLR 2026
Table 3: Statistics of Dataset. ACB: AutoCodeBench; Langs: Languages; Prob: Problem; Solu:
Solution; Len: Length; E/M/H: Easy/Mediun/Hard. The difficulty level is determined by the number of passes in ten samplings of Seappeeee-ooee-v// are classified as hard, 1-5 correct solutions medium, and those with more than five as easy.
Problems with zero correct solutions
#Prohlems
#Test Cases
#Iaans
Prob Len
Solu Len
Difficulty (E/M/H)
ACB-Full 3,920
37,777
498.2
487.5
646/846/2428
ACB-Lite 1,586
15,341
20
517.2
469.3
263/421/902
ACB-Complete
1,000
9,608
20
505.2
461.2
169/265/566
approximate language translation approach due to their limited data resources and lack of diversity. We extract unused data from the dataset generated in Section 2.1.3 and translate them into the target low-resource language, as shown Table 2 This ensures a sufficient and diverse datasct, which is further refined through the Data Filtering process in Section 2.1.4
2.2
AUTOCODEBENCH
2.2.1
DATA OVERVIEW
As shown in Table D and AutoCoodBeench(Ful) is large-scale, high-difficllyy multilingual benchmark. Over 60% of the problems are classified as hard problems, with each problem averaging 498.2 characters and accompanied by 9.6 cases, providing a challenging and comprehensive evaluation standard. The 20 languages are as follows: Python, C++, Java, JavaScript(JS), Go, Shell, C#, Dart, Elixir, Julia, Kotlin, Perl, PHP, Racket, R, Ruby, Rust, Scala, Swift. TYpeScript(TS).
To analyze the diversity and language coverage of AutoCodeBench, we first use Claude Sonnet to generate 20 language-agnostin task categories, and then employ DeepSeek-V3 classify each problem accordingly. Categories with less than 2% representation are merged into the "Other" group. AutoCodeBench covers 14 categories, demonstrating comprehensive coverage of practical programming scenarios. Besides, analyze the distribution of problems across the 20 programming languages. AutoCodeBench exhibits a relatively balanced distribution across languages, with significant bias toward any specific one, further validating its completeness and representativeness as a multilingual benchmark. Detailed category and language distribution are provided in Appendix B.3
to
2.2.2 AUTOCODEBENCH LITE AND AUTOCODEBENCHI COMODDBBBEENN-CC-LLTEE CONSTRUCTION
To facilitate quicker and more efficient model evaluations, we create AutoCodeBeneh--Lnn simplified subset of AutoCodeBench. Specifically, we collect the problem-solving results from all models and sort the problems in ascending order based on the number of passes. After discarding problems with fewer than 2 passes, we select approximately 1,500 problems based on their pass count in ascending order. These problems, which have been solved correctly by existing models at least twice and have certain level of difficulty, are selected to amplify the differences between the models. use these problems as the set for the Lite version.
To enable evaluating base models, further present MelllooodeeeechCCoooll.e based version of ACB. Concretely, we select 1,000 data points from ACB-Lite to ensure balanced distribution of 50 problems per programming language and use 3-shot demonstrations to evaluate the performance base models. ACB-Cos ACB-Complete can serve as a comprehensive benchmark for evaluating the multilingual code generation capabilities of base models.
completion-
EVALUATION
3.1 EVALUATION SETUP
We the Pass @ I (%) (Chen al. 2021 the default evaluation metric. In terms of inference parameters, for proprictary models and open-source models that provide APIs, we access them through direct API calls. Other models evaluated with official parameters when available, or
5

---

Published as a conference paper at ICLR 2026
Averege
Pedhes 65.7
Oye
Jam
JS
G+
Stefl
Chhrrp
Dert
Jella
Kotlin
Pert
HEP
Recket
Rabry
Rust
Sosla
Swift
15
Carrer Cpper Seand
22.3
75.3
60.3
66.7
76.7
72.9
23.4
75.0
27.5
70.5
56.3
64.5
$3.4
20.2
75.3
81.0
42.5
75.4
78.5
67.3
Preprietary Maddk
2140
Meddlk
Claade Opuc 41 (2820488))
13.3
19.5
13.6
26-9
56.5
11.3
714
15.6
618
Chande Opuc 4.1 (2820
52.6
183
181
513
613
16.
16.6
76-
15.0
88.3
47.5
0
+5.0
66.
11.3
96.5
763
518
510
(13
GPE5 (NC59997)
25.3
44.
515
40.0
44.
40.0
43.5
13.5
13.3
56.5
23.0
54.
43.
58.3
46.
x3-ash (2835-16-11
361
433
210
18.3
12.7
413
412
131
60.3
900
080
13
16.
05.4
11.
16.3
96.5
134
663
Grok-4
Gonalit2.55 Pee
46-7
483
435
55.1
310
48.2
54.0
66.7
5441
12.0
41.3
287
52A
413
565
56.5
45.7
413
4.3
DeppSest- 228821
353
433
53.3
350
364
71.9
286
52A
560
362
233
44.
56.3
283
283
18.3
84.
286
536
43
408
43
Hatvean Torted- 1829771
45.
142
349
413
10.0
146
19.3
64.
48.
762
610
920
420
185
639
300
913
313
913
422
46.
163
383
473
370
161
61.4
68.3
53.5
784
18.5
66.5
41.
165
856
61
113
463
465
03
218
Qwcad-
41.2
313
23.
86.7
599
12.9
48.
550
20.
368
450
350
434
442
Creec Coder
364
01.
511
31.
36-4
13.
71.
16.5
631
41.
133
961
131
515
431
135
11.
Scall Soolla--Thokkeee-55
400
483
43.1
90.0
15.2
382
15.0
K3.5
36.3
611
510
510
410
261
510
445
363
27.
312
46.3
153
Swaill www.iiiddddddddoo (356615)
42.4
353
653
06.8
13.6
346
15
169
17.5
66/7
450
67.0
96.5
23.
195
13.1
06.5
741
331
GLMS-5--A Altenoble
403
363
376
794
31.
785
76.7
66.3
38.0
61.5
43.0
510
40.5
21.1
403
-70
251
314
314
137
Qwenk-Neat-30
20.5
233
48.1
12.0
293
12.9
15.3
65.1
3500
41.
79.
190
34.
402
41.7
318
353
33.0
36.1
318
44.5
-70
Owwnd-10 1m
161
62.1
181
166
15.5
10.
370
174
121
Qread-220
310
255
23.
21.
150
25.
24D
29.5
21.3
2001
450
56.5
21.
25A
22.1
56.5
16.1
25.
318
351
Qhead-af
23.3
1133
25.0
12.
183
4222
23.5
16.1
271
23.
Qpead-177 Att fnstrect
14
15
83
0.5
112
15.5
4.6
74
16.3
58
346
350
Oweiil D-okle--Cookee/7 77h fistnct
22.5
199
56
233
113
126
16.7
185
400
180
150
235
15
260
13
265
10
111
178
346
Desppoeb- Cunpleeb-Caaee3333 Sarinact
25.1
250
263
263
385
286
335
545
135
657
16.
130
265
161
261
187
330
343
168
261
Deppbekk
15.
10
213
Seal Cako--ii --------- fisinest
32.3
235
28
33.5
225
548
38.5
300
525
360
967
26
135
215
260
112
OpenCodee-ll Enstrrctt
19.3
345
67
1611
135
175
312
155
363
155
300
365
146
158
155
365
60
150
313
Table 4: @ 1 (%) performance of different models for AutoCodeBench. Current Upper Bound
represents the @ass@1 1 value calculated by taking the union of problems correctly solved by all models. Blue and Green denotes reasoning and non-reasoning modes.
with greedy decoding otherwise. All models are provided with our custom system prompt, which standardizes the output format: You are an expert programmer. Your task is to provide a code solution within single Markdown code block for the given programming problem. Do not include any direct execution commands, test cases, or usage examples within the code block.
3.2 MAIN RESULTS
We comprehensively evaluate the performance on ACB, with results across different programming languages shown in Tables 4
The results of ACB-Lite and leaderboards are shown in Table Figure
and Figure g
Results show that ACB is highly challenging, as no model surpasses 55.5 average score, indicat- ing that current LLMs still struggle with complex, practical multilingual problems. Among all models, Claude Opus 4. 1 consistently achieves the best performance both reasoning and non- reasoning modes, confirming its strength across diverse coding tasks and aligning with observations from SWE-bench (Jimenez et al. 2024). Finally, while individual models perform moderately, their combined upper bound reaches 75.3, revealing complementary strengths and substantial room for improvement, as no single model dominates across all languages.
3.3 PERFORMANCE ACROSS POPULAR AND LOw-RESOUE PROGRAMMING LANGUAGES
We select five models with similar performance levels (ranging from 47.7 to 49.3) and evaluate their performance differences across popular and low-resource languages. As shown in Figure 2 the difference in average Pass@1 1 scores among the models for popular languages is small 3.1). However, the performance gap between models widens (A6.3) in low-resource languages, suggesting that low-resource programming languages may have received insufficient attention in model development. Besides, since we use the moderately capable DeepSeek - SatsSSeee-Coeer-eeee C2der-V2-Li3
as a filter to remove simple problems, the Pass scores of top models on popular languages relatively low. However, because the filter itself performs poorly on low-resource languages, many problems that appear trivial to top models are not filtered out, resulting in higher Pass @ I scores in these languages than in popular ones. This further highlights the pronounced disparities in
6

---

Published as a conference paper at ICLR 2026
60
2937
Langusgee
g
Pan
1t. Oarh cot lagh
Figure 2: The performance comparison of differ- ent models across two language sets.
Figure 3: Performance drop of models on multi- logic problems (1,622) compared to full dataset.
low-resource language capabilities across models and underscores the need for greater community attention to this issue.
3.4 PERFORMANCE ACROSS MULTI-LOGIC PROGRAMMING PROBLEMS
A key feature that distinguishes ButcCodeBench from prior benchmarks is the inclusion of multi- logical problems. These problems require models implement multiple distinct functions or classes within single task, challenging their ability to handle multiple core demands simultancously. We use DeepSeek-V3 V3-000300000000 0324 to identify all multi-logical problems AutoCodeBench and evaluate model performance on them. The results, shown in Figure reveal significant performance drop for all models when faced with multi-logical tasks. Among them, Claude Opus 4. 1 and DeepSeek-V2 exhibit relatively smaller declines, while the other models show larger drops. These findings highlight a key limitation: current models still struggle with multi-logical problem solving, an ability that is particularly critical for real-world code agent applications.
3.5 PERFORMANCE ANALYSIS OF MULTI-TURN REFINEMENT WITH SANDBOX FEEDBACK
As shown in Figure we evaluate how models leverage execution error messages to iteratively refine their code solutions. The results highlight the substantial value of our multilingual sandbox error feedback across all evaluated models. 2 5-Coderr-3322 Inst ruct achieves remarkable improvement from 35.8% to 47.4% after three refinement turns, while 3wen -8B shows consistent progress from 23.3% to 30.2%. The most significant performance gains occur during the first refinement turn, with diminishing returns in subsequent iterations. This pattern suggests that models can effectively leverage execution feedback to identify and correct common coding errors, though the complexity of remaining problems increases with each iteration. The consistent improvement across different model scales indicates that multi-turn refinement with sandbox feedback is a valid strategy for enhancing code generation quality.
3.6 AUTOCODEBENCH CEW.AIBEEENCCC-CCLEEEEE EVALUATING BASE MODEL CAPABILITIES
Table 5 presents performance comparison between base models on ACB- ACB-Complete and chat models on ACB-Full, Among models with 8B parameters or fewer, Seed-Coder- 8B demonstrates superior performance in ACB-Complete, consistent with its strong showing on ACB-Full. This consistency suggests that the pretraining process effectively equipped multilingual programming capabilities, enabling them to handle diverse coding scenarios across mul- tiple languages. Besides, an interesting observation arises when comparing Qwen2. 5-Coder-78 and OpenCoder-8B. While it outperforms OpenCoder-82 ACB-Complete. This suggests that Owen2.5- 5-Coder-777 73 may have undergone more effective post-training multilingual code generation data.
8B models with strong
ACB-Full, the trend reverses on
7

---

Published as a conference paper at ICLR 2026
Table 5: The pass@1 values of chat models (ACB-Full) and base models (ACB-Complete).
ACB-Full ACB-Complete
30B+ Models
www.ea
KeepSeek-Coder-
37.7
39.0
Qwen2.5-72B
34.3
35.9
Qwen2.5-Coder-322
35.8
35.5
8B Models
Sced-Coder-8B
31.6
Qwen3-8B
23.3
22.6
OpenCoder-888 -B
19.3
26.1
Qwen2.5-Coder-711
22.5
24.6
Figure 4: Performance improvement across multi- turn refinement with sandbox feedback.
DeepSeck-Coderr-.. 7B
20.5
22.9
Table Results two-stage GRPO and SFT with AutoCodeInstruct.
Model
ACB-Full
ACB-Lite
LiveCodeBench- V6
FullStackBench
McEval
Wwen2.5-Coder-7H5
22.5
21.5
18.3
41.1
57.2
first-stage GRPO
25,0,2.6
24.8F3.3
18.3
46.9ta.8
58.6;1.4
second-stage GRPO
27.4
27.6p6.1
17.1
47.7
58.4
SFT
28.9
29.0p6.5
17.7
47.7
63.1
Wwen..S-Coders 3-e---BBBiniin- Instruct
35.8
37.4
24.0
57.1
64.5
first-stage GRPO
38.3r2.6
39.5r2.1
25.1
58.3+1.2
65.4,0.9
sccond-stage GRPO
41.6ra.8
45.3,7.9
28.0p4.0
59.7+2.6
66.1
SFT
41.9
46.2,8.8
30.3
58.7
69.5g5.0
AUTOCODEINSTRUCT
AutoCodeInstruet Besides the evaluation benchmarks, we further construct AutoCodeInstruct, training dataset of comparable quality AutoCodeBench. Specifically, we collect data generated during the AutoCodeGen process that does not overlap with AutoCodeBench, and repeatedly sample DeepSeek-V3 -0324. Problems with excessively high pass rates (>80%) or low pass rates (<40%) are filtered out to ensure both solvability and appropriate difficulty. We further apply a two-stage deduplication strategy (MinHash LLM-as-Judge) across existing code benchmarks. The resulting dataset contains 37K verifiable problems spanning 20 programming languages.
Training Setup We conduct RL experiments via a two-stage GRPO (Shao et al. 2024) training strategy to unleash the potential of AutoCodelnstruct, based on the Qwen2. SC 32B-Instruct models. Concretely, we apply a data filtering strategy O by sampling the responses 15 times from the Instruct models and filter out easy problems with pass rates above 0.6. The remaining problems are divided into solve-partial and solve-none parts depending on whether the pass rate zero. In the first stage, only the vereepataall problems join training, with a rollout size of 8. In the second stage, we incorporate both solve-partial and solve-none problems, and increase the rollout size 16 enable better exploration for harder problems. Besides, for all colve-partial and solve-none problems, we additionally obtain correct code solutions from DeepSeek-V3- 022-43244 and perform SFT on Owen2, 5-Coder- 32//322B-2 Instruct. Details of the training configurations are presented in Appendix [
Results As shown in Table 6 after the first-stage GRPO, both models achieve noticeable gains on in-domain benchmarks (ACB-Full and ACB-Lite), which suggests that they begin to learn how stably consolidate existing knowledge. The second-stage GRPO enables the models to tackle harder problems, effectively pushing their multilingual capability boundaries and leading to significant performance improvements. Surprisingly, the models also show consistent gains on out-of-domain multilingual benchmarks such as FullStackBench and McEval. The performance of Qwen2. 5-Coder- 32B-Instruch on the programming contest task LiveCodeBench- V6 (20250201-2 00250201-20250500 also improves by 4 points. In addition, after the SFT stage, both models
8

---

Published as a conference paper at ICLR 2026
achieve even larger performance improvements than those obtained from GRPO, thanks to the distilled correct code solutions from DeepSeek-V3-0322 These results indicate that AutoCodcInstruct enhances the comprehensive code generation capability of models and demonstrate the effectiveness of our approach in synthesizing high-quality training datasets for code LLMs..
5 RELATED WORK
Code Generation Benchmarks The rapid evolution of code LLMs, ranging from open-source mod-
els (Roziere et al. 2023, Zhu al., 2024] Hui et al. 2024b) to proprietary LLMs (Anthropic 2025a OpenAl| 2024 2025a Gemini 2025) series, has reshaped code generation, create a demand for robust and contemporary code generation benchmarks. Pioneering benchmarks like HumanEval (Chen et al.)
2021) and MBPP (Austin et al. 2021) established foundational correctness on small Python tasks but suffer from contamination and limited language coverage. Later benchmarks target more complex
settings, such competition-leve challenges (Hendrycks et al.) [021] Li et al. 2022 Jain et al.
2025 Wang et al. 22025b] Zheng et alla,2222200 2025) and multilingual scenarios
Cassano et al. 2022 Peng
et al. 2024 Zhang et al. 2024 Jimenez al. 2024, Chai et al.
2025 Bytedance 2025| Zhang
et al.
ab). McEval (Chai et 2025)) is a massively mul lingual benchmark covering 40
languages for generation, explanation, and completion tasks. FullStackBench (Bytedance) (0255
assesses LLMs in realistic, multi-domain scenarios across 16 languages, employing a novel execution environment. However, due to the challenges of manual annotation, these benchmarks suffer from issues such as limited diversity and insufficient difficulty, making them difficult to scale in line with the evolving demand for high-quality evaluation. By comparison, our AutoCodeBench series adopts a fully automated and scalable approach to create realistic, diverse, and high-difficulty tasks. A recent trend, exemplified by the SWE--Bench series (Jimenez et al., 20244 Zan et al., 2025 Rashid et al., 2025 He et al.) 2025), focuses on evaluating LLMs in real-world software engineering tasks such as GitHub issue solving, thereby assessing models' comprehensive capabilities beyond atomic-level code generation. By comparison, ButcCodeBench specifically targets LLMs' atomic-level code generation abilities, which remain a crucial foundation for overall model performance.
Code Data Synthesis To reduce dependence manually curated data, a growing body of research explores automatic data synthesis to augment the training of Code LLMs (Luo et al. 2024
Wei et al. 2024b Zheng et al.) 2024 Wu et al. 2024 Yu et al., 2024 (Ahmad et al. 2025 Xu al. 2025). For instance, Evol-Instruct (Luo et al. 2024) uses heuristic prompts to guide LLMS in evolving existing programming problems, thereby increasing their diversity and difficulty. OSS-Instruct (Wei et al.) 2024b) prompts LLMS to generate new coding problems and solutions from raw, open-source code snippets. KodCode (Xu el. 2025) synthesizes a broad spectrum of Python coding including questions, solutions, and test cases and ensures correctness through a systematic selffverificatiio procedure. Some other methods focus on model self-improvement (Wu et al. 2024 Wei et al.. 2024a Chen et al. 2025b Zhou et al., 2025 Zhang et al. 2025c). For instance, Inverse- Instruct ( (Wu et al. 2024) is a self self-improvement technique that generates new instructions by maakktranslltinng code from an LLM's own training set, reducing the need to distill from more powerful proprietary models. Collectively, these data synthesis methods significantly reduce the reliance on manual curation and enable the continuous expansion of the problem space for training. Our work extends this paradigm of automation from data augmentation to the benchmark creation process. By leveraging extensive LLM-sandbox interaction, our pipeline not only automates the synthesis of verifiable test problems but can also be naturally repurposed for synthesizing high-quality training datasets.
6 FUTURE WORKS
Ensuring high-quality and reliable code data synthesis remains fundamentally challenging. Although the sandhox provides strong guarantees regarding the correctness of code solutions and their align- ment with test functions, the intrinsic quality of the synthesized programming problems and the completeness of their test coverage cannot be fully ensured. AutoCodeGen incorporates an LLM-as- Critic stage and careful prompt engineering to mitigate such risks, yet these mechanisms inevitably operate under uncertainty. To further improve reliability, future iterations of AutoCodeGen will incorporate repeated verification rounds using increasingly capable LLMs. Double- or triple-check validation loops are expected to improve accuracy, albeit at the cost of increased computational overhead and reduced pipeline efficiency.

---

Published as a conference paper at ICLR 2026
Moreover, repository-lewe evaluation and data synthesis represent more realistic and demanding setting, while the current multi-logic analysis provides only an initial bridge between fine-grained function behavior and higher-level software engineering reasoning. Moving forward, we aim to extend AutoCodeGenn toward SWE-Bench- and ferminal-Bench-stylee domains, enabling automated synthesis of repo-level tasks together with the stateful sandbox environments required for such scenarios, This represents a crucial step toward scalable and fully autonomous code evaluation frameworks.
7 CONCLUSION
In this paper, explored the large-scale and automated construction of code generation benchmarks. We introduce AutoCodeGen, an automated workflow based on LLM-Sandbox interaction, designed to generate multilingual verifiable code data without any manual annotation. Through this novel approach, we have successfully built AutoCodeBench, a large-scale, human-free code generation benchmark. AutoCodeBench contains 3,920 problems, evenly distributed across 20 programming languages, and is characterized by its high difficulty, practicality, and diversity. We also provide AutoCodeBenchh-Lite and autoCodeBennh--Copppee both chat and base LLMs. Our evaluation of more than 40 open-source and proprietary LLMs reveals that even the most advanced models still face challenges when confronted with the complex and diverse multilingual tasks set by Bench..eBench. Besides, we construct AutoCode large-scale, high-quality multilingual training dataset, and validate its effectiveness through GRPO.
for efficient and high-quality evaluation of
REFERENCES
Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi. Somshubra Majumdar, and Boris Ginsburg. Opencodeinstruct: A large-scale instruction tuning dataset for code llms, 2025. URL https: http://///wxwv.oog/////// abs wtt.://gg//.///..// 04030
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL pattps://arxivv http:////wwww...../a//////////// 20.0
Anthropic. Claude code, 2025. URL
Anthropic. Introducing claude 4, 2025a. URL.
u
Anthropic. Claude opus 4.1, 2025b. utt.htteee//ww..attrrricc..o///t 1-opu--41
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL htth::::/// http:::///xxxvv.rr//... 2108.11000 07732
Baidu. Fmiiex11turbo-322, 2025. URL. http:ttss//coooo..aad...... #HOP/s/Nm.ccvy6r.
WENX VENXINWORK NWORK
Bytedance. Fullstack bench: Evaluating 11ms as full stack coders, 2025. URL org/aba/2412. 00535
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022. URL httts:/ /xr.vv...vv//////////aa//aaaa wttpg/g///////////ae 082
27
10

---

Published as a conference paper at ICLR 2026
Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, JinKe, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang. Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang. Sufeng Duan, Zhaoxiang Zhang. and Zhoujun Li. Mceval: Massively multilingual code evaluation. In The Thirteenth International Conference on Learning Representations, 2025. URL ht.tps://openrevieew net forum Sitrrummiid-UunnnPPPPP0
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel feerbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021. URL htt.:htttt::///ttt 2/tp///////t////////////////////
74
Wei-Lin Chen, Zhepei Wei, Xinyu Zhu, Shi Feng, and Yu Meng. Do Ilm evaluators prefer themselves for a reason?, 2025a. URL httttt://aaxiii......mm////// org/ /bs/ 03846
Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Xinyu Zhang, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, and Zhi Jin. Revisit self-debugging with self- generated tests for code generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 18003- Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-99176-251-0. URL https
//aclantho wowpthhooooy..oooo ogw.ooo.......o................. wwg.y.rgg/025..0 acl wow.ellooonn.....
Cursor. The ai code editor, 2025. URL
DeepSeek-AL. Deepseek-v3 technical report, 2025a. URL [ttpps:/aarxiv.orrgaas///// 19437
DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning,
2025b. URL httts:://axxivu org/ abs 12948
DeepSeek-Al, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang. Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi. Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Decpseck-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. URL httph:ttt//tatit http://wxxx.......aaaaa 2400 11931
Gemini. Gemini 2.5: Pushing the frontier with advanced reasoning, multimooalimmiiiill..aao long context, and next generation agentic capabilities,
06261
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the the rise of code intelligence, 2024. URL https::/aaxiie org httpa/////211111990
Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Zejian Yuan, and Zejun Ma. Swe-perf: Can language models optimize code performance real-world repositories?, 2025.
URL httpps///aaai..ootaaaaas/a/////p
wttp:r/ggabb////o/ae 12415
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Mcasuring coding challenge competence with apps. arXiv preprint a wrXiv22105009933 2021.
11

---

Published as a conference paper at ICLR 2026
Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang. Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang. Zili Wang. Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models, 2025. URL
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tanyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng. Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren. Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024a. URL https
7/arxiv. org/ abs/
Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12166 2024b.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL httttp/s//ppeneeeeeee../. htaptt/foommmmmidodmmm
C
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL httpt://aaxxiii. wwwpaaawwwvvrraaa/aaaaaaaaaaaa/a
00515
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
htttps://arxiv. http:///axiii.ooo///////////////
Kimi-Team. Kimi k2: Open agentic intelligence, 2025. URL/ittpes//arxxvvorr///bb/////p
225534
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' l'Attmme, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378(6624) D December 2022. ISSN 1095-9203, doi: 10. 10.1126/science.abg) 158. URLhttp:: /x wwtpdxx..oooooooooooooooooooo.o. www.:oggg/o....... 1126 science abq1158
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang. Qian Liu. Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder and the stack v2: The next generation, 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference Learning Representations, 2024. URL
https://openrevia net wtt.://eeeettooouoooooooonuuunnn
OpenAl. Hello GPT-4o, 2024. URL Utt.atteer//geeaaa..cmm///e/eeer
OpenAl. Introducing gpt-4.1 in the api,
F/
OpenAI. 2025b. URL http://ttt///g/aaaaaaoo////////a
12

---

Published as a conference paper at ICLR 2026
OpenAl. Introducing openai o3 and o4-mini, Eoducing-o3-anee -4-in//
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. L.M evaluators recognize and favor their own generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htRtt:::: htttps://ppenzevierrr http:/ewwwwwweeeeeeeeeeeeeeeeeee feet/forummideee
Qiwei Peng. Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization, 2024.
02. 16694
Qwen. Qwen3-coder: Agentic coding in the world, 2025. URL /blog//wen3--coer/
Qwen. Qwen3-next: Towards ultimate training & inference efficiency, 2025. URL ht https //qwen ai
Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. Swe-polybench: A multi-languaae benchmark for repository level evaluation of coding agents, 2025. URL httpt:://taxi...oooooo /
.08703
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Seed. Introduction to techniques used in seed1.6, 2025. URL
om htm/en/seed! 6
ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong. Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang. Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Shen, Liang Xiang, and Yonghui Wu. Seed-coder: Let the code model curate data for itself, 2025. URL BtLLttts::/ waw.vv.vvvv..//..../...aa...aav. orggabs/////ooooooooot...p....rr abs/ 2506. 03524
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL
/bs/
2402.00 03300
Tencent. Hunyuan turboss Advancing large language models through tamba-tra synergy and adaptive chain-of of-thought, 2025. URL ptttpp::/aaxxv.oor//bbbss/
Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, and Bin Shi. Software development life cycle perspective: A survey of benchmarks for code large language models and agents, 2025a. httl:ttts///xx...oott
22500 05283
Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, et al. Ojbench: A competition level code benchmark for large language models. arxiv preprint arXiv:2506. 16395, 2025b.
Yuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding, Naman Jain, Zachary Mueller, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. Selfcodealign: Self-aligmment for code generation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 62787- 62874. Curran Associates, Inc., 2024a. URL httpa////t/////pooeeeeo....eooea er +Conference.pde
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empower- ing code generation with OSS- ASS-instruct. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 52632-5635550 52657. PMLR, 21-27 Jul 2024b. URL httpss:/prrceeddnng mlr. press v235/22555//// /24h html
13

---

Published as a conference paper at ICLR 2026
Yutong Wu, Di Huang, Wenxuan Shi, Wei Wang, Lingzhe Gao, Shihao Liu, Ziyuan Nan, Kaizhao Yuan, Rui Zhang, Xishan Zhang, Zidong Du, Qi Guo, Yewen Pu, Dawei Yin, Xing Hu, and Yunji Chen. Inversecoder: Self improving instruction-tumed code Ilms with inverse-instruct, 2024. URL abs 200.:////////222005500
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging. and verifiable synthetic dataset for coding, 2025, URL http:::/aaxiv..orggggggg
5/2503302951
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0925 009888 2025.
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning, 2024. URLhttps::/
/ttp://awivvvv/.aa wwt.:bsg////ooooooooooo..ooo..rt 2312 14187
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong. Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: A multilingual benchmark for issue resolving, 2025. htL.ttt:::// /ttpaaaaiivv//..///vv//ee..aaaaa org/ abs////////// 2504 02605
Alexander Zhang, Marcus Dong. Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang. Ge Zhang. Tianyu Liu, Zhongyuan Peng. Yingshui Tan, Yuanxing Zhang. Zhexu Wang. Weixun Wang, Yancheng He, Ken Deng. Wangchunshu Zhou, Wenhao Huang, and Zhaoxiang Zhang. Codecriticbench: holistic code critique benchmark for large language models, 2025a. URL httth::t//axiii/
g/abs/2502. 16614
Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Liu, Shihui Hu, Dengpeng Wu, Guanhua Huang. Kejiao Li, Qi Yi, Ruibin Xiong, Haotian Zhu, Yuanxing Zhang, Yuhao Jiang, Yue Zhang, Zenan Xu, Bohui Zhai, Guoxiang Hc, Hebin Li, Jie Zhao, Le Zhang. Lingyun Tan, Pengyu Guo, Xianshu Pang, Yang Ruan, Zhifeng Zhang. Zhonghu Wang, Ziyan Xu, Zuopu Yin, Wiggin Zhou, Chayse Zhou, and Fengzong Lian. Artifactsbench: Bridging the visual-interactive gap in Ilm code generation evaluation, 2025b. Uttpsstss://axxii http:///rrr/rbbb//..//00/0
Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng. Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodehemch: Examining coding performance mismatch on humaneval and natural user prompts, 2024. URL https: org/ httg//bs//44005 04520
Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, and Mengdi Zhang. Scoder: Iterative self self-distillation for bootstrapping small-scale data synthesizers to empower code 1lms, 2025c. (Rtt::tts:
org/ abs 2509. 07858
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodelnterpreter:: Integrating code generation with execution and refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, pp. 12834-12859, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1tt55/3/11/222.fnndini ptttps://aclanthoooog org/ 2024, ftn.i.ii...ii....... win.ingnnnacili 762/
URL
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang. Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, and Saining Xie. Livecodebench pro: How do olympiad medalists judge Ilms in competitive programming?. 2025. URL htttt::t//attttil www.voorggaaaee abs 6506. 11928
Zhipu. Glm-4.5: Reasoning, coding, and agentic abililties, 2025. URL htttt:://..aa//lll/ g1m-4.3
Changzhi Zhou, Xinyu Zhang, Dandan Song. Xiancai Chen, Wanli Gu, Huipeng Ma, Yuhang Tian, Mengdi Zhang, and Linmei Hu. Refinecoder: Iterative improving of large language models via adaptive critique refinement for code generation, 2025. URL attps: http///xxvvv.//// org. orgg/bs/25 abs
02. 09183
14

---

Published as a conference paper at ICLR 2026
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseck-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint 1IX1v:2406.11911, 2024.
Yaoming Zhu, Junxin Wang, Yiyang Li, Lin Qiu, Zong Yu Wang, Jun Xu, Xuezhi Cao, Yuhuai Wei, Mingshi Wang, Xunliang Cai, and Rong Ma. Oibench: Benchmarking strong reasoning models with olympiad in informatics, 2025. URL https:t/p////m. /attte/:/arrxv..ooo///////......
15

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
Manual Verification
17
Setup of Benchmark Comparisons
17
B.1
Molii-Logic
17
B.2 Difficulty
17
B.3 Category Distribution and Language Distribution
18
B.4 Problem Length
19
The Results of AutoCoodeBench--iit
19
D The Results of AutoCoaleeeeencc-CCCopppeeee
19
Leaderboards
19
F
Performance Analysis of Scaling Laws
19
Baselines
20
Hypotheses on Model Bias the Generation Process
21
AutoCodeInstruct Experimental Details
22
J Multilingual Code Sandbox Service
23
Prompts for Automated Workflow
24
L Multi-Logic Task Analysis
24
M LLM-based Error Analysis
26
The Order of 1/O Format
28
16

---

Published as a conference paper at ICLR 2026
Table Comparison of Accuracy, Upper Bound, and Model Performance.
Average
Python
C++
Java
JS
Go
Shell
Problem Accuracy
87.6
83.5
88.0
86.0
89.0
86.0
93.3
Current Upper Bound
66.9 6207
61.7
71.5.16.5
76.1
58. a....
65.4.220.6
68 6.24
Claude Opus (Reasoning)
44.6
4..3
44.1 44.1
$5.9
38.6
37.2
51.6241.7
A MANUAL VERIFICATION
Although our pipeline enforces quality control through specifications and an LLM-as- mech- anism, we further validate AutoCodeBench with human annotators. We employ a Human-LLM collaboration approach for data quality validation. Specifically, we design prompts in the native lan- guages of the annotators and use the DeepSeek-R1 -152 generate detailed reasoning processes and checklist- based annotation results. The prompt is shown Figure 10 During the annotation process, assume that the programming problems are completely correct. The primary task of the annotators is to assess the correctness of the test functions and their alignment with the programming problem, based the LLM's output. We allow for test cases that may not cover all boundary condi- tions, focusing primarily on the correctness of the test functions rather than their comprehensiveness. The annotators pay particular attention the following aspects:
Whether the function names, class names, variable definitions, and return types are consistent with the problem description;
Whether the test cases exhibit randomness or non-reproduciiiilty:
Whether the test cases contradict the logic presented in the problem statement;
Whether there are any precision issues with the test cases;
Whether the test functions include test cases that are not addressed the problem description.
We calculate the problem accuracy rates for different programming languages (Python, C++, Java, JavaScript, Go, Shell), as shown in Table 7 The results indicate that, despite the presence of some noisy data, our benchmark model still demonstrates high accuracy (87.6%). Furthermore, even after removing the noise, the current SOTA model shows significant room for improvement (A43.0), further validating the high difficulty level of our benchmark. The performance of Claude Opus (Reasoning) in these six languages are only 44.6 (4.6(443.0) highlighting the significant potential for improvement. Besides, we find that, compared to logic errors in the problem description and errors in the test functions, the most frequently occurring issue is incomplete problem descriptions. For example, some test functions reference class or function names that are essential but not explicitly mentioned in the problem description, they require natural language outputs for edge cases that are not explicitly specified the problem statement, leading to mismatches between the generated code and the test functions. Interestingly, we observe similar issues in manually annotated benchmarks, highlighting the significant challenge of creating comprehensive and accurate programming problems for annotators.
B SETUP OF BENCHMARK COMPARISONS
B. MULTI-LOGIC
AutoCodeBench contains tasks that demand executing multiple functionalities, such implement- ing both an addition and a multiplication function simultaneously. In contrast, the tasks in other benchmarks are almost exclusively focused implementing single core functionality.
B.2 DIFFICULTY
We rate the difficulty of each benchmark based the performance of DeepSeek-V3-033 Specifically, benchmarks with pass@ below 40 are assigned five stars; those between 40-50 receive four stars (e.g., LiveCoddeBench-v6: 46.9, AutoCodeBench: 48.1); between 50-60 receive three
17

---

Published as a conference paper at ICLR 2026
(a) AutoCodeBench
(b) AutoCodeBench-Lic
(c) FullStackBench
(d) McEval
Figure Category Distribution of Different Benchmarks.
(a) AutoCodeBench
(b) AutoCodeBench-Lin Lite
(c) FullSttckBench
(d) McEval
Figure 6: Language Distribution of Different Benchmarks.
stars; between 60-80 receive two stars (e.g.. FullStackBench: 67.0, McEval:
72.3); and those
above 80 receive one star. HumanEval and MBPP are excluded from evaluation due to extensive data leakage and its overly simple problems. For reference, DeepSeek-Ve (October 2024) already achieves pass @ 1 of 91.5 in HumanEval.
B.3 CATEGORY DISTRIBUTION AND LANGUAGE DISTRIBUTION
We prompt Claude
Sonnet 4 to generate 20 language-agnostic category labels for classification:
Language Fundamentals, Functions & Modules, Object- Oriented Programming. Functional Programming. Memory Management & Performance. Error Handling & Debugging
Core Programming Concepts:
Data and Algorithms: Data Structures & Collections, Algorithms & Problem Solving, String & Text Processing, File & 1/O Operations, Concurrency & Async Programming
Network Programming & Communication, Database Operations & Persistence, Web Development & Frameworks, Mobile & Cross-platform Development, Systems Programming & Low-level Development
Application Domains:
Advanced Topics and Tooling: Data Science & Analytics, Machine Learning & AI, Testing & Quality Assurance, Development Tools Ecosystem
In addition to BuncCodeBench, we conduct task tagging and language distribution analysis for AutoCodeBench-LLiee BullStackBench, and McEval. The results are presented in Figures and FullStackBench demonstrates comparable category diversity to AutoCodeBench( (eeeecn-hh(-te) but suffers from an imbalanced language distribution. In contrast, McEval exhibits well-balanced multilingual distribution but lacks diversity and balance its category coverage. Our ButoCodeBench( (Lite) achieves the most comprehensive category coverage while maintaining balanced multilingual distribution, enabling thorough and accurate evaluation of LLMs' multilingual code generation capabilities.
18

---

Published as a conference paper at ICLR 2026
Table 8: Pass @1 I (%) performance of different base models for 3-shot AutoCooeBBnnh-Comppppl
Average
Pytben
Cyp
Jona
IN
Ge
Caterp
Dart
Julla
Koriin
Por
PHP
Racket
K
Hulty
Bost
Scala
Swith
15
368+ Madec
Deeptale
S-Cale--735
16.5
15:11
710
12:0
1K31
SED
10V3
12.0
35:1
25.3
310
180
220
3/.
41.3
Queid 5-28
359
32.9
22.0
25.0
400
22,0
34.0
62.0
22.0
42.0
SRL
46.0
42.3
450
200
260
35.0
25.0
25.0
300
$4.0
Medak
Scal Sall-Cola-dB 68- Bese
MAA
25.0
22.0
400
368
33.0
13.0
SL.
24.0
43.0
50.5
48.0
25.9
35.0
12.0
260
32.0
18.0
20.0
560
68.0
Uweeg 5- Urdae-78
34A
3331
10.0
22.11
2600
8.0
32.0
3411
2113
180
16.0
34.
14.
54.3
DecpSedu Ooder 5.78 Seae
22.9
2309
14.0
368
368
18.0
18.0
0
3.0
44.0
20.5
15.0
25.0
18.0
12.0
14.0
34.0
6.0
10.0
4.
42.9
Table 9: Pass @ (%) performance of different models for A
Polsin
Jana
Cateryy
Dart Elerr
Koils
Porl
Hacket
Haky
Coael
37
135
55
Open
Clande Upens (201
60.5
63.9
65.6
18.1
66.3
31.5
60:2
(9:1
16.3
13
71.5
01.1
55.0
50-3
68.5
12.5
61.5
56-3
65.4
17.4
Clende (302N0091
65.8
135
18.
40.5
11.1
00.3
13.3
48.3
26-7
66.0
44%
970
643
11.
783
75h
613
73.5
68.3
19.8
adledd 2015--00 du 3o
015
(3
15.1
516
534
153
05.5
1633
16.0
11.8
16.1
67.0
16.3
104
73-1
16.5
96.6
10.3
13
630
624
611
653
61.3
6011
13.5
15.1
960
08
05.
41.3
194
136
66.0
605
541
Geeble4 Flack
114
681
56.4
526
153
45.
15.
964
150
140
595
565
861
516
922
663
614
231
430
960
43.
484
261
Hononen
963
431
346
236
560
585
361
413
361
453
955
Kint- 3.3 2.3.50000000
352
461
563
46.
485
518
Qweab-2300- 7aa7kng-207
373
351
543
62.5
261
333
600
484
500
463
516
Qread- 2300 2230 Sa
413
233
253
433
56.
18.8
312
459
261
238
438
234
311
331
260
01
210
253
352
123
313
238
325
533
545
513
280
43.
413
518
Sredll
433
350
226
325
300
400
1888 Opm ware Make
514
Grees--hea 180- Tleeking
46.3
329
4.0
459
343
454
26.
425
603
413
293
515
543
1611
9.1
441
47A
418
46
413
45A
362
357
61
44.1
M03
43.
258
641
43
361
463
M43
(hwad-85
359
252
244
153
15.
261
212
1611
28.1
233
23.
248
183
300
(wead-323
323
232
13.
111
165
124
25.5
361
16.1
333
36.
235
230
21
23
115
(wead-88
114
282
64
26.
394
365
16.3
160
348
111
235
361
233
185
153
250
113
116
45
(wead Cudee-33--Baae Bt-32--bwttratt
374
339
268
386
81
350
366
11.
206
135
151
464
S31
433
163
148
465
26.
233
358
484
Owedd Dadge- tacined
162
123
54
715
138
184
163
94
111
12
145
255
53
37
67
125
31
32
110
193
380
113
313
26
253
167
482
106
135
155
383
263
163
115
161
255
115
41
123
Opeal SB-laanad
341
135
1:3
203
335
153
26
185
170
325
122
22.
361
233
151
153
255
75
165
143
363
B.4 PROBLEM LENGTH
We use Qwen2. 5- 32B-Instzuct tokenizer to calculate the problem length. For Multi-SWE. Bench, problem length is not reported, its tasks depend on extremely large multi-file repositories.
C THE RESULTS OF AUTOCOORBEENH--LTTE
The pass@1 1 values of AutoCodeBench-Linn is shown in Table
D THE RESULTS OF AUTOCOOSBBENCH-COPPLEE
The pass@1 values of AutoCoodeBench-Conplee is shown in Table
E LEADERBOARDS
The leaderboards of ACB-Full and ACB-Lite are shown in Figure and
F PERFORMANCE ANALYSIS OF SCALING LAWS
Figure compares parameter scaling and test-time sampling scaling across different models. The parameter scaling law (left) shows significant variation between models, with Owen3 (Think) Series demonstrating the steepest scaling curve, indicating that chain-of-thouse reasoning particularly
19

---

Published as a conference paper at ICLR 2026
AutoCodeBench: Overall Performance LLMs
Claude Opus 4.1 (Think)
55.4
GPTS (20250807)
53.6
Claude Opus 4.1
52.6
Claude Sonnet (Think)
(high)
50.0
Cluude Sonnet
49.3
Gemini2.5 Pro
48.7
DespSeole13.1 (Think)
46.2
GP74. (2025-04-44)
48.0
Qevno-25B-422--hninn---222
47.7
46.8
Gueale5-erable
Seed é-enabled
45.3
Seed1, 25075
45.0
Deen3--Cddn--66844458------
44.8
Humyunn//uuosS-2225501
43.8
43.1
Seed1. 6-clsabled (250615)
Qwer3-328 (Think)
40.6
ERNIX Turbo-32K
39.6
39.6
DeeeSeok Caw.eeek---eee/iiiiiaaaeeee -nnstrrtrntttuuttt
37.7
Ower3-148 (Think)
37.6
Qwen2.5 5 Ceder 328 instruct
Seed-Doder-B-iistruuc
Qwen3-328
Qwen3-148
Qwen3-88
DeepSeek-Coder- 10-instruct
28.5
Qwen3-88
23.3
Qwen2 5 Coder 7B Ri-instruct
22.5
DeepSeet-Coderaa 78-instruct
20.5
Rea soning
Open BB-instrcct
19.3
Non Reasoning
20
Average Pass@]
Figure 7: AutoCodeBench leaderboard showing Pass@1 performance of various LLMs.
benefits larger models. The test-time sampling scaling law (right) reveals more uniform behavior, with three models showing similar improvement rates from increased sampling during inference. These results suggest that while test-time sampling provides consistent benefits regardless of model size, reasoning capabilities scale more aggressively with model size.
G BASELINES
We evaluate diverse set of open-source models with sizes ranging from 1.5B parameters, as well as leading proprietary models. These models are classified based their families:
OpenAl: GPT-5
OOeeAAl 2025b), o3 and 04 -mini OOpenAl 2025 and GPT4. Ope
nAI 2025a).
Claude: Claude Opus 4.1 Anthroppc 2025b) and Claude Sonnet 4 (Anthropic 2025a).
Gemini: Gemini 2.5 Pro and Gemini 2.5 Flash (Gemini 2025
DeepSeek: DeepSeek-V3, (DeepSeek-A) Al 2025a and SeepSeek-Coder Se-
ries (DeepSeek-Al et 2024 Guo al. 2024)
20

---

Published as a conference paper at ICLR 2026
AutoCoodeBench-Lte:: Overall Performance of LLMs
Claude Opus 4.1 (Think)
69.9
GPTS (20250807)
67.0
Claude Opus 4.1
63.8
03 (high)
Sonnet (Think)
60.5
Cluude Sonnet
59.8
Gemini2.5 Pro
59.1
DeapSeal-33.1 (Think)
57.5
Owen3 Thiniinn 2507
57.3
GP741 1202 5-04-14)
56.9
55.0
Seed1. 6-1hinking-550155
53.7
Seed]
53.2
Gemini2.5 Flash
52.9
De.pSeek-V311
52.2
51.5
Humyunn-laboss-22255111
50.3
49.8
Seed1. 6 6-clsabled (250615)
Qwen3-3228 (Think)
Qwen3-Next-80B-44
46.3
GLMe---Aiceerablle
46.2
ERNIE Turbo-32K
44.4
Deen--Nex--BB----BBB 3B-instruc
42.6
QwenB-148 1-1 (Think)
40.7
DeepSeeck-Ced Coder V2-instruct
40.5
Qwen2.5 5 C-Cer 328 instrrct
Seed-Code--BB-inrtte
32.7
Qwen3-328
(Think)
27.8
DeepSeek-Coder-3 J0-instruct
27.7
21.5
Qwen3 s3
21.4
OpenCader-RB-iinttt -ibiisttrt
20.1
Rea soning
DespSeek.Codrrs 7.inntrtt
19.9
Non Non-Reasoning
30
Average Pass
Figure 8: LutoCoddeBench-Lttee leaderboard showing Pass performance of various LLMs.
Hunyuan: Hunyuan-Turbos Tencent 2025
Qwen: - Bwen3-233B-1222--Lssoot-----
Next 808 A3B Qwen 2025),
Sreen3-235B-A22--11
2507,
and Owen3 Series (Yang et al. (025).
(wwn 2025).
Owen2. 5-Coder Se-
ries (Hui al. 2024a)
Seed:
Seed
2025
Seedl. 6
Seed
2025
and
Seed-Coder-8B (Seed 2025).
GLM: GLM-4. and GLM-4 5-Air
Zhipu 2025).
Other Models:
32K
Baidu 2025
Kimi-i -2
Kimi-Team 2025
and OpenCoder-83 (Huang et 2025).
H HYPOTHESES ON MODEL BIAS IN THE GENERATION PROCESS
is well-known that models exhibit inherent biases, particularly their tendency to favor their own outputs common phenomenon in automated data synthesis and evaluation tasks (Panickssery ctt al., 2024; Chen et al. 2025a). Our automated workflow is no exception to this issue. While completely eliminating such bias is challenging, employ several mitigation strategies. Specifically,
21

---

Published as a conference paper at ICLR 2026
Parameter Scaling Law
Test-time Sampling Scaling Law
Cabrrssaa Ower3 (Ttind Sento
45
13
14
Oven2,5 Cat.2a5CCeeeaaaaarr tnitrct
Owee 48
25
73
225
10
Figure 9: Scaling laws for different models.
Table 10: The average pass @1 scores and rankings of models at different stages.
Initial Stage (Rank)
After Simple Problem Filtering (Rank)
After Critic Filtering (Rank)
DeepSeek V3-0324 DeepSeck-R1-0526
25.7 21.4
31.6 +a. (4)
48.9 (2)
28.7 -203 (2)
36.2 +7.5 (2)
46.4 (4)
28.1 1883 (3)
34,9 +6.8 (3)
Gemini2.5 Pro 0wen2,5-Coder-33 Coder-32B Instruct
51.4
31.6 1
38.7 +7.0
(5)
17.1 22088 (5)
22.0 +4u (5)
we intentionally only use DeepSeek series models in the workflow to prevent bias from affecting other model families. We hypothesize that using DeepSeek-v3-0324 for code generation and DeepSeek- -R--eekk-R1022 R1-0528 8 for the Critie process may introduce favorable bias toward DeepSeek families. Seappeeee--oee---eee
To counteract this, we employ phase, creating a "push-and-pull" mechanism that balances potential biases across different stages.
te during the simple problem filtering
To quantitatively assess bias, sampled 3,600 data points across six programming languages (Python, C++, Java, JS, Go, and Shell) and tracked performance changes at each generation stage, as shown in Table I0 The results reveal nuanced bias patterns: simple problem filtering negatively impacts smaller models (Qwen2. Co33er3ee----- Inst Instrrct) more than DeepSeek series, while the Critic process benefits DeepSeek- Ra1--R1-00 R1-0528 but surprisingly provides greater improvements to reasoning models (o3 and Gemini 2.5 Pro) than to SeepSeek-V33-332
This suggests that model bias depends not only on model family but also on factors like model size and reasoning modes. Furthermore, as mutual distillation between models from different families continues, this bias becomes increasingly difficult to measure. In conclusion, we believe that our automated process may introduce favorable bias toward the DeepSeek family of models, but the impact is minimal.
AUTOCODEINSTRUCT EXPERIMENTAL DETAILS
We apply separate data filtering for the two Instruct models, as the performance varies across models. This results in 8684 solve-partial (0 < pass rate < 0.6) prompts and 4882 solve-none (pass rate = 0) prompts for Qwen2. -- solve-none prompts for Qwen2. the batch size is set to 128 and 64, respectively for 32B and 7B experiments. The learning rate is set to x 10 and the maximum input/output lengths are 8192/8 192. The 32B model is trained for 60 steps in the first stage and 80 steps in the second stage, while the 7B model is trained for 70 steps in both stages. During SFT, the batch size is set to 64, the gradient accumulation steps set to 2, the learning rate is 1 x 10 and both the 32B and 7B models are trained for two epochs. For evaluation, adopt greedy decoding for all the models and the maximum output length set to 16384.
10518 solve-partial and 3294 During the two-stage GRPO training,
22

---

Published as a conference paper at ICLR 2026
#CONTEXTA
** **iggggugge ** test developrent expert.
two 1. ** **/OEETIONN The problen description, outlining the functionality and constraints that the code under test should fulfill,
will provide
2. ** FICCTOO The test script, containing severol test
Your task is rigoruusly review the
** ensure it truly veri fies the requirerents of the ** (QUESTTION"** ** and provide structured review results for the deta annotators.
DEJECTIVE x
Revies the **|TEST_
according to the following oudit rules (in fixed order), providing boolesn value and 20-48 word justiFication for each:
1. Bariing/Sigretum Mi seotch Function nores, classes, variables, and return types in the test do not natch the ones described in the quesston.
Randomeesz hanaas/on--nennniiiie
foctors,
results unstable (e.g.. colling
wi thout
or relying system tine).
Incorrec Torget
functiorality that described
Hand ing
precision requirerents then preciston reeded.
iscloseO sirilar aporoxinate
Exception Swallowing
Inevocutable ooiing default environsent exceptions except) which obscure actual Irrelevant Requirenents The exceptions).
okay, the entry (like rame__
executable. checks for functionality required the problen (reasonablee edge
8. Other Tssues
Any defects rot covered by the above rules.
STYLE
Structured, concise, ongineering tone
Standard JSON formt: fields should use snalie case Justification should be 28-40 words, in Chirese
Professional, objoctive, direct
AUDIENCE
Date amnotators sentor develogment experience, who judge quality scripts
#RESPONSED
Only output the follaxing 1SON structure (without Markdown block tags):
jjson
"rule results": [
"rute": "Kariig//iggatuu Signnatuee Miseotch",
"result": true,
"reason": "Exomple: Functionn nare "odd" does not notch "sun' in the problee"
iters in total in the fixed order
Sumaary'
ovverall.puss"" folse,
Foi Led_rule"" Key points":
["Noming/S (inninn/Signuture Mismotchh", "Rundboresss Barekenness//on---eee Saaisss//nn-Dnnnnnn Mann--etermnissr" foctors
results; test target deviates probles's
reqvirement
results:
(result should true folse).
summary. evenal poss: false if ony rule result tolse; otherwise true. summany. fai rules: List of all foiled rule descriptions; empty if all poss.
sumwary. key points: 250 words sumorizing the rain flons.
USER INPUT #
-QUESTION:
(question)
<'QLESTION
FTEST_RACTION-
(test_ (test_Furctioon: Function)
_/TEST_FUNCTION
Figure 10: The English prompt of annotation and critic.
MULTILINGUAL CODE SANDBOX SERVICE
This service offers secure and hegh-performance environment for the compilation and execution of code in over programming languages. supports large-scale code data validation, making suitable for high-volume, automated testing scenarios. Our multilingual sandbox has the following features:
Multilingual Support: The service supports more than 30 programming languages, includ- ing popular ones like Python, JavaScript, Go, Java, C++, and Rust, providing versatility for various use cases.
Security Isolation: Code execution is isolated within Docker containers, ensuring that each execution environment is separate. Additionally, iptables firewall rules are applied to maintain high level of security, preventing unauthorized access interference.
23

---

Published as a conference paper at ICLR 2026
Table 11: Statistics of the 1,622 multi-logic programming problems.
#Problems
#Test Cases
#lanngs
Prob Len
Solu Len
Difficulty (E/M/H)
MI.PP
1,622
16,131
20
576.4
610.0
238/315/1069
Smart Code Integration: The system automatically manages the integration of function code with testing code. It adapts to language-specifee syntax, ensuring seamless execution without requiring manual intervention for code merging.
High Performance: Powered by Gunicorn multi-process architecture, the sandbox sup- ports concurrent execution of multiple code instances, making it capable of handling high volume of requests efficiently.
The service provides a clean and easy-to-use HTTP-based API, allowing developers to interact with the sandbox programmatically, whether for integrating into larger applications or automating tasks.
RESTful API:
Extensive Language Support: Beyond the mainstream languages, the sandbox also sup- ports emerging and niche languages, allowing cater to wide variety of development environments and user needs.
Custom Execution Environments: Users can configure specific environments for their tasks, enabling tailored execution conditions based on their unique requirements.
K
PROMPTS FOR AUTOMATED WORKFLOW
The prompt of generating code solution is shown in Figure
The prompt of generating test function is shown in Figure 12
The prompt of generating programming problem is shown in Figure
13
The prompt of LLM-as- Criiii is shown in Figure 10
The prompt of translating languages is shown in Figure 14
L MULTI-LOGIC TASK ANALYSIS
First, we provide clear definition of multi-logic programming tasks:
Multi-Logic Programming Problem (MLPP). MI.P is a programming task whose correct solution requires implementing and coordinating multiple core logical units-such as functions, classes, or modules. Each logical unit corresponds to independent semantic responsibility or algorithmic objective.
We further divide Multi-Logic Problems into two categories:
Intra-Logic Problems: multiple logical units collaborate to achieve single overarching functional goal.
Inter-Logic Problems: multiple logical units address distinct functional goals, each inde- pendently contributing to the full solution.
Following the structure of Table 3 perform detailed analysis the 1,622 multi-logic problems in AutoCodeBench. As shown in Table II multi-logic problems exhibit substantially longer prob- lem descriptions and longer canonical solutions compared the overall dataset, reflecting richer instruction structure and greater intrinsic difficulty.
In addition, we used DeepSeek in each lolti-logic problem, and (2) the distribution of Inter- Logic and Intra-Logic types. On average, each problem contains 3.37 logical units. Among them, 389 are Inter-Logic, 1,223 are Intra-Logic,,
to compute (1) the number of logical units contained
24

---

Published as a conference paper at ICLR 2026
V Code Ber chrnit Oon corsioncion Task
under coce tenchmenk, fneed help me creata functin,,, on teet functions,, Later conatruct programming problemm board
laae Diveneey
Ulven cods Mis coda smpputt unyy programmeet language, "weef contaied" "hwritable", ""oomeghh Python funation You silo need creans two teeting tu nxtions for reating deme teeinggf and "uul teeing)" Rellow the disa ledrouurimenta ta.
"rdlin and avotre ts lage and nemove core lage recusaary createe
Provide ""duce distinct blookgr"
block meontiinn tython funattores that conbent oone lngic. Two blocks torthe rescfuncdonee rodonn (' demo, besting] and (Lll, beod highiigg"" the traces urspped with python
The lundtion ahould contaan "heo more than casss It be used "ceenorstrstt"" the incur-cutput format.
esch case, "hard code the input values" prin both the repu and comroponnigg ourput witien demootestigg) ceocuted. Avand of crese inttis inthiiitnnteee function, and fro on haver teet cress for de demonoataiinn parproos.
Codo "Mul bestingt
wormpre
should des
pooughhtttte functional by converted
inclucig:
""ittess caase"" teot
used demoo feating"
"* "BBuuddory cases* (n g. minimiii innrimmmmmmdmmm un values, emety inpurst
LLaag cussu tm that tast the rabustness of the cocs de abnormat valua, sour dayyy condition cheec
ench teat case, "Tharhoode the rpur values" and prin borh the inpur and the commoponddngg ing sumpur when teetings" noncuned
agel the funccionia pronde compretionnnne hoemrne inputs and out notion being to obnain output, Input handking ttal, validation, edge cosos)
should done the
tselt,
4. <iteee Standand and Thins- Python ircareeet
You are allowed us stancord lyrtoni loa mmath" wiit' janddtiidd pant Mavorios 3f roquu oct for implemendin the core loglo.
Ersoue oll depondono is clearly oxplail commo
"Evocurable Codee*;;
Phovede "haal contained and enecusabte inynoon coater with hand coddd tust impurs.
Rerrov any non--oon logle nush no plats, the Io, undiud funnttion.
the or ginal cods not cirecity ceocutable ledi, due depencency on speci tie busil logis or onternal systerma), adupt the code to sotf staatanne testable farms semuu the mey lind contret or simpliyynng the lngio whine prosersing the cone functiorally by. ovarty simptat tike fulunciion
bly bwo nun is, puease exoluc logio more chotlengingg rignng everty flluu compltcx like legics
hert, simplity logic modtentely linging.
enCodee Styde and Nariing:
fellow "Pythonec"" narning convertooe and eroure that funstan rammen sew "ddeeristtoe" "celear" in camagying ther purpose.
Provide "hlloar and complse comminess" oxplaining the logio and purpose eocch notion, especisily the steps..
Ourgut Requinementa
vil 1, Corve Python Furctiono
""Bppenderreee Lise standterd Pyther librrrees ge methi thirdsparty llomueee when necaeooery
"AAdppration Excoucabillan* i origin coplands external
spocific business infowo, modily to be contained hardooodding purs,
on* Eliminnn zapi, essental suchas fl funotions.
"RRo Non
pracecaes, inctudi indertatn tation, antaatttt commentann, and sutrerence to Moins gueaatanss comments exptein the notionality by the code andiaar steps
"oade styte" clioe gyttt
VW Exocutabra Test Functions
"Function Narmer
demo twetingl"
'ull teeting)
furatione.
""noorghh Hardzode the tnstisputs withar the tost funttiacions, Do parametios into the domoo bestingl or Tull testinn)) functions,
Daatpuenn Diiectty prin the west inpart and their copppppendingg rpups to comeutes Finure cilputt sew in eorl--eeieliaclle typee je strings, Integees, firers, ists). nonot include the word Cuup-t"indhe plimecrresuto.
""eppedderran caser" Use conntrriiyttto litrrmee ii math print) and speryp tiometes, mended,
lagle.
TTssing Adaprasion" trineeiiginn coce's lo part uage systom, sdaptthe domo cestingt' tootingg functions croate a seff concained beating
Codde Srylerh Follow Python testing process.
functions, with proper indertation
Include
TTeee FnrttonFFrope The euuchre wndfforntt of demoo sweting) and "ulll twottng)" should ioentiia,, whth thh difieence baing the number of teet cases.
Struourr
Figure 11: The prompt of generating code solution. Due to the excessive length of the prompt, have omitted the latter part.
and 10 fall into both categories. Opus 4.1 achieved 50.6% and 51.5% performance on the Inter-Logic and Intra-Logic types, respectively.
Figure 16 provides an example of an Inter-Logic problem. The task requires implementing three separate classes, each serving a different functional purpose-clearly corresponding to independent logical goals. Figure 17 shows more complex hybrid case. The problem requires implementing Rational class containing multiple internal logical units that jointly support the overarching goal of "rational number representation and operations," " making them Intra-Logic. At the same time, the auxiliary function "parse_rational" rationals" handles separate goal of "string parsing," forming an Inter-Logic relationship with the Rational class.
The results of Opus 4. 1 and these cases indicate that, in multi-logio problems, more complex instructions, greater logical demands, and intricate relationships among logical units pose greater challenges for L.LMs.
25

---

Published as a conference paper at ICLR 2026
Fll prccede yourith e Pytton codi, cost laner on call (nnlusing inpera tietuses this Pythonfu rction, and the tesk cutput ooncarcd ahtr excouting soatt lnncion coll. diatomnmms
provided pipuss surpons
place those bssort"
Youcanonty inpuus
Roosa ganoreti a fythor test funetion us assert stane morros Youwill be provided witt:
Python nediooesi repungg
lipurte.
Test You receto actcklidd
function
assert
Avoid ussentiore that offter listing prezes scross mach ung nat cutput invotves tasting-oset numbers, and the prob kem con sugaestts consicer using maingmmath mith saaa) usserting thin ressonabee ubsa toleranee the test casis onforently invove such compurioons proceatin concem, Howeve, peleetine direet acpuxitg ity the proutred nurpues wowe
Al essent statements must be ploced withi fundionn pomod besg).
Please "ww acparatio Lunctions oach
test fundiinni provic, crounnng ng
o They
Hore aus coumple:
"Oodder"
hommmtthimeer
der prrmmentt
i'n F:2:
interpptort)
Si mn
return True
primeety
ruturn
Turctioon
bothen
test,
for innneesssssess printf (N7)
WrOon lngic rcsult ithh primetri
serfoliaadee
(poau)))
.ramaa main demo seeting?
Iepul: Tothee prtmenmmeett::
Figure 12: The prompt of generating test function. Due the excessive length of the prompt, have omitted the latter part.
M LLM-BASED ERROR ANALYSIS
In this section, we present an error type analysis. As shown in Appendix fully automated pipeline inevitably introduces certain amount of noise. To distinguish noisy data from genuinely difficult tasks, we designed a checklist- based template to systematically categorize emror types. The checklist is as follows:
Model Solution Errors
Misunderstanding Problem: The model fails to correctly interpret the problem statement, leading to fundamentally incorrect solution.
Failure to Comprehend Complex Instructions: The model fails to correctly interpret or decompose multi-step or nuanced requirements in the problem statement, resulting in conceptually incomplete irrelevant solutions.
26

---

Published as a conference paper at ICLR 2026
You are on dxportooeed progarming seccaratie, and comeet Pytten cartent avang with more funitionn designed varty commecrreo, Your base "Iperamite programmm www.engggetnnnn proditmm Pyston
Button Butoo, odopt at cratting mosoar, conciss, and colcctiooall piogrommic pe probiems "h Fill supply with "*goff conta nod.
diocely comossionds
Hew the codla and tuncheras
|Pythen codk))
cocondenco
Pythen code and]
jlent function demmaend)
Hest tunstion]
Placos srcure the protitem you gresentte achtenee to motimeiig cothce cottinngottii regurmmeettt memente:
Lengu age Specificaton: Explic dy stote that solutions must plomented Pyther,
Picblom Desoriction: Dany
probiom ciicc nomioiiuuueel ig age. tocinicc gon, unnocessary desails, nimts
Curtpri semat: Delone finhs, a llowelidee rideta nyoost
format typea, atructure,
Uetoo expected output
constrantt inourt
Exammie Usagee Use the lust cass the aiorementin taot fundtio damoo to conatruct csample usage, The number caaza in this cusuaaty mosuuttyynonn more than threa. modiity euptain the conee-jjur copy them wertattm.
The probiem ipoin test caraaa bepond
any solutton andiany
prow ded examples.
Ploaso ono soo the gone doppoononngngiiingg probien wi equestioe and siquestione lags.
Figure 13: The prompt of generating programming problem.
Table 12: Top-10 10 error types.
Type
Count
Unreasonable Test Cases
313
Algorithmic Logic Error
218
Missing Functionality
157
Misunderstanding Problem
151
Algorithmic Logic Error Misunderstanding Problem
121
Insufficient Fundamental Knowledge
108
Algorithmic Logic Error Missing Functionality
106
Algorithmic Logic Error + Unreasonable Test Cases Algorithmic Logic Error Insufficient Fundamental Knowledge
73
57
Omission of Edge Cases
55
The model fails to handle extreme special input conditions. Missing Functionality: The model understands the task but fails to implement all required components or subtasks in the generated code.
Omission of Edge Cases:
Algorithmiic Logic Error: The algorithmic approach or control logic contains mistakes leading to incorrect results.
Insufficient Fundamental Knowledge:
The model demonstrates a lack of understand-
ing of basic programming syntax, data structures, or language semantics.
Problem Design Errors
Logical Contradiction in the Problem:
The problem statement contains internal in- consistencies or mutually exclusive constraints that make correct solution impossible.
Ambiiuuus Problem Statement: The problem description is unclear or lacks explicit constraints, leading multiple valid interpretations.
Test Design Errors
Unreasonable Test Cases: The test data deviate from the problem definition.
We applied this checklist to analyze all 1,745 tasks that Claude Opus 4. 1 failed to solve, using DeepSeek-v3 2-Exp the judge L.MM. If an was attributed to Problem Design Test Design, the task was classified as low-quality. Otherwise, we considered it a valid high-difficulty task. The Top-10 error types shown in Table
In total, 538 tasks contained Problem Design Errors or Test Design Errors, accounting for 13.7% of all tasks. The remaining 1,207 tasks were judged be high-quality and genuinely challenging.
27

---

Published as a conference paper at ICLR 2026
You are on dxporionens programing morninggggnnn muster en oxperise mutipic longuggs,, portcularly in Pho. livl provid youreeh:
retersnce astution another typicuity Pythaniiawwt wir (onllaaattenphttrrttt Cameapanding cusus functions inctuae damo test cussca
cases
Your taskis to:
Accapately
Phpp
fullowe the consentione
Uses appropriate data strutures from Prip's standard Jonery
vullowe thip ruam any conventiona
sesting
Manntannng tast coverage origiaal
the case main function, de not unit test Murltlbrrrg
Additionna tenal requirmmnnes
oigoritii prodiems,
necopace complositt
design Includio documennatioo
axcloiiinng
Your answer ahould be like
stramiatted, problemo
new probblem
situnsianed probtomp
stranslaned, ceforence, solationo
phop
your code
videro Juni, cunNes
pho
your codo
D0003P
dtlll
your
Here probdam, reference aclution case [onnheem]]
cocprob amppoo
jomsiieem wnd]
coscodecco
Inteernnce and]
Idemmo cane)
Jeati-o
Hull teer coos) cocull, postooo
Hual coon end]
Figure 14: The prompt of translating languages.
Among these, the dominant failure modes were Algorithmic Logic Errors and Missing Functionality. Figures 18 and 19 illustrate representative cases. After excluding low-quality tasks, Opus 4 still achieves only 64.3%, demonstrating that the remaining tasks remain substantially challenging.
N THE ORDER OF I/O FORMAT
In this section, we elaborate on the design choice of generating test inputs prior producing the programming problem description in the AutoCodeGeen pipeline. Although human annotators typically define the input-output (I/O) format before writing tests, our automated workflow adopts the reverse order. We show that this alternative workflow is both feasible for LLMs and beneficial in practice. There are two valid sequences for synthesizing test cases and problem descriptions:
Generate the test inputs first, followed by the programming problem (including its I/O specification);
Generate the I/O specification first, followed by the test inputs.
Both workflows be handled by modern LLMs. However, the first approach offers additional advantage: the public test cases produced during this stage can be directly embedded into the final problem description, improving clarity and reducing ambiguity in the specification. Below is example that illustrates why generating test inputs first a reasonable design choice.
Consider code solution such
28

---

Published as a conference paper at ICLR 2026
Please iuploment the following theee setwosk-rcloted functions In Go:
ReversePPaddeees: Accepts adcrees type net.ip returns seversed string representation (e.pir 1.1..558.222")
invalid,
apply stzing.
Repetationtaolup: Accepts IPed adexess as stxing and zecuzns its seputation value strlig (0.8., (100"). II the teput is snvalid, return an orror.
It's HostaFowicM:: Accopes range (eccluding
stzing (e-R., 292.158.2.0/227 retuzes stzing ieput
Ieput/ /Oriput sepuieeents:
All IF adoresses rust be in IPVC forstt.
Inwalid irputa inclade aal foared or passaalle aditesses.
The CIDR range rust valld and contain at least one available host sodress.
Enampl Usage:
// Twast rase 1: Bevennn TP wddrees io ParssePPPPP (1292 ..."" soversed ReverscSPddaaess should
// Test case 2: Queay asputattion tpötr to "0.8.8.8 sop, ezx t- Recutationkochhaaappp
nep should be '100", err should be mil
Test
1192.169, 169.1.0/25
hoata, wer tis fal torTT////c)) i/ hosts should be (" 168.1.1", "152.168.4.2 198.119. 392.198.1.47
168.1.5.. "192.168.1.91
eEE should
Figure 15: The prompt generating code solution. Due to the excessive length the prompt, we have omitted the latter part.
def cal_two_sun (a, b) return a
Using explicit specificatiins and few-shot examples, the model may generate test-input function such as:
def private_test teeettttt_____eeeeeeeeeeeeeeeeee tet___ttss_tt_tttttt_eeeee input ()
test_cased =
[1, 21,
[0, 01, boundary test
[99999939993999999, 3333333333] stress test
When constructing the programming problem, the model infers the input domain from the generated test cases and produces description such as: Write a function named cal two sum that computes the sum of two non-negative integers. If instead the model generates test cases containing floating- point values:
def private_test. input (): test_cases
[1, 21,
[0, 01,
14.243, 4.2221, stress test [-0.45, 8881, # boundary test
The model naturally adapts the problem description accordingly: Write a function named cal two sum that compules the sum of two Houting-point numbers. Therefore, When test inputs are generated first, model derives the 1/O format by analyzing the input domain represented the test cases. When the I/O format specified first, the model constructs matching test cases based the declared constraints. Both approaches are logically consistent, but generating test inputs first leverages the model's s ability to generalize from concrete examples and simplifies the subsequent problem construction stage.
Finally, regardless the workflow, AutoCodeGen employs an LLM-as as-Critic verification stage to ensure strict alignment between the generated programming problem and the test function. Any inconsistent, ambiguous, or underspecified cases are filtered out during this procedure. This ensures the final benchmark maintains high quality and internal coherence.
29

---

Published as a conference paper at ICLR 2026
30 Carcro System Emplerentation
bescription
implesen-ing
The should incluce:
"Vccoo" class for 30 satheratical oseza Lions
2. An MEB (Ads Aligned Bouncing Box) class for collision desection
S. A Cameee class that reprasents vime frustum in in spacn
caneza
bounding
Delazs inw jects (comseett ASBs) aithin its vim fruston
Class Requisments
U8a Vector
class Vector
{
publ he:
r1oot y. 2:
Vector operator- (conct Vectors other)
Vector operatoz- (cons) Vie took other) corst;
Vector (perators scalar) const:
float const Vectoak othex) const;
Vector croestconnt Wechors other) const;
msgnibude() const;
vold print() coew;;
U6A AA38 Class
class AABB
public:
Vector nún, Dax;
M4B(VVes ndn-Moctez().
rox-iectoz());
paint)
vold print()
2:.
UMA Canaza
class Caneza
public:
Canesal(cons: stzingg gg name- "Cafuuttaaessa' (orr)
const Vectops sptater-Voctoxx))
#1oat 5faz-1800.00 8t.
Tlaat 2lwar=0, 12,
2loat 4d0th-000 6t.
21oat height=660.335) 34);
Tr.
Eoappl Usagl
de Proolet en Specifications
Notes
Figure 16: Example of an inter-logic problem. The full content is partially omitted due to length.
30

---

Published as a conference paper at ICLR 2026
Bational Mamber QuecaLion
Implesent class 'Rational" to repzedent and nanipulate rational rumsers (tractions) with the follordng expobcisties:
den Private Merbers
'oon long nuserstor": The nunezator of the traction
lonn long denceinstor" The cenos instor of the fraction
"long long ged(long Long a. long long b) const A helser function to compute the greates: commn divisor of two nurbors sedica Srcction sisplast form wisur denosinetor poxilive
memuliiw() () helpps
Public
Constructon
pational(long long nun long long dence 21"; atoz 2010.
Crpates Rational numbez sith given nuserator and danoninator
(delaults
"swalld acqurent
Arichactic Operators
l operators (canst Rational other) const' is Adds tan Bet-ional mutters
*Rational operator- (const Rationall other) const:: Subtracto tio Rational, numbers
Beiid spesator Balien athec) canal Haltipl les Loo Rai ional nun bereess
*Rational operator/ (const Rationall other) const": Divides two Rational rumbers (thrwws "domstn orror" it dividiigg by 2010)
Corparison Oporators
"bool operanorin (canet Rationsll other) const's Checes
of Rationsl numbers
"bol opezaroz- (const Rationals othez) const Checks if this National less than another boo1 opezaccss (conct Rationals othez) sonct Checks if this Rational is greater thar snothex
Conoesiion
double damble() conat': Convests the Bat iinel mumber dndble
stzzngg ststingbbool shoa, parenss talse) const" Retusns string representation of the Ratiioaal nunber, When show, sasana is Loue, regative nurbers should be unclosed in patenthesss The string should be in nised nurbeu loce when topropriate (big.. 1/3" for 7/3).
Returns
"long long denedee notor) const" fittures the denceinstor
29 Helpex Functson
veclascRal ionals passe (conat wtrings irput)':
space-sepazaled string
(wither
"a/b" Soan narbess) seluans vectoz Rational mubbus.
""epp
Rational
<cc www.to,.. siritin // Outguts '5/6"
Rational c17, 3); cout <c 1otosstrngg); // Ourguts "2 1/3"
Bational d(-1, (tr.nn // Ditputs "(-1/8)"
string irput 19/9 3/4 -1/ 8";
voctor-Fatiooall> nurbers parse sasteenstattn))s)rrrtt)) numoers)
(const st_nnnnng
21 Outaiis: 112 3/4 -1/3
Specifications
Conttraints
Notss
solution will tested
various including axithweiic
convessions.
Figure 17: Example of problem exhibiting both inter-logic and intra-logic components. The full content is partially omitted due to length.
31

---

Published as a conference paper at ICLR 2026
[programming problem]
Neite JoveSerist function octolTosinazy" that converts given octal number string into its corzespending binazy stzing.
Irput requirements:
inpa sizing commenting an digit
my empty
6-7),
Valid octal
incluce
"0630")
Outout secuizerents:
toor oslie conal string, return torrespanding hinary string representation,
eepty srring.
"Ezror: Invalid stzing".
The sinary whould not ceit leading (e.g.. octal. should rehurn "511" and not "151").
Eswaple usage:
javascriot
const assest suquise( "assast");
const de doscessingg
assert strictiiial foctal Tahineryt (123"), "161001");
assert. www..ttttittaaaaa TotaaTTeinayyy '7), "asi");
11.
[Private Test Function] 'assert");
const full Teating ing m
assert. swr.cttaualllloco.o (octalic 1123")
"1810812");
assert. alsitt.
inery
strictetual oetalloBsnarr) 9*)
assest. aastt.ataiilll slpie toctalTooiaannn LalToBinery! 199*))
"111111:1"";;
assert stzacttqual (octalToBinayy 1000) "300000");
assert stractEqual loctal Tobsnary! 11246678"). 1018110111111111811
assezt. stsictEuull talloBinasyy "", "Erzo:: Expty input");
assest. stsicttqualllecccocc..co.o (octalToBinasy"" 'abe").
string";;
string");
assert striciiiaal foctal totaaanne 19833) "08660000"1);
[Code Solution from Opus 4.1]
"Jaascriot
Check
if [octa15)s ")]
scturn 'Exooz: Erpty inout":
digste
octal5t2)
'g" char
retuen "Euzor: Inwaldd octal string":
Convert octal lel. hinasySiz to, for (let digst ot octallsz) {
const octalbigit paseintiaiggi))
const biraryligit actalligit wtigiitttttttnn2/2.. toString(2), podStart(3, 101):
binanyStt straryDigth:
Logic Error!
!/ Renove Leading zeroe, but keep at 1east one digit
while (ninesydtr. lengtn 1 as hinerySt:ft) - '01l bonsxyStt binaxyStz, suttttssstttee cubstring());
zeluzn binasySix;
Figure 18: A case of Algorithmic logic errors.
32

---

Published as a conference paper at ICLR 2026
[prograrning problen]
drite function Toaest ancestoz binsry looeetCaaarrr aonoraannnnnneee.o
fiven the Livest ancestor (LGA) these lowest conmon sncestoz defiied losest mode descendants (whozo node descendant of itself)).
inslerent
The funsttion sigrature as follows:
ATraelids
The "Treohade" is deffned do:
Treebock struct
int
Left ATreeNode
Right *TzooNodo
All sode values the
Both uniat the glown birway Lase
The sumber or rodes in the thee ranges iron [2. 18"5).
Enarple
"teatring" import
Zunc Testbenett *testing, T)
1/ Coeete trme nucture text caee
47xceMode57, mil,
STaeeMode node7, node4)
nodes (TzeeMode ail, A[l]
Noteo:
including arcector of being antashor of and bwing the sare node, who. not rocidyy the tiee structure.
hanclles
fffrretnt bwing
complcciy should
as possible.
Missing the definition of TreeNode!
[Code Solution from Opus 4.1]
func JevertComonnncestee (root, P. a TTeohloce) TTzolNods
17 case:
zoot the target nodes
1/ Recursivaly search p
right subtaows
Innetoomonnecettrrii lowsstCam dikeessstoo (ooot. Right,
P.
1/ I1 both left and right are non-nil, cuzzent nede is the LOk 1/ This enans and are in difffeentt sibtress
11 left nil 66 right i- ndll octuen soot
A/ II only subtzue contains beth nodes, zeluan that sesult
1/ 1t lett: as non-nil, both nodes are lett sibtres
11 lest nil
return left
1/ Otherwise, both roden ri subtree for both nil) retuzn right
Figure 19: A case of Missing functionality.
33


Reviews:
[{"review_id": "BrNfopRjlN", "reviewer": "Reviewer_6ERu", "summary": "Existing benchmarks for evaluating code generation have key limitations: they rely on manual annotation, which is time-consuming and small in scale; they focus on Python code, while benchmarks for other languages have limited difficulty and low coverage.\nTo address these issues, the authors propose a fully automated framework called AutoCodeGen for building high-difficulty, multi-language code generation datasets. The framework first generates code solutions and test inputs from code snippets, then executes the code solutions in a sandbox to obtain test outputs. Finally, it uses LLMs to combine these three components into test functions, and reversely generates programming problem descriptions based on the solutions and test functions. This process ensures the quality of the benchmark through a three-stage filtering (difficulty, quality, diversity).\nBased on this framework, the authors introduce AutoCodeBench, a large-scale benchmark containing 3920 problems across 20 programming languages. Experiments show that even the most advanced models still face significant challenges in the complex and diverse multi-language tasks defined by this benchmark. In addition, the authors also release a supporting training set, AutoCodeInstruct.", "questions": "1. (Regarding Weakness 1) Given that the relatively simple difficulty filter is insufficient in distinguishing high-difficulty problems and filtering simple problems in low-resource languages, have you considered adopting improved methods? For example, could you use a more capable model as an evaluator to try more reasonable filtering for problems in different languages? Or, similar to quality control, could you incorporate more LLM-based analysis (e.g., using LLMs to analyze unsolved problems and solutions) to assist in filtering truly high-difficulty problems?\n2. (Regarding Weakness 2) Could you supplement more detailed analysis of multi-logic tasks? Could you provide more detailed statistics on the complexity of these problems? For instance, how many functions do they require to implement on average? Is there a specific reflection of the low pass rate of advanced models on high-difficulty problems?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The greatest contribution of this paper is the AutoCodeGen framework, which automatically generates benchmark data and attempts to overcome the reliance on expensive and time-consuming manual annotation. Its highlight lies in ensuring the correctness of the generated data: first, generating test inputs and then obtaining outputs through sandbox execution to avoid potential errors when LLMs directly generate test cases; second, reversely generating problems based on answers and test cases to ensure the actual solvability of the problems.\n\n2. To obtain a high-difficulty and diverse dataset, the paper implements multi-stage filtering. In the problem generation stage, the authors use LLMs as evaluation models to ensure alignment between problem descriptions and test functions (quality control). After generating problem descriptions, they use a \"moderately capable\" model (DeepSeek-Coder-V2-Lite) to filter out overly simple problems (difficulty control); they also ensure broad data coverage through problem classification and cyclic sampling (diversity control).\n\n3. AutoCodeBench, derived from the AutoCodeGen framework, features high difficulty, multi-language support (20 languages), and balance. Its versions of different sizes also meet the diverse needs of different researchers. Meanwhile, the paper shows that models trained with AutoCodeInstruct (Section 4) have comprehensively improved code generation capabilities, indicating that the data generated by the AutoCodeGen framework is of high quality and generalizability.", "weaknesses": "The paper states that AutoCodeBench aims to conduct strict evaluations of large language models on diverse, high-difficulty, and realistic multi-language programming tasks. However, to achieve this goal, there are areas where the paper can be improved:\n1. Insufficient Difficulty Assessment in the AutoCodeGen Framework\nIn Section 2.1.4, the authors mention the method for difficulty control: \"we employ a moderately capable code model, DeepSeek-Coder-V2-Lite, to filter out too easy problems. Specifically, we sample answers for each problem ten times using the model and validate the correctness via sandbox execution. We discard problems that are solved in all attempts.\" This is a filtering method designed to eliminate simple problems, but its limitation is that it is difficult to directly identify high-difficulty problems. The authors later mention in Table 3 how to further distinguish easy/medium/hard problems in the dataset using pass rates. The problem is that while high difficulty equals a low pass rate, a low pass rate does not necessarily mean the problem has real high difficulty (e.g., multi-logic/multi-task). Difficulty assessment based solely on the pass rate indicator requires support from other indicators. In addition, this difficulty assessment imposes requirements on the evaluation model. The paper reports in Section 3.3 that the Pass@1 scores of advanced models on low-resource language problems are relatively higher than those on mainstream languages, which to some extent indicates that the evaluation model is insufficient in filtering simple problems in low-resource languages, leading to differences in the consistency of benchmarks across different languages.\n2.  Vague Description of Multi-Logic Tasks in AutoCodeBench\nIn Section 3.4, the authors mention the concept of multi-logic tasks: \"A key feature that distinguishes AutoCodeBench from prior benchmarks is the inclusion of multilogical problems. These problems require models to implement multiple distinct functions or classes within a single task, challenging their ability to handle multiple core demands simultaneously.\" In subsequent experiments, the authors show that advanced models perform poorly on such tasks. As a representative of high-difficulty problems, multi-logic tasks can indeed reflect a model's comprehensive ability to handle complex code tasks. However, the description of multi-logic tasks in the paper is somewhat brief, and such problems are not generated through active control; the analysis of such problems is more inclined to data mining.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "856nqtaxsr", "reviewer": "Reviewer_To2A", "summary": "This paper presents an automated framework called AutoCodeGen for constructing high-difficulty, multilingual code generation datasets without manual annotations, which contains 20 programming languages with balanced coverage to rigorously evaluate LLMs on diverse, challenging, and realistic multilingual programming tasks. Experiments reveal that even state-of-the-art models struggle on these tasks, particularly in low-resource languages. The paper also gives complementary training and evaluation resources, including a large-scale, verifiable multilingual training set.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The automation of the benchmark is good, which eliminates the need for manual annotations. This is a significant advantage for scaling code generation evaluation across languages and problem complexities.\n\n2. AutoCodeBench contains 20 languages and high-difficulty problems, which makes it a robust benchmark for multilingual code generation tasks. \n\n3. The paper is easy to follow.", "weaknesses": "1. The authors should pay attention to other low-resource languages.\n\n2. The system heavily relies on LLMs for generating code solutions and test cases, which can introduces biases or errors inherent to the models being used, especially if those models are trained on flawed data.\n\n3. The paper focuses on evaluating the code generation of models but does not address the broader issue of how benchmarks extends to other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "quiiehF3ti", "reviewer": "Reviewer_LmDg", "summary": "This paper proposes AutoCodeGen, a automated pipeline for synthesizing multilingual code generation benchmarks. The key idea is to use LLMs to generate test inputs, execute them in a secure sandbox to obtain ground-truth outputs, and then combine these elements to create verifiable test cases. Based on this pipeline, the authors introduce AutoCodeBench, a benchmark of 3,920 problems spanning 20 programming languages, which they claim is more difficult, diverse, and realistic than existing benchmarks. The paper also releases AutoCodeInstruct, a training dataset. The authors' experiments suggest that current LLMs still struggle significantly on this new benchmark.", "questions": "1. The paper reports pass@1 results. Could the authors also provide pass@k results for k > 1 (e.g., k=5, 10)?\n2. The paper uses GRPO with the AutoCodeInstruct. What would be the performance if standard Supervised Fine-Tuning (SFT) were used instead? An ablation study comparing GRPO with SFT would help disentangle the benefits of the dataset itself from the specific training strategy employed.\n3. The pipeline relies on LLM to generate test inputs. Have the authors considered integrating or comparing this approach with established automated test case generation tools, such as CYaRon or property-based testing libraries (e.g., Hypothesis for Python)?\n4. I am slightly confused about the problem generation workflow detailed in Section 2. Specifically, why are the test functions generated before the input/output format specifications? Could you elaborate on how boundary and stress tests are systematically generated within this workflow? It seems that defining the I/O format first would provide a clearer structure for generating comprehensive test cases.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors have conducted a comprehensive evaluation across a very large number of models. This provides a broad and valuable snapshot of the current landscape of code generation capabilities as measured by their proposed benchmark.\n2. The strategy of generating test inputs first and then using a sandbox to obtain ground-truth outputs is a clever and effective method for ensuring the correctness of the generated test cases.\n\nI am looking forward to the authors' response and would be happy to reconsider my evaluation upon successful clarification.", "weaknesses": "The paper's central claim of high difficulty is not sufficiently deconstructed or justified. The methodology for ensuring difficulty involves a post-hoc filtering step that removes any problem solvable by a \"moderately capable model\" (Line 195). This approach risks conflating genuine, meaningful difficulty with other confounding factors. Specifically, the source of difficulty remains ambiguous: \n- Is a problem difficult because it requires complex algorithmic reasoning or deep domain knowledge?\n- Is it difficult due to the need to understand obscure language features?\n- Or is it \"difficult\" simply because the problem description is ambiguous, underspecified, or even flawed?\n\nThe authors' own finding of a 12.4% error rate in the benchmark (Appendix A) lends significant weight to the third possibility. If a substantial portion of the benchmark is poorly specified, then it may be evaluating a model's ability to \"guess the user's intent\" rather than its true code generation capabilities. This ambiguity undermines the benchmark's validity as a measure of progress in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svuHudoFEB", "reviewer": "Reviewer_5r4r", "summary": "This paper introduces AutoCodeGen, a fully automated pipeline for generating multilingual code generation benchmarks without manual annotation. The core of the proposed method is generates test inputs using LLM, which are then executed in a secure sandbox to obtain ground-truth outputs. These verified input-output pairs are subsequently used to formulate a complete programming problem. Using this pipeline, the authors present AutoCodeBench, a large-scale benchmark comprising 3,920 problems across 20 programming languages, designed to be both difficult and diverse. The authors report that the benchmark reveals significant weaknesses in current state-of-the-art models, particularly on low-resource languages and multi-logic tasks. In addition, they release a companion instruction-tuning dataset (AutoCodeInstruct) to support future research.", "questions": "1. The benchmark was generated using a model from the DeepSeek family. To what extent are the benchmark's characteristics and the resulting model performance metrics biased by this choice of generator? It would be insightful to conduct a small-scale experiment where the key pipeline steps are run using a completely different model family (e.g., Llama 3, Claude 3, or Qwen). A comparison of the resulting problems (e.g., types, styles) and model performance on this new subset would help quantify the potential for generator bias. Meanwhile, I agree Line 1064.\n2. The concept of \"multi-logic\" problems is interesting and presented as a key contribution. However, its definition in the paper feels somewhat informal. Could the authors provide a more rigorous definition? Furthermore, a deeper analysis of what makes these problems challenging would be valuable. Is the difficulty primarily a function of longer context and following multiple discrete instructions, or does it require a more fundamental type of compositional reasoning that current LLMs lack?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The cost and scalability limitations of manual annotation are well-known, and automating this process is a valuable research direction. \n2. The public release of the benchmark, a large-scale training dataset, and a high-concurrency multilingual sandbox are significant assets that could benefit the code generation research community.", "weaknesses": "1. While the engineering effort is commendable, the core concept of using LLMs for data synthesis in the coding domain is not new and builds upon a line of existing work (e.g., Evol-Instruct, OSS-Instruct). \n2. A significant concern is the methodology used to generate problems for 14 of the 20 languages via \"approximate language translation (sec. 2.1.5)\". However, code translation is not a solved problem, and often fails to capture language-specific idioms, standard library conventions, or type system nuances. This approach raises questions about the quality and naturalness of the generated problems in these low-resource languages. It is unclear what the quality gap is between these \"translated\" problems and problems that could have been generated \"natively\" using the full pipeline. The paper would be strengthened by an analysis quantifying this gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)