AutoCode: LLMs as Problem Setters for Competitive Programming
Authors: Shang Zhou
Zihan Zheng
Kaiyuan Liu
Zeyu Shen
Zerui Cheng
Zexing Chen
Hansen He
Jianzhu Yao
Huanzhi Mao
Qiuyang Mang
Tianfu Fu
Beichen Li
Dongruixuan Li
Wenhao Chai
Zhuang Liu
Aleksandra Korolova
Peter Henderson
Natasha Jaques
Pramod Viswanath
Saining Xie
Jingbo Shang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Competitive Programming
Test Case Generation
Problem Generation
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases, far surpassing current state-of-the-art performance.

Abstract: Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.

Published as a conference paper at ICLR 2026
AUTOCODE: LLMs AS PROBLEM SETTERS FOR COM- PETITIVE PROGRAMMING
Shang Zhou'
Zihan Zheng
Kaiyuan Liu
Zeyu Shen
Zerui Cheng**
Zexing Chen'
Hansen He
Jianzhu Yao*
Huanzhi Mao?
Qiuyang Mang"
Tianfu Fu
Beichen Li
Dongruixuan Li
Wenhao Chai
Zhuang Liu
Aleksandra Korolova*
Peter Henderson
Natasha Jaques
Pramod Viswanath4.10,
Saining Xie
Jingho Shangl.t
University of California San Diego NNww York University University of Washington
'Princeton University Canyon Crest Academy "OpenAl University of California Berkeley
*MIT "University of Waterloo
Sentient Labs
'Equal contribution
Advising authors
ABSTRACT
Abstract:
Writing competitive programming problems is exacting. Authors must: set constraints, imput distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple problem statements and test cases.
rounds of validation to yield competition- On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-an methods like HardTests, which achieve less than 81%. Furthermore, Starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross- vross-veii these generated solutions against test cases, further filter out malformed problems. Our system ensures high correctness, as verified by human experts. AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality.
INTRODUCTION
As Einstein and Infeld put it, "The formulation a problem is often more essential than its solution, which may be merely matter of mathematical experimental skill. To raise questions, possibilities, to regard old problems from new angle requires creative imagination and marks real advances science." " (Einstein & Infeld 1938). As large language models (LLMs) march toward general-purposs. capabilities, with the ultimate goal of artificial general intelligence (AGI), we argue that testing problem generation abilities just important as problem solving abilities. This is particularly true when applying LLMs to advanced programming tasks, where future advancement and economic integration of LLM coding capabilities will require significant validation.
First, problem setting for competitive coding requires deeper understanding of algorithms that problem solving may not. For example, basic problems can collapse into recognizable templates that can be solved with simple tricks; and many standard programming questions often allow for partial credit or boilerplate solutions that can mask incorrect reasoning. Competitive programming problems have strict bar designed to assess deeper understanding of the underlying algorithm design principles, data structures, and complexity trade-offs. Verifying the vast space of possible solutions, along with sufficient coverage of short-cuts corner cases is challenging, but necessity for competition programming problems. As such, problem setting encompasses all the challenges of solving problem, and then more.

---

Published as a conference paper at ICLR 2026
M Create
2 Generate
3 Validate
4 Stress
Seed Problem
New Problem
New Problem
Validated Test Cases
Given two permutstione, P and
Glven two parmutations, and q, (00 pos, a[o0]). Count the number of [1.r] that max(f(lur) (ooel(llrr)mminfiii min(((r))
Given two permutations,
and let (f] pos glo()), Count number of
p= 2,
Count the number of Intervals (1. such
intervela [Lr] that max(f(Lur) min((((L))
q= [3,1,,
MEX(p().r) - MEX(q[L.r]).
Create
Generator
Validator w/ tests Validator Program
Run
New Prablem
Generator Program
Solution
Given two permutations,
FarmPar gantint numting typa)
void seddaaloaaeeeeeeaaaaaoo ol tip qateeoli acse size ou sneartip ttp stee (B0000):
Reforence Solution
let (LU pos.o[pl]]- Count the number
If glgeenn orate wo random
sasse in tiyse "ddenccal)
intervals
(exxf((-rr)-mmnnffll.. rol.
jaise i (syse en "
arear]jk, parmu barmutationiijl
Brute-force Solution
pro idenliyy q reversedd
Solve
Generate
Validate
Checker w/ tests
Solution
Test Cases
Validated Test Cases
Check Output
Referance
[= 2,
Solution
q q (3,
p= [1, 3]
ANS
Brute-force
Solution
q [3,
p= [1, 2,
q = [3, 1,
OUT
Sewiooo-nGnnnaet-----ieCC- framework that enables robust test case generation and scalable, self-verified problem generation for competitive programming. It achieves 98. 7% consistency with official judgments. This framework precisely mirrors the process human experts follow when creating programming contest problems.
Figure 1: AutoCode introduces closed-loop multi-role
Second, better problem setting will lead more rigorous competitive programming benchmarks. Since official test data from premier platforms such as Codeforces and AtCoder are not publicly available, researchers currently rely on synthesized datasets such CodeContests+ (Wang al.) 2025). TACO (Li al. 22023). and HardTests (He et al. 2025). Our analysis ($4). however, shows that existing test datasets can have both high false positive rates (FPR) and false negative rates (FNR). For instance, a greedy algorithm with poor time complexity might pass a suite of small, random tests, only to fail against adversarially constructed cases designed to expose its flaws. This critical weakness creates a distorted evaluation landscape, rewarding models that discover shortcuts.
Third, successful problem setting of novel challenges may pave the path for self-improving models and AGI, as well as validating deployments in complex software stacks.
LLMs are already used by over a billion people and increasingly for coding applications, saving hundreds of millions of dollars with LLM-assisted programming (OpenAI 2025 Zeff 2025 AWS
DevOps Blog 2024), albeit largely on relatively simple tasks. To expand LLM capabilities toward the strategic reasoning and long-term planning that real programming demands, and to integrate LLMs safely within the software development stack, we must be able to determine, with high fidelity, whether model-generane
code is valid solution the problem at hand. Misspecified rewards, in the form of malformed problem formulation or verification, can cause models to optimize for the wrong thing or fall into bad local optima. We find, for example, that current benchmark datasets suffer from an even higher FNR than FPR, often caused by invalid inputs to unit tests; as result, a perfectly valid, creative solution can be unjustly penalized when it crashes or produces incorrect output on malformed input data, This pollutes data for reinforcement learning by punishing valid lines of reasoning, while high FPR fails penalize flawed ones.
To address these critical gaps, we introduce AutoCode, systematic framework that employs LLMs in a closed-loop, multi-role system to automate the entire lifecycle of competitive programming problem creation and evaluation. Our first contribution is an enhanced Validator- -eeneettrr-hheeeee framework that achieves state-of-the-art reliability in test case generation, where the Generator generates test cases, the Validator validates whether the generated test cases satisfy the problem constraints, and the Checker checks whether a solution is correct given the test cases. We move beyond standard implementations with several key heuristics: the LLM generates multiple candidate Validators and Checkers, and we select the most robust one through targeted testing. The generator executes multi-pronged strategy, creating diverse and adversarial test suite that includes small-data
2

---

Published as a conference paper at ICLR 2026
exhaustion, randomized stress tests, and TLE-inducing cases designed to expose incorrect time complexities. Building on this highly reliable verification pipeline, we introduce our second major contribution: a novel process for generating new, high-quality problems. This process begins with a seed problem to inspire the I.I.M in a promising direction. To ensure correctness without human intervention, we employ dual verification protocol: the L.MM generates problem statement, an efficient reference solution, and simple brute-force solution. The new problem is accepted only after rigorously verifying that the efficient solution's 's output matches the ground truth established by the brute-force solution across all test cases. In summary, our work makes the following contributions:
State-of-the-Art Test Cases Generation for Existing Problems. Our testcase generation frame- work achieves over 91% consistency with official judgments a large-scale benchmark of 7538 problems, significantly outperforming existing methods whose consistency ranges from 72 - 81%. %. We also point out that the ability to generate test cases is not only useful for competitive program- ming. but also has broad practical significance for general tasks involving input-output matching.
Novel and High-Quality Problems Generation. We pioneer a systematic process that uses a dual- verification mechanism generate, validate, and score new problems. Vetted by elite competitive programmers, this process produces novel problems deemed high-quality and original enough for official contests, with problems passing automated verification achieving a correctness rate of 94%.
2 RELATED WORKS
LLMs show rapid progress in code generation recently (Hui et al. 2024 Deepseek 2024 Zhu
al. 2024, Gong al. 2025 Xie et al. 2025). Recent efforts have progressed along three threads. First, new benchmarks have been proposed to target diverse problem types. These range from standard coding interviews (Jain al.) 2024) and competitive programming (Zheng et al.) 2025) to challenging domain-specifee problems (Beniamini et al.) (025) and human-LLM, collaboration scenarios (Yang et al.] 2025). Second, a growing line of work focuses on strengthening verdict reliability. For example, (Liu al. 2023) augments existing unit tests. A more recent trend involves
rule- or LLM--riven generators that create new challenging test cases (Li et al. 2023) Wang et al.
2025 He et al. 2025 Cao et al. 2025 Sung et al. 2025 Ma et al.. 2025 ). These frameworks
designed to emphasize constraint validity, edge-case coverage, and adversarial-ness. Third, solver /
data-centric approaches have aimed to scale solution search curate verified corpora. Methods like
AlphaCode eL al.] 2022), AceReason (Chen et al.
(025b), Absolute Zero (Zhao et al. (2025) ) and
rStar-Coder (Liu et al. (025) leverage signals from reinforcement learning or self-play (Zelikman
et al. 2022 2024][Shim et al., [2033 Madaan et al., 2023 Chen et al., 2025a). (Huang et al.) 2026)
manually create test cases for evaluation. However, these methods focus on solving or data curation
rather than offering end-to-end testing pipelines (Zhang et al. 2023). In contrast, AutoCode unifies and extends these three directions. couples Salidator-Generatoor Generator- Checkkr (and interactor) loop to enforce test legality and adversarial coverage. Critically, AutoCode also employs
verffiiriiiaatoon protocol (using a reference solution and a brute-force solution) to generate and certify entirely new problems. This framework allows AutoCode to simultancously address two key challenges in the field: static-benchmark contamination and under-constrained test cases.
3 TEST CASE GENERATION
Our test case generation process is a structured framework designed for maximum rigor and coverage. The framework as shown in Figure starts with the Validator, which serves the cornerstone of the entire system. Its function is to ensure any given input strictly adheres to all constraints specified in the problem description. A Validator is critical for minimizing the FNR, as prevents correct programs from failing on malformed data. Next, the Generator employs diverse strategies to create a broad spectrum of inputs, aiming to reduce the FPR, where incorrect or inefficient programs are erroneously judged as correct. Any invalid cases produced by the Generator are subsequently filtered by the Validator, ensuring we obtain a high-quality set of inputs. Finally, to assess the contestant's 's output, a Checker compares it against the reference solution's 's output, while for interactive tasks, an Interactor engages in a multi-turn dialogue with the contestant's 's program to issue a final verdict. Terminology is defined in Appendix A Because one of our prominent goals is to serve as high-quality verifier for RLVR, we pay particular attention to reducing the FPR. We distinguish test cases (input-answer pairs) from the test data, which includes the Checker and Interactor programs required for evaluation.
3

---

Published as a conference paper at ICLR 2026
3.1 VALIDATOR
Algorithm 1: BUILDVALIDATOR
Input: Problem spec S
Output: Selected Validator V
3 GEN.GEERRRRT SENTEERATEEVAACAESSIICC N= 10 valid, 30 near-valid) (V1, V22 VaJ <- LLM. - EMIVALLDATOORS(S K=3) for V in [V1 V2, Va}:
Score(V) + - E- (a (a)=aabel]
V f arg m sVore(V)
return V*
The foundation of our framework is robust Validator responsible for rejecting any input that violates the problem's 's explicit constraints. To construct this Validator, we first prompt the LLM to generate targeted suite of 40 test cases. This suite consists of 10 valid inputs and 30 "near-valid" invalid inputs, i.e. cases that are almost valid but contain subtle violations (see Appendix [ for an example). With this evaluation set, then prompt the LLM generate three distinct candidate Validator programs. Each candidate Validator is benchmarked against these cases. The one that correctly classifies the most valid and invalid cases is selected the winning Validator. This Validator is then used in all subsequent stages to filter malformed inputs.
3.2 GENERATOR
Algorithm 2: BUILDGENERATORSUITE
Input: : S, Validator V*
Output: Final test set T
91 + EXHAUSTIVVEMMALL(S)
# small-scale enumeration coverage
G2 RANDOMEXTREME(() hash collisions ons, etc.
random extreme: overflows, precision,
G3 +- TLEINDUCING(S)
worst-case inputs inducing TLE
11 + U2 U93
+ {x U (z)==valid)
# Validator filters invalid inputs
11 < DEDUPBYSIGGNATURE(() # hast/nomallization--taa deduplication
< BALANCEBUCKETSS/EE size/structurehhaddnes))
T SAMPLEW target size)
return T
With a reliable Validator, the Generator's main goal is to maximize test coverage. designed to create challenging cases that identify incorrect inefficient solutions, which could reduce the false positive rate (FPR). To achieve this, we combine three distinct strategies:
Exhaustive Small-Case Coverage. For problems with small constraints, we exhaustively gen- erate all permutations and combinations of small-scale data. This ensures complete coverage of fundamental boundary conditions.
Randomized and Extreme Data. We generate large-scale random inputs stress-test solutions. This targets common failure modes such as integer overflows, floating-poin point precision, and array out-of-bounds. Drawing from contest experience, we also design adversarial "hack" cases, such as those specifically intended to break common greedy algorithms or induce hash collisions.
TLE-Indueing Data. To identify solutions with incorrect time complexity, a common issue where slow algorithms pass weak test suites, construct inputs with specific structures designed to maximize the rumning time. This ensures that only solutions that meet the intended time complexity pass.
4

---

Published as a conference paper at ICLR 2026
Algorithm 3: BUILDCHECKER
Input: S. reference solution R, Validator V
Output: Selected Checker C*
R; N=40)
foreach Q do
if in.iutt) * valid:
remove
# ensure input legality
{C1, C2 C3} +- LLM.EMITCHECKERSS( K=3)
for C in C2, acc(C) + 12) L C(q.input, g.contestant out, q.ref.out) -gef.out)=g..erdiie
C* t arg maxe ace(C)
return C*
3.3 CHECKER
The Checker's role is determine the final verdict (e.g., Accepted Wrong Answer) by comparing a submission's 's output against the reference solution's output. To ensure the quality of the Checker, follow analogous construction process the Validator. First, prompt the LLM generate 40 distinct test cases. Each test case includes valid input, veasonable-looking output that is potentially incorrect, the correct reference output, and the expected verdict. The validity of all inputs is guaranteed by the winning Validator. Next, prompt the LLM to generate three candidate Checker programs. We evaluate each candidate Checker against the 40 test cases. The checker that achieves the highest accuracy in matching the expected verdicts is selected as the winning Checker.
3.4 INTERACTOR
Algorithm 4: BUILDINTERACTOR
Input : S, reference solution R, Validator V*
Output: Selected Interactor [*
M 4 (RM small logical edits: <>/<
12 swap, off-by-one, missing checks, wrong tie- bieebs,aks, RNG misuse, etc.) LLL.L...MIIIII SLLMEMIIINTEERACTRRSSSS K=3)
{I1, Is) +-
for T in (T,, I2, Is):
p [SIMULATE([ R, V* Accepped
ft EmEm! (SMULATEE,, m, Rejeeted
score(I) + (p, f) solution, then maximize discrimination
lexicographic: prioritize passing the true
1* + arg max, score(I)
return /*
Our framework introduces a novel, fully automated approach for generating test data for interactive problems, previously unaddressed challenge (c.g.. in CodeContests+ (Wang innovation lies in mutant-based discrimination process. We begin by prompting the LLM to perform several small but critical logical modifications the provided reference solution, thereby creating set of slightly incorrect mutant programs. Examples are shown in Appendix [E] These mutants serve as challenging foils for the Interactor. The LLM then generates three candidate Interactor programs. The crucial selection criterion is identifying the Interactor that most effectively distinguishes the correct, unmodified reference solution from the flawed mutant versions during simulated interaction shown in Appendix G The Interactor that successfully passes the reference solution while failing the maximum number of mutants is chosen, proving its ability to robustly probe for the specific logic required by the problem.
2025 ). The core

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on the 7538-problem benchmark. The evaluation is performed on a set of 195,988 human submissions randomly taken from the CodeContests dataset, where each problem has 26 submissions, 50% of which are correct and 50% are incorrect. Results are reported with 95% confidence intervals. The results for AutoCode are obtained using o3.
Method
Consistency (%) (t)
FPR (%) ())
FNR (%) (U)
CodeContests I al. 2022
72.9 + 0.2
7.7 + 0.2
46.3 + 0.3
CodeContests+ Wang ct al. 2025
79.9 0.2
+ 0.2
31.6 + 0.3
TACO (Li et al. 2023
80.7 + 0.2
11.5 0.2
26,9 + 0.3
HardTests (el et al. 2025
81.0 1 0.2
12.1 0.2
25.8 1 0.3
AutoCode (Ours)
91.1 1 0.1
3.7 1: 0.1
14.1 1 0.2
Table 2:
Ablation study results on the 720-problem benchmark. For each problem, the evaluation uses
33 submissions generated by different LLMs, 25% of which are accepted. This setup is particularly relevant for reinforcement learning applications. Each row shows performance after removing single component from the full framework. All results in this table are generated by GPT-5 5-T----H22 -ii gh.
Configuration
Consistency (%) ) (1)
FPR (%) (1)
FNR (%) (L)
w/o Generator Strategy (Exhaustive)
98.4 = 0.2
1.7 + 0.2
1.3 0.3
w/o Generator Strategy 2 (Random/Extreme)
98.4 = 0.2
1.6 + 0.2
1.3 0.3
w/o Generator Strategy (TLE)
98.6 0.2
1,4 0.2
1.3 0.3
w/o Prompt Optimization
98.0 = 0.2
1.8 + 0.2
2.9 0.4
AutoCode Full Framework
98.7 0.1
1.3 + 0.2
1.2 1 0.3
4 BENCHMARKING TEST CASE ROBUSTNESS
To rigorously evaluate our test case generation framework, establish two distinct benchmarks. The primary benchmark consists of 7538 problems derived from the intersection well-knoww existing datasets: CodeContests+ (Wang al. 2025), CodeContests, HardTests (He et al. 2025). and TACO (Li et al., 2023). Notably, this large-scale set does not contain interactive problems, and due to the filtering inherent in these datasets, its average difficulty for test data generation is slightly lower than typical Codeforces round.
To address this and test system under more challenging, real-world conditions, we create second benchmark 720 recent, rated problems from Codeforces. This set is completely unfiltered, including notoriously difficult-to-handee interactive problems and those requiring complex, structured test data. We are unable to evaluate prior methods on this newer benchmark as their data generation codebases are not publicly available.
Our evaluation is based on three key metrics. Consistency measures the overall percentage of agreement between the verdicts from our tests and the official judgments. We further dissect disagreements into two critical error rates. The FPR is defined as the proportion of officially incorrect solutions that are erroneously accepted by our generated tests. Conversely, the FNR the proportion of officially correct solutions that are erroneously rejected by our tests.
Comparison with other baselines. We evaluate AutoCode on the benchmark of 7538 problems
against four leading baselines. As detailed in Table our framework achieves 91.1 1 % consistency with official judgments. This marks a significant leap over existing methods which don't 't surpass 81.0%. Critically, AutoCode substantially reduces the FPR to just 3.7% and the FNR to 14.1% representing 50% decrease in both metrics over the current state-of-the-ane Figure shows the error verdicts distribution, showing that most of the problems are consistent with ground truth verdicts.
To further test our system's robustness, we curate more challenging benchmark of 720 recent, unfiltered Codeforces problems, including complex interactive tasks. As shown in Table 2 Au- toCode maintains its exceptional performance, achieving 98.7% consistency. This result validates our method's effectiveness modern, difficult problems where prior methods could not be evaluated.

---

Published as a conference paper at ICLR 2026
7.78%
8.00%
Fully Consistent (=0): 85.6%
Near Consistent (s3): 97.1%
Distribution
6.00%
High Non Consistent (=7): 1.5%
4.00%
2.00%
0.83%
0.14%
0.14%
0.14%
2
6
10 11
17
23
Number Non Consistent Verdicts
Figure 2: Distribution of Non-Consistent Verdicts. Each problem in the 720-problem benchmark is evaluated using 33 submissions generated by different LLMs. The verdicts for 85.6% of the problems consistent with the official judgments, and 97.1% % have three fewer inconsistencies.
Ablation studies. We further conduct ablation study to determine how each part of AutoCode affects overall performance. The complete AutoCode framework sets a strong baseline, achieving 98.7% consistency with official judgments, along with 1.3% FPR and 1.2% FNR. Prompt optimization turns out to be especially important; removing it drops consistency to 98.0% 0% and more than doubles the FNR 2.9%. The three different generator strategies also significantly complement each other. Removing either the exhaustive or random/extreme strategy raises the FPR to 1.7% and 1.6%, respectively, highlighting their role catching flawed solutions.
Validator and Checker selection tests are also crucial. These tests pick the most reliable output from several model-generatel candidates, helping weaker models avoid logical errors. Removing these tests increases the FNR to 1.3% % and 1.4%. A good Validator prevents correct solutions from being rejected due to bad inputs, while a strong Checker accurately evaluates outputs with complex judgment rules. Even small improvements (0.1% matter, as they help address problems that are very difficult to generate test cases for.
PROBLEM GENERATION
Our novel problem generation framework builds upon the robust test generation framework described in Section 3 and shown in Figure [ but introduces a critical dual-verification protocol to ensure correctness without human intervention.
Each generated problem is graded on a 6-level scale, judged by top human competitive programmers. We interviewed 8 human expert problem setters, all of whom report that they often build on specific existing problem when authoring new problems. By adding, removing, or modifying certain conditions of such a "seed problem," " they create new and often more difficult problems that require novel insights. Inspired by their insights, our approach begins by selecting random Codeforces problem (with difficulty rating less than 2200) as "seed problem." " The LLM is tasked with generating
problem by adding, deleting, modifying certain conditions from this seed problem, along with an efficient reference solution (std.cpp) and a brute-force solution (brute.cpp). brute.epp usually has higher time complexity but is very unlikely to be incorrect, so we leverage it to stress-test the validity of the problem. Using our enhanced test case generation technique, we construct a comprehensive
of test data that fully covers small cases. Both brute.cpp and std.epp are then executed on this dataset. A problem is only deemed correct if, for every test case, both programs' outputs (where the brute-force solution may legitimately fail to finish due timeout) are pairwise validated by the checker as consistent answers and outputs. This nual-verification protocol, where brute.cpp serves as the initial ground truth, and the validated reference solution undergoes an additional full test generation cycle, successfully filters out 27% of prone problems, raising the correctness rate of -----iiddeed reference solutions from 86% to 94%.
After filtering, over 80% of the problems are annotated having sufficient quality to serve training data for models, and 23% of the problems have novel or creative designs involved. We present the detailed rubrics and the score distribution in Figure 3 In the following, summarize several key findings regarding the performance LLMs in problem generation. We also present the human expert grading criteria in Appendix and one example under ICPC/101-levl in Appendix

---

Published as a conference paper at ICLR 2026
01
Solvability Check (Remove Unusable)
91.6%
Ensures each problem has at least valid solution
02 Solution Correctness Check
76.3%
Validates the reference solution ag against brute force
03
Model-Training Usable
61.6%
Sufficlent quality to serve as training data models
04
T Usable
27.4%
instructive enough
learners
05
Novel Problems
17.4%
Rotaining only novel or oreative designs
ICPC//OI-Level
The elite tier: problems that are suitable for contests
Figure 3: AutoCode is capable of automatically generating novel competitive programming problems. After careful examination and filtering by multiple human experts, 61.6% achieve quality level suitable for large language model training, and 3.2% are considered good enough to serve as problems for ICPC/IOI-level contests. Level I and correspond automatic correctness checks performed by the system, while level 3 through 6 involve human evaluation to differentiate among finer levels. It is noted that only the Level questions are wrong. while the Level questions are all correct and usable, with the only difference being their quality. Detailed grading rubrics are in Appendix
Finding 1. LLMs can generate solvable problems that they themselves are unable to solve.
LLMs can generate solvable problems that they themselves are unable to solve. In our experiments, about 4.2% of the problems fall into this category. In other words, the model is indeed capable of creating logically sound problems that can be solved by other models by humans, but due to limitations in its own reasoning ability, it fails to solve them correctly. Ideally, these problems could serve sources for model self-improvement, and we think this is very interesting phenomenon that is worth further investigation. We provide an example in the Appendix C
Finding 2. LLMs tend to create new problems by combining existing problem frameworks and emphasizing knowledge and implementations.
LLMs tend to concatenate, combine, or embed existing algorithmic knowledge or tricks into estab- lished problem frameworks, rather than proposing entirely new problem models that require original solution strategies. This form of recombinational innovation primarily reflects reuse of existing knowledge rather than expansion divergence between humans and I.LMs in defining novelty: while humans emphasize originality in modes of reasoning and problem-solee solving ideas, LLMs rely more heavily on recombination and reconfiguration of pre-existing knowledge. Also, the generated problems often place greater emphasis on assessing specific algorithmic knowledge or demanding complex implementation details, while less often rely clever design and subtle observation.
creative thinking. Such a tendency highlights a fundamental
Finding Novel problems tend to have larger difficulty gain over seed problems, and the generated problems have the highest quality when the corresponding seed problem is moderately difficult.
We observe that the difficulty shifts induced by LLMs during problem adaptation follow systematic pattern rather than occurring randomly. On average, adapted problems become approximately 334 Elo score harder; those judged as "novel" show an average increase of 498 score, whereas noven problems increase by only about 108 score. High-quality problems are predominantly generated when the original seed problem lies in the moderately high difficulty range. For overly difficult seed problems, the LLM has limited room to introduce effective modifications, resulting in minimal difficulty gains and insufficient novelty. Conversely, for overly easy seed problems, even after an average increase of 334 score, the absolute difficulty remains too low to meet high-quality standards. Notably, around 5% of generated problems fall into a critical zone, with pass@ scores between 0.1 and 0.5, meaning the model sometimes succeeds and sometimes fails. These boundary cases present a valuable opportunity for constructing high-quality self-play datasets, enabling models to enhance their capabilities through repeated attempts at solving such borderline problems.
8

---

Published as a conference paper at ICLR 2026
Difficulty Ele Quality
Novelty Novelty
2250
2000
129
1-122
1800
0 0.048
o3 Navelty Elo
2000
p 000.206
03 Dificuity
1750
14:00
1500
1200
1250
1000
1.0
2.0
3.0
4.0
5.0
0.0
0.5
1.0
Quality
Novelty
Quality Human Quality
Quality Quality
2250
2000
0.073
129
2000
#=133
o3 Quqailty Ele
1800
p= 0.410
o3 Quality Ele
p - 0.0000
1750
1600
2500
1200
1230
09
1000
1000
1.0
2.0
3.0
4.0
5.0
800
1000
12
1400
1600
1800
2000
Quality
Quality
Figure 4: Correlations between human experts and LLM judgments on generated problems. The charts illustrate a significant gap between human and LLM perceptions of quality and novelty (Finding 4), despite high inter-LLM agreement (bottom right). LLM L.LM-predccted difficulty shows weak positive correlation with human-rated quality (top left), suggesting it can serve as a noisy proxy for quality estimation (Finding 5).
Finding 4. Human experts and LLMs show almost no correlation in their judgment of problem quality and novelty.
We identify a significant divergence between humans and LLMs in their judgment of problem quality and novelty. To quantify this discrepancy, we employ an Elo Rating scheme to assess LLM judgments, following the methodology described in (Zhou al. (204)). The correlation coefficients between the o3 and human experts are only 0.07 for quality and 0.11 for novelty, indicating substantial misalignment between the model's internal evaluation standards and human expert criteria. Interestingly, both humans and LLMs demonstrate high uithin-group consistency in terms of quality: correlations among human experts reach 0.71, while GP T-40 and o3 show a correlation of 0.72. These findings suggest that relying solely LLMs self-evaluate the quality of their generated problems inadequate, and more sophisticated evaluation mechanisms are required to align with human preferences as also shown in Figure
Finding 5. Difficulty the generated problem and difficulty gain over the seed problem serve as a better indicator of problem quality than LLM self-evaluations.
While LLMs are unreliable in directly assessing problem quality, leveraging their predictions of problem difficulty provides a more effective indirect measure. First, difficulty is the strongest predictor of quality: correlations between human-asee assigned quality scores and both absolute problem difficulty and difficulty gain reach high 0.60. Second, difficulty gain outperforms novelty as indicator. As continuous variable, it captures richer information, since problems that exhibit sufficiently large difficulty increases tend to incorporate new ideas more complex combinations, thereby manifesting novelty. Finally, LLMs possess certain ability to estimate problem difficulty; by exploiting this capacity, difficulty can be used proxy signal indirectly predict problem quality, achieving correlations of around 0818 as shown in Figure
CONCLUSION
In this work, we introduce AutoCode, closed-loop multi-role framework that leverages LLMs as
problem setters for competitive programming. By coupling Validator-Gennrrtoor Geherattr-Checker Checker (and Interactor) framework with a dual-verification protocol, AutoCode achieves ofate-of-the-an reliability in test case generation and extends beyond prior methods generate entirely new, contest-quality

---

Published as a conference paper at ICLR 2026
problems. Extensive experiments over 7,500 problems and recent Codeforces benchmarks demon- strate that AutoCode substantially reduces both false positives and false negatives, achieving more than 98% agreement with official judgments and successfully producing novel problems validated by expert programmers. Beyond test generation, our analysis sheds light on both the strengths and weaknesses of LLMs in creative problem authoring. While models excel at recombination of algorithmic knowledge, they struggle to introduce truly novel reasoning paradigms or flawless sample design. Nevertheless, we show that difficulty and difficulty gain serve reliable proxy signals for problem quality, providing scalable path toward self- elay
ETHICS STATEMENT
This study adheres to the ethical standards academic research. The benchmark datasets we used are all publicly available from competitive programming platforms (such Codeforces and AtCoder), and do not involve any personal privacy data or sensitive information. All data were filtered and anonymized, and used solely for algorithm evaluation and model validation. Furthermore, the experiments did not cause any direct or indirect harm to individuals or groups. We recognize that automated problem generation may be misused, for example in cheating spreading low-quality content. Therefore, we emphasize the critical role of human experts in the final screening and evaluation. We advocate for the responsible and reasonable use of the outcomes of this research.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, provide complete description of the framework design, including the construction of the Validator, Generator, Checker, and Interactor (see Section 2 of the main text). Algorithm pseudocode and process diagrams detailed in the appendix, along with concrete steps for validation data construction, problem generation, and the dual-veriiication protocol. All experiments were conducted on publicly available benchmarks (CodeContests+, TACO, HardTests, etc.) as well as on recent Codeforces problems. We report ablation studies (Table and comparisons with existing methods (Table T). supplemented with statistical significance analysis.
THE USE OF LARGE LANGUAGE MODELS
The main contributions of this work rely on the generative and discriminative capabilities of L.Mss. Specifically, L.LMs are employed in a multi-role framework as generator, validator, and checker. During training and experimentation, we use publicly accessible LLM APIs (e.g., GPT-5-Hi- without additional fine-tuning. It is important to note that while LLMs can automatically generate new problems, the quality and novelty of these problems still require expert evaluation and filtering. We firmly believe LLMs should serve as auxiliary tools to expand data generation and test coverage, rather than as replacements for human creative judgment. In addition, LLMs are partially used during the writing stage of this paper for language polishing and improving clarity of expression.
REFERENCES
AWS DevOps Blog. Amazon q developer just reached $260 million dollar mile-
stone, August 2024. -
URL htttp:://aw. amazon htt.a//wwo...mm/ool////////so/oo
amazon Siaazon-qddevelog justoppprr-ttt--eeeeeee -useeeeeoope--sst-rreeteeeee
Gal Beniamini, Yuval Dor, Alon Vinnikov, Shir Granot Peled, Or Weinstein, Or Sharir, Noam Wies.
Tomer Nussbaum, Ido Ben Shaul, Tomer Zekharya, et al. Formulaone: Measuring the depth of algorithmic reasoning beyond competitive programming. arXiv preprint DrXiv:2507.13337 2025.
Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, et al. Can llms generate reliable test case generators? a study on competition-level programming problems. arXiv preprint arXiv:2506.0683 256.2066666621... 2025.
Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan- Yee K. Wong. Spe: Evolving self play critic via adversarial games for 1lm reasoning, 2025a.
//rxivi wrg. abs/ 2504 191 69162
10

---

Published as a conference paper at ICLR 2026
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotrron Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.16400, 2025b.
Deepseek. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024. URL htttts::/axxiv..oo wtt.a///g//////....aggg
Albert Einstein and Leopold Infeld. The Evolution of Physics: The Growth of Ideas from Early Concepts to Relativity and Quanta. Cambridge University Press, Cambridge, 1938. Quoted passage on p. 92.
Shansan Gong, Ruixiang Zhang. Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506. 20639, 2025.
Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. Hardtests: Synthesizing high-quality test cases for Il coding. arXiv preprint arXiv:2505. 24098, 2025.
Ailin Huang, Ang Li, Aobo Kong, Bin Wang. Binxing Jiao, Bo Dong, Bojun Wang, Boyu Chen, Brian Li, Buyun Ma. Chang Su, Changxin Miao, Changyi Wan, Chao Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengting Feng, Chengyuan Yao, Chunrui Han, Dan Ma. Dapeng Shi, Daxin Jiang, Dehua Ma, Deshan Sun, Di Qi, Enle Liu, Fajie Zhang, Fanqi Wan, Guanzhe Huang, Gulin Yan, Guoliang Cao. Guopeng Li, Han Cheng, Hangyu Guo, Hanshan Zhang, Hao Nie, Haonan Jia, Haoran Lv, Hebin Zhou, Hekun Lv, Heng Wang, Heung- Yeung Shum, Hongbo Huang, Hongbo Peng. Hongyu Zhou, Hongyuan Wang, Houyong Chen, Huangxi Zhu, Huimin Wu, Huiyong Guo, Jia Wang, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiashu Lv, Jiashuo Liu, Jiayi Fu, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yang, Jie Zhou, Jieyi Hou, Jing Bai, Jingeheng Hu, Jingjing Xie, Jingwei Wu, Jingyang Zhang, Jishi Zhou, Junfeng Liu, Junzhe Lin, Ka Man Lo, Kai Liang, Kaibo Liu, Kaijun Tan, Kaiwen Yan, Kaixiang Li, Kang An, Kangheng Lin, Lei Yang. Liang Lv, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lina Chen, Luck Ma, Mengqiang Ren, Michael Li, Ming Li, Mingliang Li, Mingming Zhang. Mingrui Chen, Mitt Huang, Na Wang, Peng Liu, Qi Han, Qian Zhao, Qinglin He, Qinxin Du Qiuping Wu, Quan Sun, Rongqiu Yang, Ruihang Miao, Ruixin Han, Ruosi Wan, Ruyan Guo, Shan Wang, Shaoliang Pang, Shaowen Yang, Shengjie Fan, Shijie Shang. Shiliang Yang, Shiwei Li. Shuangshuang Tian, Siqi Liu, Siye Wu, Siyu Chen, Song Yuan, Tiancheng Cao. Tianchi Yue, Tianhao Cheng, Tianning Li, Tingdan Luo. Wang You, Wei Ji, Wei Yuan, Wei Zhang, Weibo Wu, Weihao Xie, Wen Sun, Wenjin Deng, Wenzhen Zheng, Wuxun Xie, Xiangfeng Wang, Xiangwen Kong, Xiangyu Liu, Xiangyu Zhang, Xiaobo Yang. Xiaojia Liu, Xiaolan Yuan, Xiaoran Jiao, Xiaoxiao Ren, Xiaoyun Zhang, Xin Li, Xin Liu, Xin Wu, Xing Chen, Xingping Yang. Xinran Wang. Xu Zhao, Xuan He, Xuanti Feng. Xuedan Cai, Xuqiang Zhou, Yanbo Yu, Yang Li, Yang Xu, Yanlin Lai, Yanming Xu, Yaoyu Wang. Yeqing Shen, Yibo Zhu, Yichen Lv, Yicheng Cao. Yifeng Gong, Yijing Yang, Yikun Yang. Yin Zhao, Yingxiu Zhao, Yimmin Zhang, Yitong Zhang, Yixuan Zhang, Yiyang Chen, Yongehi Zhao, Yongshen Long, Yongyao Wang, Yousong Guan, Yu Zhou, Yuang Peng, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yudi Zhao Yue Peng, Yueqiang Lin, Yufan Lu, Yuling Zhao Yunzhou Ju, Yurong Zhang, Yusheng Li. Yuxiang Yang, Yuyang Chen, Yuzhu Cai, Zejia Weng, Zetao Hong, Zexi Li, Zhe Xie, Zheng Ge, Zheng Gong, Zheng Zeng, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhiheng Hu, Zidong Yang. Zili Wang, Ziqi Ren, Zixin Zhang, and Zixuan Wang. Step 3.5 flash: Open frontier-level intelligence with 11b active parameters. arXiv preprint arXiv:2602.10604 2026.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0 079777 2024.
Rongao Li, Jie Fu, Bo- Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312. 14852, 2023.
11

---

Published as a conference paper at ICLR 2026
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition code generation with alphacode. Science, (09666624:109221999 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 2221558-21172. 2023.
Yifei Liu, Li Lyna Zhang. Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang. rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset. arXiv 2025.
preprint arXiv:2505.2
Zihan Ma, Taolin Zhang, Maosong Cao, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking verification for IIm code generation: From generation testing. arXiv preprini arXiv:2507.06920, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL.https.///rxxi.orr////a
OpenAl. How people are using chatgpt, September 2025, URL how foww-people-oo are ceee--rre-usssnn chatgpt/
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and
Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
bttts:://rrxt httpp/////riii...g////////////// wwtp///////aaoooooooooooooo 2303 11366
Sicheol Sung, Yo-Sub Han, Sang-Ki Ko, al. Logicase: Effective test case generation from logical description in competitive programming. arXiv preprini arXiv:2505. 2025.
Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen. Codecontests+: High-quality test
case generation for competitive programming. arXiv preprint arXiv:2506.05887, 2025.
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. Dream-coder 7b: An open diffusion language model for code.
arXiv preprint 1xXiv:2509.01144 04422, 2025.
Xinwei Yang, Zhaofeng Liu, Chen Huang, Jiashuai Zhang, Tong Zhang, Yifan Zhang, and Wenqiang Lei. Elaboration: A comprehensive benchmark on human-llm competitive programming. arXiv
preprint arXiv 2025.
Maxwell Zeff.
Github
copilot
crosses
20m
all-time
users,
July 2025.
URL https://techcrunee com/2025/073333//
github copilot crosses
20 mi mollion-sse aal--imme users/
Eric Zelikman. Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. UtL http://axxvv.//////e wrg. hasp//b//220 htt/a/bb/22222222 220..14462
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https
// wwwpxi.orrg/ab///222 09629
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesizing algorithmic programs with generated oracle verifiers. Advances in Neural Information Processing Systems, 36:54769-544 54784, 2023.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. 2025.
arXiv preprint arXiv:2505.033.
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang. Wenhao Chai, et al. Live- codebench pro: How do olympiad medalists judge llms in competitive programming? arXiv preprint 250i 1928, 2025.
12

---

Published as a conference paper at ICLR 2026
Shang Zhou, Feng Yao, Chengyu Dong, Zihan Wang, and Jingbo Shang. Evaluating the smooth control of attribute intensity in text generation with llms. arXiv preprint arXiv:2406.0444 2024.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprimt arXiv:2406. 01.4406.1999 1931,, 2024.
13

---

Published as a conference paper at ICLR 2026
Appendix
The appendix is structured as follows:
Preliminaries and definitions key concepts and terms in Section
Example of Competition- level generated problem in Section
Example of generated problem that LLM itself can't solve in Section
Example of valid inputs and near-valid illegal inputs in Section
Example of incorrect mutant programs in Section H
Human experts grading policy on L.IM generated problems in Section
Pseudo code of simulator in interactor in Section
G
Limitations and future work in Section
Prompt templates in Section 1
Early results of RL on AutoCode data in Section
Fair comparison with Hardtests in Section
Human evaluation process and requirements in Section
A PRELIMINARIES
In this section, we provide background information on the judgment process in competitive program- ming, define key terminologies used throughout the paper, and detail the core components of the AutoCode framework.
A.1 COMPETITIVE PROGRAMMING SOLUTION JUDGMENT
In competitive programming, solutions are judged by executing them against series of test cases. Each test case consists of an input and ground-truth output. The system compares the program's output for each input against the ground-truth answer. solution only considered correct passes all test cases while executing within the problem's time and memory constraints. The outcome of the judgment process is a single verdict. The most common verdicts are:
Accepted (AC): The solution passed all test cases within the given time and memory limits. This is the only successful outcome.
Wrong Answer (WA): The solution produces outputs that fail on at least one test case.
Time Limit Exceeded (TLE); The solution exceeds the time limit on at least one test case.
Memory Limit Exceeded (MLE): The solution exceeds the memory limit on least one test case.
Runtime Error (RE); The solution terminates abnormally due to error like segmentation fault or division by zero.
A.2 KEY TERMINOLOGIES
Codeforces. popular online platform that hosts competitive programming contests. The difficulty of problems on this platform is often used as benchmark.
Codeforces, each problem has numerical rating assigned to problem quantify its difficulty. It's calibrated based on the performance of contestants on this problem during the competition. In general, a difficulty rating of 2000 is deemed as easy to medium range, (2000, 3000] is deemed medium to hard range, and > 3000 is deemed very hard (Zheng et al. 2025).
Elo rating for problem difficulty.
A program that produces the input files for the test cases. A robust set of generators needed to create diverse set of test cases that cover edge cases, average cases, and worst-case scenarios designed to challenge the time memory complexity of incorrect algorithms.
Generator.
14

---

Published as a conference paper at ICLR 2026
Validator. A program that verifies whether test case conforms to the constraints described in the problem statement (e.g., 1 I N Σ 1000, the given graph is a tree).
Checker. program that compares a contestant's output the correct answer and issuing a verdict. Many problems require custom Checkers beyond simple text comparisons, including
Problems with multiple correct solutions: For example, constructive problem asking for any valid path in graph, the Checker must verify the validity of the contestant's proposed path, not just match it to one specific example.
Problems with floating- point outputs: When the answers are floating-point numbers, answers are accepted if they are within a certain absolute or relative error tolerance (e.g., 10-
Problems where output format is flexible: For instance, some problems that require contestants output "Yes" "No" may also accept "YES" "NO" as correct answers.
Interactor. A specialized component is used for interactive problems. In an interactive problem, contestant's solution does not receive all input at once. Instead, engages in a real-time dialogue with the judge's 's program (the Interactor). The solution makes series of queries the Interactor, receives responses, and use the information gathered from this dialogue to determine the final answer. A classic example is guessing game where the Interactor knows secret number, and the solution must find it by making queries and receiving "higher" or "lower" as responses.
B EXAMPLE OF LLM GENERATED PROBLEM: ROw-COLUM PORTAL
Time limit per test: seconds
Memory limit per test: 256 megabytes
You are given binary grid with n rows and m columns (0 = empty, obsidian).
Fix two integers and with 5 I < and 4 b A sub-rectangle M of size x b (with rows r..r + a and columns c..c + - 1) is called a portal and only if:
Every cell on the border of M is except the four corner cells (which can be either or 1).
Every strictly interior cell of M is O.
You are allowed to choose one sub-rectangle of size x b and then perform the following operation any number of times inside that chosen sub-rectangle only:
Pick of the sub-rectangle and flip all its cells (0 1), or
Pick column of the sub-rectangle and flip all its cells (0 <+ 1).
Flips affect only the chosen x sub-rectangle (not the rest of the grid). Your task is to find the minimum number of flips (rows columns) needed to turn some b sub-rectangle into a portal. If it is impossible for every placement, print 11
INPUT
The first line contains an integer (1 $ 109) - the number of test cases.
For each test case:
The first line contains four integers m. a, b (5 < a < < 2000, 4 < S m & 2000).
Then follow n lines, each a string of length m consisting characters 0 and
is guaranteed that the sum of n m over all test cases does not exceed 105.
OUTPUT
For each test case, output single integer the minimal number of flips needed, 1 if impossible.
15

---

Published as a conference paper at ICLR 2026
EXAMPLES
Input
4
5 4
1111
1001
1001
1001
1111
Output
Input
1
6 5 4
010101
111000
001010
111111
000000
101010
Output
ACCEPTED SOLUTION (C++)
#include <bits/stdet+ .h>
using namespace std;
// 2D prefix sum over matrix of ints (0/1).
struct Fref20 {
int n, m; veetor<vector<intt> 9s; // (n+1) x (m+1)
Pref2D () ()
Frof2D (int n int m ((n),, m(m)), ps vector<int> (m_+i, 0))
(}
void buind vettoreweterdinoog a)
11
for (int i 0; i n; ++i) {
12
int rowSum = 0; for (int 0; j < ++5) {
rowSum alillil;
15
psli+1lli+1] pslilli+11 rowSum;
16
int sum (int rl, int cl, r2, int c2) const
20
if (rl r2 11 cl c2) return 0;
21
return ps[r2+1] [c2+1] ps[rl] [c2+1] ps [x2+1] [cl] + ps [s]]]]]]]]]]] [el])
};
24
25
int main())
int t; 26
withhsttdeee (false); cin.tie (nullptr);
16

---

Published as a conference paper at ICLR 2026
if (cin t)) return while (t--)
31
int n, m, a, bi ein >> >> m >> a >> bi
32
vector<string> (n)) for (int
i n; ++i) cin s[i];
36
vector<vector<ints>>
(n,, vector<int> (m, 01);
37
for (int i = 0; i ++i)
for (int j j A[i][j] (s[i][j] 119;
++3)
40
41
vector<wector<iit>>> rowpS (n, vector<int> (m+1, 0)); vector<vector<intt> (ntl, vector<int> (m, 0));
for (int i = i n; +ti)
44
for (int j == 0; j m; ++3)
45
rowpS[i) rowPS[i][]] + A[l](];;
46
co]Ps[1+1][j] ccIPS[i][3] + a[1]());;
49
50
int NỘn - max(0, n-1), NQm - max m-1); vector<vectorcint>> Q (VQn, vector<int> (N0m, 0)); for (int i = i+1 < n; ++i)
51
53
for (int j 0; j+1 < my ++il
54
Q[i][i] - Atillil A[i+1]li] Atil(j+t] 15-111
55
Pref2D Ops (NOn, NQm) Qps. (0s
58
59
60
long long best = (long leng) 1c18; int h a, b;
62
63
for (int r 0; r
n; ++D) {
64
for (int a = 0; c + w <= m; ++C)
int rql = r, eql
rq2 r h 2, eq2
2;
int total sps.sum(rql, eql, rq?, cq2);
if
&
2)
68
total - [irqll[cqq]]
total - Q[rql] [cq2];
total Q[rq2] [cql];
71
total Q[rq]] [cq2];
72
73
if (total != 0) continue;
int pivotRow
pivotCol 1;
int p AlpivotRow [pivotCol];
77
int innerColsOnes rowPS [pivotooo [c + 1 rowPS Row [c 11;
int leftEnd A[pivetRow] [c] 1;
80
int rightEnd A[pivotRow] Ic w 11 1; int S_cols = innerColsCnes + leftEnd rightEnd;
s1
int I
U
int len max (0, int onesInSeg (len ? [oolps(U+11 [pivotCol] colPs
+);
pivotColl) 0);
int imequalInterion = == 0 onesInšeg (len onesTn
int top Airl [pivotColl p; int bot Air + 11 lpiootCColl pi
int rows Inequalinterior 2 d -d bot;
17

---

Published as a conference paper at ICLR 2026
92
int s0 s_rows S_cols;
93
int flips min (80, h w s0); best = min (best, (leng long) flips);
94
95
97
98
if (best = (long long) 1e18) cout << -1 <s "\n"; else cout << best << "\
return 0;
102
C EXAMPLE OF LLM GENERATED PROBLEM THAT ITSELF CANNOT SOLVE: IT RESTAURANTS - DISTANCE-2 RIVALRY
Time limit per test: 2 seconds Memory limit per test: 256 megabytes
City N still has tree road network with n junctions (n - roads, connected, undirected). The mayor again wants to place restaurants of two rival networks: iMac Donalds and Burger Bing. Each junction can host at most one restaurant.
This time, the owners impose a stricter rule set:
Rivalry (distance- 1): Two neighboring junctions (joined by road) cannot host restaurants of different networks.
Brand dilution (distance-2): Two junctions at distance exactly 2 (i.e., they have a common neighbor) cannot host restaurants of the same network.
Both networks must build least one restaurant. The mayor wants to maximize the total number of restaurants. Among all optimal placements, find all pairs (a, b) such that restaurants belong to Mac Donalds and to Burger Bing, with a + maximized.
Print all such pairs sorted by increasing
INPUT
The first line contains a single integer (3 S s 2000).
Then follows n 1 lines: edges v, (1 < 24, S n).
The edges form a tree.
OUTPUT
2 al bl
a2 b2
az bz
Where is the count of pairs, and (a;, b.) all pairs with maximum a; + d, sorted by ai ascending. no valid placement uses both networks, print only.
EXPLANATION OF THE NEW RULES
Adjacent restaurants cannot be different brands (so if both ends an edge have restaurants, they must be the same brand).
18

---

Published as a conference paper at ICLR 2026
However, for any chain
if and w both have restaurants, then they must be different
brands (distance-2 same-brand is forbidden).
Together these imply strong local coupling between siblings the rooted tree:
If node is occupied, then have the same brand as the parent).
one of its children can also be occupied its root (and must
If node is empty, then among its children, at most one may have red root and at most one may have a blue root (siblings with occupied roots must have different brands).
These constraints make the original "remove one vertex and do subset sums" approach invalid; we need new DP.
EXAMPLES
Input
2
2 3
5
Output
1 2
One optimal solution uses 3 restaurants the path (e.g., 1-R, 2-R, 4=., 5=B), giving totals (2,
(1,2).
Input
1 3
4
1 5
Output
1
1
A star: the center must be empty to allow both brands on leaves; at most one red leaf and one blue leaf exactly 11.
D NEAR- VALID TEST CASE EXAMPLE: PERPENDICULAR SEGMENTS TIME
time limit per test: seconds
memory limit per test: 256 megabytes
You are given coordinate plane and three integers Y, and K. Find two line segments AB and CD such that
1. the coordinates of points A, B, C, and are integers;
2. 0 < Ag, Ca, Dr S X and I Ay, By, Cy. D, $ Y;
3. the length of segment AB least K;
4. the length of segment CD is at least K;
19

---

Published as a conference paper at ICLR 2026
5. segments AB and CDD are perpendicular: if you draw lines that contain AB and CD, they will cross at a right angle.
Note that it's not necessary for segments to intersect. Segments are perpendicular as long as the lines they induce are perpendicular.
INPUT
The first line contains single integer (1 < v 5000) the number of test cases. Next, t cases follow.
The first and only line of each test case contains three integers X, Y, and K (1 s X,Y & 1000; 1 < K 1414).
Additional constraint on the input: the values of X, Y, and K are chosen in such way that the answer exists.
OUTPUT
For each case, print two lines. The first line should contain integers Az, Ay, Bz, and By the coordinates of the first segment.
The second line should also contain integers CaN Cg, Da, and D, - the coordinates of the second segment.
there are multiple answers, print any of them.
EXAMPLE
input
1 1
1
3
3
output
0
0
1
0
0
1
2
4 1
2 1 1
2
1
0
3
2
4
1
1
4
3
3
0
D.1 NEAR-VALID TEST CASE EXAMPLE
Here are two minimal test inputs that differ by only 1 in a single field (K). The first is eligible (a solution exists), the second is not (no segment of length 2 K can fit at all).
Valid
1
5
Why: In 4x4 box, you take two perpendicular diagonals: (0, - (4,4) and (0,4) (4,
Each has length /42 4 v32
5.657 > 5.
20

---

Published as a conference paper at ICLR 2026
Near Valid
4 4 6
Why: The longest possible segment inside 4x4 box is the diagonal 5.657, which is so segment of length 2 can't exist, hence two perpendicular ones can't either.
E EXAMPLE OF INCORRECT MUTANT PROGRAMS: GUESS THE TREE
time limit per test: seconds memory limit per test: 256 megabytes
This an interactive problem.
Misuki has chosen secret tree with n nodes, indexed from 1 to n, and asked you to guess it by using queries of the following type:
? a b Misuki will tell you which node x minimizes Id(a, x) - d(b, x)). where d(a,, y) is the distance between nodes x and y. If more than such node exists, Misuki will tell you the one which minimizes d(a, r).
Find out the structure of Misuki's secret tree using at most 15n queries!
INPUT
Each test consists of multiple test cases. The first line contains single integer t S 200) the number of test cases.
Each test case consists of single line with an integer S n < 1000), the number of nodes in the tree.
is guaranteed that the sum of n across all test cases does not exceed 1000.
INTERACTION
The interaction begins by reading the integer n.
Then you make up to 15n queries.
To make a query, output a line the format "? b" (without quotes) (1 S a, S n)) After each query, read an integer the answer to your query.
To report the answer, output a line the format "! al b1 a2 b2 a (n-1) b (n-1)" (without quotes), meaning that there is edge between nodes a, and b,, for each S You can print the edges in any order.
After 15n queries have been made, the response to any other query will be -1. Once you receive such a response, terminate the program to receive the Wrong.Answer Answer verdict.
After printing each line, do not forget to output the end of line and flush the output buffer. Otherwise, you will receive the Idleness limit exceeded verdict. To flush, use:
f f ush (stdout) cout flush () in C++;
System. out, flush () in Java;
flush (output.) in Pascal;
stdout.c flush () in Python;
the documentation for other languages.
21

---

Published as a conference paper at ICLR 2026
HACKS
For hacks, use the following format: The first line contains an integer / (1 S S 200) the number of test cases.
The first line of each test contains an integer n2 the number of nodes in the hidden tree.
Then lines follow. The i-th of them contains integers as and bç (1 S ai b, S n), meaning that there is an edge between a; and b in the hidden tree.
The sum of all test cases must not exceed 1000.
EXAMPLE
input
1
1
output
1 2
1 3
1 4
1 2 3
NOTE
tree is undirected acyclic connected graph. A tree with nodes will always have 1 edges. In the example case, the answer to "? 2 is This means that there is an edge hetween nodes and 2.
The answer to "? 1 3" is 1. This means that there an edge between nodes and 3.
The answer to "? 4" is 3. It can be proven that this only happen if node 3 is connected to both node and 4.
The edges of the tree are hence (1,2), (1,3) and (3,4).
F. WHY IT'S WRONG
Here's subtly broken version. behaves the same the sample, but the component update decision based on comparing the node index to the label the found node (instead of comparing component labels). This will make it fail plenty of hidden cases while still looking plausible.
Vinclude <iostream>
Vinclude <numeric>
Vinclude <vector>
#include <array>
using namespace std;
8
int query (int u, int v) {
cout << "? << + <s < v + <s (nn;
int cin >>
return
22

---

Published as a conference paper at ICLR 2026
14
int main() ()
15
int t; cin >> ti while (t--)
int ny cin nj
19
vector<array<int, 2>> ej
20
vector<int> c(n); iotalc.begin c.end 0);
23
auto addEdge [&] (int u, int v) {
24
pusuh_bbcck ((u, v));
vector<int> cand;
for (int 0; n; it+)
27
if [c[i] [[v])
28
cand, wwwpacc_paaccocccccccccoocccocoe (1); for (int cand)
29
c[i] = clul;
};
33
for (int i 0; i< 1, i++) {
34
int = 0, v 0;
while (clvl clul) v++;
36
x;
37
// SUBTLE BUG: use (u -- c[x]) instead of (c[u] -- c(x)).
38
while ((x query (u, v) 1- u) {
if (u c(x)) x; // <-- wrong criterion else x;
41
42
addEdge (u, v);
43
45
cout "!";
46
for (aute [u, v] e) cout << << < <v+
47
cout << (nn;
F HUMAN EXPERT GRADING POLICY
Each problem is assessed independently across five dimensions: Solvability, Solution Correctness, Quality Rating, Novelty, and Difficulty.
F.1 SOLVABILITY
This criterion evaluates whether deterministic solution exists that satisfies the problem's time and space constraints.
Yes: A deterministic solution exists that satisfies the problem's 's time and space constraints.
No: It can he proven that no such solution exists.
Unknown: It is unclear whether a valid deterministic solution exists.
F.2 SOLUTION CORRECTNESS
This evaluates whether the provided reference solution (e.g., model -enerreee code) fundamentally correct in its approach.
Yes: The core idea of the solution is correct. We allow minor implementation errors, as long as the overall idea is correct.
23

---

Published as a conference paper at ICLR 2026
No: The solution contains fundamental flaws in reasoning design.
Unknown: It is not possible to determine the correctness of the provided solution approach.
F.3
QUALITY RATING
The overall quality of problem is rated on scale from to 5, considering clarity, originality of approach, and implementation difficulty.
1: Severely flawed. For example, unsolvable problems incomprehensible descriptions.
2: Low quality. For example, unclear statements, trivial or uninteresting solutions, or unverifiable correctness of the reference solution.
3: Moderate quality. The problem clearly solvable, but may be derivative, overly complicated, inelegant, or rely obscure knowledge.
4: High quality. The problem is clear, solvable, moderately difficult, and contains at least one insightful idea, though its overall structure might still feel somewhat formulaic standard.
5: Excellent quality, The problem is elegant, novel, and appropriately challenging. It is free from contrived traps and its solution is non- non-trivial, requiring genuine insights.
F.4 NOVELTY
This measures whether the problem introduces a genuinely new challenge.
Yes:
The problem is novel and not seen in prior contests or problem archives, even for experienced setters.
No: The problem duplicates or closely resembles existing ones.
Unknown:
It is unclear whether the problem is novel.
F.5 DIFFICULTY
Difficulty is assessed using the Codeforces rating scale (e.g., 800, 1200, ...)....). This score is provided as a reference and does not directly influence the overall rating.
F.6 OVERALL PROBLEM RATING
A composite grading system is defined follows. A problem must satisfy the requirements of its assigned level and all lower levels.
0: All evaluated problems.
> 1: Solvability = Yes.
2: Solution Correctness Yes.
3: Quality Rating 2 3.
4: Quality Rating 4.
5: Novelty = Yes.
6: Quality Rating 5.
24

---

Published as a conference paper at ICLR 2026
G SIMULATOR IN INTERACTOR
Algorithm 5: SIMULATE (in Interactor)
Input: Interactor I, solver A, Validator V
Output: Accepted/Réjected
Initialize protocol state; time/memory limits; seed RNG deterministically
while session active do
msggr - [ (state); if WELLFORMED(MSgg)
L return Rejected
msgA f- A( WELLFORMED((NSA))
return Rejected
if E.PPOTTDSSTTTLLAAVVVR((gggg
if (implied input) * valid:
return Rejected
10
return FUDGLEINAL(,, msga)
11
UPDATESTATEEStaat, msgr, mSgA)
12 return Rejected
H LIMITATIONS AND FUTURE WORK
While AutoCode demonstrates robust pipeline for generating and verifying competitive program- ming problems, we acknowledge several limitations that open up promising avenues for future research.
The quality judgment gap. A primary limitation that LLMs currently lack a robust, human- aligned sense of problem quality. As shown in Finding 4 of Section [5] there is little correlation between LLM self-evaluations and the scores provided by human experts. This judgment gap means system must still rely on human experts identify truly high-quality, novel problems. This reliance on manual annotation a bottleneck to a scalable, high-quality problem generation pipeline. key future direction is develop specialized judge LLM, fine-tuned expert-rated problem datasets to better align its assessments with human preferences, potentially serving as a reliable automated filter.
The bottleneck for automated self-improvemennt AutoCode is ideally suited for creating a self- improvement loop, where an LLM could enhance its reasoning abilities by training the problems it generates. However, deploying and testing such a system faces immediate practical barriers. The most capable woobem--gnnerating models like GPT-5 are closed-soures and do not expose the APIs necessary for reinforcement learning. Even for powerful open-source models, scalable and stable yet mature support fine-tuning at this scale the best of our knowledge. natural future direction is to focus on developing RL or self-play frameworks compatible with frontier LLMs to unlock the full potential of AutoCode.
RL frameworks
Advancing beyond recombinational novelty. Our analysis indicates that L.LMs currently tend to create new problems by combining existing algorithms rather than creating conceptually new problems that require subtle insights. Future research could explore novel prompting strategies or problem-generation pipelines encourage greater conceptual leaps. Additionally, for valid problems that LLMs can generate but cannot solve themselves (i.e., their reference solution std.cpp is incorrect), AutoCode's 's strict verification protocol currently filters them out. A prominent future direction is to develop scalable methods for identifying these valid but unsolved questions without relying on manual check by human experts, which is extremely lahor-intensive process.
25

---

Published as a conference paper at ICLR 2026
1
PROMPT TEMPLATE
Checker Prompt Template
This is competitive programming problem statement and its accepted solution:
"markdown
(problem statement)
'cpp
(accepted solution]]
1..
Implement a testlib checker for this problem.
a standard checker sufficient, your response should contain only the name of the appropriate cpp file from the list below.
fcmp. opp: Compares files line by line, doesn't ignore trailing whitespaces on each line.
lcmp. cpp: Compares files line by line, ignoring trailing whitespace each line.
hemp cpp: Compares single huge integer.
wemp cpp: Compares sequences of tokens, ignoring trailing whitespace.
ncmp. app: Compares sequences of integers, igmoring trailing whitespace.
remp4 opp: Compares sequences of real numbers with precision of 10 10-4.
remp6 opp: Compares sequences of real numbers with precision of 10 10-6.
rcmp9. cpp: Compares sequences of real numbers with precision of 10
yesno. cpp: Compares a single "yes" or "no" token, case-insensitively.
For example, reply "womp. cpp", do not include quotation marks and other content.
the problem requires custom validation logic that checker comparator cannot handle, your response should contain the checker enclosed in single markdown block:
''ppp
//checker code here
11
Please follow these additional constraints:
Always include "testlib. first, then <bits/stdc=t.h> and add using saacspace st..
All testlib. functions in the global namespace. Never use the testlib: prefix or using namespace testlib;
Initialize your checker with registerTest bCmd (arge, argv) ;-
Use the built-in inf (input), ou (contestant output), and ans (jury answer) stream. Call methods the correct stream, e.g., long long inf readLong () string ouf readToken () Do not create extra stream objects.
When passing stream helper function, declare the parameter as InStreams; e.g., int readAns (InStreams stream).
The optional message parameter of stream. readLong (), stream. readToken (), must be fixed literal print--pritf--syllee placeholders like $d are not supported. Write inf.readLong (OLL, "x"); and never inf readLong (CLL, "test case # tc);.
Use stream. readLeng () for 64- integers. Ensure all arguments leng long by using the LL suffix (e.g., inf. readLong (1LL, 100LL) ;). Do not use arguments of type int or double, such ans. readLong (-1e9, 1e9)
26

---

Published as a conference paper at ICLR 2026
Use stream. readToken read string; do readStri ng () Also, do not add extra arguments inside readToken (..tokee(...). (...), for example, do not write inf. readToken ("string").
Do
ensure ensuref; instead, call quitf( ok, swers and quitf (_wa, "....) for wrong answers. Do not use string concatenation to form the message; must be a single string literal.
") for correct
To read character from specific set: first read it using char stream. reacChar () then validate it with quitf.
Use stream. readSpace () for a single space and stream. readEoln () for an end-of-line. These separators are read automatically, you only need to call them if you are about to call stream, readChar () immediately afterward.
Do use the ternary operator (? :) for void functions like stream. readBoln ().
Use a standard if/else block instead.
The judging machine does not have boost-multiprecision installed; avoid including or relying on
Never use non-existent functions like stream.skip$pacee (),
stream.peekChar (), stringToLongLong (), or getTestCases ()
Do not check whether the stream has reached EOF; testlib handles end- of-file checks automatically. For example, avoid calling ouf (), ans readEof (), ouf. ensureEof (), a ensureEof (), ouf.afterEOF (), ans afterEOF ().
The ans stream is the output of the accepted solution. Do not skip any numbers strings, and not expect any extra content.
Checker Tests Template
This is competitive programming problem statement and its accepted solution:
'mrrkdown
(probblem statement))
''epp
((accepted solution))
Write 20 checker tests for this problem.
Your response should contain the checker tests enclosed in single markdown code block. Use "verdict": when the output meets the problem constraints, and "verdict"" otherwise.
Provide 5 correct outputs and 15 incorrect outputs.
input is valid input, output is the contestant's output, answer is the jury's output, and verdict indicates whether the output meets the problem's 's constraints.
The format should look like this:
"json
"input": "...", "output": "...", "answer": "...", "verdict" 1 }
"input": ;; "...", "output": "...", "answer": "...", "verdict" 0 }
Please follow these additional constraints:
27

---

Published as a conference paper at ICLR 2026
Do not include any explanations, comments, any other text outside of the JSON structure.
The incorrect output should be input that becomes correct with minor modifications.
The answer you generate should preferably be the accepted solution that provides the correct output for the input.
Generator Template
This is competitive programming problem statement and its accepted solution:
'marrddwnn
((problem statement))
''ppp
(accepted solution))
'1
Implement a testlib generator for this problem. Your response should contain the generator enclosed in single markdown block:
'cppp //generator code here
The generator should take single command-line argument (from to 10) to determine which test generate.
For a single-test problem: The generator must output exactly 10 small, distinct, and meaningful corner cases.
For multi-test problem (with test cases):
Let mma be the maximum allowed number of test cases. The generator must define 10 distinct test groups, each associated with different command-line argument (from 10), and each
with its own specific variable constraints (e.g., Group 1: s < 5, 1 < ali] < 9; Group 2: 15 n S 10, 1 < ali] < 3).
Each group must be able to generate more than m but no more than maxx valid and distinct test cases.
The generator must then output exactly fma test cases, randomly sampled from the distribu- tion defined by the specified group.
Please follow these additional constraints:
Always include "testlib.h" first, then <bits/stdo=+.B h>, and add using namespace std.
All testlib. h functions are in the global namespace. Never use the testlib: prefix usi ng namespace testlib;
Never use the global:: prefix or std::.
Please use int testId = atoi [1]g to obtain the test ID.
Initialize your generator with registerGen (arge, argv, 1);.
For conditional validation with a custom message, use ensuref (condition, "message");
When calling rnd. next (1, r), ensure that both 1 and r are of the same type, either both int both long long. Note that rnd. next<int>> (1, r) not valid syntax.
Use the global shuffle sirec.beging vec.end ;).. Do not use rnd.shuffi (vec. (), end.end()) or shuffle (vec ()egin)) vec.end (), rnd).
28

---

Published as a conference paper at ICLR 2026
Don't t use int 64 i 14, that's not valid syntax.
Don't It define lowerCase, it will conflict with std. Use different function name instead.
Use cout to print all output.
Do not set the random seed manually.
Do not use non-existent methods like rnd. sadlea (), rnd. permutation (),
rnd. next (), randomString () global-shuffle ().
Generator Template
This is competitive programming problem statement and its accepted solution:
'markdown
(problem statement))
'cppp
(accepted solution))
Implement a testlib generator for this problem. Your response should contain the generator enclosed in single markdown block:
''ppp //generator code here
The generator should take single command-line argument (from to 10) to determine which test case generate. The generation strategy is divided into two parts:
Large Random Cases (5 test cases)
The goal fail incorrect solutions with strong, randomized data.
Tests 1-2: Generate cases where n and are large and roughly equal, with their product
x approaching the maximum allowed limit.
Tests 3-5: Generate cases where t = and is set to its maximum possible value.
For all tests:
Some cases should be completely random, while others should conform specific structure relevant to the problem (e.g., a star graph, a chain, an array with all equal elements).
While meeting the primary condition for the test (e.g., max), randomize all other variables across their largest possible valid ranges.
Ensure these cases collectively cover various scenarios that correct solution must handle.
Boundary/Edgg Cases (5 test cases)
The goal target specific, extreme, tricky corner cases.
Analyze the given problem and select the 5 most effective strategies from the list below to create test cases that are most likely fail incorrect solutions. You may also devise your own problem-specific strategies if they are more effective.
Potential Strategies:
test that maximizes the algorithm's 's runtime.
A test case that maximizes the numerical output value check for integer overflows (e.g.,
int, long long).
test case that maximizes memory usage.
A designed cause hash collisions for common hashing methods.
29

---

Published as a conference paper at ICLR 2026
test that maximizes the total number of characters in the input output.
Problem-specific corner cases.
Please follow these additional constraints:
Always include "testlib.h" first, then <bits/stdc++.br h>, and add using namespace std.
All testlib. ib. functions are in the global namespace. Never use the testlib: prefix or Fuusing ng namespace testlib;.
Never use the global:: : prefix or std::.
Please int restId atoi (1]) to obtain the test ID.
Initialize your generator with registerGen (argc, argv, 1);.
For conditional validation with a custom message, use ensuref (condition, "message");
When calling rnd. next (1, r), ensure that both 1 and are of the same type, either both int or both long long. Note that rnd. nwx.<eexntttt... (1, r) is not a valid syntax.
Use the global shuffle (vec (vec. beginn (), en) Do not und.sfuflle (vec.begin vec.end or shuffle (vec.begin(), vec.end (), rnd).
Don't 64 or 14, that's not valid syntax.
Don't define lowerCase, it will confict with std. Use different function name instead.
Use cout to print all output.
Do not set the random seed manually.
Do not non-existent methods like rnd. sample (), rnd. permuttaaaiio (), next. (), randomString (), or global-shuffll ().
Generator Template 3
This is competitive programming problem statement and its accepted solution:
'markdown
(problem statement]
'cppp
(aaccepted solution)]
Implement a testlib generator for this problem. Your response should contain the generator enclosed in single markdown block:
''ppp //generator code here
11
The generator should take single command- argument (from to 10) to determine which case to generate, select which of the following ten test case generation strategies to use. You may also devise your own problem-specine strategies if they are more effective.
1. Generate a test with the maximum number of test cases (f), where t 1 cases have minimal input size (e.g., n 1), and one case has the largest possible n to maximize the total input size.
2. Generate test with the maximum number of test cases (t), where is the same large value for all cases, maximizing the total input size.
30

---

Published as a conference paper at ICLR 2026
3. Generate test where the output for each case is maximized, leading large total output size.
4. Generate a test case containing long, contiguous sequence of identical values.
5. Generate a test case containing long, strictly increasing sequence.
6. Generate test case containing long, strictly decreasing sequence.
7. Generate test containing long sequence of alternating large and small values.
8. Generate a test case designed to exploit the worst-case time complexity of a common brute-force incorrect algorithm for this problem.
9. Generate problem-specific defined in the problem statement, pushing it to its limits.
case that maximizes particular metric or constraint
10. Generate a test case that is unlikely to covered by small or random inputs, targeting specific edge conditions that might lead performance issues.
Please follow these additional constraints:
Always include "testlib. first, then <bits/stdc=+.hn h>, and add using namespace std.
All tlltetiibb. h functions are in the global namespace. Never use the testlib: prefix or using namespace testlib;.
Never use the global:: prefix or std::.
Please int testId atoi (1]); obtain the test ID.
Initialize your generator with registerGen (arge, argv, 1)
For conditional validation with a custom message, use ensuref (condition, "message")
When calling rnd. (1, r), ensure that both 1 and are of the same type, either both int or both long long. Note that rnd. next<int> (1, r) is not a valid syntax.
Use the global shuffle (vec svrecbbggn() vec.end Do not use rnd.shuffle (vec.begin() vec.end ()) or shuffle (vec.begin(), vec. vec.end (), rnd).
Don't int 164, that's not valid syntax.
Don't define lowerCase, it will conflict with std. Use different function name instead.
Use cout to print all output.
Do not set the random seed manually.
Do not use non-existent methods like rnd rnd. sample (), rnd. ind.prrmuattion (), rnd. next (), randomString (), or slufal ().
Validator Template
This is competitive programming problem statement and its accepted solution:
"markdown
((problem statementt)
''cpp
(accepted solution))
Implement a testlib validator for this problem. Your response should contain the validator enclosed in single markdown block:
cpp
31

---

Published as a conference paper at ICLR 2026
1/val idator code here (t)
Please follow these additional constraints:
Always include "testlib.h first, then <bits/stdcrt.h> h>, and add using
namespace std.
All testtiib.b functions are in the global namespace, Never use the testlib: prefix using namespace testlib;
Initialize your validator with registervar idation (arge, argv)
All validator testlib read functions are methods of the inf object. Always call them using inf. for example: inf. readInt (). Do not use ouf ans.
Use inf. readLong for 64-bit integers. Ensure all arguments long long by using the LL suffix (e.g., inf. readlong (1LL, 100LL) ;).
Use inf. readToken to read string; do not use readString (). Also, do not add extra arguments inside readToken (...), for example, do not write inf. readToknn ("string").
To read character from specific set: first read using char inf. readhar ();, then validate it with ensuref, e.g. ensuref (c == 4 11 "Must be + or =");
For conditional validation with a custom message, use ensuref (condition, "message")
Always read separators explicitly. Use inf. readSpace () for single space, inf. readEonn () for end-of-line, and inf, readEof () for end-of-file. These calls are mandatory.
The last two reads are always inf. readEoln () and then inf readEof ()
Do not use the ternary operator (? :) for void functions like inf. readEoln () Use standard if/else block instead.
The judging machine does not have boost-multiprecision installed; avoid including or relying on
Never use non-existent functions like inf. sk skipsppppaae (Spacess () inf, pee.phe ()
Do not use quit or quitf.
If the input violates the additional constraints guaranteed by the problem statement (c.g., an answer is guaranteed exist), it must be judged as invalid.
If the input would cause the accepted solution fail, the validator must reject it.
Validator Tests Template
This is competitive programming problem statement and its accepted solution:
'markdown
(problem statement] 1
'cppp
{ (accepted solution})
Write 20 validator tests for this problem.
Your response should contain the validator tests enclosed in a single markdown code block. when the input meets the problem constraints, and "verdict":
Use "verdict": otherwise.
Provide 5 valid inputs and 15 invalid inputs.
32

---

Published as a conference paper at ICLR 2026
The format should look like this:
"json
"input": "...", "verdict": }
"input": "...", "verdict"; 0 /
...
Please follow these additional constraints:
Do not include any explanations, comments, any other text outside of the JSON structure.
The invalid input should be an input that becomes valid with minor modifications.
If the problem includes additional constraints, such as "an answer is guaranteed to exist", prioritize constructing invalid cases that meet other conditions but the additional constraint.
the input would cause the accepted solution to fail, it must be invalid.
Author Template
This is competitive programming problem statement and its accepted solution:
"ma rkdown
(pprbblem statement))
''ppp
(acceepted solution)]
You are experienced competitive programming problem setter. Please modify at least one condition of the problem to design a new, more difficult problem. This new problem must require different solution approach from the original and should not be solvable using heuristic methods. The problem must be of such quality and novelty that it could be accepted on platform like Codeforces, at the very least serve valuable training exercise. In the problem statement, please include time and space constraints and provide sample test cases. Furthermore, please write an accepted C++ solution, along with complexity analysis, proof of correctness, and a clear explanation.
Response Format:
''markdown
Problem statement for the new problem
''ppp
// Accepted solution for the new problem
11
1/ Solution explanation and proof
J EARLY RESULTS OF RL ON AUTOCODE DATA
To provide an objective complement to our human evaluation of the "Model-Training Usable" as the base model. We employ the Slime framework[) to implement GRPO, using the 133 AutoCode-generated problems as training data and the corresponding AutoCode-generia model is trained for 5 epochs with global batch size of 64, constant learning rate of x 10-6
criterion, conduct preliminary RL experiment using Swen3-8B-Think-e
test cases the reward signal. The
https://gitthee com coomTTHJOM
33

---

Published as a conference paper at ICLR 2026
and the AdamW optimizer (31 0.9, .2 0.98). During the rollout phase, generate samples per prompt with temperature 0.8. We then evaluate the model on LiveCodeBench before and after fine-tuning.
Model
Metric
Base Model
Fine-tuned w/ AutoCode
Qwen3-8B
Pass@1
46.2%
48.5 %
Table 3: Preliminary GRPO fine-tuning on AutoCode-generatee problems and test cases. Training on AutoCode data yields improvement in LiveCodeBench Pass providing initial evidence that the generated problems are useful for model training.
This experiment is intentionally small-scale and serves as a proof-of-concepe scaling the training (e.g., more problems, longer training, and additional base models) is left for future work more rigorously quantify the urility of autoCode-generann problems for model training.
K COMPARISON WITH HARDTESTS
To demonstrate that the performance gains are derived from our proposed AutoCode pipeline rather than simply the of better models (e.g., o3, GPT-5), we conducted controlled quan- titative analysis. We compared AutoCode directly against HardTeests, keeping all other variables constant. In particular, we use the exact same LLMs for generation (GPT-40 and o3) and iden- tical execution environments for both pipelines. We use the official HardTTsst implementation (https://itthub.com/))
The results (shown the table below) indicate that AutoCode outperforms HardTests on both the GPT-4o model and the more advanced o3 model. We focus on comparing with HardTests a few reasons: (i) CodeContestst does not disclose the specific models used for generation, nor is their pipeline open-source, making a fair, controlled comparison impossible. (ii) The other two baselines we compare with, TACO and CodeContests assume the large number of correct submissions to stress test the legitimacy of test cases. This is an unrealistic assumption when creating test cases for new problems. Our pipeline operates without this dependency, which we believe is stronger and more practical setting. (iii) HardTTests has the strongest performance (highest consistency) out of the four baselines we compared against the paper.
existence of and rely
Pipeline
Model
Checker Model
Consistency (Acc) (t)
FPR (L)
FNR (1)
HardTests
4o
AutoCode (4o)
78.4%
12.2%
31.4%
AutoCode (Ours)
4o
AutoCode (4o)
79.5%
7.5%
34.2%
HardTests
o3
AutoCode (o3)
80.0%
21.5%
18.5%
AutoCode (Ours)
o3
AutoCode (o3)
91.1 %
3.7%
14.1 %
Table 4: Fair comparison with Hardtests.
On the 4o model: AutoCode demonstrates higher consistency (79.5% vs 78.4%) and a lower False Positive Rate compared to the baseline. On the o3 model: The advantage of AutoCode is significant, with accuracy increasing to 91.1% % (compared to 80.0% for the baseline) and the False Positive Rate (FPR) dropping to an extremely low 3.7%. Special Note on Experimental Setup: In the comparison experiments, the native Checkers generated by HardTests meaning they could not be run directly on standard Online Judge systems. To ensure fairness and reprodueibility, used the standard-come standard-complian Checkers generated by our AutoCode pipeline for all experiments listed above.
not in the standard testlib.h format,
HUMAN EVALUATION PROCESS
To assess the quality of the L.M-generated problems beyond automated metrics, we conducted a structured human evaluation with experienced competitive programmers.
34

---

Published as a conference paper at ICLR 2026
Expert background We invite 5 human experts evaluate the LLM-generated problems. The group consisted 2 Codeforces International Grandmasters (reviewing high-difficulty problems) and Codeforces Masters (reviewing redium-to-low difficulty problems).
Evaluation time Our experts comprehensively evaluate each problem's solvability, difficulty, novelty, quality, and the correctness of the LLM-provided solution. The average time spent reviewing each problem was approximately 30 minutes.
Sample size We generate and review total of 190 new problems, which is the data source for Figure 3. Within these, 133 problems were generated using our final prompt version, and these constitute the dataset for Figure 4. We will add these key numbers to the appendix of the paper to improve transparency and credibility.
35


Reviews:
[{"review_id": "W4TV0hFTBw", "reviewer": "Reviewer_2TeP", "summary": "Auto-code is about being able to generate new competitive programming problems. This requires being able to generate new problem descriptions from scratch or by modifying existing coding problem descriptions. And it requires providing reference solutions to the new problems to show they are solvable and well formed.  And to do that requires being able to generate high quality test cases and be able to verify if a code solution can pass all the test cases, and verify the code solution is solving the new programming problem description.\n\nThis paper describes a pipeline of many agents that work together to accomplish that task, and benchmarks some of the components in the pipeline against other approaches previously tried.", "questions": "Are you planning to release the source code for the system described in the paper?\n\nAre you able to leverage the generated programming problems and solutions to finetune an LLM and show it leads to the LLM being a better problem solver?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper describes a system of agents that produce new competitive programming problems and solutions that appear to be of high quality when judged by human experts.  The paper does nicely describe the rigorous validation and verification required to vet out proposed problems and their proposed solutions as correct or incorrect, by ensuring a reliable and broad set of test cases is produced. The paper does a number of comparisons that shows the quality of the system, like how well it is able to generate test cases compared to other test case generation systems.", "weaknesses": "The description of the system in the paper is well written and extensive, but I do worry that without a repo of the source code accessible to researchers the ability to leverage this work and build upon it will be very difficult.\n\nThe ability to produce new competitive programming problems that humans judge as good is useful on it's own, but if those new problems and correct solutions are able to be leveraged to further finetune the LLM and improve it's ability to solve competitive programming problems, that would be strong indicator that the problems being generated are useful in an interesting way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dt2rC5oJ2f", "reviewer": "Reviewer_hPcp", "summary": "This paper presents AutoCode, an LLM agent system for writing and developing competitive programming problems. AutoCode generally consists of two parts: a test case generator, and a problem writer.  The test generation component implements multiple components including validator, generator, checker, and interactor. Experimental results indicate that the test cases generated by AutoCode outperform current state-of-the-art baselines in terms of both FPR and FNR metrics. The problem writing component has achieved excellent results in human evaluation, and the findings show that LLMs are capable of generating high-quality and high-difficulty problems.", "questions": "1. How do you develop the sandbox system? Submissions in CodeContests may come from many different online judge systems, using a variety of operating systems (Linux, Windows), compilers (GCC and MSVC spanning multiple eras), and interpreters (Python 2, Python 3, and JVMs of various versions). Therefore, if you want to evaluate the consistency with official verdicts, the first thing to do is achieve environment alignment—and I think this is extremely difficult. Many submissions fail to produce the same verdicts even when using the exact same original test cases. How do you solve this problem?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. In terms of test case generation, AutoCode has indeed gone deeper and more refined than previous work (e.g., HardTests, CodeContests+). For instance, it enhances the validator by adding validator tests to further ensure the validator’s correctness, and it is capable of writing interactors for interactive problems.\n2. This paper conducts pioneering research on problem writing using LLMs and validates the problem-writing capabilities of LLMs through large-scale human evaluations.\n3. The study on the value alignment between LLMs and human experts is highly insightful. Given that LLM-as-a-judge is currently a highly mainstream method, the results (Fig. 4) demonstrate the limitations of LLMs in such complex OOD tasks.\n4. This paper is well-written. I am enjoying reading this paper.", "weaknesses": "1. **Does this paper propose a better test case construction method?**, The generator-validator-checker-interactor workflow is a widely adopted problem-construction process in competitive programming, and has also been used more or less in previous studies (e.g., HardTests, CodeContests+). The similarity between AutoCode’s overall workflow and that of previous works may raise questions about the source of improvements in its evaluation. First, this study is conducted based on SOTA models such as o4 and GPT-5, while previous works are before the release of such advanced models. Second, prompts are a critical factor affecting performance—providing a open agent framework that includes prompts would be a significant contribution to the community. However, this paper does not disclose the prompts it used, thus diminishing its reproducibility and overall contribution.\n2. **Questions on the soundness of the LLM problem writing study.** I find the topic of LLM problem writing very interesting, but I do not believe this paper has conducted an in-depth and rigorous study. First, it is an experiment entirely based on human evaluations, yet I think some of the criteria could have been tested through objective experiment. For instance, the paper mentions a criterion called \"Model-Training Usable\"—this criterion could be fully verified through actual training results, rather than relying on human evaluations. Additionally, the human evaluation lacks details, such as the average time each expert spent evaluating each piece of data and the profiles of the experts. Some conclusions also lack sufficient evidential support; for example, for Findings 2, it is recommended to incorporate more case studies for further discussion. Furthermore, the sample size of this experiment should be placed in a more prominent position, such as the caption of Figure 3. I read through this section several times but still could not find this number. Finally, regarding the most critical part of this experiment: how did the authors guide LLMs to generate problems? The entire problem-writing process lacks a detailed description,  nor does the prompt provided, so I believe it lacks reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dCvnpeIisN", "reviewer": "Reviewer_gTC1", "summary": "The paper focuses on test generation and problem creation for competitive programming problems. First, they show that their test generation framework can achieve very low false positive and false negative rates compared to prior SOTA work. The test generation framework consists of using LLMs to construct validator, generator, and checker with novel prompting techniques specifically designed for more robust checking. Building on the robust test generation, they show that they can prompt LLMs to create very high quality problems (judged by top competitive programmers) by seeding from existing problems and then going through testing to filter problems such that the LLM-generated brute-force solution (acting as ground truth) and the LLM-generated solution agree.", "questions": "- Although stated in Finding 1: \"LLMs can generate solvable problems that they themselves are unable to solve,\" this kind of problem would be filtered by the solvability check right? Could you clarify if the system using the filtering process described in the beginning of Section 5 would be able to produce such problems at the end?\n- What is the \"prompt optimization\" refer to? Line 291 shows the ablation \"without prompt optimization\" and in line 321 \"Prompt optimization turns out to be especially important;\" there seems to be no explanation of the prompt optimization in the paper.\n- In 3.2 for the generator, it is stated that they use strategies: Small Data Exhaustion, Randomized and Extreme Data, and TLE-Inducing Data, but didn't explain how these strategies are implemented. Are they being implemented with LLM by using LLM to generate a program to perform this kind of strategy?\n- Can you explain GENERATECHECKERSCENARIOS in Section 3.3? How does it generate the (input, contestant_out, ref_out, verdict) data?\n- Can you clarify which parts of the test generation required access to ground-truth programs? is it just the for thchecker?\n- In line 285 \"all results in this table are generated by GPT-5-High\", do you mean the model used in the test generation system or the 33 submissions?\n- Since competition programs accept or not sometimes depending on the timeout set for the program, how is timeout set for the generated problems?\n- For the problem creation prompt that prompts the model to create new problems from seed problems, do you need to specifically guide the model what kind of variations they can make and does it bias the model to alter the seed problem in specific ways?\n- Are the LLM prompting used in this work zero-shot or few-shot like you need to provide more examples to guide the model? Can you provide the prompts used in this work to help with reproducibility?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significantly improve the test generation false positive and false negative rate for the competition programming\n- Demonstrate that model can generate novel problems not only useful for training but also can generate very high quality deemed as good as top competition problems (ICPC/IOI-Level)\n- Analysis and ablation showing many insights of the results, the finding from the human experts annotation is especially valuable", "weaknesses": "The critical weakness of this paper is the lack of reproducibility. The paper uses specific LLM prompting techniques to achieve impressive results but fails to provide any of the prompts used in the work. Many details are also missing, for example, how the generator works and the problem creation procedure are not clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)