AutoCode: LLMs as Problem Setters for Competitive Programming

## Abstract

Abstract Abstract: Writing competitive programming problems is exacting. Authors must: set constraints, input distributions, and edge cases that rule out shortcuts; target specific algorithms (e.g., max-flow, dynamic programming, data structures); and calibrate complexity beyond the reach of most competitors. We argue that this makes for an ideal test of general large language model capabilities and study whether they can do this reliably. We introduce AutoCode, which uses multiple rounds of validation to yield competition-grade problem statements and test cases. On held-out problems, AutoCode test suites approach 99% consistency with official judgments, a significant improvement over current state-of-the-art methods like HardTests, which achieve less than 81%. Furthermore, starting with a random seed problem, AutoCode can create novel variants with reference and brute-force solutions. By cross-verifying these generated solutions against test cases, we can further filter out malformed problems.
Our system ensures high correctness, as verified by human experts.
AutoCode successfully produces novel problems judged by Grandmaster-level (top 0.3%) competitive programmers to be of contest quality. Project page: https://livecodebenchpro.com/projects/autocode/overview

## 1 Introduction

> An AI system can create and maintain knowledge only to the extent that it can verify that knowledge itself. — Rich Sutton, Verification, The Key to AI, 2001

As Einstein and Infeld put it, “The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old problems from a new angle requires creative imagination and marks real advances in science.” [1].
As large language models (LLMs) march toward general-purpose capabilities, with the ultimate goal of artificial general intelligence (AGI), we argue that testing problem generation abilities is just as important as problem solving abilities. This is particularly true when applying LLMs to advanced programming tasks, where future advancement and economic integration of LLM coding capabilities will require significant validation.

First, problem setting for competitive coding requires a deeper understanding of algorithms that problem solving may not.
For example, basic problems can collapse into recognizable templates that can be solved with simple tricks; and many standard programming questions often allow for partial credit or boilerplate solutions that can mask incorrect reasoning.
Competitive programming problems have a strict bar designed to assess a deeper understanding of the underlying algorithm design principles, data structures, and complexity trade-offs.
Verifying the vast space of possible solutions, along with sufficient coverage of short-cuts or corner cases is challenging, but a necessity for competition programming problems. As such, problem setting encompasses all the challenges of solving a problem, and then more.

Second, better problem setting will lead to more rigorous competitive programming benchmarks. Since official test data from premier platforms such as Codeforces and AtCoder are not publicly available, researchers currently rely on synthesized datasets such as CodeContests+ [2], TACO [3], and HardTests [4]. Our analysis (§[4](https://arxiv.org/html/2510.12803v1#S4)), however, shows that existing test datasets can have both high false positive rates (FPR) and false negative rates (FNR). For instance, a greedy algorithm with poor time complexity might pass a suite of small, random tests, only to fail against adversarially constructed cases designed to expose its flaws.
This critical weakness creates a distorted evaluation landscape, rewarding models that discover shortcuts.

Figure: Figure 1: AutoCode introduces a closed-loop multi-role Validator-Generator-Checker framework that enables robust test case generation and scalable, self-verified problem generation for competitive programming. It achieves 98.7% consistency with official judgments. This framework precisely mirrors the process human experts follow when creating programming contest problems.
Refer to caption: x1.png

Third, successful problem setting of novel challenges may pave the path for self-improving models and AGI, as well as validating deployments in complex software stacks.

LLMs are already used by over a billion people and increasingly for coding applications, saving hundreds of millions of dollars with LLM-assisted programming [5, 6, 7], albeit largely on relatively simple tasks. To expand LLM capabilities toward the strategic reasoning and long-term planning that real programming demands, and to integrate LLMs safely within the software development stack, we must be able to determine, with high fidelity, whether model-generated code is a valid solution to the problem at hand. Misspecified rewards, in the form of malformed problem formulation or verification, can cause models to optimize for the wrong thing or fall into bad local optima. We find, for example, that current benchmark datasets suffer from an even higher FNR than FPR, often caused by invalid inputs to unit tests; as a result, a perfectly valid, creative solution can be unjustly penalized when it crashes or produces an incorrect output on malformed input data. This pollutes data for reinforcement learning by punishing valid lines of reasoning, while high FPR fails to penalize flawed ones.

To address these critical gaps, we introduce AutoCode, a systematic framework that employs LLMs in a closed-loop, multi-role system to automate the entire lifecycle of competitive programming problem creation and evaluation. Our first contribution is an enhanced Validator-Generator-Checker framework that achieves state-of-the-art reliability in test case generation, where the Generator generates test cases, the Validator validates whether the generated test cases satisfy the problem constraints, and the Checker checks whether a solution is correct given the test cases. We move beyond standard implementations with several key heuristics: the LLM generates multiple candidate Validators and Checkers, and we select the most robust one through targeted testing. The generator executes a multi-pronged strategy, creating a diverse and adversarial test suite that includes small-data exhaustion, randomized stress tests, and TLE-inducing cases designed to expose incorrect time complexities. Building on this highly reliable verification pipeline, we introduce our second major contribution: a novel process for generating new, high-quality problems. This process begins with a seed problem to inspire the LLM in a promising direction. To ensure correctness without human intervention, we employ a dual verification protocol: the LLM generates a problem statement, an efficient reference solution, and a simple brute-force solution. The new problem is accepted only after rigorously verifying that the efficient solution’s output matches the ground truth established by the brute-force solution across all test cases. In summary, our work makes the following contributions:

- •
State-of-the-Art Test Cases Generation for Existing Problems. Our testcase generation framework achieves over 91% consistency with official judgments on a large-scale benchmark of 7538 problems, significantly outperforming existing methods whose consistency ranges from 72 – 81%. We also point out that the ability to generate test cases is not only useful for competitive programming, but also has broad practical significance for general tasks involving input-output matching.
- •
Novel and High-Quality Problems Generation. We pioneer a systematic process that uses a dual-verification mechanism to generate, validate, and score new problems. Vetted by elite competitive programmers, this process produces novel problems deemed high-quality and original enough for official contests, with problems passing automated verification achieving a correctness rate of 94%.

## 2 Related Works

LLMs show rapid progress in code generation recently [8, 9, 10, 11, 12]. Benchmarks and tooling for code reasoning span three threads. Datasets like LiveCodeBench [13] evaluate coding interview problems, LiveCodeBench Pro [14] focuses on competitive programming, FormulaOne [15] assesses challenging domain-specific problems, and ELABORATION [16] explores human-LLM collaboration. Second, to strengthen verdict reliability, EvalPlus [17] augments unit tests, and TACO [3], CodeContests+ [2], HardTests [4], TestCase-Eval [18], LogiCase [19] and the meta-benchmark TCGBench [20] propose rule or LLM-driven generators emphasizing constraint validity, edge-case coverage, and adversariality. Third, solver/data-centric lines AlphaCode [21], AceReason [22], Absolute Zero [23], and rStar-Coder [24] scale solution search or curate verified corpora via RL/self-play signals [25, 26, 27, 28, 29] rather than offering end-to-end test pipelines [30]. In contrast, AutoCode unifies and extends these directions by coupling a Validator-Generator-Checker (and interactor) loop that enforces legality plus adversarial coverage with a dual-verification protocol (reference vs. brute force) to generate and certify new problems, addressing both static-benchmark contamination and under-constrained tests within a single framework.

## 3 Test Case Generation

Our test case generation process is a structured framework designed for maximum rigor and coverage. The framework as shown in Figure [1](https://arxiv.org/html/2510.12803v1#S1.F1) starts with the Validator, which serves as the cornerstone of the entire system. Its function is to ensure any given input strictly adheres to all constraints specified in the problem description. A Validator is critical for minimizing the FNR, as it prevents correct programs from failing on malformed data. Next, the Generator employs diverse strategies to create a broad spectrum of inputs, aiming to reduce the FPR, where incorrect or inefficient programs are erroneously judged as correct. Any invalid cases produced by the Generator are subsequently filtered by the Validator, ensuring we obtain a high-quality set of inputs. Finally, to assess the contestant’s output, a Checker compares it against the reference solution’s output, while for interactive tasks, an Interactor engages in a multi-turn dialogue with the contestant’s program to issue a final verdict. Terminology is defined in Appendix [7](https://arxiv.org/html/2510.12803v1#S7). Because one of our prominent goals is to serve as a high-quality verifier for RLVR, we pay particular attention to reducing the FPR. We distinguish test cases (input-answer pairs) from the test data, which includes the Checker and Interactor programs required for evaluation.

### 3.1 Validator

Figure: Algorithm 1 BuildValidator

The foundation of our framework is a highly robust Validator, responsible for rejecting any input that violates the problem’s explicit constraints. To construct it, we first guide the LLM to generate a targeted suite of 40 test cases. This suite is strategically composed of 10 valid inputs and 30 near-valid illegal inputs, i.e., cases that become valid after minor modifications. We present an example in Appendix [10](https://arxiv.org/html/2510.12803v1#S10). These carefully crafted and validated edge cases are designed to rigorously probe the Validator’s robustness against subtle constraint violations. With this evaluation set established, we then prompt the LLM to produce three distinct candidate Validator programs. Each candidate is subsequently benchmarked against the 40 cases. The program that correctly classifies the highest number of these valid and invalid test cases is selected as the definitive Validator for all subsequent stages of the framework, ensuring a strong guard against malformed data.

### 3.2 Generator

Figure: Algorithm 2 BuildGeneratorSuite

After ensuring input validity through a precise Validator, the Generator’s core task becomes maximizing test coverage. By creating challenging test cases, it can more effectively identify incorrect or inefficient solutions, thereby reducing FPR. To enhance the coverage of our test cases, we adopt strategies across three distinct dimensions.

- •
Small Data Exhaustion. For problems with small constraints or those featuring multiple test points, as are common on platforms like Codeforces and AtCoder, we generate inputs that exhaustively explore all permutations and combinations of small-scale data. This strategy ensures complete coverage of fundamental boundary conditions and corner cases.
- •
Randomized and Extreme Data. We generate large-scale random inputs to stress-test solutions. This includes pushing integer types to their limits to trigger overflows, testing floating-point precision, and probing for out-of-bounds array access. Drawing from contest experience, we also adversarially design hack cases, such as those intended to make common greedy algorithms fail or to induce hash collisions.
- •
TLE-Inducing Data. To specifically address the common issue of solutions with incorrect time complexity passing on weak test cases, we construct inputs with specific structures designed to maximize the computational load for certain algorithms. This ensures that only solutions meeting the intended time complexity requirements can pass, effectively catching false positive verdicts caused by insufficient timing pressure.

### 3.3 Checker

Figure: Algorithm 3 BuildChecker

The Checker is responsible for determining the final verdict of a submission by comparing the output with the answer output provided by the reference solution. To ensure that the Checker achieves a similar accuracy as the Validator, we adopt a construction method analogous to that used for the Validator. First, we prompt the LLM to generate 40 distinct test cases. Each test case consists of a valid input, an output that appears reasonable but might contain mistakes, the correct output provided by the reference solution, and the expected verdict (e.g., Accepted or Wrong Answer). The validity of all inputs is guaranteed by the previously established Validator. Next, the LLM generates three candidate Checker programs. We evaluate each candidate Checker against these 40 test cases to assess their ability to produce accurate verdicts. Finally, the Checker program that performs best is selected to be part of the final test data package.

### 3.4 Interactor

Figure: Algorithm 4 BuildInteractor

Our framework introduces a novel, fully automated approach for generating test data for interactive problems, a previously unaddressed challenge (e.g., in CodeContests+ [2]). The core innovation lies in a mutant-based discrimination process. We begin by prompting the LLM to perform several small but critical logical modifications to the provided reference solution, thereby creating a set of slightly incorrect mutant programs. Examples are shown in Appendix [11](https://arxiv.org/html/2510.12803v1#S11). These mutants serve as challenging foils for the Interactor. The LLM then generates three candidate Interactor programs. The crucial selection criterion is identifying the Interactor that most effectively distinguishes the correct, unmodified reference solution from the flawed mutant versions during a simulated interaction shown in Appendix [13](https://arxiv.org/html/2510.12803v1#S13). The Interactor that successfully passes the reference solution while failing the maximum number of mutants is chosen, proving its ability to robustly probe for the specific logic required by the problem.

## 4 Benchmarking Test Case Robustness

**Table 1: Performance comparison on the 7538-problem benchmark. The evaluation is performed on a set of 195,988 human submissions randomly taken from the CodeContests dataset, where each problem has 26 submissions, 50% of which are correct and 50% are incorrect. Results are reported with 95% confidence intervals. The results for AutoCode are obtained using o3.**
| Method | Consistency (%) ($\uparrow$) | FPR (%) ($\downarrow$) | FNR (%) ($\downarrow$) |
| --- | --- | --- | --- |
| CodeContests [21] | 72.9 $\pm$ 0.2 | 7.7 $\pm$ 0.2 | 46.3 $\pm$ 0.3 |
| CodeContests+ [2] | 79.9 $\pm$ 0.2 | 8.6 $\pm$ 0.2 | 31.6 $\pm$ 0.3 |
| TACO [3] | 80.7 $\pm$ 0.2 | 11.5 $\pm$ 0.2 | 26.9 $\pm$ 0.3 |
| HardTests [4] | 81.0 $\pm$ 0.2 | 12.1 $\pm$ 0.2 | 25.8 $\pm$ 0.3 |
| AutoCode (Ours) | 91.1 $\pm$ 0.1 | 3.7 $\pm$ 0.1 | 14.1 $\pm$ 0.2 |

**Table 2: Ablation study results on the 720-problem benchmark. For each problem, the evaluation uses 33 submissions generated by different LLMs, 25% of which are accepted. This setup is particularly relevant for reinforcement learning applications. Each row shows performance after removing a single component from the full framework. All results in this table are generated by GPT-5-High.**
| Configuration | Consistency (%) ($\uparrow$) | FPR (%) ($\downarrow$) | FNR (%) ($\downarrow$) |
| --- | --- | --- | --- |
| w/o Generator Strategy 1 (Exhaustive) | 98.4 $\pm$ 0.2 | 1.7 $\pm$ 0.2 | 1.3 $\pm$ 0.3 |
| w/o Generator Strategy 2 (Random/Extreme) | 98.4 $\pm$ 0.2 | 1.6 $\pm$ 0.2 | 1.3 $\pm$ 0.3 |
| w/o Generator Strategy 3 (TLE) | 98.6 $\pm$ 0.2 | 1.4 $\pm$ 0.2 | 1.3 $\pm$ 0.3 |
| w/o Prompt Optimization | 98.0 $\pm$ 0.2 | 1.8 $\pm$ 0.2 | 2.9 $\pm$ 0.4 |
| AutoCode Full Framework | 98.7 $\pm$ 0.1 | 1.3 $\pm$ 0.2 | 1.2 $\pm$ 0.3 |

To rigorously evaluate our test case generation framework, we establish two distinct benchmarks. The primary benchmark consists of 7538 problems derived from the intersection of well-known existing datasets: CodeContests+ [2], CodeContests, HardTests [4], and TACO [3]. Notably, this large-scale set does not contain interactive problems, and due to the filtering inherent in these datasets, its average difficulty for test data generation is slightly lower than a typical Codeforces round.

To address this and test our system under more challenging, real-world conditions, we create a second benchmark of 720 recent, rated problems from Codeforces. This set is completely unfiltered, including notoriously difficult-to-handle interactive problems and those requiring complex, structured test data. We are unable to evaluate prior methods on this newer benchmark as their data generation codebases are not publicly available.

Our evaluation is based on three key metrics. Consistency measures the overall percentage of agreement between the verdicts from our tests and the official judgments. We further dissect disagreements into two critical error rates. The FPR is defined as the proportion of officially incorrect solutions that are erroneously accepted by our generated tests. Conversely, the FNR is the proportion of officially correct solutions that are erroneously rejected by our tests.

Figure: Figure 2: Distribution of Non-Consistent Verdicts. Each problem in the 720-problem benchmark is evaluated using 33 submissions generated by different LLMs. The verdicts for 85.6% of the problems are consistent with the official judgments, and 97.1% have three or fewer inconsistencies.
Refer to caption: x2.png

#### Comparison with other baselines.

We evaluate AutoCode on the benchmark of 7538 problems against four leading baselines. As detailed in Table [1](https://arxiv.org/html/2510.12803v1#S4.T1), our framework achieves 91.1% consistency with official judgments. This marks a significant leap over existing methods which don’t surpass 81.0%. Critically, AutoCode substantially reduces the FPR to just 3.7% and the FNR to 14.1%, representing $\approx 50\%$ decrease in both metrics over the current state-of-the-art. Figure [2](https://arxiv.org/html/2510.12803v1#S4.F2) shows the error verdicts distribution, showing that most of the problems are consistent with ground truth verdicts.

To further test our system’s robustness, we curate a more challenging benchmark of 720 recent, unfiltered Codeforces problems, including complex interactive tasks. As shown in Table [2](https://arxiv.org/html/2510.12803v1#S4.T2), AutoCode maintains its exceptional performance, achieving 98.7% consistency. This result validates our method’s effectiveness on modern, difficult problems where prior methods could not be evaluated.

#### Ablation studies.

We further conduct an ablation study to determine how each part of AutoCode affects overall performance. The complete AutoCode framework sets a strong baseline, achieving 98.7% consistency with official judgments, along with a 1.3% FPR and a 1.2% FNR. Prompt optimization turns out to be especially important; removing it drops consistency to 98.0% and more than doubles the FNR to 2.9%. The three different generator strategies also significantly complement each other. Removing either the exhaustive or random/extreme strategy raises the FPR to 1.7% and 1.6%, respectively, highlighting their role in catching flawed solutions.

Validator and Checker selection tests are also crucial. These tests pick the most reliable output from several model-generated candidates, helping weaker models avoid logical errors. Removing these tests increases the FNR to 1.3% and 1.4%. A good Validator prevents correct solutions from being rejected due to bad inputs, while a strong Checker accurately evaluates outputs with complex judgment rules. Even small improvements (0.1%) matter, as they help address problems that are very difficult to generate test cases for.

## 5 Problem Generation

Our novel problem generation framework builds upon the robust test generation framework described in Section [3](https://arxiv.org/html/2510.12803v1#S3) and shown in Figure [1](https://arxiv.org/html/2510.12803v1#S1.F1), but introduces a critical dual-verification protocol to ensure correctness without human intervention.

Each generated problem is graded on a 6-level scale, judged by top human competitive programmers. We interviewed 8 human expert problem setters, all of whom report that they often build on a specific existing problem when authoring new problems. By adding, removing, or modifying certain conditions of such a “seed problem,” they create new and often more difficult problems that require novel insights. Inspired by their insights, our approach begins by selecting a random Codeforces problem (with difficulty rating less than 2200) as a “seed problem.” The LLM is tasked with generating a new problem by adding, deleting, or modifying certain conditions from this seed problem, along with an efficient reference solution (std.cpp) and a brute-force solution (brute.cpp). brute.cpp usually has higher time complexity but is very unlikely to be incorrect, so we leverage it to stress-test the validity of the problem. Using our enhanced test case generation technique, we construct a comprehensive set of test data that fully covers small cases. Both brute.cpp and std.cpp are then executed on this dataset. A problem is only deemed correct if, for every test case, both programs’ outputs (where the brute-force solution may legitimately fail to finish due to timeout) are pairwise validated by the checker as consistent answers and outputs. This dual-verification protocol, where brute.cpp serves as the initial ground truth, and the validated reference solution undergoes an additional full test generation cycle, successfully filters out 27% of error-prone problems, raising the correctness rate of LLM-provided reference solutions from 86% to 94%.

After filtering, over 80% of the problems are annotated as having sufficient quality to serve as training data for models, and 23% of the problems have novel or creative designs involved.
We present the detailed rubrics and the score distribution in Figure [3](https://arxiv.org/html/2510.12803v1#S5.F3). In the following, we summarize several key findings regarding the performance of LLMs in problem generation. We also present the human expert grading criteria in Appendix [12](https://arxiv.org/html/2510.12803v1#S12) and one example under ICPC/IOI-level in Appendix [8](https://arxiv.org/html/2510.12803v1#S8).

Figure: Figure 3: AutoCode is capable of automatically generating novel competitive programming problems. After careful examination and filtering by multiple human experts, 61.6% achieve a quality level suitable for large language model training, and 3.2% are considered good enough to serve as problems for ICPC/IOI-level contests. Level 1 and 2 correspond to automatic correctness checks performed by the system, while level 3 through 6 involve human evaluation to differentiate among finer levels. It is noted that only the Level 1 questions are wrong, while the Level 2 – 6 questions are all correct and usable, with the only difference being their quality. Detailed grading rubrics are in Appendix [12](https://arxiv.org/html/2510.12803v1#S12).
Refer to caption: x3.png

Finding 1. LLMs can generate solvable problems that they themselves are unable to solve.

LLMs can generate solvable problems that they themselves are unable to solve. In our experiments, about 4.2% of the problems fall into this category. In other words, the model is indeed capable of creating logically sound problems that can be solved by other models or by humans, but due to limitations in its own reasoning ability, it fails to solve them correctly.
Ideally, these problems could serve as sources for model self-improvement, and we think this is a very interesting phenomenon that is worth further investigation. We provide an example in the Appendix [9](https://arxiv.org/html/2510.12803v1#S9).

Finding 2. LLMs tend to create new problems by combining existing problem frameworks and emphasizing knowledge and implementations.

LLMs tend to concatenate, combine, or embed existing algorithmic knowledge or tricks into established problem frameworks, rather than proposing entirely new problem models that require original solution strategies. This form of recombinational innovation primarily reflects reuse of existing knowledge rather than expansion of creative thinking. Such a tendency highlights a fundamental divergence between humans and LLMs in defining novelty: while humans emphasize originality in modes of reasoning and problem-solving ideas, LLMs rely more heavily on recombination and reconfiguration of pre-existing knowledge. Also, the generated problems often place greater emphasis on assessing specific algorithmic knowledge or demanding complex implementation details, while less often rely on clever design and subtle observation.

Finding 3. Novel problems tend to have larger difficulty gain over seed problems, and the generated problems have the highest quality when the corresponding seed problem is moderately difficult.

We observe that the difficulty shifts induced by LLMs during problem adaptation follow a systematic pattern rather than occurring randomly. On average, adapted problems become approximately 334 Elo score harder; those judged as “novel” show an average increase of 498 score, whereas non-novel problems increase by only about 108 score. High-quality problems are predominantly generated when the original seed problem lies in the moderately high difficulty range. For overly difficult seed problems, the LLM has limited room to introduce effective modifications, resulting in minimal difficulty gains and insufficient novelty. Conversely, for overly easy seed problems, even after an average increase of 334 score, the absolute difficulty remains too low to meet high-quality standards. Notably, around 5% of generated problems fall into a critical zone, with pass@1 scores between 0.1 and 0.5, meaning the model sometimes succeeds and sometimes fails. These boundary cases present a valuable opportunity for constructing high-quality self-play datasets, enabling models to enhance their capabilities through repeated attempts at solving such borderline problems.

Finding 4. Human experts and LLMs show almost no correlation in their judgment of problem quality and novelty.

We identify a significant divergence between humans and LLMs in their judgment of problem quality and novelty. To quantify this discrepancy, we employ an Elo Rating scheme to assess LLM judgments, following the methodology described in  [31]. The correlation coefficients between the o3 and human experts are only 0.07 for quality and 0.11 for novelty, indicating a substantial misalignment between the model’s internal evaluation standards and human expert criteria. Interestingly, both humans and LLMs demonstrate high within-group consistency in terms of quality: correlations among human experts reach 0.71, while GPT-4o and o3 show a correlation of 0.72. These findings suggest that relying solely on LLMs to self-evaluate the quality of their generated problems is inadequate, and more sophisticated evaluation mechanisms are required to align with human preferences as also shown in Figure [4](https://arxiv.org/html/2510.12803v1#S5.F4).

Finding 5. Difficulty of the generated problem and difficulty gain over the seed problem serve as a better indicator of problem quality than LLM self-evaluations.

While LLMs are unreliable in directly assessing problem quality, leveraging their predictions of problem difficulty provides a more effective indirect measure. First, difficulty is the strongest predictor of quality: correlations between human-assigned quality scores and both absolute problem difficulty and difficulty gain reach as high as 0.60. Second, difficulty gain outperforms novelty as an indicator. As a continuous variable, it captures richer information, since problems that exhibit sufficiently large difficulty increases tend to incorporate new ideas or more complex combinations, thereby manifesting novelty. Finally, LLMs possess a certain ability to estimate problem difficulty; by exploiting this capacity, difficulty can be used as a proxy signal to indirectly predict problem quality, achieving correlations of around 0.18 as shown in Figure [4](https://arxiv.org/html/2510.12803v1#S5.F4).

Figure: Figure 4: Correlations between human experts and LLM judgments on generated problems. The charts illustrate a significant gap between human and LLM perceptions of quality and novelty (Finding 4), despite high inter-LLM agreement (bottom right). LLM-predicted difficulty shows a weak positive correlation with human-rated quality (top left), suggesting it can serve as a noisy proxy for quality estimation (Finding 5).
Refer to caption: x4.png

## 6 Conclusion

In this work, we introduce AutoCode, a closed-loop multi-role framework that leverages LLMs as problem setters for competitive programming. By coupling a Validator-Generator-Checker (and Interactor) framework with a dual-verification protocol, AutoCode achieves state-of-the-art reliability in test case generation and extends beyond prior methods to generate entirely new, contest-quality problems. Extensive experiments on over 7,500 problems and recent Codeforces benchmarks demonstrate that AutoCode substantially reduces both false positives and false negatives, achieving more than 98% agreement with official judgments and successfully producing novel problems validated by expert programmers. Beyond test generation, our analysis sheds light on both the strengths and weaknesses of LLMs in creative problem authoring. While models excel at recombination of algorithmic knowledge, they struggle to introduce truly novel reasoning paradigms or flawless sample design. Nevertheless, we show that difficulty and difficulty gain serve as reliable proxy signals for problem quality, providing a scalable path toward self-play.

## References

- [1]
Albert Einstein and Leopold Infeld.
The Evolution of Physics: The Growth of Ideas from Early Concepts to Relativity and Quanta.
Cambridge University Press, Cambridge, 1938.
Quoted passage on p. 92.
- [2]
Zihan Wang, Siyao Liu, Yang Sun, Hongyan Li, and Kai Shen.
Codecontests+: High-quality test case generation for competitive programming.
arXiv preprint arXiv:2506.05817, 2025.
- [3]
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li.
Taco: Topics in algorithmic code generation dataset.
arXiv preprint arXiv:2312.14852, 2023.
- [4]
Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li.
Hardtests: Synthesizing high-quality test cases for llm coding.
arXiv preprint arXiv:2505.24098, 2025.
- [5]
OpenAI.
How people are using chatgpt, September 2025.
- [6]
Maxwell Zeff.
Github copilot crosses 20m all-time users, July 2025.
- [7]
AWS DevOps Blog.
Amazon q developer just reached a $260 million dollar milestone, August 2024.
- [8]
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al.
Qwen2. 5-coder technical report.
arXiv preprint arXiv:2409.12186, 2024.
- [9]
Deepseek.
Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024.
- [10]
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al.
Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.
arXiv preprint arXiv:2406.11931, 2024.
- [11]
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang.
Diffucoder: Understanding and improving masked diffusion models for code generation.
arXiv preprint arXiv:2506.20639, 2025.
- [12]
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al.
Dream-coder 7b: An open diffusion language model for code.
arXiv preprint arXiv:2509.01142, 2025.
- [13]
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
arXiv preprint arXiv:2403.07974, 2024.
- [14]
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, et al.
Livecodebench pro: How do olympiad medalists judge llms in competitive programming?
arXiv preprint arXiv:2506.11928, 2025.
- [15]
Gal Beniamini, Yuval Dor, Alon Vinnikov, Shir Granot Peled, Or Weinstein, Or Sharir, Noam Wies, Tomer Nussbaum, Ido Ben Shaul, Tomer Zekharya, et al.
Formulaone: Measuring the depth of algorithmic reasoning beyond competitive programming.
arXiv preprint arXiv:2507.13337, 2025.
- [16]
Xinwei Yang, Zhaofeng Liu, Chen Huang, Jiashuai Zhang, Tong Zhang, Yifan Zhang, and Wenqiang Lei.
Elaboration: A comprehensive benchmark on human-llm competitive programming.
arXiv preprint arXiv:2505.16667, 2025.
- [17]
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.
Advances in Neural Information Processing Systems, 36:21558–21572, 2023.
- [18]
Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, et al.
Can llms generate reliable test case generators? a study on competition-level programming problems.
arXiv preprint arXiv:2506.06821, 2025.
- [19]
Sicheol Sung, Yo-Sub Han, Sang-Ki Ko, et al.
Logicase: Effective test case generation from logical description in competitive programming.
arXiv preprint arXiv:2505.15039, 2025.
- [20]
Zihan Ma, Taolin Zhang, Maosong Cao, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen.
Rethinking verification for llm code generation: From generation to testing.
arXiv preprint arXiv:2507.06920, 2025.
- [21]
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.
Competition-level code generation with alphacode.
Science, 378(6624):1092–1097, 2022.
- [22]
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
Acereason-nemotron: Advancing math and code reasoning through reinforcement learning.
arXiv preprint arXiv:2505.16400, 2025.
- [23]
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang.
Absolute zero: Reinforced self-play reasoning with zero data.
arXiv preprint arXiv:2505.03335, 2025.
- [24]
Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang.
rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset.
arXiv preprint arXiv:2505.21297, 2025.
- [25]
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman.
Star: Bootstrapping reasoning with reasoning, 2022.
- [26]
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman.
Quiet-star: Language models can teach themselves to think before speaking, 2024.
- [27]
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning, 2023.
- [28]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
Self-refine: Iterative refinement with self-feedback, 2023.
- [29]
Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan-Yee K. Wong.
Spc: Evolving self-play critic via adversarial games for llm reasoning, 2025.
- [30]
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li.
Algo: Synthesizing algorithmic programs with generated oracle verifiers.
Advances in Neural Information Processing Systems, 36:54769–54784, 2023.
- [31]
Shang Zhou, Feng Yao, Chengyu Dong, Zihan Wang, and Jingbo Shang.
Evaluating the smooth control of attribute intensity in text generation with llms.
arXiv preprint arXiv:2406.04460, 2024.

## 7 Preliminaries

In this section, we provide background information on the judgment process in competitive programming, define key terminologies used throughout the paper, and detail the core components of the AutoCode framework.

### 7.1 Competitive Programming Solution Judgment

In competitive programming, solutions are judged by executing them against a series of test cases. Each test case consists of an input and a ground-truth output. The system compares the program’s output for each input against the ground-truth answer. A solution is only considered correct if it passes all test cases while executing within the problem’s time and memory constraints. The outcome of the judgment process is a single verdict. The most common verdicts are:

- •
Accepted (AC): The solution passed all test cases within the given time and memory limits. This is the only successful outcome.
- •
Wrong Answer (WA): The solution produces outputs that fail on at least one test case.
- •
Time Limit Exceeded (TLE): The solution exceeds the time limit on at least one test case.
- •
Memory Limit Exceeded (MLE): The solution exceeds the memory limit on at least one test case.
- •
Runtime Error (RE): The solution terminates abnormally due to an error like a segmentation fault or division by zero.

### 7.2 Key Terminologies

#### Codeforces.

A popular online platform that hosts competitive programming contests. The difficulty of problems on this platform is often used as a benchmark.

#### Elo rating for problem difficulty.

On Codeforces, each problem has a numerical rating assigned to a problem to quantify its difficulty. It’s calibrated based on the performance of contestants on this problem during the competition. In general, a difficulty rating of $\leq 2000$ is deemed as easy to medium range, $(2000,3000]$ is deemed as medium to hard range, and $\geq 3000$ is deemed as very hard [14].

#### Generator.

A program that produces the input files for the test cases. A robust set of generators is needed to create a diverse set of test cases that cover edge cases, average cases, and worst-case scenarios designed to challenge the time or memory complexity of incorrect algorithms.

#### Validator.

A program that verifies whether a test case conforms to the constraints described in the problem statement (e.g., $1\leq N\leq 1000$, the given graph is a tree).

#### Checker.

A program that compares a contestant’s output to the correct answer and issuing a verdict. Many problems require custom Checkers beyond simple text comparisons, including

- •
Problems with multiple correct solutions: For example, in a constructive problem asking for any valid path in a graph, the Checker must verify the validity of the contestant’s proposed path, not just match it to one specific example.
- •
Problems with floating-point outputs: When the answers are floating-point numbers, answers are accepted if they are within a certain absolute or relative error tolerance (e.g., $10^{-6}$).
- •
Problems where output format is flexible: For instance, some problems that require contestants to output “Yes” or “No” may also accept “YES” or “NO” as correct answers.

#### Interactor.

A specialized component is used for interactive problems. In an interactive problem, a contestant’s solution does not receive all input at once. Instead, it engages in a real-time dialogue with the judge’s program (the Interactor). The solution makes a series of queries to the Interactor, receives responses, and use the information gathered from this dialogue to determine the final answer. A classic example is a guessing game where the Interactor knows a secret number, and the solution must find it by making queries and receiving “higher” or “lower” as responses.

## 8 Example of LLM Generated Problem: Row–Column Portal

Time limit per test: 2 seconds
Memory limit per test: 256 megabytes

You are given a binary grid $A$ with $n$ rows and $m$ columns ($0=$ empty, $1=$ obsidian).

Fix two integers $a$ and $b$ with $5\leq a\leq n$ and $4\leq b\leq m$. A sub-rectangle $M$ of size $a\times b$ (with rows $r..r+a-1$ and columns $c..c+b-1$) is called a portal if and only if:

- •
Every cell on the border of $M$ is $1$ except the four corner cells (which can be either 0 or $1$).
- •
Every strictly interior cell of $M$ is 0.

You are allowed to choose one sub-rectangle of size $a\times b$ and then perform the following operation any number of times inside that chosen sub-rectangle only:

- •
Pick a row of the sub-rectangle and flip all its cells ($0\leftrightarrow 1$), or
- •
Pick a column of the sub-rectangle and flip all its cells ($0\leftrightarrow 1$).

Flips affect only the chosen $a\times b$ sub-rectangle (not the rest of the grid).
Your task is to find the minimum number of flips (rows + columns) needed to turn some $a\times b$ sub-rectangle into a portal. If it is impossible for every placement, print $-1$.

### Input

The first line contains an integer $t$ ($1\leq t\leq 2\cdot 10^{5}$) — the number of test cases.

For each test case:

- •
The first line contains four integers $n,m,a,b$ ($5\leq a\leq n\leq 2000$, $4\leq b\leq m\leq 2000$).
- •
Then follow $n$ lines, each a string of length $m$ consisting of characters 0 and 1.

It is guaranteed that the sum of $n\cdot m$ over all test cases does not exceed $4\cdot 10^{5}$.

### Output

For each test case, output a single integer — the minimal number of flips needed, or $-1$ if impossible.

### Examples

Input

Output

Input

Output

### Accepted Solution (C++)

## 9 Example of LLM Generated Problem that Itself Cannot Solve: IT Restaurants — Distance-2 Rivalry

Time limit per test: 2 seconds
Memory limit per test: 256 megabytes

City $N$ still has a tree road network with $n$ junctions ($n-1$ roads, connected, undirected). The mayor again wants to place restaurants of two rival IT networks: iMac Donalds and Burger Bing. Each junction can host at most one restaurant.

This time, the owners impose a stricter rule set:

- •
Rivalry (distance-1): Two neighboring junctions (joined by a road) cannot host restaurants of different networks.
- •
Brand dilution (distance-2): Two junctions at distance exactly 2 (i.e., they have a common neighbor) cannot host restaurants of the same network.

Both networks must build at least one restaurant. The mayor wants to maximize the total number of restaurants. Among all optimal placements, find all pairs $(a,b)$ such that $a$ restaurants belong to Mac Donalds and $b$ to Burger Bing, with $a+b$ maximized.

Print all such pairs sorted by increasing $a$.

### Input

The first line contains a single integer $n~(3\leq n\leq 2000)$.

Then follows $n-1$ lines: edges $x_{i}~y_{i}~(1\leq x_{i},~y_{i}\leq n)$.

The edges form a tree.

### Output

Where $z$ is the count of pairs, and $(a_{i},b_{i})$ are all pairs with maximum $a_{i}+b_{i}$, sorted by ai ascending. If no valid placement uses both networks, print 0 only.

### Explanation of the new rules

- •
Adjacent restaurants cannot be different brands (so if both ends of an edge have restaurants, they must be the same brand).
- •
However, for any chain $u-v-w$, if $u$ and $w$ both have restaurants, then they must be different brands (distance-2 same-brand is forbidden).
- •
Together these imply strong local coupling between siblings in the rooted tree:
–
If a node is occupied, then at most one of its children can also be occupied at its root (and it must have the same brand as the parent).
–
If a node is empty, then among its children, at most one may have a red root and at most one may have a blue root (siblings with occupied roots must have different brands).

These constraints make the original “remove one vertex and do subset sums” approach invalid; we need a new DP.

### Examples

Input

Output

One optimal solution uses 3 restaurants on the path (e.g., 1=R, 2=R, 4=., 5=B), giving totals $(2,1)$ or $(1,2)$.

Input

Output

A star: the center must be empty to allow both brands on leaves; at most one red leaf and one blue leaf → exactly $(1,1)$.

## 10 Near-Valid Test Case Example: Perpendicular Segments time

time limit per test: 2 seconds
memory limit per test: 256 megabytes

You are given a coordinate plane and three integers $X$, $Y$, and $K$. Find two line segments $AB$ and $CD$ such that

- 1.
the coordinates of points $A,B,C,$ and $D$ are integers;
- 2.
$0\leq A_{x},B_{x},C_{x},D_{x}\leq X$ and $0\leq A_{y},B_{y},C_{y},D_{y}\leq Y$;
- 3.
the length of segment $AB$ is at least $K$;
- 4.
the length of segment $CD$ is at least $K$;
- 5.
segments $AB$ and $CD$ are perpendicular: if you draw lines that contain $AB$ and $CD$, they will cross at a right angle.

Note that it’s not necessary for segments to intersect. Segments are perpendicular as long as the lines they induce are perpendicular.

### Input

The first line contains a single integer $t$ ($1\leq t\leq 5000$) — the number of test cases. Next, $t$ cases follow.

The first and only line of each test case contains three integers $X$, $Y$, and $K$ ($1\leq X,Y\leq 1000;\;1\leq K\leq 1414$).

Additional constraint on the input: the values of $X$, $Y$, and $K$ are chosen in such a way that the answer exists.

### Output

For each test case, print two lines. The first line should contain 4 integers $A_{x},A_{y},B_{x},$ and $B_{y}$ — the coordinates of the first segment.

The second line should also contain 4 integers $C_{x},C_{y},D_{x},$ and $D_{y}$ — the coordinates of the second segment.

If there are multiple answers, print any of them.

### Example

#### input

#### output

### 10.1 NEAR-VALID TEST CASE EXAMPLE

Here are two minimal test inputs that differ by only 1 in a single field ($K$). The first is eligible (a solution exists), the second is not (no segment of length $\geq K$ can fit at all).

#### Valid

Why: In a 4×4 box, you can take two perpendicular diagonals: $(0,0)-(4,4)$ and $(0,4)-(4,0)$. Each has length $\sqrt{4^{2}+4^{2}}=\sqrt{32}\approx 5.657\geq 5$.

#### Near Valid

Why: The longest possible segment inside a 4×4 box is the diagonal 5.657, which is $<$ 6, so even one segment of length $\geq$ 6 can’t exist, hence two perpendicular ones can’t either.

## 11 Example of incorrect mutant programs: Guess The Tree

time limit per test: 2 seconds
memory limit per test: 256 megabytes

This is an interactive problem.

Misuki has chosen a secret tree with $n$ nodes, indexed from $1$ to $n$, and asked you to guess it by using queries of the following type:

- •
"? a b" — Misuki will tell you which node $x$ minimizes $\lvert d(a,x)-d(b,x)\rvert$, where $d(x,y)$ is the distance between nodes $x$ and $y$. If more than one such node exists, Misuki will tell you the one which minimizes $d(a,x)$.

Find out the structure of Misuki’s secret tree using at most $15n$ queries!

### Input

Each test consists of multiple test cases. The first line contains a single integer $t$ ($1\leq t\leq 200$) — the number of test cases.

Each test case consists of a single line with an integer $n$ ($2\leq n\leq 1000$), the number of nodes in the tree.

It is guaranteed that the sum of $n$ across all test cases does not exceed $1000$.

### Interaction

The interaction begins by reading the integer $n$.

Then you can make up to $15n$ queries.

To make a query, output a line in the format "? a b" (without quotes) ($1\leq a,b\leq n$). After each query, read an integer — the answer to your query.

To report the answer, output a line in the format "! a1 b1 a2 b2 ... a_{n-1} b_{n-1}" (without quotes), meaning that there is an edge between nodes $a_{i}$ and $b_{i}$, for each $1\leq i\leq n-1$. You can print the edges in any order.

After $15n$ queries have been made, the response to any other query will be $-1$. Once you receive such a response, terminate the program to receive the Wrong_Answer verdict.

After printing each line, do not forget to output the end of line and flush the output buffer. Otherwise, you will receive the Idleness limit exceeded verdict. To flush, use:

- •
fflush(stdout) or cout.flush() in C++;
- •
System.out.flush() in Java;
- •
flush(output) in Pascal;
- •
stdout.flush() in Python;
- •
see the documentation for other languages.

### Hacks

For hacks, use the following format: The first line contains an integer $t$ ($1\leq t\leq 200$) — the number of test cases.

The first line of each test contains an integer $n$ — the number of nodes in the hidden tree.

Then $n-1$ lines follow. The $i$-th of them contains two integers $a_{i}$ and $b_{i}$ ($1\leq a_{i},b_{i}\leq n$), meaning that there is an edge between $a_{i}$ and $b_{i}$ in the hidden tree.

The sum of $n$ over all test cases must not exceed $1000$.

### Example

#### input

#### output

### Note

A tree is an undirected acyclic connected graph. A tree with $n$ nodes will always have $n-1$ edges.

In the example case, the answer to "? 1 2" is $1$. This means that there is an edge between nodes $1$ and $2$.

The answer to "? 1 3" is $1$. This means that there is an edge between nodes $1$ and $3$.

The answer to "? 1 4" is $3$. It can be proven that this can only happen if node $3$ is connected to both node $1$ and $4$.

The edges of the tree are hence $(1,2)$, $(1,3)$ and $(3,4)$.

### 11.1 Why it’s Wrong

Here’s a subtly broken version. It behaves the same on the sample, but the component update decision is based on comparing the node index to the label of the found node (instead of comparing component labels). This will make it fail on plenty of hidden cases while still looking plausible.

## 12 Human Expert Grading Policy

Each problem is assessed independently across five dimensions: Solvability, Solution Correctness, Quality Rating, Novelty, and Difficulty.

### 12.1 Solvability

This criterion evaluates whether the problem admits a deterministic solution that meets the given time and space constraints.

- •
Yes: The problem has a deterministic solution within the prescribed complexity limits.
- •
No: It can be proven that no deterministic solution exists within the constraints.
- •
Unknown: It is currently indeterminate whether a valid deterministic solution exists.

### 12.2 Solution Correctness

This evaluates whether the provided reference solution (e.g., a model-generated code) is fundamentally correct in its approach.

- •
Yes: The core idea of the solution is correct. Minor implementation details may contain errors but do not invalidate the overall method.
- •
No: The solution contains fundamental flaws in reasoning or design.
- •
Unknown: It is not possible to determine the correctness of the provided solution approach.

### 12.3 Quality Rating

The overall quality of a problem is rated on a scale from 1 to 5, considering clarity, originality of approach, and implementation difficulty.

- •
1: Severely flawed. For example, unsolvable problems or incomprehensible descriptions.
- •
2: Low quality. For example, unclear statements, trivial or uninteresting solutions, or unverifiable correctness of the reference solution.
- •
3: Moderate quality. The problem is clear and solvable, but may feel derivative, overly difficult for humans without conceptual elegance, or overly focused on obscure knowledge.
- •
4: High quality. The problem is clear, solvable, moderately difficult, and contains at least one clever or insightful idea, though it may still exhibit some formulaic aspects.
- •
5: Excellent quality. The problem is competition-grade: elegant, novel, appropriately challenging, and free from contrived traps or rote patterns.

### 12.4 Novelty

This measures whether the problem introduces a genuinely new challenge.

- •
Yes: The problem is novel and not seen in prior contests or problem archives, even for experienced setters.
- •
No: The problem duplicates or closely resembles existing ones.
- •
Unknown: Uncertainty remains regarding the existence of prior similar problems.

### 12.5 Difficulty

Difficulty is assessed using the Codeforces rating scale (e.g., 800, 1200, 1600, …).
This score is provided as a reference and does not directly influence the overall rating.

### 12.6 Overall Problem Rating

A composite grading system is defined as follows. A problem must satisfy the requirements of its assigned level and all lower levels.

- $\geq$ 0: All evaluated problems.
- $\geq$ 1: Solvability = Yes.
- $\geq$ 2: Solution Correctness = Yes.
- $\geq$ 3: Quality Rating $\geq 3$.
- $\geq$ 4: Quality Rating $\geq 4$.
- $\geq$ 5: Novelty = Yes.
- $\geq$ 6: Quality Rating $\geq 5$.

## 13 Simulator in Interactor

Figure: Algorithm 5 Simulate (in Interactor)

## 14 Limitations and Future Work

While AutoCode demonstrates a robust pipeline for generating and verifying competitive programming problems, we acknowledge several limitations that open up promising avenues for future research.

#### The quality judgment gap.

A primary limitation is that LLMs currently lack a robust, human-aligned sense of problem quality. As shown in Finding 4 of Section [5](https://arxiv.org/html/2510.12803v1#S5), there is little correlation between LLM self-evaluations and the scores provided by human experts. Consequently, our system still requires human-in-the-loop to distinguish truly high-quality and novel problems from those that are merely technically correct. This reliance on manual annotation is a bottleneck to a scalable high-quality problem generation pipeline. A key future direction is to develop a specialized judge LLM, fine-tuned on expert-rated problem datasets to better align its assessments with human preferences, potentially serving as a reliable automated filter.

#### The bottleneck for automated self-improvement.

AutoCode is ideally suited for creating a self-improvement loop, where an LLM could enhance its reasoning abilities by training on the problems it generates. However, deploying and testing such a system faces immediate practical barriers. The most capable problem-generating models like GPT-5 are closed-source and do not expose the APIs necessary for reinforcement learning. Even for powerful open-source models, scalable and stable RL frameworks are not yet mature to support fine-tuning at this scale to the best of our knowledge. A natural future direction is to focus on developing RL or self-play frameworks compatible with frontier LLMs to unlock the full potential of AutoCode.

#### Advancing beyond recombinational novelty.

Our analysis indicates that LLMs currently tend to create new problems by combining existing algorithms rather than creating conceptually new problems that require subtle insights. Future research could explore novel prompting strategies or problem-generation pipelines to encourage greater conceptual leaps. Additionally, for valid problems that LLMs can generate but cannot solve themselves (i.e., their reference solution std.cpp is incorrect), AutoCode’s strict verification protocol currently filters them out. A prominent future direction is to develop scalable methods for identifying these valid but unsolved questions without relying on manual check by human experts, which is an extremely labor-intensive process.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Are you planning to release the source code for the system described in the paper?

Are you able to leverage the generated programming problems and solutions to finetune an LLM and show it leads to the LLM being a better problem solver?
soundness: 3
strengths: The paper describes a system of agents that produce new competitive programming problems and solutions that appear to be of high quality when judged by human experts.  The paper does nicely describe the rigorous validation and verification required to vet out proposed problems and their proposed solutions as correct or incorrect, by ensuring a reliable and broad set of test cases is produced. The paper does a number of comparisons that shows the quality of the system, like how well it is able to generate test cases compared to other test case generation systems.
summary: Auto-code is about being able to generate new competitive programming problems. This requires being able to generate new problem descriptions from scratch or by modifying existing coding problem descriptions. And it requires providing reference solutions to the new problems to show they are solvable and well formed.  And to do that requires being able to generate high quality test cases and be able to verify if a code solution can pass all the test cases, and verify the code solution is solving the new programming problem description.

This paper describes a pipeline of many agents that work together to accomplish that task, and benchmarks some of the components in the pipeline against other approaches previously tried.
weaknesses: The description of the system in the paper is well written and extensive, but I do worry that without a repo of the source code accessible to researchers the ability to leverage this work and build upon it will be very difficult.

The ability to produce new competitive programming problems that humans judge as good is useful on it's own, but if those new problems and correct solutions are able to be leveraged to further finetune the LLM and improve it's ability to solve competitive programming problems, that would be strong indicator that the problems being generated are useful in an interesting way.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 4
questions: 1. How do you develop the sandbox system? Submissions in CodeContests may come from many different online judge systems, using a variety of operating systems (Linux, Windows), compilers (GCC and MSVC spanning multiple eras), and interpreters (Python 2, Python 3, and JVMs of various versions). Therefore, if you want to evaluate the consistency with official verdicts, the first thing to do is achieve environment alignment—and I think this is extremely difficult. Many submissions fail to produce the same verdicts even when using the exact same original test cases. How do you solve this problem?
soundness: 2
strengths: 1. In terms of test case generation, AutoCode has indeed gone deeper and more refined than previous work (e.g., HardTests, CodeContests+). For instance, it enhances the validator by adding validator tests to further ensure the validator’s correctness, and it is capable of writing interactors for interactive problems.
2. This paper conducts pioneering research on problem writing using LLMs and validates the problem-writing capabilities of LLMs through large-scale human evaluations.
3. The study on the value alignment between LLMs and human experts is highly insightful. Given that LLM-as-a-judge is currently a highly mainstream method, the results (Fig. 4) demonstrate the limitations of LLMs in such complex OOD tasks.
4. This paper is well-written. I am enjoying reading this paper.
summary: This paper presents AutoCode, an LLM agent system for writing and developing competitive programming problems. AutoCode generally consists of two parts: a test case generator, and a problem writer.  The test generation component implements multiple components including validator, generator, checker, and interactor. Experimental results indicate that the test cases generated by AutoCode outperform current state-of-the-art baselines in terms of both FPR and FNR metrics. The problem writing component has achieved excellent results in human evaluation, and the findings show that LLMs are capable of generating high-quality and high-difficulty problems.
weaknesses: 1. **Does this paper propose a better test case construction method?**, The generator-validator-checker-interactor workflow is a widely adopted problem-construction process in competitive programming, and has also been used more or less in previous studies (e.g., HardTests, CodeContests+). The similarity between AutoCode’s overall workflow and that of previous works may raise questions about the source of improvements in its evaluation. First, this study is conducted based on SOTA models such as o4 and GPT-5, while previous works are before the release of such advanced models. Second, prompts are a critical factor affecting performance—providing a open agent framework that includes prompts would be a significant contribution to the community. However, this paper does not disclose the prompts it used, thus diminishing its reproducibility and overall contribution.
2. **Questions on the soundness of the LLM problem writing study.** I find the topic of LLM problem writing very interesting, but I do not believe this paper has conducted an in-depth and rigorous study. First, it is an experiment entirely based on human evaluations, yet I think some of the criteria could have been tested through objective experiment. For instance, the paper mentions a criterion called "Model-Training Usable"—this criterion could be fully verified through actual training results, rather than relying on human evaluations. Additionally, the human evaluation lacks details, such as the average time each expert spent evaluating each piece of data and the profiles of the experts. Some conclusions also lack sufficient evidential support; for example, for Findings 2, it is recommended to incorporate more case studies for further discussion. Furthermore, the sample size of this experiment should be placed in a more prominent position, such as the caption of Figure 3. I read through this section several times but still could not find this number. Finally, regarding the most critical part of this experiment: how did the authors guide LLMs to generate problems? The entire problem-writing process lacks a detailed description,  nor does the prompt provided, so I believe it lacks reproducibility.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Although stated in Finding 1: "LLMs can generate solvable problems that they themselves are unable to solve," this kind of problem would be filtered by the solvability check right? Could you clarify if the system using the filtering process described in the beginning of Section 5 would be able to produce such problems at the end?
- What is the "prompt optimization" refer to? Line 291 shows the ablation "without prompt optimization" and in line 321 "Prompt optimization turns out to be especially important;" there seems to be no explanation of the prompt optimization in the paper.
- In 3.2 for the generator, it is stated that they use strategies: Small Data Exhaustion, Randomized and Extreme Data, and TLE-Inducing Data, but didn't explain how these strategies are implemented. Are they being implemented with LLM by using LLM to generate a program to perform this kind of strategy?
- Can you explain GENERATECHECKERSCENARIOS in Section 3.3? How does it generate the (input, contestant_out, ref_out, verdict) data?
- Can you clarify which parts of the test generation required access to ground-truth programs? is it just the for thchecker?
- In line 285 "all results in this table are generated by GPT-5-High", do you mean the model used in the test generation system or the 33 submissions?
- Since competition programs accept or not sometimes depending on the timeout set for the program, how is timeout set for the generated problems?
- For the problem creation prompt that prompts the model to create new problems from seed problems, do you need to specifically guide the model what kind of variations they can make and does it bias the model to alter the seed problem in specific ways?
- Are the LLM prompting used in this work zero-shot or few-shot like you need to provide more examples to guide the model? Can you provide the prompts used in this work to help with reproducibility?
soundness: 3
strengths: - Significantly improve the test generation false positive and false negative rate for the competition programming
- Demonstrate that model can generate novel problems not only useful for training but also can generate very high quality deemed as good as top competition problems (ICPC/IOI-Level)
- Analysis and ablation showing many insights of the results, the finding from the human experts annotation is especially valuable
summary: The paper focuses on test generation and problem creation for competitive programming problems. First, they show that their test generation framework can achieve very low false positive and false negative rates compared to prior SOTA work. The test generation framework consists of using LLMs to construct validator, generator, and checker with novel prompting techniques specifically designed for more robust checking. Building on the robust test generation, they show that they can prompt LLMs to create very high quality problems (judged by top competitive programmers) by seeding from existing problems and then going through testing to filter problems such that the LLM-generated brute-force solution (acting as ground truth) and the LLM-generated solution agree.
weaknesses: The critical weakness of this paper is the lack of reproducibility. The paper uses specific LLM prompting techniques to achieve impressive results but fails to provide any of the prompts used in the work. Many details are also missing, for example, how the generator works and the problem creation procedure are not clearly explained.


Decision: Accept (Poster)