rStar-Coder: Scaling Competitive Code Reasoning  with a  Large-Scale Verified  Dataset
Authors: Yifei Liu
Li Lyna Zhang
Yi Zhu
bingcheng dong
Xudong Zhou
Ning Shang
Fan Yang
Cheng Li
Mao Yang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Code Reasoning
Test Case Generation
Primary area: deep_learning
TL;DR: We introduce rStar-Coder to train advanced code reasoning LLMs, with our 14B model achieving comparable performance to QWQ-32B.

Abstract: Advancing code reasoning in large language models (LLMs) is fundamentally limited by the  scarcity of high-difficulty datasets, especially  those with verifiable input-output test cases necessary for rigorous solution validation at scale. We introduce rStar-Coder,  which significantly improves LLM  code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition-level code problems,  580K long-reasoning solutions  along with rich test cases of varying difficulty. This is achieved through three core contributions: (1) we curate competitive programming code problems and solutions  to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline  that decouples the  generation  into a three-step input generation method and a mutual verification mechanism for effective output labeling; (3) we augment problems with high-quality, test-case-verified   long-reasoning solutions. Extensive experiments on Qwen models (1.5B-14B) across various  code reasoning benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with significantly smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2.5-7B from 17.4% to an impressive 57.3%,  and Qwen2.5-14B from 23.3% to 62.5%, surpassing o3-mini (low) by 3.1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass@1 accuracy of 16.15%, outperforming the frontier-level QWQ-32B.  rStar-Coder dataset is publicly available at https://huggingface.co/datasets/microsoft/rStar-Coder.

rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
Yifei Liu' Li Lyna Zh Yi Zhu
Bingcheng Donga.3 Xudong Zhou
Ning Shang Fan Yang* Cheng Li Mao Yang"
University of Science and Technology of China
Microsofft Research Asia
Dalian University of Technology SShangaii Jiao Tong University
Abstract
Advancing code reasoning large language models (LLMs) is fundamentally limited by the scarcity of high-diffculty datasets, especially those with verifi- able input-outpul test cases necessary for rigorous solution validation at scale. We introduce rStar-Codee which significantly improves LLM code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition- level code problems, 580K long-reasoning solutions along with rich test cases varying difficulty. This achieved through three core contributions: (1) we curate competitive programming code problems and solutions to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline that decouples the generation into stepe-step input generation method and a mutual verification mechanism for effective output labeling; (3) we aug- ment problems with high-quality, fest-case-verffied long-reasoning solutions. Ex- tensive experiments on Qwen models (1.5B-14B) across various code reason- ing benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with significantly smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2.5-7B from 17.4% to an impressive 57.3%, and Qwen2.5- -4BB from 23.3% to 62.5%, surpassing o3-mini (low) by 3.1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass @ accuracy of 16. 15%, outper- forming the frontier-level QWQ rStar-Coder dataset publicly available
63.4 65.9
rSlar-Coder-7B
57.3
57.2
57.5
rStar Coder-145 14B
DeepSeek-R1 Distill
DeepSeek-R1 Distill 32B
37.6 8
DeepSeek-R1 Distill 708
32B
DeepSeek-R1
21.87
16. 16.1517.19
11.98 54 15.62
LiveCodeBench (v5)
USACO 2025
Figure @ass@1 accuracy on code reasoning benchmarks.
"Yifei Liu, Bingcheng Dong, and Xudong Zhou completed this work during their internships at MSRA. Project leader: Izhani @ microsoft.com 9 "Correspondigg author: chengli7 wttp:///tstttuuucnnn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent large language models (LLMs) have made significant strides in reasoning, with models like OpenAl o1/o3 [17] and DeepSeek-RI [9] showing strong performance on complex code problems. However, further improve advanced code reasoning, a persistent challenge is the searcity of large- scale, high-quality datasets that contain verifiable, high-difficuty programming problems requiring both algorithmic thinking and efficient code implementations.
Unlike datasets on math word problems, where solution correctness can be verified through simple rule-based matching against reference answers [9, 8], code reasoning requires executing solutions against diverse test cases to detect logical and implementation errors. These input-output test cases must differ in content, scale, and complexity while meeting problem-specific constraints. Critically, each test output must correctly label the expected result for its corresponding test input.
Obtaining such reliable and verifiable input-output test cases at scale remains highly challenging. Hlmann-curtte datasets, such as CodeContests [22] and TACO [20], provide high-quality problems but typically lack comprehensive test coverage, often including only a few simple test cases that fail to capture diverse input scales of edge conditions. Synthetic datasets such as WizardCoder [24], Magicoder [30], and KodCode [3 ]] primarily target iunction-level code generation, where correctness can be verified with minimal or even no test cases. These datasets are therefore insufficient for advanced competitive-
code reasoning. While frontier LLMs may offer scalable way for test case synthesis, two substantial challenges arise: generating semantically valid, constraint- inputs varying in scale and difficulty; and labeling correct outputs without ground truth solutions, particularly since synthetic problems inherently lack reference implementations.
In this paper, we introduce rStar-Coder, novel approach that reliably constructs a large-scale, high-difficulty dataset for training advanced code reasoning LLMs. This dataset includes 418K unique competitive programming problems and 580K long-reasoning solutions, each verified through diverse synthetic test cases across difficulty levels. rStar-Coder incorporates three key components.
First, we curate and clean 37.7K expert-written problems with oracle solutions from competitive programming platforms (e.g., IOI, Codeforces) and use them as seeds to synthesize new, solvable problems. Unlike prior works [31, 24] that emphasize diversity, we prioritize solvability and correct- ness, since the seed set already spans a wide range of topics and difficulty levels. However, these problems are often too difficult for even frontier models like GPT-4o, and directly prompting the LLM with only the problem statement often results in invalid or unsolvable outputs. To address this, we design a structured prompt that incorporates both the problem statement and its oracle solution, guiding the model to understand core algorithmic concepts and generate novel yet solvable problems.
Second, we address the challenge of generating reliable and diverse input-output test cases for solution validation by decoupling the process into two stages. (1) First, we propose a three-step approach to generate valid test inputs of varying scale and complexity. We prompt GPT-4o to generate two utility functions for each problem: one for synthesizing semantically valid inputs with exposed scale-controlling parameters, and another for validating whether the inputs satisfy problem-specifie constraints. We then sample input scale values (10" 10a) for these scale-controlling parameters to cover a diverse range of complexities. Executing the two utility functions with these sampled scales yields diverse, constraint- satiiiiinniiiyiig test inputs. (2) Second, to reliably label outputs, we introduce mutual verification mechanism. We sample multiple long-CoT solutions from strong reasoning model (QWQ-32B), and accept both the test outputs and solutions a majority produces consistent results across all test imputs. This mechanism is effective because incorrect solutions tend to diverge in errors, while correct solutions converge. Thus, consistent agreement among multiple solutions on identical outputs across diverse test inputs, particularly those of varying complexity, serves strong evidence for the correctness of both the solution logic and the resulting outputs. Our ablation study demonstrate the effectiveness of this approach, achieving 96.8% accuracy in output labeling.
Finally, we augment curated expert- problems with verified long-reasoning solutions. While these problems are of high-quality, their original solutions often lack detailed reasoning steps. Generating them directly with frontier LLMs is unreliable, as few simple original tests are insufficient for validation. To address this, we use our test generation method to produce diverse, constraint-awarc inputs. Since curated problems include oracle solutions, we run them to get ground tround-truth outputs. We then prompt QWQ-32B for long- aangrreaonning solutions, keeping only those that pass all generated tests.
2

---

Extensive experiments across different-sizeee sized LLMs (1.5B-14B) and diverse code reasoning bench- marks demonstrate the effectiveness of iStar-Coder, consistently improving the code reasoning capabilities of all base models to state-of-the-arr levels, even with significantly smaller sizes. On Live- CodeBench, rStar-Coder improves the 14B model from 23.3% to 62.5%, surpassing RI-distill-70B and o3-mini (low) by +5.0% and 3.1 %. And even the 1.5B model reaches 40.1%. 10,, outperforming both RI RI-distill-7B and GPT-4o. On the highly competitive USA Computing Olympiad, "Star-Coder- and rStar-Coder- 14B outperform the frontier reasoning model QWQ-32B. Furthermore, rSStarCoode also generalizes remarkably well on standard code generation tasks like HumanEval and MBPP.
2 Related Works
Instruction Code Data Synthesis. Prior to the rise of cansoning-focuseee LLMs, code data synthesis
methods [24, 30, 32] primarily targeted scaling instruction data to improve LLMs capabilities to generate code aligned with use intent. Efforts like Code Alpaca [4], WizardCoder [24]. Magicoder [30] and WaveCoder [32] focused on synthesizing diverse and complex prompts by collecting seed code snippets and prompting LLMs for generating instructions and solutions. Recently, KodCode [31] scaled this paradigm by synthesizing 447K prompts from 12 sources using five distinct methods to ensure diversity and complexity. While effective for improving general instruction- -ooloooiinn.ooooiingg these datasets show limited gains on reasoning-heavy code tasks [31, 16].
Code Reasoning LLMs and Datasets. Recent advances in code reasoning LLMs have garnered attention, with two main approaches: distilling long-reasoning solutions from frontier models like R1 and 03-mini [26, 2, I]. and applying reinforcement learning to high-difficulty code problems [25, 35]. Both face limitations. Distillation-ben -based methods build datasets [26, 2, 28] by scaling solutions per problem but suffer diminishing returns due to limited diversity. RL methods struggle with unreliable evaluation, as comprehensive test cases are often lacking. To our knowledge, prior work has systematically scaled algorithmic problem sets while providing robust, verifiable test case generation.
Code Solution Verification. Verifying the correctness of LLM -LM-eeeertted code solutions, especially for competition-lever problems, remains highly challenging. A common approach using generated test cases for validation. Prior work often uses the same LLM to produce both solutions and test cases, relying on self-consistency to select the most agreed-upon output [5, 11, 33], or comparing outputs to oracle solutions derived via exhaustive brute-force search [34]. However, the validity and diversity of test cases across difficulty levels have not been extensively explored, yet both are essential for robust evaluation. Our work addresses this challenge through three-step input generation method and mutual-verification mechanism that reliably labels both solutions and test cases.
3 Methodology
3.1
Collection of Competitive Code Problems
Collection of expert-designed competitive coding problems. We curate a seed dataset from pub- licly available resources, including programming competition websites and open datasets, where the problems and test cases are designed by human experts. This includes original problems, reference solutions, and available test cases from TACO [21], APPS [10]. CodeContests [22], CodeContests- [14], CodeFroces from the OpenR I project [26], and USA Computing Olympiad 2011 1-20- (USACO) [27]. We also gather problems from the International Olympiad Informatics (IOI) spanning 2002-2023. As IOI problems are published in PDF format, we follow Numina- Math [19] and use Mathpix convert them into La'exx. In total, we collect 57,215 problems (see Table 1). To ensure high quality, we remove duplicate problems across datasets and discard problems lacking reference solutions, resulting in 37,754 unique problems with at least one reference solution.
Synthesis of new solvable and high-difficulty code problems. Then, we use the 37,754 expert- written problems as seeds to synthesize new ones. These competitive programming problems span a wide range of algorithmic topics and data structures, but each type is sparsely represented. Therefore, unlike prior works that focus maximizing the diversity between generated and seed problems [31, 24, 13], our focus is on generating problems that are both solvable and high-diffccull
However, directly prompting LLMs (e.g., GPT-40) with seed problems alone often fails to generate valid new problems, as even GPT-4o struggle to solve competition code problems. Without a deep understanding of the knowledge and reasoning skills being tested, the LLM lacks the necessary grounding to create problems that preserve both difficulty and solvability. To address this, we
3

---

Table I: Summary of competitive-level programming problems. In total, we collect 418K verified problems, including 37.7K expert-desinned and 380K synthetic problems.
Original Seed Questions
Synthesized Questions
Total
Source
# Original
Verified
Synthesized
# Verified
Verified
AIZU
4302
2151
100712
17386
19537
AtCoder
2824
2080
88103
29096
31176
CodeChef
5232
3765
164646
46749
50514
CodeWars
4975
2515
108565
8520
11035
GeeksForGeeks
2680
2680
116809
37060
39740
HackerEarth
3657
1411
166385
48185
49596
HackerRank
854
35959
11707
12561
LeetCode
1516
754
32794
2282
3036
CodeForces
30049
20616
706289
170713
191329
International Olympiad Informatics (IOI)
626
444
24760
1767
2211
USA Computing Olympiad (USACO)
484
484
20610
7095
7579
Total
57,215
37,754
1,565,632
380,560
418,314
Test case exampleat
case example42
case examples8
Test case examples4
Input fword, locallzation]
Input 4graph, 20-array
Input 40, 3,
2 tiscale: number of words
Input (groph, the xzy axes]
2 scale: scale:number of rows and cols
i stale: the size of the amay
word a string
scale: number points
..... locatiion shoep, walf
132 elemmenss of the arra
localiation the string
11 the coordinates of the 1 point
.S.9. . Alocalion of sheep, wolf
12 the coordinates of the 2 point
Output YES
Output
Output
word
Outout
Yes whether protect all sheep
110n
llocaion
.SDW. location
Figure 2: Examples of standard input- output test case pairs from competitive programming datasets. The first line indicates the scale of the test input (e.g., the size of ID array), followed by each subsequent line representing the specific values for the input elements.
design structured prompts that include both the seed problem, its reference solution, and step-by-step synthesis instructions. The reference solution helps the LLM internalize the key algorithmic reasoning concepts involved in the seed problem. Specifically, we instruct the model to: (1) understand the seed problem and solution; (2) identify the reasoning and core knowledge being tested from the solution; (3) synthesize a new problem that test similar skills. Full prompt details are in Appendix Fig. 8.
In practice, incorporating reference solutions proves crucial for achieving high-quality synthesis. Even smaller models such as Qwen2.5-725 Instruct perform well under this setup, though we adopt GPT-4o as the main generator to ensure higher reliability and success rates.
In totoal, we synthesize 1,565K new code problems as shown in Table 1. Compared the original seeds, the synthesized problems still exhibit novelty by: (1) creating new contexts while maintaining the original algorithmic strategy (e.g., dynamic programming): (2) modifying adding constraints to alter difficulty or computational complexity; (3) changing both context and constraints. These synthesized problems lack verifiable test cases and reference solutions, and some may be unsolvable. We address this through test case generation and mutual verification, as detailed in the next sections.
3.2 Test Case Generation
Checking whether code solution runs without syntax or runtime errors is a straightforward way to verify correctness, but it's not sufficient. A correct solution must also be logically sound and bug-free. The standard approach is to use input-output test cases: given test input (as shown in Fig. 2), valid solution should produce the expected test output exactly. Effective test cases, therefore, should meet two key criteria. First, test inputs should cover a range of scales and complexities to evaluate both correctness and efficiency. For example, real-world code competitions often include large inputs (c.g.. size 10²) to test whether a solution satisfies the algorithmic efficiency constraints and runs within time limits. Second, the test outputs must be accurate to support reliable validation.
However, obtaining such high high-quality test cases is highly challenging. Curated datasets, though sourced from major coding platforms, typically provide only public test cases, which are often too simple to catch deeper logical errors. Synthetic problems, on other hand, lack test cases entirely. Moreover, without ground-truth solutions, it becomes even harder to accurately label outputs, making reliable test case generation especially difficult. To address these challenges, we decouple the test case generation process into two stages: input generation and output labeling. Instead of prompting

---

Input C Constraints
The first line contains single Integer "n" (1<=56="" <=sc=100000) the size of array. The second line contains 'nn integers' "a"1, """2."" "a""n" "arrscc"""""ieess"1"00"" - the elements of the array.
Pusage: printgenneate, test input(5))
uusage print (validate test ingut 5 #True def validate, test, inputiinput stringl:
2 64 не possible generated test
from cyaron import*
def generate test, Inputin]:
lines input, string spliit))
Constraint check
lendlines] I- 2: return False intilines(0)
If 12 <c 100000):
not if lenía) I- return False
100000):
é Generate an arryy of size with values between 1 and 1e9
=1al)mappiin, line [sse[sssrlr))))))R))
Vector.andommin [O1, int(1e9])), mode-1)
flat_ sum(a, (I) & flation oi integers
aayyi <11 10"9 for return True
return False
Format string
input, lines = =]st(n), " "jpin(mapistr, flat
except: return False
return "nn'jcin((nput_ lines!
Figure 3: An example of LLM-generated utility functions for test input generation and validation.
an LLM to directly produce input-output pairs, we first introduce a three-step input generation method to create valid and diverse inputs of varying complexity that satisfy the problem's constraints. We then use a mutual verification mechanism to reliably label the corresponding test outputs.
3.2.1 Valid Test Input Generation with Varying Complexity
Test case input format. To generate valid test imputs across varying complexities, we first introduce the typical format of test inputs. As shown in Fig. 2, a test input in competitive programming is usually a multi-line string. The first line encodes size or grid dimension), and the following lines specify the actual input content. These strings are passed directly to the solution code, which must implement custom parsing logic interpret them.
or more scale-controlling parameters (e.g., array
On top of that, generating valid inputs across different scales, however, is non-trivial due to two key challenges. First, although all inputs are tringgencoded, their structure and semantics vary significantly across problems. For instance, a string might represent a ID integer array, a list of strings, coordinates, or 2D grid. Without understanding these structures, it is difficult to produce valid inputs. Moreover, problem statements often include constraints on valid input ranges, making naive prompting likely produce invalid test inputs. Second, we aim generate inputs of different complexities, yet naive prompting of LLMs tends to result in random and overly simple inputs ( see Fig. 5). To address these challenges, we propose three-step approach as illustrated in Algorithm 1.
Step Generating utility functions for input generation and validation. To produce high-quality test inputs that satisfy both the semantics and constraints of each problem, we prompt a frontier LLM (GPT-40) to generate two utility functions per problem: one for test input generation and one for input validation (Fig. 3). This serves two purposes: (1) automatically producing well-structured inputs that satisfy problem constraints, and (2) exposing scale-controlling parameters (i.c., the first string line in Fig. 2) to enable flexible input sizing. Notably, direct LLM generation of input values often causes hallucinations. To reduce this, we allow GPT-4o to use CYaRon 9, a reliable input data generation toolkit. As shown in Algorithm 1, given the problem description and CYaRon documentation, the LLM is asked to generate GENERATE. parameters call CYaRon for input construction, and the resulting input string and checks for constraint satisfaction. Prompt details are in Appendix Fig. 9.
INPUT function that uses scale function that parses
Step2: Defining input scale ranges. From the scale-controlling parameters exposed by the GENERATE_ TEST TEST_INPUT function in Step I, we define value scales for each parameter to control test case difficulty (e.g.. 119x11900 for easy cases, up to 10 for hard cases). For example, in ID array, the scale denotes the number of elements (e.g.. 10 means 100K elements); in a 2D grid, has two scale-controlling parameters for the number of rows and columns. We instantiate each parameter across its range and input the values into the generation function.
Step3: Executing utility functions to produce valid test inputs. Finally, for each instantiated scale-controlling parameters from Step 2, we invoke the GENERATE_ TEAATE_TEST__NGE function to generate a test input string. We then use the VALIDATE_ TEST__TT_____ INPUT function to verify whether each generated input string meets the constraints outlined in the corresponding problem statement. Only the inputs that pass validation are retained as valid test inputs.
5

---

Algorithm Three-Step Test Input Generation Algorithm
Step I: GPT-4o generates test input and validation functions with applying the CYaRon library.
1:
function CEEERRATTTTTTTSNNUTTTTOO poreo
purom......)
param, param...... paraaat,
controls
the imput scales
2:
return None If parameters do not satisfy the problem constraints
3: Generate input using the CYaRon library
4: return input_string
5:
end function
1: function VALIDATE TEST INPUT(input___ string)
2:
Check whether input_string satisfies all constraints in the problem statements.
3: return True if valid; otherwise False
4: end function
Step 2: Define varies input scales
1: for each (param,, parama.......................... param,.) extracted from GENERATE_TEST INPUT:
2: C (1,2....,99 9} U {10'' Osise)
Step Execute functions produce valid test inputs
1: for each (param,, parama, param,)) E C do
2: inapul_srinng GENERATE TEST (NPPT_TUPUUUA,,,
paraaaar parama....)
3: valid VALIDATE TEST _EST_INPUT (inpu(iinuu_ssrinn
4: Retain intpt_ssting string If valid is True
5:
end for
Augmenting difficult test inputs for seed problems. addition to generating test inputs for synthetic code problems, we also enhance the test inputs for original seed problems, which often feature either simple public test cases or none at all. By applying the above pipeline, we generate more diverse and challenging test inputs for these problems.
3.2.2 Mutual Verification for Test Output and Solution Labeling
Inputs
(varied difficulty levels)
Test Input,, Test Inpufs..... Test Input,
Majority Voting
Candidate Solution,
Test Oulput,s. Test Outpul a..... Test Outpul
If majority solutions culputs
Candidate Solution
Output,, Oulpuly,....
Output,
Label (solutions, outputs) correct
Candidate Solution,,
Test Outputar. Test Outputaz.... Test Output
Figure 4: Mutual verification between candidate solutions and test outputs. Given diverse set of inputs, if most solutions return the same outputs, both the solution and outputs are considered correct.
The next step is to reliably label each test input with the correct output. Since synthetic test inputs come from two sources (i.e., the original seed problems and synthesized problems), we apply different strategies to achieve optimal labeling reliability for each.
Labeling test output for seed problems. For our augmented test inputs from seed problems, we simply execute the provided oracle solution on the input. Since the reference solution is assumed correct, its output serves as a the ground-truth label.
Mutual verification for synthetic problems. However, labeling test outputs for synthetic problems quite challenging as there are no oracle solutions. To address this, propose simple yet effective mutual verification mechanism that identifies both correct test outputs and the solutions that produce them. As illustrated in Fig. 4, for each problem, we first sample long-reasoning candidate solutions using a frontier reasoning model (QWQ-32B [29]). We then sample diverse set of at least 50 test inputs with varying complexities. Each candidate solution executed on this shared set test inputs to generate the corresponding outputs. If majority of these candidate solutions produce identical sets of outputs across this entire of test inputs, then both these consistent sets of outputs and the candidate solutions that generated them are considered correct.

---

The effectiveness of mutual verification stems from the fact that incorrect solutions are more likely to diverge in errors than converge on the exact same incorrect answer across multiple test inputs. If majority of independently generated solutions produce identical results for diverse set of inputs, it suggests they are not just randomly failing or misunderstanding the problem in different ways. Instead, their consistent agreement indicates successfully solved the problem, providing strong evidence for the correctness of both the solutions and their generated outputs.
3.3 Augmentation and Post-processing
Seed problems augmentation. Although our seed problems expert-designed and high quality, their original oracle solutions often lack detailed reasoning steps. To address this, we rewrite the solutions to include rich reasoning pattems, such as self-reflection which are essential for training advanced code reasoning L.Mss. To verify correctness, we augment each seed problem with diverse test cases generated by our method, which used for solution verification and filtering. Specifically, we use QWQ-32B to generate 16 long CoT' solutions per problem and retain only those that pass all tests. For particularly challenging problems where QWQ-32B fails to produce a correct solution, we follow prior work [26, 2] and retain all generated solutions to include more diverse and potentially correct intermediate reasoning steps in the training data.
Post- Post-Processing for synthetic data. We also clean the synthetic data to ensure high quality. First, we remove unsolvable overly difficult problems, which may result from hallucinated generation or limitations of the frontier reasoning model. The mutual verification mechanism naturally acts as an effective filter. Specifically, if fewer than 60% of solutions agree on the outputs, the problem is discarded. For problems synthesized based on Codeforces problems, we use the original cf_rating to identify hard problems acf_rating 1600) and adjust the threshold to 40% to include harder problems. This process effectively filters out unreliable problems, as shown in Table 1. After filtering, we retain 380K verified synthetic problems. For these 380K problems, we initially have 2.25M long CoT solutions, which are too large for efficient fine-tuning. We reduce this by executing all solutions and retaining only the fastest one per problem based on CPU execution time.
Decontamination. To ensure fair and unbiased evaluation, we follow prior works [15] and perform data decontamination by removing any problems that overlap (16-gram) with evaluation benchmarks: HumanFval, HumanEval+, MBPP, MBPP+, LiveCodeBench, and USACO 2025. We adopt a 16-gram threshold because shorter n-grams (e.g.. 8- 10-gram) often produce false positives in competitive programming problems, which naturally share common phrases in I/O descriptions and problem statements (e.g.. "the sum of n overall all test cases").. After decontamination, dataset contains 418K problems with extensive test cases, totaling 580K question-solution pairs.
Experiments
4.1 Setup
Training setup. Using our 580K dataset, we fine-tune Qwen2.5-Codee instruct models [16] at 1.5B, 7B, and 14B sizes for 6 epochs using the AdamW optimizer, batch size of 96, and max sequence length of 16k. The learning rate is 4e-5 with cosine decay. Training is accelerated with FlashAttention-22 [7] and DeepSpeed ZeRO-O. Specifically, the 1.5B and 7B models are trained 8 MI300X AMD GPUs, while the 14B model uses 32 MI300X GPUs
Evaluation benchmarks. We evaluate diverse benchmarks, including LiveCodeBenchh [18] v5, which features new problems from LeetCode, AtCoder, and Codeforces (2024.8-208) 2-225.2).. and a challenging new benchmark from the USA Computing Olympiad 2025 (USACO), containing 12 Olympiad problems across bronze to platinum tiers. These problems test a broad spectrum of algorith- mie and commonsense reasoning. In addition, test generalization abilities by evaluating popular code generation benchmarks: HumanEval [6], HumanEyal+ [23], MBPP [3], and MBPP+ [23].
Baselines and Inference Settings. We compare against two strong baselines: (1) leading frontier LLMs, including GPT-4o, Claude 3.5 Sonnet, DeepSeek-RL, and QWQ-32B; and (2) state-of- the-art code reasoning models fine-tuned on large-scale long-CoT code datasets. Specifically, in addition to DeepSeek-RI distilled models, we compare with Bespoke-Stratoe (fine-tuned on 17k traces) [28], OpenThinker (1 144k [28], OlympicCoder (100k) [26], Nvidia OCR-Owen (736k) [2], and OpenThinker (1M) [28]. Notably, OCR and OpenThinke22 are trained with significantly larger datasets than ours. For inference, set temperature to 0.6, maximum output length to 32k tokens.
7

---

Table 2: Results of rStar-Coder and frontier reasoning LLMs on diverse benchmarks. We outperform on both reasoning-heat and general code generation tasks with significantly smaller model sizes. Gray text in indicates our absolute score improvements over the base model.
Model
LiveCodeBench Hvmannvvll
HumanEval+
MBPP
MBPP+
GPT-4o
30,0
92.7
87.2
87.6
72.2
Claude3.5-Sonnct
32,0
92.1
86
91
74.6
OpenAl 03-mini-2025001-11 (low)
59.4
QWQ-32B
63.4
95.6
89.8
92.2
76.5
OpenAl ol
63.4
DeepScck-RI Gemini-2.5-Pro
65.9
96.3
90.9
95.3
81.2
69.2
1,5B Long-Cor CoT coder reasoning LLMs
Binieed-RR---iiiil------sess 1odar-Coder-1.5B
16.9
66.3
61.8
57.1
48.5
40.1 (+33.6)
88.4 (+17.7)
81.7
(+16.5)
74.1
(+8.9)
60.8 (+1.5)
Bespoke-Stratos Marratos7B(BBe)) 7BBBesBoke-SSreneeeeeee Sartnokk--SSrtts-1111 Stratos-17k)
16.2
75.1
69.1
73.6
59.8
OpenThinkkrt-71 (OpenTTougghts 114k)
25.5
80.5
75.7
80.9
68.2
OpenThhinker2779
37.4
92.7
87.8
86.9
73.9
DeepSeek-R 1-Distill- Qwen-7B
37.6
89.6
83.7
78.4
66.7
DeepSeek-R
39.6
85.9
79.7
62.7
OlympicCuder::BB (egeenl--codffrotn 100k)
40,9
82.1
76.9
80.0
66.4
OCR- SCR-Qwen-7BB -B-nstrutttttt (OCR-736k)
51.3
rStar-Coder-7B
57.3 (+39.9)
95.9 (+7.5)
90.8 (+6.7)
87,9 (+4.4)
74.0 (42.3)
14B-70B Long-CoT reasoning LLMss
DeepSeek-R R 1 D--Diiill--QO Distill-Qwen-Iae Qwen-14B
53.1
96.1
89.8
87.4
74.1
OCR-Qwen-1ee @0wen-144-Instteie 14B-Inttruct (OCR-736k)
59.4
Strnke--traatss335
48.9
95.5
89.8
93.8
77.5
DeepSeek-HR R L--issiil--Quee Qwen-32R
$7.2
95.9
89.9
92.8
78.7
OpenTTinnker- ThpenTThuuhhts 114))
54.1
94.8
89.2
94.1
78.4
OlympicCoder-332B -oodenn--ooeeoorress--eoo
57.4
90.0
85.2
86.7
71.3
OCR-Qwen:32B-Insteee (3CR-736k)
61.7
DeepSeck-R -. Ri-----ill---LLMMMAAAA
57.5
96.5
90.7
91.9
77.1
rStar-Coder-14B
62.5 (439.2)
95.9 (+6.3)
89.6 (+2.4)
91.4 (+5.2)
77.3 (44.5)
To mitigate performance variance inherent in single-run evaluations, we sample 16 solutions per problem and report the average pass @ accuracy for all benchmarks.
4.2
Main Results
Results on diverse code benchmarks. Table 2 presents the performance SSaarCooder compared to faat-off-hheaat reasoning models. We highlight three key observations: (1) By scaling with high-quality dataset, rStar-Coder significantly improves LLMs code reasoning capabilities, achieving performance comparable frontier reasoning LLMs with substantially smaller model size (1.5B- 14B). For instance, Swee2.5-Coder-11550 originally at only 6.5% accuracy on LiveCodeBench improved dramatically to 40.1% with rStar-Coder even outperforming GPT-4o and RI-Distill-QUus -wenn 1.5B. Morcover, larger models see even greater gains: rStar-Coder- 7B achieves 57.3%, surpassing #Star-Coder-Ile
reaches 62.5%, outperforming all open-source basclines. including OCR-32D 32B and RI-Distill- and even surpass 03-mini by 3.1%. (2) Dataset quality matters more than size. Though OCR [2] and Open'Thinker-2 [28] use much larger datasets (736K and IM vs. our 580K), rStar-Codes performs significantly better-by by +6% and 19.9% at 7B, and even surpasses their 32B models 14B. (3) Despite not being tailored for general code generation, rStar-Coder generalizes remarkably well. rStar-Coder consistently improves the performance on HumanEval, HumanEval+, MBPP, and MBPP+ all base models to state-of-the-ard levels. Notably, rStar-Coder-7BB achieves performance par with Claude3.5-Sonnee showing that strong reasoning data can generalize effectively beyond its original domain.
L1-Distille--Wwen-
Results on challenging Olympiad programming. We further evaluate on USACO 2025, which features highly challenging algorithmic problems across four tiers: Bronze Platinum. Unlike standard competitive questions, these problems require grounded and often creative reasoning, especially at the Gold and Platinum levels, where deep algorithmic insight and creativity are essential. As shown in Table 3, the benchmark is extremely difficult. Even OpenAl o3 scores only 32.03% and fail on all Platinum problems. Despite the difficulty, our 7B and 14B models perform competitively, outperforming the QWQ-32B. Notably, QWQ-32B generated the long-reasoning solutions in our dataset. That even Star-Coder-7B surpasses highlights the strength of our data: diverse, competitive problems and verified high-quality reasoning enable smaller models to rival far larger ones.

---

Table 3: rStar-Coder performs competitively on USACO 2025 with much smaller model sizes.
Model
Avg.
Bronze
Silver
Gold
Platinum
OpenAl-o3
32.03
72.91
28.12
27.08
DeepSeck-RI
21.87
58.33
22.91
6.24
QWQ-32B
15.62
43.75
12.5
6.25
7B-8B Long-CoT coder reasoning LLMs
OpenThinker-7B
0
0
0
OlympicCoder-7B
0,52
2.08
OpenThinker2-77
4.16
16.67
0
DeepSeek- RI -i-sstill-Qwen-7 Qwen-7B
4.68
18.75
0
rStar-Coder-7B
16.15
47.92
4.17
12.5
14B-70B Long- CoT coder reasoning
LLMs
- Qwen---eennn- 14B
8,85
33.33
2.08
0
OpenThinker-32B
9.37
35,42
OldrpicCoder-322B
9.89
35.42
4.17
OpenThinker2-322
14.06
39.58
12.5
4.17
DeepSeck-R
11.98
39.58
4.17
4.17
13.54
43.75
8.33
2.08
rStar-Coder-14B
17.19
47.92
12.50
8.33
Table 4: Ablation curated-only synthetic-only subsets proves the value of each source.
Model
LiveCodeBench
HumanEval
HumanEval+
MBPP
MBPP+
DoepSeck-RI-Distill
37.6
89.6
83.7
78.4
66.7
Synthetic-only subset 7B
46.8
89.6
83.6
86.5
72.7
Seed-only subset 7B
49.7
93.7
88.2
91.0
73.7
rStar-Coder-7B
57.3
95.9
90.8
87.9
74.0
4.3
Ablation Study and Analysis
Quality of seed and synthetic data sources. To evaluate the contribution of each data source, we conduct SFT on scale using only expert-written seed problems and only synthetic problems with their corresponding solutions. As shown in Table 4, our 7B model finetuned on either the curated or synthetic subset significantly outperforms the R1-Distill-7BB model on both reasoning-intensive and general code generation benchmarks, showing both sources are highly effective. While each subset individually underperforms compared to our full dataset, the results indicate that curated and synthetic data provide complementary benefits, and their combination yields the strongest performance.
Effectiveness of mutual verification. Mutual verification enables reliable labeling of test outputs without oracle solutions. To evaluate its effectiveness, we randomly sample 64 expert-written seed problems with oracle solutions and collect all their test inputs (3,150 in total). For each input, we label test outputs using mutual verification: long CoT solutions from QWQ-32B are executed to produce outputs, and majority voting is applied to determine the final label. These outputs are then compared against ground-truth outputs obtained by running the oracle solutions.
As baseline, prompt GPT-4o directly generate input-output pairs, following prior work [31, 12] (see Prompt in Appendix Fig. 10). As shown in Table 5 (Left), mutual verification achieves 96.8% accuracy, while the GPT-4o baseline yields only 12.7%. highlighting the reliability from our method. To further validate scalability, we extend the evaluation to 1,024 problems (27,613 test cases in total), where mutual verification still maintains 92.8 2. accuracy, confirming robustness at larger scales.
Table 5: Ablation study the reliability of synthetic test cases. Left: Our mutual verification ensures the high accuracy test output labeling without oracle solutions. Right: Compared to directly generating test inputs with LLMs, our three-step approach significantly improves input quality.
Method
Accuracy
Method
LiveCodeBench
Average
Easy
Medium
Hard
GPT--4 verification 12.7%
GPT-4o prompting
42.9
87.3
10.6
Mutual verification
96.8%
Three-step input generation
44.6
87.7
56.5
12.6
Effectiveness of test input generation. key component in rStar-Coder is the three-step test input generation method, which produces constraint-satisfying and diverse inputs critical for mutual

---

GPT Generated
rSarrCooder
50
13.8%%
Percentages (96
40
36.51%
30
25.07%
20
11.98%
8.24%
1,58
09%
100
10
102
10a
10
Scale distribution of test inputs
Figure rStar-Coder generates more diverse and larger-scale test inputs, while directly LLM prompting yields smaller and simpler cases.
verification and accurate labeling. To evaluate its impact, we sample 150K synthetic problems and generate inputs using (I) our method and (2) GPT-4o prompting baseline (Appendix Fig. 11). We then apply the same mutual verification process and fine-tune Swen2.5-Coder-7B-mee both resulting datasets for 3 epochs. As shown in Table 5 (Right), method yields much higher results on LiveCodeBench across all difficulty levels, demonstrating the importance diverse and complexity-avee inputs for stronger verification, which further evaluate the next experiment.
on
Ablation on the test input scales. To verify the correctness of code solutions, test inputs need to span wide range of difficulty. Our three-step generation method explicitly controls input scales to achieve this. To evaluate the effectiveness, we sample IK problems from our dataset and plot the distribution of test input scales. For comparison, we also generate inputs using direct GPT-4o prompting (Appendix Fig. II). As shown in Fig. our method produces inputs that evenly cover the range from easy (10") very hard (10"), while GPT-4o generated inputs concentrated in the easier range 110)) with no input scales exceeding 103. This demonstrates the superiority of our method in generating more challenging and diverse test cases.
5 Conclusion
In this work, we present yStar-Coder to construct a large-scale, high-quality dataset for advancing LLMs in competitive code reasoning. By introducing reliable test case generation method, we address the core challenge in generating verified solutions for high-difficllty code problems. Built upon expert-curated seeds, approach enables large-scale synthesis and augmentation. resulting in 418K verified competitive-level problems and 580K long-rcasonnn solutions. Extensive experiments across diffcrent-sizee LLMs (1.5B-14B) 14B) and diverse code reasoning benchmarks demonstrate the superiority of -Star-Coder, achieving performance comparable to QWQ-32B while consistently outperforming existing code reasoning models the same model scale. As future work, plan to further expand the dataset by curating more problems and scaling up synthesis and verification.
Acknowledgement
This work was supported by grant from the National Key R&D Program of China (No. 2024YFB4505701).
References
[1] Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4- reasoning technical report. arXiv preprint -xXii:2504.213// 2025.
[2] Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereassoinn:: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.019433 2025.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis
10

---

with large language models, 2021.
[4] Sahil Chaudhary. Code alpaca: An instructionfoolowiig llama model for code generation. GitHub repository, 2023.
[5] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guam Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv: 10397, 2022.
[6]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hehgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
[7] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
[8] Xinyu Guan, Li Lyna Zhang. Yifei Liu, Ning Shang. Youran Sun, Yi Zhu, Fan Yang, and Mao Yang, rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.045// 2025.
[9] Daya Guo. Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12945 2025.
[10] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint axXiv:2105.00 09938, 2021.
[11] Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi-perspective self-consistency. arXiv preprint arXiv:2309. 17272, 2023.
[12] Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04900 2024.
[13) Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint 2rXiv:2403.02333 2024.
[14] Huggingface. Codeforces wtwp:eeeee-pyttoo-ouusbmiiee
[15] Huggingface. Open rl.
[16] Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng. Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report. 2024.
[17] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Bi-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey,, et al. Openai ol system card. arXiv preprint arXiv:2412.16770 2024.
[18] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2tt::227037774 2024.
11

---

Tan. Deepcoder: A fully open-source http:: [19] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul. Longhui Yu. Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numina- math. o http://hhgggggaee.oeoooooo/o/aan
/ //ggthbb...b.....////gg com/ pdf),
2024.
[20] Rongao Li, Jie Fu, Bo-Wen Zhang. Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312. /4852, 2023.
[21] Rongao Li, Jie Fu, Bo- Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2212. 14852, 2023.
[22] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, 378(6624) 2022.
[23] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[24] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
[25] Roy Huang Ameen Patel Alpay Ariyak Qingyang Wu Xiaoxiang Shi Rachel
Xin Colin Cai Maurice Weber
Ce Zhang Li Erran Li Raluca Ada Popa
Ion Stoica Michael Luo, Sijun
14b
coder at 03-mini level.
Auulll-er---llly--pee---------ee Notion Blog.
2025.
[26] Guilherme Penedo. Lewis Tunstall, Anton Lozhkov, Hynek Kydlicek, Edward Beeching, Loubna Ben Allal, Quentin Gallouedec, Leandro von Werra, Agustín Piqueres Lajarín, and Nathan Habib, Open rl: Update 3, 2025.
[27] Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024.
[28]
OpenThought, Team. Open Thoughts. https://open-thougtts.aa ai, January 2025.
[29] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[30] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with OSS-instruct. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52632-52657. PMLR, 21-27 Jul 2024.
[31] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025.
[32] Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction wning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5140-5153, 2024.
[33] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.017 010.00118. 2025.
12

---

[34] Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthe- sizing algorithmic programs with generated oracle verifiers. Advances in Neural Information Processing Systems, 36:54769-54784, 2023.
[35] Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang. Yinghan Cui, Chao Wang, Junyi Peng. et al. Srpo: A cross-domain imple- mentation of scage-scale reinforcement learning Ilm. arXiv preprint arXiv:2504.1423 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of our paper accurately reflect its contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A.I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose all the needed information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We have included the code in the supplementary materials for transparency and reproducibility. We have open-sourced our dataset https: haggss/ggggiigaec..... co/ da wtttasts/iiirrsooo/tttttrrrrreee
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co Teeii//guddee/ooeeeieeee wwteeebbbissoooooiioooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ht / //nips. cttpccccccbllccccccc cuuubblc///a c/g des/CodeSubmii cetoommssssopPaai siionPooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and inference details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide detailed explanations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the hardware and software information about experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpp::/nnce /tt////neurriii http://rripp...pppiiit/tcccciiie
Answer: [Yes]
Justification:
We follow the Code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license and terms.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
for Please refer our LLM policy (https: http:///errriiecccc////eeeee//// what should or should not be described.
20

---

A Appendix
A.1
Additional Results and Discussions
Table Ablation on scaling dimensions for the 7B model shows that expanding problem diversity is more effective and efficient than only increasing the number of solutions per problem.
Dataset
Unique Problems
Data size
LiveCodeBennc
USACO 2025
Seed Problemsx 1 solutions
37.7K
37.7K
40.8
0.52
Seed Problemsx 8 solutions
37.7K
302K
51.1
3.64
Seed Problemsx 16 solutions
37.7K
603K
54.7
10.41
rStar-Coder dataset
480K
580K
57.3
16.15
Analysis of different scaling dimensions. Most existing code reasoning datasets focus scaling the number of long CoT solutions per problem based on a limited set code problems [2, 26, 28]. In contrast, rStar-Codee emphasizes not only scaling expert-curated solutions but also expanding the number of unique code problems. To compare the effectiveness of these two scaling dimensions, conduct controlled experiment using 37.7K expert-designed problems. We vary the number of long-CoT solutions per problem (1, 8, and 16), while keeping the problem set fixed. The 16-solution setting yields 603K examples, already surpassing the total size of rStar-Coder-580k,
As shown in Table 6, both scaling solution count and problem diversity improve reasoning perfor- mance. However, scaling only the number of solutions yields diminishing returns and becomes less efficient. For example, our 580K dataset, with broader problem coverage, achieves significantly better results reasoning-heee heavy benchmarks like LiveCodeBench and USACO than the 603K dataset derived from scaling solutions alone. Notably, for training efficiency, IStar-Coder-33 580K includes only one solution per synthesized problem, and we plan to scale this further in future work.
Table 7: Ablation on scaling up synthetic data size.
Dataset
LiveCodeBench HumanEval HumanEval+
MBPP
MBPP+
Sced+50K synthetic
54.0
94.4
88.3
91.2
76.5
Seed+ 100K synthetic
55.1
94.2
87.9
90.6
75.4
Seed+200K synthetic
56.2
93.8
88.0
89.2
74.5
Seed+all synthetic (rStar-Coder dataset)
57.3
95.9
90.8
87.9
74.0
Scaling the size of synthetic data. We further analyze the impact of scaling the amount of synthetic data. Specifically, we incrementally add synthesized problems to fixed set of seed problems while keeping other factors constant. As shown in Table reasoning performance on LiveCodeBench improves consistently with larger synthetic datasets, demonstrating the effectiveness and scalability of our data generation pipeline.
Table 8: Core skills evaluated at each tier of USACO, from https://usaco.guide
Difficulty
Core Skills Evaluated
Bronze
simulation, complete search, sorting, greedy
Silver
binary search, comparators, graphs, trees, floodfill, prefix sums, bitwise operators dynamic programming, disjoint set union, spanning trees, Euler tour, combinatorics segment tree, range queries, binary jumping, sweep line, convex hull, flows
Gold
Platinum
USACO 2025 Benchmark. The USA Computing Olympiad (USACO) is a prestigious algorithmic programming competition for high school students in the United States, consisting four difficulty levels: Bronze, Silver, Gold, and Platinum. Each level contains set of challenging problems that test algorithmic thinking and implementation skills, making USACO a valuable benchmark for evaluating the reasoning and problem-solving capabilities of large language models. We adopt the USACO 2025 problem set as a benchmark, which includes 12 problems, with three from each level. Following [27], we evaluate zero-shot pass @ I for each difficulty level and report the average pass @ I across all four levels as the final score.
Discussions and Limitations. Our approach relies on substantial GPT resources for synthesizing code problems and test inputs. Many generated problems are discarded after mutual verification due to being invalid or unsolvable. Additionally, we have observed that some competition problem
21

---

statements do not explicitly provide constraints but instead imply them through context. Since our current method primarily depends on frontier LLMs to interpret the problem statements. it is not yet capable of handling such cases. Furthermore, our pipeline is currently specialized for competitive programming problems, and adapting more open-ended software engineering tasks (e-g.. bug fixing, pull request generation) would likely require additional design and modifications. We leave these challenges to future work.
Broader Impact. rStar-Coder enables the development of stronger code reasoning models by providing lcage-scale, fighdifficulty problems with verifiable test cases. This supports progress in algorithmic reasoning, and Al-assisted programming. However, similar to the other reasoning LLMs, our
could also generate misleading, harmful or hallucinated outputs. We recommend careful consideration of potential misuse during training and deployment, and encourage future work on improving the reliability and safery of code reasoning systems.
A.2 Code Problem Types
According to the input-output format, competitive programming problems can be divided to two types: standard input-output based problems and function-basee problems.
A.2.1
Standard Input-Output Based Problems
Standard Input-Ouiput based problems require the solution code to read from standard input, parse the actual input content, and write results to standard output. As shown in the example problem the first line indicates the number of test cases, and each of the subsequent lines contains two integers representing a single test case.
Code provides an example of evaluation logic for this type of problem, utilizing Python's subprocess module to execute the solution code under time and memory constraints. Since the input is read from standard input, it can be treated as plain string.
To generate input samples of varying scales for these problems, we employ GPT-4o and DeepSeek- V3 to synthesize two utility the prompt shown in 11. The sewroonns-ggeereteeeeeeeeeeeeeee testte_ttst_iputttt function takes the specified problem scale as input and produces a formatted input string, which is subsequently checked for validity by selidate_test_inne
and validate__at seattee_ess_iiutttteeeee
22

---

Figure 6: Standard input-ouput based problem example
Standard Input/Output Problem Example
You have two positive integers a and You can perform two kinds of operations:
La/b]
.
-b b+1
Find the minimum number of operations required to make a = 0. Input Format
The first line contains a single integer (1 < t 100) - the number of test cases.
Each of the following lines contains two integers a and b S b S 10%).
Output Format
For each test case, print single integer: the minimum number of operations required make O.
Example Input
9
1337 1
1
50000000
991026972 997
1234 5678
Example Output
12
3
Solution Code:
= intpin)
Cest_cases=[tuple (map (int, input () split()) for in range (t)
def min_operations (a, b):
if b 1:
operations
else
operations min_ops float('inf')
for increment in range range (100):
current_a =
current_b
increment
current_operations operations increment
while current_a > 8:
current_a //= current_b current_operatioon *=
min_ops min minnmin_ps current.operations)
return min_ops
for a, in test_cases: print (min_operaaiions( (a, b))
23

---

Figure 7: snaectionbasedd problem example
Function-based Problem Example
Given a number (in string form), find the smallest number (without leading zeros) that can be obtained by rearranging its digits.
Example I: Input: "846903"
Output: 304689
Example 2: Input:
"55010" Output: 10055
Starter Code:
class Solution:
def mini mum num_ermberrraeeeeeeeeueueeaaarre (self, s)
# Code here
Listing Evaluation for Standard Input/Output Problems def orr.sollutio(soouutioooccoc input_data):
try:
def set_couLaffinity():
Resource.serrLiitrreeeee RLIMIT (MAX_MEMORY, MAX_MEMORY
process = subprocess (open
""python3'.,-C" solution_code),
stdin=subbroces..PPPS
stabut=ssubrooces..Frr PIPE,
stderr=subrrocess.Prr
text=True,
prexxee_insstt..ppeeffiiiig
stdout, itterraproces..oomniicee communicate
timeout=TIMEOUT
success = process. returncode ==
return stdout if success else stderr success
except:
return "Exception", False
finally:
if process and process. p)ll() is None process, process.kill()
A.2.2
Function-Based Problems with Starter Code
A.2.3 Function-Based Problems
Function- based problems provide a starter function class definition as part of the problem statement. The solution is written by completing the specified function, which is then invoked with deserialized inputs. As shown in the example problem 7, each test input represented as a parameter dictionary, and the expected output is compared against the return value of the function.
Code presents example of evaluation logic for this type of problem. During evaluation, the input string-stored in a serialized format-is is first deserialized into structured data (e.g., integers, arrays, or strings). which is then passed as arguments the solution function through dynamic code execution. The return value is captured and compared to the expected output to determine correctness.
To generate inputs of varying scales for these problems, employ GPT-4o and DeepSeek- V3 to synthesize two utility functions. The generate_ cemerrae_test_iiee test_input function produces a list of arguments in serialized format, which is then deserialized and validated by validate_ted tesiatt__et__ttee teseteittttttttttttteeee to ensure correctness and consistency.
Listing 2: Evaluation for Function- -Based Problems
24

---

from pyext import RuntimeModule method_name in_outs["fn_nami inputs [on loads (line) for line in inputs. split ("
mod = RuntimeModule "rom_string(" ("mmp_ss",,, "", sol) obj mod if "class__Sluutoo"" not in test else mod. Solution method getattr (obj, method_name) output teo (* (xinputs)
25

---

B Prompts
Figure 8: New code problem synthesis prompt.
New Code Problem Synthesis Prompt rStar-Coder
will provide you with programming prohlem along with its solution. Your task progrumming problem based on the original one.
new, transformed
You need to complete the following steps:
understand the original problem and solution, Identify the reasoning steps (e.g., Step I, Step 2, Step 3) and summarize the knowledge points tested the original problem.
Analyze
2. Design a new problem that similar to the original one and can be solved using the same knowledge points.
you reference any conditions descriptions from original problem, rewrite them clearly avoid phrases like "as in the original problem".
Provide example
demonstrate the problem.
Ensure that the complexity of the problem is well-designed by specifying appropriate input constraints.
Your output should follow this format:
## Part I: Original Problem and Solution Analysis
Step I: [Describe the first step of reasoning] [Describe the second step of reasoning]
Step 2:
Knowledge Points: [Summarize knowledge points tested, separated by commas if there are multiple]
## Part 2:
New Problem Problem Description: [Describe the new problem clearly in natural language. Ensure doesn't directly copy from the original problem description, Avoid phrases like "as the original problem"
Input Format: [Specify the input format]
Output Format: |Specify output format]
## Part 3:
Example Test Cases
Input: [Input test case !]
Output: [Expected output for
Input: [Input test case 2]
Output: [Expected output for
Given Problem:
(question)
Given Solution: (solution)
26

---

Figure 9: Test Input Generation Prompt for Standard Input/ Output based Problems
rStar-Coder: Test Input Generation Prompt for Standard Input/Ouuppe uupput based Problems
I will provide you with programming problem description, and your task is generate standardized test input samples using the CaaRon library.
need to complete the following steps:
1. Parse the constraints on the input from the problem description, such as the range of input data,
specific imput constraints, etc.
Write a function generate_ seatte_tett_iie te__ttsssttttttttteeeeeee using the CYaRon library randomly generate test inputs based specified problem size. The function should validate that the parameters fall within the specified constraints. If any parameter is out range, the function should return None. If the parameters are valid, generate a random test input and return an input string (input_string
3. Writc a function requirements specified in the problem description. This includes checking the input data type and constraints parsed step such as range and other conditions. The function should take input_string as input and return boolean (True/Falss) Part 1: Parse Input Constraints
tesp_inputttttttttteeee verify whether the generated test input satisfies the
Specify the input constraints deseribed in the problem.
Part 2: Code for Test Input Generation
import cyaron
cy
def generate
<param2>, ...):
Check if parameters meet constraints if not (<condition! >) or not (ccondition2 >):
return None
Generate input using CYaRon input data [
return "In". join (map (tr. input_data)
Part 3: Code Validate Test Input
def validate_ree _et_etttsttuuuutt (input_string" Validation logic return <boolean>
Given Problem: (question)
Figure 10: Ablation study: Prompt for directly generating test input-output pairs with GPT-4o
Ablation study: Prompt for directly generating test input-output pairs with GPT-4o
I will provide you with a programming problem description, and your task is to generate test inputs und outputs for the problem.
You need to generate 50 test inputs and outputs pair that effectively verify the correctness of the core logic and assess the time complexity of the solution. Ensure that your test cases cover diverse range of problem seales, including edge cases, small inputs, and large inputs that push the problem's 's constraints. Your output should follow this JSON format:
"test_ing inputs".
"idx" 0.
"input_see seriisssting" <test input 0> "output_string": <test output 0>
Given Problem: (question)
27

---

Figure Ablation study: Prompt for directly generating test input with GPT-4o
Ablation study: Prompt directly generating test input with GPT-4o
I will provide you with programming problem description, and your task generate inputs for the problem.
You need to generate test inputs that effectively verify the correctness of the core logic and assess the time complexity of the solution. Ensure that your test cases cover díverse range of problem scales, including edge cases, small inputs, and large inputs that push the problem's constraints. Your output should follow this JSON format:
"tsst inputs" : [
" idx" : 0. "input_string": : <simple test input 0>,
1.
Given Problem: (question)
28


Reviews:
[{"review_id": "uxlxGw6Asg", "reviewer": "Reviewer_6WNE", "summary": "The paper introduces a large scale pipeline to synthesize large scale difficult problem and solution pairs. It includes proper seeding with selection, carefully prompted model for problem synthesis, test generation and solution generation. Many steps include quality based filtering with careful design. The authors produced 580k examples, trained on which models of different sizes attain very good performance in the weight size class.", "questions": "- The final datasets is very large. Would the author be able to provide some taxonomy or other global description of the data so it's easier to understand what is going on.\n- [nit] Dual verification in Table 5 is \"mutual verification\" i believe?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is well-written, easy to follow and intuitive. A full pipeline for generating large set of code reasoning problems are carefully designed and well designed. The final synthetic dataset has been shown to substantially improve coding capabilities of LLMs. The ablation results are also interesting in understanding the utility of the synthesis pipeline stages.\n\nWhile it is obvious that synthesizing coding problems with seeds and solutions with strong proprietary models, e.g. magicoder, for learning is a relatively well-known approach, the specific design in the paper and the strong performance generally outweigh the novelty issue.\n\nThe use of GPT-4o may limit the generality of the method as any other methods that uses frontier models.", "was_revised": true, "final_justification": "As discussed, I am keeping my original score accept, as I find the paper despite not the most novel, quite useful and solid.", "rebuttal": "Thank you for your thoughtful and positive feedback on our work! We sincerely appreciate your insights and your recognition of our contributions. Below, we address your specific comments. \n\n> Would the author be able to provide some taxonomy or other global description of the data so it's easier to understand what is going on.\n\n**Response**: Thank you for the helpful suggestion.  In the final version, we will include a detailed taxonomy in the appendix to provide a clearer global view of our dataset. Below is a preview,  organized along three key dimensions: algorithmic task types, file size breakdown, and solution token length distribution.\n\n|Algorithmic Task Types| \n| :--:   |\n| 2-sat, binary search, bitmasks, brute force, chinese remainder theorem, combinatorics, constructive algorithms, data structures, dfs and similar, divide and conquer, dp, dsu, expression parsing, fft, flows, games, geometry, graph matchings, graphs, greedy, hashing, implementation, interactive, math, matrices, meet-in-the-middle, number theory, probabilities, schedules, shortest paths, sortings, string suffix structures, strings, ternary search, trees, two pointers |\n\n\n|Total file size (compressed)| seed problems&solutions | seed test cases| synthetic problems&solutions|synthetic test cases| \n| :--:   |:--:   |:--:   |:--:   |:--:   |\n|**442.1G** | 12G|167G |6.1G | 257G|\n\n|Token length distribution (Solution Traces)| # of solutions  | percentage|\n| :--:   |:--:   |:--:   |\n|<4k|118593|20.3%|\n|4k-8k|111421|19.1%|\n|8k-16k|233002|39.9%|\n|>16k|120748|20.7%|\n\n\nWe hope this taxonomy provides a helpful overview of the dataset's scope and structure. We are committed to releasing the full dataset to support open research and advance progress in code reasoning.  \n\n> Dual verification in Table 5 is \"mutual verification\" i believe?\n\n**Response**: Thank you for pointing out this. Yes, \"dual verification\" in Table 5 should be consistently referred as \"mutual verification\". We will revise this and carefully proofread the final version for consistency. \n\n> The use of GPT-4o may limit the generality of the method as any other methods that uses frontier models.\n\n**Response**: Thank you for your question. While we used GPT-4o in our main experiments, several key steps in the pipeline (such as new problem synthesis) are model-agnostic. We have verified that smaller open-source models like Qwen2.5-72B-Instruct can also generate high-quality, solvable problems using our synthesis method. This indicates that our approach generalizes beyond GPT-4o and can be reproduced or extended using publicly available alternatives.  We will include this discussion in the revision for further clarity. \n\n\nBelow is an illustrative example comparing a seed problem and a new synthetic problem generated by Qwen2.5-72B-instruct:\n\n>**Original seed problem**:\n\n>We have a grid with H rows and W columns of squares. Snuke is painting these squares in colors 1, 2, ..., N. Here, the following conditions should be satisfied:\n\n>* For each i (1 ≤ i ≤ N), there are exactly a_i squares painted in Color i. Here, a_1 + a_2 + ... + a_N = H W.\n>* For each i (1 ≤ i ≤ N), the squares painted in Color i are 4-connected. That is, every square painted in Color i can be reached from every square painted in Color i by repeatedly traveling to a horizontally or vertically adjacent square painted in Color i.\n\n>Find a way to paint the squares so that the conditions are satisfied. It can be shown that a solution always exists.\n\n>Constraints\n\n>* 1 ≤ H, W ≤ 100\n\n>* 1 ≤ N ≤ H W\n\n>* a_i ≥ 1\n\n>* a_1 + a_2 + ... + a_N = H W\n\n>Input\n\n>Input is given from Standard Input in the following format:\n\n>H W\n\n>N\n\n>a_1 a_2 ... a_N\n\n>Output\n\n>Print one way to paint the squares that satisfies the conditions. Output in the following format:\n\n>c_{1 1} ... c_{1 W}\n\n>:\n\n>c_{H 1} ... c_{H W}\n\n>Here, c_{i j} is the color of the square at the i-th row from the top and j-th column from the left.\n\n>Examples\n\n>Input\n\n>2 2\n\n>3\n\n>2 1 1\n\n\n>Output\n\n>1 1\n\n>2 3\n\n>**New synthetic problem by Qwen2.5-72B-Instruct**:\n\n>You are given a rectangular garden with R rows and C columns. Each cell in the garden can be planted with one of K types of flowers. The goal is to plant the garden such that:\n\n>\tEach type of flower is planted exactly b_1, b_2, ..., b_K times, where the sum of b_1, b_2, ..., b_K equals R * C.\n\n>\tEach type of flower forms a contiguous block, meaning that all cells with the same flower type are connected either horizontally or vertically.\n\n>Your task is to provide a planting scheme for the garden that meets the above conditions. It is guaranteed that a valid solution exists.\n\n>Constraints:\n\n>\t1 ≤ R, C ≤ 100\n\n>\t1 ≤ K ≤ R * C\n\n>\t1 ≤ b_i ≤ R * C\n\n>\tb_1 + b_2 + ... + b_K = R * C\n\n>Input: The input consists of three lines:\n\n>\tThe first line contains two integers R and C, the number of rows and columns in the garden.\n\n>\tThe second line contains an integer K, the number of flower types.\n\n>\tThe third line contains K integers b_1, b_2, ..., b_K, the number of times each flower type should be planted.\n\n>Output: Output R lines, each containing C integers representing the flower type planted in each cell. The flower types are numbered from 1 to K.\n\n\n>Input:\n\n>3 5\n\n>5\n\n>1 2 3 4 5\n\n>Output:\n\n>1 4 4 4 3\n\n>2 5 4 5 3\n\n>2 5 5 5 3"}, {"review_id": "S04il1qUyk", "reviewer": "Reviewer_gnwi", "summary": "The authors present rStar-Coder, a new pipeline and dataset aimed at pushing code-generation and code-reasoning abilities of LLMs. They start from 37.7 K expert-written competitive-programming problems and, through structured prompting that includes oracle solutions, synthesize an additional 380 K solvable tasks—yielding 418 K problems with 580 K long chain-of-thought solutions in total.\n\nA key innovation is a two-stage test-case workflow: (i) a three-step input generator that exposes scale-controlling parameters and validates constraints, and (ii) a mutual-verification scheme where multiple solutions must agree on every output—achieving 96.8 % labeling accuracy.\n\nFine-tuning models on rStar-Coder dramatically boosts performance. For example, the Qwen2.5 7B model improves from 17% to 57.3% pass@1 on LiveCodeBench.", "questions": "1. Beyond the manually curated seed problems, your pipeline still relies on transforming existing tasks to create new ones. Could you clarify why this dependence is necessary? Is the method infeasible for problems that lack any reference solution, and does this imply that fully unconstrained scaling-up of code datasets is out of reach?\n2. Could you explain how your approach could be extended to other domains—­for instance, to SWE-bench bug-fixing tasks or front-end/UI coding challenges—­where the problem format and test-case design differ markedly from competitive-programming questions?\n3. Did you perform deduplication between the training corpus and each evaluation set to prevent train–test leakage? If so, please describe the procedure and its impact on the reported results.\n4. How much of the observed performance gain stems from the presence of chain-of-thought supervision versus the increase in data number? Have you evaluated models trained on the same problems without CoT traces, and if so, what were the outcomes? If CoT is the dominant factor, does generating additional problems without reasoning traces yield any benefit at all?\n5. The `validate_test_input` function itself could contain errors. What safeguards or verification steps do you employ to ensure that this component is correct?", "limitations": "The authors did not include a Limitations section in the paper. My suggestion is to refer to the Questions part above.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The authors present a robust, parameter-controlled input generation toolkit that produces diverse and constraint-valid test cases, going beyond previous approaches that merely rely on LLMs to hallucinate inputs. This allows candidate solutions to be evaluated over a much broader range of scenarios. Their ablation study further validates the effectiveness of this wide-ranging test coverage.\n2. rStar-Coder contributes a large-scale, publicly released corpus—many of its solutions include chain-of-thought reasoning—which is likely to become a valuable resource for both training and evaluating code-reasoning LLMs.\n\nWeaknesses:\n1. Beyond the 37.7 K manually-curated seed problems, additional examples are produced by transforming or perturbing those same seeds. Consequently, scalability still hinges on acquiring fresh, high-quality seeds; without them, the method cannot generate truly novel problem families.\n2. rStar-Coder’s problems are overwhelmingly competitive-programming style (algorithmic puzzles with full I/O control). It is unclear how well the pipeline generalises to more heterogeneous software-engineering tasks—e.g. SWE-bench bug-fixing, class-level code synthesis, or front-end/web-UI challenges—where test-case design and solution formats differ substantially.\n3. Because the seeds include well-known competitive-programming problems, some generated variants may overlap conceptually with public evaluation sets, risking train-test leakage and inflating performance numbers.\n4. Data in rStar-Coder comes with an extensive chain-of-thought trace, yet the paper never ablates (i) dataset size and (ii) presence/absence of CoT supervision separately. As a result, it is hard to know whether the reported gains are driven mainly by the additional reasoning signals or simply by more data. If CoT is the dominant factor, similar improvements might be achievable with a far smaller corpus, weakening the case for synthesising such a large number of competitive-programming problems.\n5. The validate_test_input function may also be incorrect, and there is no process in place to ensure its correctness.", "was_revised": true, "final_justification": "The rebuttal addresses most of my questions. My key concern remains that verifying test correctness with multiple candidate solutions has been covered by prior work, which limits the incremental contribution of this submission. I am keeping my score unchanged.", "rebuttal": "Thank you for your thoughtful and constructive feedback. We sincerely appreciate your recognition of our contributions. Below, we address each of your question. \n\n> Beyond the manually curated seed problems, your pipeline still relies on transforming existing tasks to create new ones. Could you clarify why this dependence is necessary? Is the method infeasible for problems that lack any reference solution, and does this imply that fully unconstrained scaling-up of code datasets is out of reach?\n\n**Response**: Thank you for your insightful question. We address each part below:\n\n1. Why rely on seed problems?\n\nUnlike general coding tasks such as HumanEval/SWEBench, where new prompts can be bootstrapped from real-world codebases, competitive programming problems are uniquely challenging. They require **significant algorithmic reasoning** and **creative thinking** capabilities, which remains challenging even for top reasoning models. \n For example, OpenAI's latest reasoning model placed second in a recent AtCoder contest (a key data source included in our dataset) [1], while human coders still dominate. As such, generating valid, solvable, and non-trivial problems entirely from scratch remains beyond current LLM capabilities.  Seed problems serve as essential scaffolds in this setting.\n\n[1] 7/27, 2025.  Human Coder Edges Out OpenAI in Atcoder Finals, but AI Gains Ground. https://shiawaves.com/english/news/science/ai-news/129745-human-coder-edges-out-openai-in-atcoder-finals-but-ai-gains-ground/\n\n2. Is the method infeasible for problems lacking reference solutions?\n\nAt present, our pipeline requires both the seed problem and the step-by-step solution. The reason is still due the high-difficulty of these competition-level problems. Without solutions to expose the underlying knowledge and reasoning skills being tested, general-purpose models (e.g., GPT-4o) lack the grounding needed to create new problems that preserve both difficulty and solvability.  \n\n3. Does this imply that fully unconstrained scaling-up is out of reach?\n\nWe view this dependence as a current model limitation rather than a fundamental barrier. As reasoning models continue to improve,  future pipelines could bootstrap new problems without explicit references by verifying candidate solutions.\n\n\nWe hope this explanation addresses your questions and provides a clearer understanding of our approach. Thank you again for your valuable feedback, and we welcome any additional questions or suggestions you might have!\n\n\n>rStar-Coder’s problems are overwhelmingly competitive-programming style (algorithmic puzzles with full I/O control). It is unclear how well the pipeline generalises to more heterogeneous software-engineering tasks—e.g. SWE-bench bug-fixing, class-level code synthesis, or front-end/web-UI challenges—where test-case design and solution formats differ substantially.\n\n**Response**: Thank you for the thoughtful question. While our current work targets competitive programming,  core components like input generation, mutual verification are domain-agnostic and can be adapted to broader code-generation tasks. \n\nThat said, competitive programming tasks differ significantly from general software engineering benchmarks. In SWE-bench-style tasks, prompts and test cases often come from natural language description or code bases. In contrast, competitive programming problems are self-contained, require **end-to-end algorithmic reasoning**, and demand **strict I/O control** with a wide range of test case difficulties.  These differences make direct generalization non-trivial and motivate our domain-specific design. \n\nWe do not claim a universal solution. Instead, our goal is to address the underexplored challenge of synthesizing \n high-quality, competition-level problems to advance LLM code reasoning capabilities, where the training data is highly scarce. To our knowledge, rStar-Coder is **the first large-scale pipeline capable of synthesizing such high-difficulty code problems with quality guarantees and diverse test cases**. \n\nWe agree that extending to more diverse domains is an exciting future direction, and leave this as our future works. \n\n\n>Did you perform deduplication between the training corpus and each evaluation set to prevent train–test leakage? If so, please describe the procedure and its impact on the reported results.\n\n**Response**: Yes, as described in Section 3.3,  we carefully performed deduplication between our training samples and all evaluation sets to minimize train-test leakage. \n\nSpecifically, we follow the widely-adopted n-gram matching approach as in prior works [1][2], removing any training problems that share a 16-gram overlap with each evaluation benchmark, The table below summarizes the  number of removed samples based on the n-gram overlap:\n\n|benchmark|# removed samples|\n| :--:   |:--:   |\n|Livecodebench | 216|\n|Humaneval+| 45|\n|MBPP+| 4|\n|USACO 2025| 47|\n|total| 312| \n\nWe will include these numbers in the revision for clarity. \n\n[1] Qwen2.5-Math technical report: toward mathematical expert model via self-improvement\n\n[2] s1: Simple test-time scaling\n\n>Data in rStar-Coder comes with an extensive chain-of-thought trace, yet the paper never ablates (i) dataset size and (ii) presence/absence of CoT supervision separately. As a result, it is hard to know whether the reported gains are driven mainly by the additional reasoning signals or simply by more data. If CoT is the dominant factor, similar improvements might be achievable with a far smaller corpus, weakening the case for synthesising such a large number of competitive-programming problems.\n\n**Response**: Thank you for the insightful question! Final code reasoning performance depends on multiple factors: overall data quality, dataset size, and the quality of CoT traces. While we didn’t include full ablations in the original paper, we now provide new results to ablate these factors.\n\n1) **Data quality > Size**: Despite having fewer examples (580K), rStar-Coder outperforms models trained on larger datasets (736K in OCR, 1M in OpenThinker2), as shown below. This highlights the importance of overall data quality (i.e., problem difficulty, diversity and reasoning quality) over raw scale.\n\n\n|model|dataset size|LiveCodeBench | HumanEval|HumanEval+|MBPP|MBPP+|\n| :--:   |:--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|rStar-Coder-7B| **580K**| **57.3**| **95.9**| **90.8**| **87.9**| **74.0**|\n|OCR-Qwen-7B-Instruct| 736K| 51.3| - | - | -|-|\n|OpenThinker2-7B| 1M | 37.4| 92.7| 87.7| 86.9| 73.9| \n\nThese results suggest that data quality, problem difficulty, and reasoning supervision play a much more significant role than dataset size alone.\n\n2) **Dataset size ablation**: We incrementally added synthetic problems to a fixed seed dataset.  As shown in the table below, reasoning performance on LiveCodeBench improves consistently as dataset size increases: \n\n|dataset| LiveCodeBench| HumanEval|HumanEval+|MBPP|MBPP+|\n| :--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|Seed only | 46.8| 89.6| 83.6| 86.5| 72.7|\n|Seed +50K synthetic |54.0 |94.4 |88.3|**91.2**|**76.5**| \n|Seed +100K synthetic  |55.1 |94.2 |87.9|90.6|75.4| \n|Seed +200K synthetic |56.2 |93.8 |88.0|89.2|74.5| \n|Seed +all synthetic (rStar-Coder dataset) |**57.3**|**95.9** |**90.8**|87.9|74.0| \n\n3) **CoT supervision ablation**: We removed the CoT traces from the seed-only set, retaining only the final Python solutions.  As expected, performance dropped sharply:\n\n|dataset| LiveCodeBench|  HumanEval|HumanEval+|MBPP|MBPP+|\n| :--:   |:--:   |:--:   |:--:   |:--:   |:--:   |\n|Seed only (long cot traces) | 46.8| 89.6| 83.6| 86.5| 72.7|\n|Seed only (no cot traces, python code solution only)| 13.6| 70.1| 64.6| 82.3| 68.3|\n\nThese findings reinforce our design: dataset scale and CoT are both important, but neither alone is sufficient. Strong performance stems from the combination of high-quality CoT reasoning traces, diverse problem coverage and difficulties, and sufficient scale.  \n\nWe will include these results in the revision. Thank you again for your suggestion.\n\n\n>The validate_test_input function may also be incorrect, and there is no process in place to ensure its correctness.\n\n**Response**: Thank you for your insightful comment. We agree that the correctness of the validate_test_input function is crucial. In practice, two types of issues may arise if this function is incorrect:\n\n1. Structurally invalid inputs (e.g., wrong types of malformed structures) are automatically filtered during execution, as they cause runtime errors.\n\n2. Inputs that violdate problem-specific constraints are harder to detect and not fully handled. However, we observe the current implementation to be highly reliable in practice.  As shown in Table 5 (Left) (original paper), across 3150 generated test cases for seed problems, **96.8%** test input-output pairs are accepted by their reference solutions, indicating that the validate_test_input function is effective in practice.  We’ll clarify this and explore stronger input validation mechanisms as part of future work.\n\n>The authors did not include a Limitations section in the paper. My suggestion is to refer to the Questions part above.\n\n**Response**: Thank you. We would like to clarify that we **do include a limitation section** in the appendix (Appendix A, page 13). To improve visibility, we'll move it into the main text. Additionally, based on your feedback, we will expand the discussion to include the following limitations:\n\n1) Our current pipeline is specialized for competitive programming problems. Adapting it to more open-ended software engineering tasks (e.g., bug fixing, pull request generation) would likely require special modifications.\n\n2) Our validate_test_input function may not fully capture all problem-specific constraints, though we see strong empirical results (96.8% pass rate) as shown in Table 5.\n\nWe sincerely appreciate your suggestions and are happy to provide further clarification if needed."}, {"review_id": "ZX4aR7r2kh", "reviewer": "Reviewer_6vKP", "summary": "The paper presents rStar-Coder, a dataset for tuning LLMs for competitive code reasoning. The paper claims a new input-output test generation method to augment existing competitive programming problems and a generated set of competitive programming codes with augmented inputs. The authors use the augmented code set to fine-tune several Qwen-based LLMs that performs better than other existing LLMs tuned for code synthesis tasks.", "questions": "$\\ $", "limitations": "Please see the comment in the weaknesses section.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The problem that the paper tackles is interesting; as competitive problems are difficult for human programmers, they represent a good benchmark for LLMs. Generating correct input/output pairs that can be used in downstream LLM tasks, if done right, can lead to improved reliability of such codes. \n\n\n* The approach uses LLMs to generate two functions for testing the algorithms: input generator and output checker. This construction is more likely to yield structurally valid inputs than directly prompting LLMs to do it. Further this approach can help human developer better understand and inspect the quality of the generated instances. \n\n* The experimental results indicate that the approach can perform better than other LLMs tuned for coding tasks. Some solutions give more solved \"Gold\" level solutions for USACO'25 than even frontier models. What are the problems that were successfully solved and what insights did you obtain from studying the solutions? Discussing these problems and solutions in greater detail in the paper would be helpful for the readers to understand the full extent of the technique. \n\n\nWeaknesses:\n\n* Incorrect use of terminology. The standard definition of verification/verified pertaining to program code in software engineering and formal methods entails that the program will output the correct result for _all_ inputs [1,2]. The paper uses the terms verified and verification to mean that the programs pass _some_ inputs, i.e., a set of generated tests, which is likely to cause confusion. Ensuring the program correctness on all inputs is a significantly more difficult problem than the one tackled here. Similarly, the term mutual verification is similarly misleading. As it corresponds to majority voting between multiple tests, it should be simply called that way. Further, Section 3.2 mixes the terms 'verify' and 'validate', which are technically separate. Getting clear definitions and aligning with the standard ones is necessary for effectively communicating the research and putting the contributions of the paper in the proper context. \n\n* The assumption that the solution of majority of generated samples is the correct solution is not fully confirmed by the paper and is still distant from the statement the paper advertises that the solutions in your dataset are verified. While the corresponding ablation study in the paper is a good starting point, and 95+% agreement is reasonably high probability for those sampled seed programs. However, confirming this hypothesis should involve both more sample seed programs (including reporting per-seed variances) and a qualitative study of generated solutions. A point of concern is that while checking independently obtained solutions could work when the sources of implementation are independent, with LLMs we cannot relay on that property, due to common training code bases so hidden dependencies cannot be excluded. \n\n* The paper's limitations section should include the detailed discussion of which tasks the proposed approach/dataset is good for and for which it is not. Currently 'Limitations' are represented as a short paragraph in the appendix, but should be clearly marked in the main paper. \n\n* Missing related work. One of the key contributions of this work is an approach to generate tests for augmenting LLM code generation. However, the paper does not cite any of the many recent techniques for test generation in software engineering. I suggest that the authors look into paper published at recent ICSE, FSE, ISSTA conferences and/or recent surveys e.g., [3,4]. Some other related techniques that use LLMs to output complex-input generators are explored in https://arxiv.org/pdf/2501.19282 .\n\n* Writing can be improved. Proof-read the paper to catch common spelling errors (e.g., \"totoal\") and stylistic errors (e.g., \"this\" without following clause). \n\n\n\n[1] https://en.wikipedia.org/wiki/Formal_verification\n\n[2] Search-based Program Synthesis. 2018. https://dl.acm.org/doi/10.1145/3208071\n\n[3] Software testing with large language models: Survey, landscape, and vision, 2024.\n\n[4] Large language models for software engineering: A systematic literature review, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "**General Response to Reviewer 6vKP and Clarifications**: \n\nWe sincerely thank the reviewer for the detailed and thoughtful feedback. Your comments, especially from a software engineering and formal methods perspective, helped us clarify our contributions and improve the paper's precision and rigor.\n\n**1. Clarifications on the use of verification terminology**: We acknowledge the reviewer’s concern regarding our use of the term “verification” and agree on the importance of aligning with established terminology in SE and formal methods, where “verification” often implies exhaustive, formal correctness across all inputs, which is a rigorous standard that we do not claim to meet.\n\n\nIn our work, \"verification\" refers to an empirical process that mimics how human beings validate correctness via execution and consensus, rather than formal proof. This usage follows common practice in the LLM literature [1,2,3], where \"verification\" often denotes execution-based validation mechanisms that offer strong empirical reliability. Our intent is not to suggest formal guarantees, but rather to describe a practical and effective validation framework for long-CoT solutions and their corresponding test cases for competitive programming tasks. In the context of LLM code reasoning, this empirical rigor already marks a significant advance over prior work[4,5]. To avoid confusion, we will explicitly clarify this distinction in the revision.\n\n\n[1] DeepCoder: A fully open-source 14B coder at o3-mini level, https://www.together.ai/blog/deepcoder\n\n\n[2] OpenAI. Introducing SWE-Bench verified we're releasing a human-validated subset of swe-bench that more reliably evaluates AI models' ability to solve real-world software issues, 2024d. https://openai.com/index/introducing-swe-bench-verified/\n\n[3] DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\n\n[4] ACL 2025, Best Paper Aware at DataWorld @ICML 2025, KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding\n\n[5] Huggingface, https://huggingface.co/blog/open-r1/update-3\n\n\n**2. Core contributions of our work**: We would like to reiterate the core contributions of our work: we present a **scalable, high-quality dataset generation pipeline** for constructing **high-difficulty competition-level** code reasoning datasets, a resource that is currently **highly scarce** and difficult to scale through human curation. \n\nUnlike general code generation tasks, competitive programming problems are fundamentally more challenging. They require multi-step algorithmic reasoning, careful planning, and precise validation against diverse edge cases. Yet, most existing datasets in this domain remain limited in both scale and quality, particularly due to the lack of **reliable and diverse test cases** that are essential for constructing high-quality training dataset.  \n\nTo address this, we introduce a pipeline that: (i) synthesizes new and solvable competition-level problems; (ii) generates parameter-controlled test inputs with varying difficulty; (iii) reliably label the test outputs and the long-cot solutions through mutual agreenment among multiple reasoning traces. \n\nOur goal is not just scaling data, but about **reliably scaling hard data**. This has been positively recognized by the other reviewers, such as **\"The authors present a robust, parameter-controlled input generation toolkit that produces diverse and constraint-valid test cases, going beyond previous approaches that merely rely on LLMs to hallucinate inputs\"**, **\"rStar-Coder contributes a large-scale, publicly released corpus, which is likely to become a valuable resource for both training and evaluating code-reasoning LLMs\"**, **\"The paper proposes innovative test case generation using GPT-4o and cross-validation for reliability\"**, \n**\"The paper demonstrates that smaller models can outperform larger ones, proving that data quality over quantity matters\"**.\n\n**3. Clarification of possible misunderstandings**: We would also like to clarify possible misunderstanding reflected in the review. \n\n1) Code reasoning ≠ code synthesis. Our work targets code reasoning for high-difficulty competitive problems—not generic synthesis. Unlike conventional coding problems, code reasoning is particularly important for competitive coding problem. As shown in this paper, general code synthesis /LLM tuned for coding tasks is much less effective in this code domain, which is of high difficulty. Framing our contribution as \"tuning for synthesis\" misses the core challenge and motivation.\n\n2) Not general test case augmentation. We also wish to correct the impression that our focus is not general test case augmentation, but a more specific, thus also more effective,  test case synthesis targeting code reasoning  validation. This differs from augmenting general downstram tasks with hallucinated inputs.\n\n3) More than a benchmark. The reviewer refers to competitive problems as “a good benchmark,” which slightly misrepresents our contribution. Our goal is not just to benchmark existing models, but to enable training and evaluation on hard, verifiable, and diverse tasks through a generative solution pipeline.\n\n\n\nWe hope our clarifications have provided a clearer explanation of the key insights and contributions of our approach. Given the novelty and demonstrated effectiveness of this approach, we believe in the value of our work. We welcome any further questions or suggestions you may have. Thank you again for your time and consideration.\n\nBelow, we address your specific comments. \n\n\n> The assumption that the solution of majority of generated samples is the correct solution is not fully confirmed by the paper and is still distant from the statement the paper advertises that the solutions in your dataset are verified...\n\n**Response**: We appreciate your insightful comment. To further validate the effectiveness of our mutual verification strategy,  we conduct an additional experiment. We sample 1024 seed problems and use our mutual verification to label the test outputs for 27613 test inputs. We then compare the labeled test output against ground-truth test output to compute the correctness ratio. As shown in the table below, we observe an even higher correctness ratio of 97.1% across these samples, further demonstrating the reliability of our approach.\n\n\n|exp| #seed problems|  #test inputs|correctness of test outputs|\n| :--:   |:--:   |:--:   |:--:   |\n|Original ablation | 64| 3150| 96.8%|\n|**Additional ablation**|**1024** |**27613** |**97.1%**|\n\nRegarding the concern about the independence of LLM-generated solutions: while outputs may share dependencies due to overlapping training data, we believe that incorrect solutions tend to diverge in their errors rather than consistently converge. This makes it unlikely for multiple erroneous traces to produce identical outputs across diverse test cases. Consequently, majority agreement remains a reliable empirical signal of correctness, even if not formally independent.\n\nWe believe this additional evidence further justifies mutual verification as a practical and effective mechanism for quality control, while recognizing that it provides empirical rather than formal guarantees.\n\n>The paper's limitations section should include the detailed discussion of which tasks the proposed approach/dataset is good for and for which it is not. Currently 'Limitations' are represented as a short paragraph in the appendix, but should be clearly marked in the main paper.\n\n**Response**: Thank you for the suggestion. We will move the discussion of limitations from the appendix into the main paper and expand it as recommended. Specifically, we will clarify that our proposed method and dataset are primarily designed for competitive programming problems, with the goal of advancing LLM reasoning capabilities. They are not intended for general software engineering tasks such as debugging, refactoring, or maintenance of real-world codebases. We appreciate the reviewer’s suggestion to make this distinction clearer.\n\n>Missing related work. the paper does not cite any of the many recent techniques for test generation in software engineering. I suggest that the authors look into paper published at recent ICSE, FSE, ISSTA conferences and/or recent surveys e.g., [3,4]. Some other related techniques that use LLMs to output complex-input generators are explored in https://arxiv.org/pdf/2501.19282 .\n\n**Response**: Thank you for the suggestion. We appreciate the reviewer’s point about better connecting our work to the broader software engineering (SE) literature on test generation. In our original submission, we focused primarily on LLM-driven data augmentation and acknowledge the omission of relevant SE research on automated test generation.\n\nWe will revise the related work section to include representative papers from ICSE, FSE, ISSTA, and the suggested work [https://arxiv.org/pdf/2501.19282], which explores LLMs for input generation.\n\nWe also wish to clarify a key distinction: our goal is not general-purpose test generation, but scalable, parameter-controlled test input synthesis tightly coupled with long-chain-of-thought solutions for competitive programming tasks. Unlike traditional SE settings, we do not assume access to complete program specs or ground-truth oracles. Our contribution lies in adapting test generation to support reliable long-cot supervision in high-difficulty code reasoning, where standard SE assumptions often break down.\n\nWe thank the reviewer again and will incorporate these additions into the final version.\n\nFinally, we hope these responses address your concerns and clarify any confusion, and we will incorporate them in the revisions. Thank you again for your valuable feedback and suggestions, and we kindly ask you to consider re-evaluating our work."}, {"review_id": "4ajANd5aCz", "reviewer": "Reviewer_H9E1", "summary": "The paper addresses the scarcity of high-difficulty, verified code reasoning datasets for LLMs. It constructs 418K competitive problems and 580K long-reasoning solutions with a three-step approach: synthesizing problems, generating diverse test cases via GPT-4o, mutual verification, and augmenting solutions. Experiments show Qwen models fine-tuned on rStar-Coder outperform larger baselines on LiveCodeBench, USACO, and general code tasks, proving data quality drives reasoning gains.", "questions": "1. Is it reasonable to use 16-gram deduplication? Do you have real examples of overlapping 15-grams?\n2. The core contribution of the paper seems to be a combination of WizardCoder and CodeT, with a larger volume. Could you explain the difference between the proposed method and WizardCoder / CodeT?", "limitations": "This article mainly discusses how to construct a dataset of long-term CoT in code, which seems to be more suitable for the NeurIPS D&B Track.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#### Strengths\n1. The paper proposes a large-scale, high-quality dataset with 418K coding problems and rigorous validation.\n2. The paper proposes innovative test case generation using GPT-4o and cross-validation for reliability.\n3. The paper demonstrates that smaller models can outperform larger ones, proving that data quality over quantity matters.\n\n#### Weaknesses\n1. As a paper with a dataset as its main contribution, the authors only uploaded some sample data.\n2. The method relies heavily on GPT resources for problem synthesis, increasing costs and dependency.\n3. Mutual verification will filter out the most difficult questions because the model can not generate the correct solution or test case.", "was_revised": true, "final_justification": "My only concern is that 16-gram deduplication may introduce unwanted contamination. Although the author frames this as a trade-off, I believe the deduplication of the test set should be enforced with the utmost strictness rather than treated as a trade-off.\n\nOverall, I recognize that this paper makes a valuable contribution to the community, and I am therefore maintaining my weak-positive score.", "rebuttal": "We sincerely appreciate your thoughtful review and your recognition of our dataset construction, test case design, and model performance results. Below we address the specific questions and comments:\n\n> As a paper with a dataset as its main contribution, the authors only uploaded some sample data.\n\n**Response**: Thank you for your question. Due to the Neurips anonymity policy, we are unable to share access to the full dataset during the rebuttal phase.  However, the complete dataset has already been carefully prepared. It has been used internally and externally in ongoing experiments, demonstrating its quality and usability in practice. In addition, we will include a detailed taxonomy in the appendix to provide a clearer global view of our dataset. Below is a preview,  organized along three key dimensions: algorithmic task types, file size breakdown, and solution token length distribution.\n\n|Algorithmic Task Types| \n| :--:   |\n| 2-sat, binary search, bitmasks, brute force, chinese remainder theorem, combinatorics, constructive algorithms, data structures, dfs and similar, divide and conquer, dp, dsu, expression parsing, fft, flows, games, geometry, graph matchings, graphs, greedy, hashing, implementation, interactive, math, matrices, meet-in-the-middle, number theory, probabilities, schedules, shortest paths, sortings, string suffix structures, strings, ternary search, trees, two pointers |\n\n\n|Total file size (compressed)| seed problems&solutions | seed test cases| synthetic problems&solutions|synthetic test cases| \n| :--:   |:--:   |:--:   |:--:   |:--:   |\n|**442.1G** | 12G|167G |6.1G | 257G|\n\n|Token length distribution (Solution Traces)| # of solutions  | percentage|\n| :--:   |:--:   |:--:   |\n|<4k|118593|20.3%|\n|4k-8k|111421|19.1%|\n|8k-16k|233002|39.9%|\n|>16k|120748|20.7%|\n\n\n We are fully committed to releasing the complete dataset to advance open-source code reasoning research and benefit the broader community. \n\n>The method relies heavily on GPT resources for problem synthesis, increasing costs and dependency.\n\n**Response**: Thank you for the comment. As acknowledged in Appendix A (page 13, discussions and limitations section), our method currently relies on GPT resources for problem and test case synthesis. However, we believe this is a cost-effective and scable alternative to fully human-curated pipelines, which are often prohibitively expensive and difficult to scale. Looking ahead, we are optimistic about incorporating smaller open-source models as their reasoning capabilities continue to improve.  \n\n>Mutual verification will filter out the most difficult questions because the model can not generate the correct solution or test case.\n\n**Response**: Thank you for the insightful comment. Indeed, mutual verification may filter out some of the most difficult problems when the model fails to generate consistent correct solutions. This reflects a current trade-off between dataset difficulty and correctness: for problems with low consistency ratios, it is challenging to reliably determine whether the issue stems from an ill-posed or unsolvable problem, or simply from high difficulty. To ensure data quality, we conservatively filter out such cases.\n\nWe view this as an opportunity for future improvement. As reasoning model capabilities continue to advance, mutual verification can become more tolerant or be complemented by stronger coding agents. These improvements will help retain harder problems with greater confidence and further advance high-difficulty code reasoning.  \n\n> Is it reasonable to use 16-gram deduplication? Do you have real examples of overlapping 15-grams?\n\n**Response**: Thank you for the thoughtful question. We take data contamination and train-test leakage very seriously and made careful design choices to minimize it. \n\nWhile many existing works (s1 for math [1], qwen2.5-coder for general code tasks [2]) use smaller thresholds like 8-gram or 10-gram for deduplication, we found that such settings lead to a high false positive rate in competitive programming problems. Many problems naturally share common patterns, especially in I/O description or problem statement. For example, common false positive cases like **\"input the input is given in the following\"**, **\"the sum of n overall all test cases\"**, **\"be considered correct if its absolute or relative\"**, **\"your task is to find the number of\"**.  As a result, short n-gram match often do not indicate true duplication.  We will include these concrete examples in the final version to illustrate this issue.\n\nAs a result, we empirically tuned several n-gram thresholds and found that 16-gram is a good balance. Based on your suggestion, we also tested 15-gram filtering, which slightly increased the number of flagged overlaps from 312 to 464 out of 580K samples.  This suggests that our current deduplication is already conservative, and the overall duplication rate remains negligible (<0.1%).\n\n|benchmark|removed samples under 16-gram| removed samples under 15-gram|\n| :--:   |:--:   |:--:   |\n|Livecodebench | 216| 336|\n|Humaneval+| 45| 49|\n|MBPP+| 4| 4|\n|USACO 2025| 47| 75|\n|total| 312| 464| \n\nWe appreciate the suggestion and will include a more detailed analysis of n-gram deduplication strategies in the final version.\n\n> The core contribution of the paper seems to be a combination of WizardCoder and CodeT, with a larger volume. Could you explain the difference between the proposed method and WizardCoder / CodeT?\n\n**Response**: Thank you for your question. While our approach also uses LLMs for dataset synthesis, it differs fundamentally from WizardCoder and CodeT in three key aspects: **task domain and difficulty**, **synthesis methodology**, and **quality control mechanisms**. \n\nWizardCoder focuses on general-purpose coding prompts, where each prompt can typically be solved with simple function-level code. In contrast, we target **competition-level programming problems**, a much more challenging domain that requires significant algorithmic reasoning and diverse test case coverage. \nThese problems are significantly harder to generate, and cannot be created via unconstrainted prompt mutation alone.  Therefore, we use a solution-conditioned generation approach: given a seed problem and its step-by-step solution, the model produces new but solvable problems of comparable difficulty. This ensures both validity and challenge, which is essential for advancing reasoning skills.\n\nCodeT directly generates test input-output pairs using LLMs. While efficient, this assumes the LLM can fully solve the problem to produce both valid inputs and correct outputs, which is unrealistic for complex algorithmic tasks and prone to hallucinations.\n\n\nOur method **explicitly decouples** input and output generation. For test inputs, we use a **three-step generation** process: the model first generates *generate_test_input* and *validate_test_input* functions, then generates inputs by varying scale parameters, this yields inputs with diverse difficulty and guaranteed validity. For test outputs, we **do not rely on the LLM to generate them directly**. Instead, we apply mutual verification: a stronger reasoning model produces multiple candidate solutions, and outputs are retained only if they achieve agreement through execution-based majority voting.\n\nThese mechanisms ensure the resulting dataset is not only larger but qualitatively superior: solvable, challenging, and robustly validated. \n\nWe hope this clarifies that our work is not merely a scaled-up combination of WizardCoder and CodeT, but a new pipeline designed specifically for high-difficulty code reasoning, with clear methodological and empircal advances.\n\n> This article mainly discusses how to construct a dataset of long-term CoT in code, which seems to be more suitable for the NeurIPS D&B Track.\n\n **Response**: Thank you for the suggestion. While our work introduces a high-quality dataset, we believe the core contribution lies in the **novel methodology** for synthesizing and verifying complex code reasoning problems at scale. Specifically, we address two fundamental challenges in competitive code reasoning:\n\n(1) how to synthesize new **solvable problems** that remain challenging and non-trivial, and\n\n(2) how to construct **diverse and reliable test cases** that cover varying input complexities while ensuring correctness.\n\nTo tackle these, we introduce a solution-conditioned problem synthesis method, a three-step test input generation pipeline, and a mutual verification mechanism based on majority voting across model-generated solutions. These innovations enable the construction of high-quality problems and test cases that go well beyond the capabilities of prior approaches such as WizardCoder and CodeT. \n\nFurthermore, our focus is on **code reasoning**, a high-level cognitive capability that remains  **underexplored yet important** for advancing LLM intelligence. This makes rStar-Coder a valuable resources for LLM open-research community. \n\nOur empirical results show that training on this dataset **substantially improves LLM code reasoning ability** across multiple strong baseline models with much smaller model sizes. This highlights not only the utility of the dataset, but also the broader significance of our methodology for advancing LLM reasoning capabilities.\n\nWe believe this integration of **methodological innovation**, **core research relevance**, and **impact on reasoning performance** makes our work well aligned with the goals of the main NeurIPS track."}]

Author rebuttal:

Meta-review: Summary of the paper: This paper presents rStar-Coder, an end-to-end pipeline that converts 37.7 K expert-written competitive-programming seeds into 418 K high-difficulty code-reasoning problems together with 580 K long chain-of-thought solutions.  It claims that (i) a two-stage, parameter-controlled test-case generator plus a mutual-verification scheme attains 96.8 % labeling accuracy without human intervention; (ii) fine-tuning even small open-weight models (e.g., 7 B-parameter Qwen2.5) on this data yields state-of-the-art or better results on LiveCodeBench, USACO, and general code tasks, sometimes surpassing frontier models; and (iii) ablations show that each component—seed selection, problem synthesis, test-case diversity, and mutual verification—contributes measurably to final performance, demonstrating that data quality can trump model scale.

Strengths of the paper:
1. Large-scale, validated resources: 418 K problems and 580 K verified solutions constitute a new public resource for training and evaluating code-reasoning LLMs.
2. Methodological novelty: a two-stage test-generation toolkit that exposes scale-controlling parameters plus a cross-checking mutual-verification protocol achieves >96 % accuracy, going beyond prior LLM-based “hallucinate-and-pray” approaches. Each pipeline stage is evaluated, giving the community clear guidance on what matters.
3. Strong empirical gains: fine-tuned 7B models jump from 17 % to 57 % pass@1 on LiveCodeBench and solve more USACO Gold problems than some frontier models, providing convincing evidence that data quality drives reasoning.
4. Clarity and reproducibility: the paper is well-written, the pipeline is intuitive, and the dataset will be released (Please do so after acceptance, I found sampled data are attached in the supplementary materials).

Weaknesses of the paper: Following the rebuttal, the authors have convincingly addressed the concerns raised by the three reviewers. Reviewer 6vKP’s remaining criticisms center on presentation—specifically, the use of misleading or inaccurate terminology, an insufficient discussion of limitations, and overlooked related work. The newly added experiments appear to resolve the concern about the unverified majority-vote assumption, and these presentation issues do not, in my view, compromise the paper’s core contribution. I therefore recommend that the authors incorporate the experimental results and clarifications from the rebuttal into the camera-ready version.

Reasons for the decision: This paper delivers both a significant resource (a 418 K-problem dataset with validated solutions) and a technically solid, well-evaluated pipeline for creating it.  The empirical improvements are significant and seem to be reproducible, and the ablations convincingly attribute these gains to design choices rather than sheer scale or proprietary tricks. Three reviewers appreciate the novelty, effectiveness, and comprehensive experiments of our approach, acknowledging its well-founded methodology and strong performance. Given the strong reviews from them and the clear community value, I think this paper meets the bar for acceptance.
Decision: Accept (poster)