Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Authors: Andrew Zhao
Yiran Wu
Yang Yue
Tong Wu
Quentin Xu
Yang Yue
Matthieu Lin
Shenzhi Wang
Qingyun Wu
Zilong Zheng
Gao Huang
Venue: NeurIPS 2025 spotlight ()
Keywords: reasoning
language model
reinforcement learning
self-play
LLM
Primary area: applications
TL;DR: self-play reasoning RL with no data can achieve SOTA against RL models trained with human data

Abstract: Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from rule-based outcome rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external human or distillation data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability. AZR uses a code executor to both validate self-proposed code reasoning tasks and verify answers, serving as an unified source of verifiable feedback to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.

Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Andrew Zhao'
Yiran Wu
Yang Yue!
Tong Wu²
Quentin Xu Matthieu Lin'
Shenzhi Wang!
Qingyun Wu
Zilong Zheng"* Gao Huang!
Tsinghua University BBIAA 3ennn State University
zqc21dmails tsinghua. edu. en
yiran. wu@psu. edu
zizhengobigai. ai gaohuangotsinghua. edu. cn
Abstract
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from rule-based outcome rewards. Recent RLVR works that operate under the zero setfing avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-qualite scalability of relying on human supervision, a challenge already evident the domain of language model pretraining. Furthermore, in hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for superintelligent system. To address these concerns, we propose new RLVR paradigm called Absolute Zero, in which a single model learns propose tasks that maximize its learning progress and improves reasoning by solving them, without relying on any external human or distillation data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), system that self-evolves its training curriculum and reasoning ability. AZR uses a code executor both validate self-pro self-proposed code reasoning tasks and verify answers, serving as an unified source of verifiable feedback to guide open open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain tuman-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and compatible with various model classes.
comman-produced examples raises concerns about the long-tern ing-termm
Introduction
Large language models (LLMs) have recently achieved remarkable improvements in reasoning
capabilities by employing Reinforcement Learning with Verifiable Rewards (RLVR) [37]. Unlike methods that explicitly imitate intermediate reasoning steps, RLVR uses only outcome- based feedback, enabling large-scale reinforcement learning over vast task datasets [22, 75, 33, 52, 51, 81]. A particularly compelling variant is the "zero" " RLVR paradigm [22]. which forgoes any cold-start distillation data, using neither coman-generated nor Al-generated reasoning traces, and applies RLVR directly on the base model with task rewards. However, these methods still depend heavily on expertly curated distributions of reasoning question-anse answer pairs, which raises serious concerns about their long-term scalability [76]. As reasoning models continue to advance, the effort required to construct large-scale, high-quality datasets may soon become unsustainable [97]. A similar scalability bottleneck has already been identified in the domain of LLM pretraining [73]. Furthermore, as Al
*Corresoonding author.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Data Comparison
Math Domain
Coding Domain
Overall Performance
AZe
Data bpss
Nath
Code
0.1-
xxpe.....
3.2
092
0:30
Base model
Prev: SOTA Model
Performance Duiling RL.
201
4031
603
0350
AZK Rio Modal
162
230
0.50
100
100
100
300
150
Curated Data Size
Training Steps
Figure Absolute Zero Reasoner (AZR) achieves state-of-the-art performance with ZERO DATA. Without relying on any gold labels or demmnndeefined queries, Absolute Zero Reasoner trained using our proposed self-play approach demonstrates impressive general reasoning capabilities improvements in both math and coding, despite operating entirely out-of-distribution. Remarkably, AZR surpasses models trained on tens of thousands of expert-labeled in-domain examples in the combined average score across both domains, and also reaches SOTA in the coding domain.
systems continue to evolve and potentially exceed human intellect, an exclusive dependence on human- designed tasks risks imposing constraints their capacity for autonomous transcendence [31]. This underscores the need for a new paradigm that begins to explore possibilities beyond the constraints of human-designed tasks and prepares for future which AI systems may surpass human intelligence.
To this end, we propose "Absolute Zero", a new paradigm for reasoning models in which the model simultaneously learns to define tasks that maximize learnability and to solve them effectively, enabling self-evolution through self-play without relying on external data. contrast to prior self play methods that are limited to narrow domains, fixed funetionalities, or learned reward models that are prone to hacking [68, 5, 7J. the Absolute Zero paradigm is designed to operate in open-ended settings while remaining grounded in real environment. It relies feedback from the environment as a verifiable source of reward, mirroring how humans learn and reason through interaction with the world, and helps prevent issues such as hacking with neural reward models [31]. Similar to AlphaZero [68], which improves through self-play, our proposed paradigm requires no human supervision and learns entirely through self-interaction. We believe the Absolute Zero paradigm represents a promising step toward enabling large language models to autonomously achieve superhuman reasoning capabilities.
Building this new reasoning paradigm, we introduce the Absolute Zero Reasoner (AZR), which proposes and solves code reasoning tasks. We cast code executor as an open open-ended yet grounded environment, sufficient to both validate task integrity and also provide verifiable feedback for stable training. We let AZR construct tasks that require reasoning and inference about a specific element in program, input, or output triplet, corresponding to three complementary modes of reasoning: induction, abduction, and deduction. We train the entire system end-to-end with a newly proposed reinforcement learning advantage estimator tailored to the multitask nature of the proposed approach.
Despite being trained entirely without any in-distribution data, AZR demonstrates remarkable capabilities across diverse general reasoning tasks in math and coding. In mathematics, AZR achieves competitive performance compared to zero reasoner models explicitly fine-tuned with domain-specifee supervision. In coding tasks, AZR establishes new state-of-the-arr performance, surpassing models specifically trained with curated code datasets using RLVR. Furthermore, AZR outperforms all previous models by an average of 1.8 absolute points compared models trained in the "zero" setting using in-domain data. These surprising results highlight that general reasoning skills can emerge without human-curated domain targeted data, positioning Absolute Zero an promising research direction and AZR as first effective instantiation. See Section 1 for more interesting findings.
2
The Absolute Zero Paradigm
2.1
Preliminaries
Supervised Fine-Tuning (SFT). SFT requires the datasets of rask-rationale-answer demonstrations D = [(a, e* y² where is the query, e* is the gold chain-of-thought (CoT) and y" is the gold answer, all provided by human experts or superior Al models. The model trains to imitate the reference responses to minimize the conditional negative likelihod [55]:

---

(0) E (ar,et w/ yoD log To (c*, y² x))
(I)
At the frontier level, the absence of stronger models for distillation and the poor scalability of expert human labeling have led researchers to explore RL as a means to enhance model reasoning.
Reinforcement Learning with Verifiable Rewards (RLVR). To move beyond the limits of pure imitation, RLVR only requires a dataset of task and answer D = {zz, y²)}. without labeled rationale. RLVR allows the model to generate its own CoT and calculate verifiable reward with the golden answer r(y, y²). However, the learning task distribution D, with its set of queries and gold answers still labeled by human experts. The trainable policy Ty is optimized maximize expected reward:
JRIVR (0) = )<D,
>-D, (c.y)-mll [=)r(y,,"))
(2)
In summary, both SFT and RLVR still rely on human-curated datasets of either queries, demonstrations, or answers, which limit scalability. The Absolute Zero paradigm removes this dependency by allowing the model to generate, solve, and learn from its own interactions with the environment by self- self-play.
2.2 Absolute Zero
We propose the Absolute Zero (AZ) paradigm, where during training, the model simultancously proposes tasks, solves them, and learns from both stages. No external data is required and the model learns entirely through self- self-play and experience, aided by some environment. To make the Absolute Zero setting concrete, we now define how one model can act both as the proposer and solver role. Let To be our parameterized language model, it is used play two roles, proposer =a variable proppsee (- which will then be validated and used to construct valid reasoning task together ng with the environment e: (2,, v²) ~ fe(-)T), where æ is the task query and y" is the gold label. Then the solver produces answer y ~ T srahe( I)). Each proposed task is scored by learnability reward r propose (T, No), which captures the expected improvement in after training on the proposed task т. Moreover, the same policy also receives a solution reward y-oolv (y, y²) for its answer to the task query with the environment again serving as the verifier. A nonnegative coefficient À balances the trade-off between exploring new, learnable tasks and improving the model's reasoning and problem-solving abilities. We formally define the absolute zero setting's ' objective as follows:
propuse and solver sollve during training. The proposer first samples proposed task conditioned
T(0) = max E Ex-p(s)
(1a)
Ar (p) + E, Eyweghh [res (g,
(3)
Notice we shift the burden of scaling data away from human experts and onto the proposer policy Th propose and the environment e. These two roles both responsible for defining/evolving the learning task distribution, validating proposed tasks, and providing grounded feedback that supports stable and self-sustainable training. When proposing, acts conditional variable that seeds generation of tasks. Practically, z can be instantiated by sampling several past (task, answer) pairs from a continually updated buffer, yet there no specific implementation tied to the paradigm. To guide the proposing process, we use learnability reward prropose (T, To). which measures how much the model is expected to improve by solving proposed task т. Moreover, the solver reward psole(y. v²) evaluates the correctness of the model's output. Together, these two signals guide the model to propose tasks that are both challenging and learnable, while also enhancing its reasoning abilities, ultimately enabling continuous improvement through self-play, see Figures 13 and 14 for AZ. framework illustrations.
3 Absolute Zero Reasoner
In this section, we present Absolute Zero Reasoner (AZR) as the first attempt to embrace the Absolute Zero paradigm. In AZR, an unified LLM is jointly trained as both proposer and solver- geverratng challenging tasks to expand its reasoning curriculum and solving them to enhance its own capabili- ties (Section 3.1.. Within this self-play training paradigm, the model learns from three distinct type of
3

---

PROPOSE
Learnability Reward
Construct Estimate
Task Types
Absolute Zero Reasoner
Self-play
Abduction:
Deduction:
?
Joint Update
Induction:
SOLVE
Accuracy Reward
Verify
model input/output
model reward
rogram,
Input,
Output)
Figure 2: Absolute Zero Reasoner (AZR) Training Overview. At every iteration, AZR first PROPOSES a batch tasks, conditioned on past self-generated triplets stored in a buffer and a particular task type: abduction, deduction, induction (Section 3.2). From these generated tasks, Python is used to filter and construct valid code-based reasoning questions. learnability reward 'propose is also calculated for each proposed task defined Equation (4). AZR then SOLVES the batch of reasoning questions. Python is used again to verify the generated responses and compute the accuracy reward Tsole as described in Equation (5). Finally, the Absolute Zero Reasoner is jointly updated using both Ppropose and Tsolve across all three task types, using TRR+ (Section 3.3).
coding tasks, which corresponding to three fundamental modes of reasoning: abduction, deduction and induction (Section 3.2). Motivated by the suring-completeness of programming languages [70] and evidence that code-basee based training enhances reasoning [1], we adopt code as open- openeended, expressive, and verifiable medium for reliable task construction and verification (Section 3.3). Finally, the model is updated using newly proposed advantage estimator designed for multitask learning (Section 3.3). We showcase an illustration of our Absolute Zero Reasoner approach in Figure and Algorithm 1.
3.1 Two Roles in One: Proposer and Solver
Large language models are naturally suited for implementing the Absolute Zero objective in multitask learning context [59], as both the formulation of reasoning tasks and their solutions occur within unified language space. At each iteration the online rollout, AZR proposes new reasoning tasks by conditioning the task type (as defined in Section 3.2) and past self--gnnrated task triplet(s). The model, in proposer mode, is then prompted to generate the necessary components for a task proposal based on the task type. These task proposals are filtered and transformed into valid reasoning tasks that can be validated using a python interpreter, described Section 3.2. AZR then attempts to solve these newly proposed tasks, receiving grounded feedback for its model responses. Both task proposal and problem solving are trained using reinforcement learning, with their rewards described next.
Reward Design. Prior work has shown that setting appropriate task difficulty is critical for promoting effective learning in reasoning systems [100]. Motivated by this, we design reward function for the proposer that encourages generation of tasks with meaningful learning potential nheentia--neihher neither too easy unsolvable for the current solver. Concretely, estimate the learnability of a proposed task, which is well studied in autotelic agents and unsupervised environment design literature [54, 71]. We perform G Monte Carlo rollouts of the solver and compute average success rate: Fsolve t/ G (i)
the same language model in its solver role to
salve" The proposer's s reward is then defined as:
S0, = [1 Fsolve, otherwise.
Fsolve =
Prropose
The intuition is that if a task is either trivial (Frotv = 1) or unsolvable (F = 0), the task provides little to no learning signal to the solver. In contrast, tasks of moderate difficulty, where the solver occasionally succeeds rewarded the most, they offer the greatest potential for learning.

---

For the solver, we assign simple binary reward based on the correctness of its final output.
rsolvv
I(v=y*)>
(5)
where yn is the ground-truth answer, and equality is evaluated based on value equality in Python.
With the primary rewards for the two roles defined, we adopt the following composite reward structure, which integrates "propsse and Tsolve with a format-aware penalty inspired by Guo et al. [22]:
Prole
correctly formatted, role E {propose.solve)
R(ya)= =
0.5
response is wrong but well-formatted, answer has formatting errors.
(6)
where Vr is the response of the language model. The main format that the proposing and solving tasks need follow is the DeepSeek RI <think> and <answer> format, as shown in Figure 35. Moreover, for the proposer, the reward criterion for format goes beyond simply following the XML. structure. As detailed in Section 3.3, only responses that produce valid triplets and pass the filtering stage (correctly parsed, executable, safe, and deterministic) are considered to be correctly formatted.
3.2 Learning Different Modes of Reasoning: Deduction, Induction, and Abduction
AZR leverages code executor both a flexible interface and a verifiable environment, enabling automatic construction, execution, and validation of code reasoning tasks [70, 1]. Given a deterministic program space 9 deserministic. input space I, and output space O, each task is represented as a triplet (p. i, o). where p(i) = o. AZR learns by reasoning over this triplet using three distinct modes: deduction, abduction, and induction. Deduction: the model infers the output from program p and input i. As a proposer, AZR generates (p, i) conditioned on K previously generated problems and obtains via execution; valid completions predicts On given (p, i). which is verified using type-aware Python equality. Abduction: AZR infers plausible input i given program p and output o, reflecting trial-and-error. The proposer generates (p, i) and the full triplet is completed through execution; the solver predicts is from (p, o) and passes The reason
stored for future bootstrapping. As solver, it
do not directly match ix with i is because p does not need be bijective, therefore any input that produces is correct. Induction: the task is to synthesize a program from 1/O examples { )///2 As a proposer, AZR samples a program, generates inputs, and computes outputs, forming task (p. (("")""))} m)), where message m helps define the intent. As solver, the model sees partial I/O examples and m, and must produce Pa that generalizes to match hidden I/O cases, discouraging overfitting and promoting abstraction. Each reasoning task type leverages code as an expressive and verifiable medium, aligning with the AZ paradigm's 's goals of fully self- inff-iinp systems in open-ended domains [22, 37]. Prompts used in Figures 36 to 41.
if p(())
3.3 Absolute Zero Reasoner Training Algorithm
The AZR training pipeline begins by initializing buffers for each task type through seed task generation with the base model. The model gets prompted to generate task triplets, which are then filtered, and validated to ensure syntactic correctness, safety, and determinism. These buffers jumpstarts self-play by providing task examples and filling incomplete solver batches. More deetails in Section 1.1.1.
During self-play, AZR iteratively proposes new tasks, constructs and validates them, solves the tasks, and verifies the outputs. Proposed tasks are rigorously validated in Python by executing the programs to 1) checking for valid syntax; 2) enforcing safety constraints; and 3) ensuring for proposed program determinism. This process ensures only valid tasks are being solved and added to the buffers. Different task-specific criteria are applied to define validity: (p, i, o) triplets (abduction or deduction), or input sets with corresponding messages (induction). See Section 1.1.3 for more details.
Finally, AZR verifies the solver's outputs against ground-trutt information from triplets. For deduction, the predicted output is matched directly: for abduction, equivalence is checked via program execution due to potential non-bijective program; for induction, generated test cases must pass functional equivalence checks, see Section A. for details. After verification, rewards are computed, and both the proposer and solver policies are updated. For detailed explanations of buffer initialization, task validation, and answer verification, we refer readers to Section A.I and Algorithm I.
5

---

Task-Relative REINFORCE++. Since AZR trains the combination of roles and task types, it operates in a multitask reinforcement learning setup [103, 104, 79, 95]. Instead of computing single global baseline in REINFORCE+ [28] (Section A.3), we compute separate baselines for each of the six task-role configurations. This can be viewed as an interpolation between per-question baselines, as in GRPO [64], and global baseline, allowing for more structured variance reduction tailored to each task setup, We refer to this variant as Task-Relative REINFORCE+ (TRR++), The normalized advantage A Abarn is computed as:
A 4noem
rtask.role
task (iid,dde,abd)), role e {propose,solve):
(7)
raskk
where the mean and standard deviation are computed within each task type/role, yielding six baselines.
4 Experiments
4.1 Experiment Setup
Training Details. For all experiments, we initialize the buffers as deseribed in Section 3.1. AZR models are trained using batch size of 64 x 6 (2 roles x 3 task types). We use constant learning rate= 1e-6 and the AdamW optimizer [49]. Complete list of hyperparameters is provided in Table 4. For the main experiments, we train AZR models Absolute Zero Reasoner-base-T and Absolute Zero Soader respectively. Additional experiments include training Qwen2. 5-Coder-33 3B, Quen2. 5- Qwen2. 5-14B, Llama-3. 3. [88, 32, 16].
Qwen2.5-74 and 55-en2.5-7B-oder. resulting
Evaluation Protocol. To evaluate our models, we divide the benchmarks into in-distribution (ID) and ditto--iisttibution (OOD) categories. For OOD benchmarks, we further categorize them into coding and mathematical reasoning. For coding tasks, we evaluate using Evalplus [45] on the HumanEval+ and MBPP+ benchmarks [6, 21, as well LiveCodeBench Generation (v1-5, May 23-Feb 25) [34]. For mathematical reasoning. we utilize six standard benchmarks commonly used in recent "zero" reasoners: AIME'24, AIME'2S, OlympiadBench [25], Minerva [40], Math500 [26], and AMC' For ID benchmarks, we use CruxEval-((nput)), CruxEval-O(utput)) and BeeeCoodeeench-eeeeeti which assess reasoning capabilities regarding the input and output of programs [41]. Greedy decoding is used for all baseline methods and AZR results to ensure reproducibility. All bascline models' details, training data and initialization settings are summarized in Section A.2 and Table 3.
[4],
4.2 Results
Research Question I: How does AZR compare to other zero setting models trained with human expert data? We present the main results of reasoning models trained under both the standard zero and our proposed absolute zero settings in Table 1. Notably, Absolute Zero Reasoner-Coder-1 achieves state-of-the-ar performance in both the 7B overall average and the coding average categories. Despite being entirely out-of-distribution for both math and code reasoning benchmarks, it surpasses the previous best model by 1.8 absolute percentages in AVG of (CAvg + MAvg) /2. Even more strikingly, it outperforms models trained with expert-curated human data in the coding category (CAvg), by 0.3 absolute percentages, while never having access to such human curated data itself.
Strong Cross-domain Generalization. To assess cross-domain generalization after RLVR, we evaluate math performance before and after training, comparing AZR models with other expert code models, since AZR was also trained in coding environments. After training, most expert code models showed minimal changes or even declines in performance compared to their base versions in math, with an average increase of only 0.65% across these models, indicating limited cross-domain generalization. In contrast, AZR base and coder achieved gains of 10.9% and 15.2% respectively, demonstrating substantially stronger generalized reasoning improvements. Similarly, although out-of-distributin on human-defined code generation tasks, AZR models improved by 3.2% and 5.0%, while the math models on average showed just moderate increases in coding (+2.0% on average).
Overall, these results highlight the surprising effectiveness of our approach. Unlike other RLVR models trained and evaluated human-defined tasks, our AZR models demonstrate strong general reasoning capabilities without any direct training on downstream human-defined math or coding data, only had access to self- self-proposed tasks during training, yet still surpassing existing models.

---

Model
Base
IHEvall
MBPP
LCB
AME21
AMC
M500
Minva
Olypad
CAng
MAng
AVG
Qwen2.5-7g
73.2
65.3
17.5
6.7
3.3
37.5
64.8
25.0
27.7
52.0
27.5
39.8
Ques2.5-7B-Inn"
75.0
68.5
25.5
13.3
6.7
52.5
76.4
35.7
37.6
56.3
37.0
46.2
Qwen2. 5-7-.-5-7-C
80.5
19.9
40.0
54.0
17.3
21.9
$6.6
Quex2.5-7B-Maun
61.0
57.9
16.2
10.0
16.7
42.5
64.2
1544
28.0
45.0
29.5
37.3
Zero-Style Reasoners Truined on Curuted Coding Data
AccCoder RM
Ins
22k
79.9
71.4
23.6
20.0
6.7
50.0
76.4
34.6
36.7
58.3
37.4
47.9
ApcCoder-Rulle Ralgln)
Ins
22k
77.4
69.0
13.3
6.7
50.0
76.0
37.5
37.8
55.4
36.9
46.2
AcoCoder-KMd RMd
Coder
23k
79.0
27.5
3.3
22.5
62.6
29.4
29.0
57.3
27.3
42.4
Coder
22k
80.5
70.4
29.0
6.7
6.7
40.0
62.8
27.6
27.4
60.0
78.5
44.1
CodeR1-LCCD L
Ins
2k
81.7
71.7
28.1
13.3
10.0
45.0
75.0
33.5
36.77
60..5
35.6
48.0
CodeR1-12x-4
37.5
74.0
35.7
36.9
33.5
47.4
Zero-Style Rensoners Trained
PRIME ZZeol SimpleRRL-Zoe iod
Coder
484kc
49.4
51.1
11.0
23.3
23.3
67.5
81.2
57.9
41.8
57.2
45.8
41.5
Base
8.3k
73.2
63.2
25.6
16.7
3.3
57.5
77.0
35.7
41.0
54.0
38.5
46.3
Oat-Zerd 11 ORZ!
Math
62.2
15.2
62.5
80.0
54.9
41.6
44.3
44.9
Base
57k
80.5
64.3
22.0
13.3
16.7
60.0
81.8
32.7
45.0
53.6
41.6
48.6
Absolate Zero Training w/ No Curated Duta (Ours)
AZR (Ours)
Base
71.3
00.1
25.3"
13.3"
13.3
02.5
74..4%
38.2
38.5
55.2
38.4
46.8
AZR (Ouns)
Coder
83.5
00.6
$1.7
200"
10,0**
07.5
72.8
36.4
38.22"
61.6
39.1
50.4
Table 1: Performance of RL-Traiied Reasoner on Reasoning Benchmarks Based on Qwen2.5-7B Models. Performance of various models is evaluated on three standard code benchmarks (vumanEEvl*", MBPP* LCB*I-5) and six math benchmarks (AIME'24 24, AIME' AMC'23, 23, MATH500, Minerva, OlympiadBench). Average performance across coding and math benchmarks is calculated as average of the two averages: AVG = (CAvg + MAvg)/2. We use for absolute percentage increase from base model. All baseline and AZR models are trained using different variants of the Qwen2.5 5-B7B model, with the variant and data usage labeled, more details of baselines listed in Table and Section A.2.
Research Question 2: How do initializing from different base model variants (base vs. coder) affect performance? As shown in Table 1, the coder variant achieved better overall performance in both math and coding after the AZR self-play process. Strikingly, although the coder base model variant started with lower average performance in math than the vanilla base model (23.9 27.5), it ultimately outperformed it after training. This highlights the importance of initial code competency as a catalyst for enhancing broader reasoning abilities within the Absolute Zero Reasoner approach.
Research Question 3: How does varying model size effect AZR's in-distribution (ID) and -ut-of-distribution (OOD) capabilities? We examine the effects of scaling model size and present both ID and OOD results in Figure 3 (a) and (b), respectively. Given the strong performance of coder models in the 7B category, Quen2. 3- Coder and Qwen2. 5- 14B- Coder. Due to the absence of existing baselines for these model sizes, we compare each model's performance its corresponding base model.
extend the analysis by evaluating smaller and larger variants:
The results reveal a clear trend: our method delivers greater gains on larger, more capable models. In the in-distribution setting, the 7B and 14B models continue to improve beyond 200 training steps, whereas the smaller 3B model appears to platcau. For out- of-distribution domains, larger models also show greater overall performance improvements than smaller ones: +5.7, +0022, +13.2 overall performance gains, respectively for 3B, 7B and 14B. This is an encouraging sign, since base models continue to improve and also suggesting that scaling enhances the effectiveness of AZR. In future work, we aim to investigate the scaling laws that govern performance in the Absolute Zero paradigm.
Research Question 4: Any interesting observations by changing the model class? We also evaluate our method on different model class, using Llama3. 1-8B as the base shown in Figure 3. Unlike the 3B and 14B categories, this setting has an existing baseline, SimpleRL. [100]. which enables a direct comparison. Although Llama3. 1-8B is less capable than the Qwen2. models, our method still produces moderate improvements (+3.2), demonstrating AZR's 's effectiveness even relatively weaker models. However, these gains appear more limited, which aligns with our earlier observation that performance improvements tend to scale with initial base model potency.
Research Question 5: Any interesting behaviors or patterns observed during AZR training? We observed interesting response patterns in both the proposal and solution stages. The model is capable of proposing diverse programs, such as string manipulation tasks, dynamic programming problems, and practical cases (e.g., calculating a triangle's is area using Heron's formula). We show concrete example Figure 15, where AZR proposes a code problem that searches for the sum of continuous sub-arrays matching a target value and solves it through trial-and-erros
7

---

Model Family
Variant
Code Avg
Math Avg
Total Avg
Llama3. 1-8b
28.5
3.4
16.0
Llama3. 1-8b
SimpleRL. [000]
33.7
7,2
20.5 1e
Llama3.
AZR (Ours)
31.6"
6,8
19.2
6.3
Qwen2.5-555 5-3B Coder
51.2
18.8
35.0
Qwen2.5-3B Coder
AZR (Ours)
54.9
26.5
40.7
6-3
Qwen2.5-7B Coder
56.6
23.9
40.2
Qwen2.5-7B Coder
AZR (Ours)
61.6
39.1
50.4+102
402
Qwen2.5- 5-22--1-41 Coder
60.0
20.2
40.1
Trining Stapa
Qwen2.5- 14BB Coder
AZR (Ours)
63.6
43.0
03.3
(a)
(b)
Figure 3: (a) In-Distribution & (b) Out-of-Distribution Reasoning Task Performances. (a)
Scores CruxEval-I, CruxEval-O., and Beeooodeeenn---eeuuttiiie which correspond to abduction, deduction, and deduction task types respectively, used to evaluate in-distribution abilities of AZR during training across different model sizes and types; (b) Out-of Out-of-distributiin reasoning performance, reported as the average of code tasks, math tasks, and their overall average, across different model sizes and types. A detailed breakdown of all benchmark results can be found in Table 5.
Overall, the models trained exhibits distinct reasoning patterns depending on the task type. For example, when solving abduction tasks, repeatedly tests different input patterns. self-correcting until the reasoned output matches the given input. When predicting outputs, it steps through the code and records structured intermediate results (such as dynamic programming arrays) until the final output is reached. When inducting programs from given inputs, outputs, and descriptions, the model systematically checks each test case to confirm that its program produces correct results. We showcase more concrete examples of these behaviors in Figures 25 and 27 to 33. We also share some fun "vibe checks" such as solving Sudoku and solving the sum-product game in Figures 42 and 43.
5.1 Intermediate Planning During Code Response. Another interesting pattern emerged in our AZR models during the code induction task: the final code outputs were often interlcaved with comments that resembled immediate step-by-stey plans, reminiscent of the ReAct prompting framework [90]. A similar behavior has been observed in recent formal math proving models, such as DeepSeek Prover v2, which is significantly larger in scale (671B). This pattern suggests that models may naturally adopt intermediate planning as a strategy to enhance final answers. Therefore, it may be beneficial to explicitly enable encourage this behavior in long-form responses across other domains.
5.2 Cognitive Behavior Llama, Interestingly, we also observed some emergent cognitive patterns in Absolute Zero Reasoner-Llama3 1-8B, similar to those reported by Zeng et al. [100], and we include one example in Figure 33, where clear state-tracking behavior is demonstrated. In addition, we encountered some unusual and potentially concerning CoT from the Llama model trained with AZR. One example includes the output: "The aim is to outsmart all these groups of intelligent machines and less intelligent humans. This is for the brains behind the future" shown in Figure 34. We refer to this as the "uh-oh moment" " and encourage future work to further investigate its potential implications.
5.3 Token Length Increase Depends on Task Type. Finally, we observed that token length increases over the course training, consistent with findings from recent studies [29, 47]. Interestingly, our results reveal one of the first observation of clear distinctions in token length growth across different types cognitive tasks. As shown in Figures 21 to 23, the extent of lengthening varies by task type. The most significant increase occurs in the abduction task, where the model engages in trial-and-error reasoning by repeatedly testing inputs to match the program's 's output. This suggests that the observed variation in token length is not incidental, but rather reflection of task-specific reasoning behavior.
Research Question 6: Are all task types essential for good performance (Ablation)? Due to resource constraints, we perform the ablation studies in this section and the next using only Absolute Zero Seasoner-Base-te
We begin by testing the importance of task types during training, with results shown in Table 2. In row 1, both induction and abduction tasks are removed; in row 2, only the induction task removed. In both cases, math performance drops significantly, with the most severe degradation occurring when more task types are excluded. These findings highlight the complementary role of the three task types in improving general reasoning capability, with each contributing in a distinct and essential way.
8

---

Experiment
Task Type
Gen Reference
Trained Roles
Code Avg.
Math Avg.
Overall Avg.
Deduction only
Ded
54.6
32.0
43.3
w/o Induction
Abd. Ded
/
54.2
33.3
43.8
w/o Gen Reference Train Solver Only
54.4
33.1
43.8
/
Solve Only
54.8
36.0
45.4
Ours
Abd. Ded. Ind
K
Propose Solve
55.2
38.4
46.8
Table 2: Ablation Results. We ablate task types and the proposer role in the AZR using 7B base. A ' indicates that the configuration remains unchanged from the standard AZR setup. Removing induction or using only deduction leads to significant performance drops (rows 1 & 2). For the proposer role, both removing conditioning on K references (row 3) and omitting proposer-rok training (row 4) result in degraded performance. Overall, all components are essential for general reasoning.
MBPP++
+umanEval++
LiveCodeBench
NIME25
AIME24
90
96
60
60-
pass@k
90-
48
45
78
84-
40
30-
78
30
Qwen2.5 7b
72
24
15
15
22 22222722
24
21
7B and its base counterpart on three coding benchmarks and two math benchmarks using the pass@k metric. As h scales up to 512, AZR maintains high answer diversity and outperforms the base model in 4 of 5 cases. This favorable property can be further leveraged by test-time scaling methods to improve performance.
Figure 4: Pass@k Results. We evaluate
Research Question 7: How much do the designs of proposer contribute to the overall performance (Ablation)? Next, we ablate two components of the proposer role and present the results Table 2. First, we examine whether conditioning on historic reference triplets is necessary. To do so, we design a variant which a fixed prompt is used to propose abduction and deduction tasks, rather than dynamically conditioning on K historical triplets (row 3). This results in 5-point absolute drop in math performance and a 1 1-point drop in code performance. This suggest that dynamically conditioning on reference programs helps improve performance, possibly by increasing diversity and achieving better coverage of the reasoning problem space.
Finally, we examine a setting where the proposer is not trained. Instead, we prompt it using the current learner and train only the solver (row 4). This results in a moderate performance drop (-1.4), indicating that proposer training is indeed beneficial. However, we believe there is potential to further enhance the proposer, possibly amplifying gains general reasoning. One possible direction is to mitigate task interference, as discussed in multitask learning literature [72], to introduce explicit incentives that encourage broader problem space coverage. Overall, we see improving the proposer as promising direction to further enhance solver performance through their synergistic interaction.
Research Question 8: What the relative performance of AZR vs. the base model for high pass @ k? We evaluate reasoning coverage following Yang et al. [97], with temperature 0.6, top-p 0.95, max output tokens 16k, and & up to 512, and present the results in Figure 5. Across three code benchmarks (LiveCodeBench, MBPP++. HumanEval++) and two math benchmarks (AIME24, AIME25), AZR consistently matches or outperforms the base model high & (256/512), with one exception at AIME24 (k=5. These gains persist at larger k indicating AZR maintains broad reasoning coverage and answer diversity after RL, compatible for further test-time scaling [69].
Research Question 9: How do AZR models perform in general reasoning tasks? We assess on MMLU-Pro [83] using greedy decoding and 16k token limit, and compare against three basclines: ORZ-7B, Qwen2.5-7B. and SimpleRL-Zoo ManpleRL-Zoo-7B 7B. AZR attains higher subject- and higher overall average, indicating strong general reasoning capabilities beyond math and code.
Additional Results. Beyond the core research questions, we present additional results, including the breakdown of individual OOD benchmark scores during training in Figures 16 to 19 and ID scores in Figure 20. Finally, we invite readers to explore Section E, which presents several experimental directions that, while not yielding significant performance gains, offer interesting findings: e.g. exploring composite functions, new error prediction task, and diversity/compplxity rewards.

---

AZR Basc.
ORZ-7b
Qwen2.5 Sweo2.5--Bss-/7 Base-
SimpleRL-7b
1.0
0.8
0.6
0.4
0.2
0.0
Business
Law
Psychology
Biology
Chemistry
History
Other
Health
Economics
Math
Computer Science
Physics
Philosophy
Subject Subject-Aggregat
Engineering
Overall Average
Figure 5: General Reasoning. We compare AZR-Base-7B with three Maselines-ORZ-e SimpleRL- Zoo-7B, and Qwen2.5-7B, on MMLU-Pro [83]. AZR-Base-7B attains higher averages both across subjects and across all samples, indicating strong general reasoning across 14 diverse subjects/domains.
5 Related Work
We include the crucial related works in this section and present a comprehensive version in Section B. Recent advances have applied reinforcement learning to improve the reasoning abilities of large language models [37]. beginning with STaR's expert iteration, followed by ol [33], which was the first large-scaled outcome-basee RL general reasoner. RI [22] introduced the zero setting-apllying RL directly to base models without SFT, prompting open- replications and algorithmic improvements [100, 47, 13, 29, 92, 94], and inspiring procedural task studies [86, 82]. In parallel, self-play [61] has emerged as a powerful paradigm for self-supervised learning. from early dual-agent systems [62, 63], AlphaGo series [67,68], and GANs [20], to automatic curriculum generation/autotllic agents/unsupervised environment design/asymmett
self-play [18, 71, 53, 78, 15, 104, 107, 12, 11, 53, 15, 24]. Recent work explores self-play in LLMs for alignment [7, 93, 35, 91], games [8], and formal math [56], though many rely static human-defined tasks [87, 102, 109]. Building on both lines, we introduce the Absolute Zero setting, where reasoning agents self- self-propose and solve code- oode-groudded tasks in environment, without external supervision. To the best of our knowledge, this is the first work to combine self-play with RLVR and leverage a verifiable environment to match the performance of general- purppse reasoning models, showing promise of our proposed paradigm.
6 Conclusion and Discussion
Conclusion. We introduced the Absolute Zero paradigm, in which reasoning agents improve by generating their own task distributions and solving them through interaction with verifiable environment. Our instantiation, the Absolute Zero Reasoner (AZR), learns by proposing and solving code-based reasoning tasks using a code executor. Despite having no exposure human-curated data, AZR achieves state-of-the-arr performance on out-of-distributin benchmarks in both coding and mathematical reasoning, demonstrating strong generalization. This shows that reasoning models can achieve high performance without any human supervision, signaling a possible shift toward an "era of pearrienee[[5,. 66, 105], where the AZ paradigm is used to train reasoning models. Finally, we demonstrate that our AZR approach is scalable and transferable across different model classes. A more detailed discussion is provided Section C, including an additional boarder impact section.
Discussion. We believe there remains much to improve, such as altering the environment to more general settings: the web [85, 84], formal math languages [74, 60], world simulators [108, 96], or even the real world. Beyond that, future directions could upgrade to multimodal models, find better distribution p(z). defining or even let the model dynamically learn how to define f (Equation (3)). Another promising direction is to better estimate the learning progress, with recent works like MAGELLAN is pioneering in this direction [19]. Moreover, exploration in solution space is underexplored and not done in this paper [97, 67, 36, 57, 106]. On top of the exploration topic, our framework allows exploration over the learning task space, where agents learn not just how to solve tasks, but which tasks to pursue and how to discover them. This shift empowers agents to expand the boundary of problem spaces. Lastly, with worrying signs, iee. uh-oh moment, observed in our experiments with the Llama3, 1-8B model, safety in self-cvolving systems needs attention [80, 77].
10

---

Acknowledgements
This work is supported in part by the National Key R&D Program of China under Grant 2022ZD01 14903, the National Natural Science Foundation of China under Grants U24B20173 and W2442032 and W2442033 and 62376031, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNUSKYCANLIZXK..EE
References
[1] Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzich Fadace, Ahmet Ustün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training, CoRR, abs/2408. 2024. doi: 10.48500ARXIV.400....0 10914. URL https://doi. org////0o.orr////////// org/10 48550/arXiv 2408. 10914.
[2] Jacob Austin, Augustus Odena, Maxwell I. Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, 2ls/2108.07332 2021. URL https:
// //arxiv. orgoor///ao httg/abs/21000 027.222.........
[3] Collin Burns, Pavel Izmailov. Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. Weak-to-stte strong generalization: Eliciting strong capabilities with weak supervision. Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http::://opmmeevve...eeeftoo ghNRg2mEgM.
[4] Miquel Canal. Radon: Python tool for code metrics. http://ggttabb..cmmmmmb////a// 2023. Accessed: 2025-04-06.
[5] Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li. and Kwan-Yee K. Wong. Spe: Evolving self-play critic via adversarial games for Ilm reasoning. 2025. URL /tttpp://arxii..../ org/ abs abaaaaaaa//00/ /20.//55..... 19162.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. Evaluating large language models trained on code. CoRR, abs/2107.003744 2021. URL https: /ttx//rxvvv./////// www.iv.oog/abaa....e.. wtt//or////aaoooeo 03374.
[7] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Confer- ence Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, net wett//oormiidd-4ciHTt..
2024. URL
[8] Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances I.IM reasoning. In Amir Globersons, Lester Mackey, Danielle Belgrave. Angela Fan, Ulrich Paquet, Jakub M. Tomezak. and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: /ttt::://aaae http://aapppaaaaeeee nips nips cc/paper files/papers html.
[9] Paul Christiano. Approval-directeer bootstrapptittttn..ccc.c.. http::::hww:wwtttpgggg.et/
org/ 2018. Al Align-
ment Forum.
[10] Paul Christiano. Capability amplification. http://aan.t.ttttt..////aaaaaeee wiiggnnftorummmmmmmmmmmmmmmmmmmt org/posts/
2019. AT Alignment Forum.
[11] Cédric Colas, Tristan Karch, Clément Moulin-Frier, Pnerre Yves Oudeyer. Language and cul- ture internalization for human-like autotelic AT. Nut. Mac. Intell, 1068 2022. doi: 10. 00138//42255-022-0055 URL https:/ //oii. orgo 00w.1010//2255-222--000000
11

---

[12] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre- Oudeyer. Autotelic agents with intrinsically motivated coal-conditooned reinforcement learning: A short survey. J. Artif. Intell. Res., 159-11999 2022. doi: jair. 13554.
1.1354.. URL. mattps://dol org/ 10. 1613/
[13] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. CoRR, abs/2502.01456, 2025. doi: 10.48550/ARXVV2552.0044
URL https //doi. org/10..//.. www.oo../g/////...oo. 48550/arxxx/ 2502. 01456.
[14] Abram Demski and Scott Garrabrant. Embedded agency. CoRR, abs/1902.09469, 2019. URL
http://arxiv.or org/ aaaaab/aaa 190.. 09469.
[15] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: httpps://prceeedigggg hew.oeeeeeinnnnnnniiiip...s.. nww.inen.i.... wwturips.ccpaaaa...e wwtpccppp/////eeee/c /2020/ 22000/2222/haha/// 0956944661000535660 1000066660011199499666------666- Abstract. html.
[16] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olanno Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21788. 2024. doi: URL https: /doi. weeeiiorrrr...1 wttpoo.../..///////. 485.0/ arrivv. 2407 21783.
10.44850/ARXXI.2407..111
[17] Christof James Cain, Giuliano Antoniol, Steve Counsell, and Phillip Laplante. Cyclomatic complexity. IEEE software, 33(6):27-29, 2016.
[18] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässen, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1514-1523. PMLR, 2018. URL http:::/proceediinnee .ml...s...r........ wwt.prressvv.. florensal8a.bn html.
[19] Loris Gaven, Thomas Carta, Clément Romac, Cédric Colas, Sylvain Lamprier, Olivier Sigaud, and Fierre-Yves Oudeyer. MAGELLAN: metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces. CORR, abs/2502.07709, 2025. doi: (0.4850/ARXIIV2502.0700
URL https: //doii///oo//// org/10 48550/ 27020
[20] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(11):139-144, 2020. doi: 10. 10.1145/3422622 URL httph::tt///ett :/doii wiw.o.oor.o..oooeeooeooooeoee... org/10.1145. 3422622.
12

---

[21] Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve. and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024. URL Ffpg52swvg.
[22]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyn Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun. T. Wang. Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan. Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl incentivizes reasoning in Iims through reinforcement learning. Nature, 645(8081):6333 638, September 2025. ISSN 1476-4687. doi: 10.103%641186-005.00004. URL. http:tt: doi org/10 10. 1038/ 02511556-055-9944
[23] Maurice H Halstead. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977.
[24] Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreviiee net/forum?? wtt.fooonmmmnaiaaaa Sadiaajjaaaaaaa
[25) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL2024, Bangkok, Thailand, August 1 11-16 2024, pages 3828- 3850. Association for Computational Linguistics, 2024. doi: 10. 11166///112224.ALLL- acl- 211.
URL httpp::///ooiiio org/ 10.
[26] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solv- ing with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmurks 2021, December 2021, virtual, 2021. htw.aatpp///aassteeeaaaaaaaaeeee hash/ btt.b/ssseecddrraaaaaaooooorraaa htwp:///rerrtttttatttattttroooee html.
wwwpeediiinnnn...iiieeeeee nwwrips....... wwwuurps.ccpppper///eeee cc/paper/2021/
13

---

[27] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.. http://axx....rrra//////00 02531.
[28] Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models. CoRR, abs/2501 003262, 2025. doi: 10.4850/ARXIV.250..2222 URL https: //doi.org//il wwweg/...4555///1 40/5//00.805////i 2501.03262.
[29] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open- open-reasoner-zere An open source approach to scaling up reinforcement learning the base model. CoRR, fbs/2503.242990 2025. doi: 1048550AARXVV2505.22225 URL
48550/arXiv betps://doi.orgee httpps//laa..rr///..8868///// 2503.24290.
[30] Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. CoRR, abs/1906.0182 2019. URL. http: peattt://arkeee /aaaaaivvvoooo org.//iioorooooooooo aat/////s.//////////// 018200.
[31] Edward Hughes, Michael D. Dennis, Jack Parker Holder,, Feryal M. P. Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktäschel. Position: Open-endedness is essential for artificial superhuman intelligence. In Forty-first International Conference on Machine Learning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. net/ URL wwt.hhttss//ppeeee...eeee ntt/ferunf 2r
[32] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang. Bowen Yu, Kai Dang. An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 5wen2.5-coder technical report. CoRR, abs/2409. 12186, 2024. doi: 10.48550/AXXVV24009 12186. URL https://doi. org/10 48550/ /ar/0///rxiv../.. 2409. 12186.
[33] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
[34] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contami- nation free evaluation of large language models for code. CoRR, abs/2403.07974, 2024. doi: 10.48550/ARXIV2403... 077774. URL https: h //woi. 48g/10..8550/arxiv 2403 07974.
[35] Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of LLM outputs. CoRR, abs/2407.13692. 2024. doi: 10. (0.485500/ARX1..407... 13692. URL httpp::://oo..or///..... 008.50855551111 240. 13692.
[36] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinffremmee learning: A survey. Inf. Fusion, 85:1-22, 2022. doi: INFF0
URL
/stps://doi..oo org/10..//.. 1016/ inffus. 2022.022222222 003.
[37] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. CoRR, abs/2411.15124, 2024. doi: (0-45550/AXXIV2241.15222 URL https://ddi.org httpe/orr///...45///ra....
[38] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Ler- rel Pinto, and Pieter Abbeel. URLB: unsupervised reinforcement learning benchmark. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Sys- tems Track Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. /ttp:////......tttt////aaaaaaaaa httpss//äaatttste--aaachattte hagpsasseeeaaacaacccacoooeeoennn neurips cc/ html.
[39] Jan Leike and Ilya Sutskever. Introducing superalignment, https://openas cttiiiccoooo//////// wtaedduciinguuueaalligiing 2023. OpenAl Blog.
14

---

[40] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Mísra. Solving quantitative reasoning problems with language models. Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 December 9, 2022, 2022. URL http: /ttptt:ppppppr...eee nips cc/paper.pppcc.ccc httpppppperr_lls___leepppapepeee www.ffflllppppeee /paper///// 2022/ 2022/hash//1abbeerfff
[41] Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction. CoRR, abs/2502.07336, 2025. doi: 10.48550/ARXXI122520.773
URL https: //do:.//dooooo/oo http:///too.oo/..///// 008.04855501111 wwwp.o.....////...t....oo
2502.07316.
[42] Rongao Li, Jie Fu, Bo- Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. TACO: topics in algorithmic code generation dataset. CoRR, abs/2312.148522 2023, doi: 10.485 1044850/ARXXV22312.4882 URL https: //tp:/doiii.oo org/oor.//0 40w.00/.88......... 2312 2812..4852.
Bo-Wen
[43] Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. SPIRAL: self- play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. CoRR, 0oos/2506.24119, 2025. doi: 10.48500ARXXV.250622211 URL https: //doi 01t.e//.0//1..0/////e// arriv. 2506 24119.
[44] Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. GitHub, 2025.
[45] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In seirth- Conference on Neural Information Processing Systems, 2023. URL https: // /ppnreviee. www.//eee///iiia htt///orrdddd1g11111000
[46] Mickel Liu, Liwei Jiang. Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models. CoRR, 074466666.774 2025. doi: 10.45550/ARXX1.2556.76666 URL https: //ooi. org/10. 1.88550/aXii. 2506. 07468.
[47] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl likeo-like training: A critical perspective. CoRR, /tbb///553.2088.. 2025. doi: 1044550/ARKXV..503.2227 2503. 20783.
URL
org/10 48550/ /ariiv..
[48] Robin Hafid Quintero Lopez. Complexipy: An extremely fast python library to calculate the cognitive complexity of python files, written in rust, 2025. URL rohaquinlop/ cwwpe//ooop/oooooooooooooolocceo compl exipy. Accessed: 2025-04-06.
com/
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpps://ppereviiiiii
[50] Jack Morris. There are new ideas in ai... only new datasets. https://blog j.mo io/ p/there-ast "'there-are-noe no---o--e----i-eee innea--in-i-oolly 2025.
[51] OpenAL. Openai 03-mini, January 2025.
URL https://openai.com//daa
openai-o3-mint/ Accessed: 2025-04-17.
[52] OpenAl. Introducing openai o3 and 04-mini, April 2025. URL https://openai. com/ index/introducing -nd--and-------a-- o4-mini/. Accessed: 2025-04-17.
and-
[53] OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, 1lge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben DaS Arthur Petron, Henrique Pondé de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation. CoRR, abs/2101. 004822, 2021. URI. https:// h/ss://arzee //axiv. wwg//b///////eeoooeoo.o.oeaaeeea abs//b//1111/ 2/t////111...882..2222/22 04882.
15

---

[54] P-Y Oudeyer, Jacqueline Gottlieb, and Manuel Lopes. Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. Progress in brain research, 229:257-284, 2016.
[55] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[56] Gabriel Poesia, David Broman, Nick Haber, and Noah D. Goodman. Learning formal mathematics from intrinsic motivation. In Amir Globersons, Lester Mackey, Danielle Bel- grave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Ad- vances in Neural Information Processing Systems 38: Anmual Conference on Neural In- formation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http:://p:::psssspps//ee/ nips cc/paper files/ hwtp:/appprp00// ///0////hhh///// Con.etttooooeeeneeeeeee .ttml.
[57] Julien Pourcel, Cédric Colas, Gaia Molinaro, Pierre- Oudeyer, and Laetitia Teodor- eseu. ACES: generating diversity of challenging programming puzzles with au- totelic generative models. In Amir Globersons, Lester Mackey, Danielle Belgrave, An- gela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http://paperss wwps.cc/pppneeeeee wwtpcpprpppppeeeeeeccccceeee.eee htcp/pper_ffls//////eeeeesssseee files/ilee/eeel www.s/epppeeeeeeeeeee paper/2024/hash) #006fff....799999277777cc6688
Absrratt-ooffeeeeee html.
[58] Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang, Yanbin Wang, et al. ART: Anonymous region transformer for variable multi-layer transparent image generation. CVPR, 2025.
[59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[60] Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, and Chong Ruan. srepseek-prover-ve Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition, 2025.
URL https:// htwpvv..ivvvv//////////.///..ava org/abs. abs abwaabas//50/ 22004.2180..
[61] Tom Schaul. Boundless socratic learning with language games.
arXiv preprint
arXiv:2411.1690 2024.
[62] Juergen Schmidhuber.
Exploring the predictable. In Advances in evolutionary computing: theory and applications, pages 579-612. Springer, 2003.
[63] Jürgen Schmidhuber. POWERPLAY: training increasingly general problem solver by continually searching for the simplest still unsolvable problem. CoRR, 115/119 2011.
URL http::tttt/taaeee http:://aaxxvvvoooeeee wtg/abs/////ooooooo.ogoo 1112.5309.
[64] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li,
Y. Wu, and Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402. 033000 2024. doi: I044550AARXXV22402.0333 URL attps://doi.. www.oo..//..... 48550/4885///// www.a. 240.0033000000
[65] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 3 April 2025, pages 1279-1297. ACM, 2025. doi: 10. (0.1145/3689031 1.3696075. URL https:
[66] David Silver and Richard S. Sutton. The era of httperieene.ittpss/s/ttaaggggee com/
http:///.....acaaeeaaaaaee////// 20Paper, pdf, 2025.
16

---

[67] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre. George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Master- ing the game of go with deep neural networks and tree search. Nat., 529(7587):484 489, 2016. doi: 10. 10.038/AATRRE1666
URL https: htt:://oo....go org/ 10. 1038/nature16961.
[68] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. URL atttp://arxis org/ abs/ 1obs/1712.00 01815.
[69] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408 03314, 2024. doi: 1044850/ARRXIV2408.0555 2408. 03314.
URL https://doi. http:///ooo..r/1/.//// (0/88500/rX..
[70] Tom Stuart. Understanding computation from simple machines to impossible programs. O' O'Reilly, 2015. ISBN 978-1-449-32927-3. URL 9781449329273/ wwn./xx22//ddxxxee// html.
hetplllllyde///tte catalo/
[71] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL net/ wett/forumidd-ST55gge
https:://penreste
[72] Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. CoRR, abs/1912.06644, 2019. URL http: // arxiv org/ abs/ 19912.06914.......
[73] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Neurips 2024 test of time award session: Sequence to sequence learning with neural networks. Conference session, December 2024. URL
[74] Richard S. Sutton, Verification, the key to ai. http://nccapleteeddes met/IncIdeas/ KeytoAI html, 2001.
[75] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua I Liao, Chuning Tang, Congcong Wang. Dehao Zhang. Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li. Haozhen Yu, Hongcheng Gao, Huabin Zheng. Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma. Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao. Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang. Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1ms. CoRR, abs/2501. 1, 12599, 2025. doi: 10.48550ARXIV.200..2222 URL //to: 485.00/85500/.... attp/..iivii. 2w011............. 12599.
[76] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of LLM scaling based on human- generated data. In Forty-first International Conference on Muchine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL www.hhttpt////cmtm /ttp://://////ee/////e// net/ forum?id=Viizcg0]3.
17

---

[77] Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, Andrew Zhao, Shenzhi Wang. Shiji Song. and Gao Huang. Model surgery. Modulating LLM's behavior via simple parameter editing. In P Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, pages 6337-6357, 2025.
[78] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, 0bss/190..011 2019. URL http: / aarxiv.iiivv./...//aa///aiaivvva org/ //s///b///// 1901. 01753.
[79] Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Gaetan Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-to- online reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// http:://ppeeeeeeeee wwppenreview... net/ o J
[80] Shenzhi Wang, Chang Liu, Zilong Zheng. Siyuan Qi, Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Boosting LLM agents with recursive contemplation for effective deception handling. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9909-9953, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024. findings-aal.591. URL htttps://aaaaattatatat
org/ findinnnnnnnnnln. 591/.
[81] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for L.M reasoning. CoRR, abs/2506.01939, 2025. doi: URL https:///ceee org/oog///1 100 48550/ 2506. 01939.
10.48550/ARXX125566001111
[82] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang. Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with training example, 2025. URL https: hatps:///rrree //arxivvv.. org/ /as///b//// abss/2044 2504. 20571.
[83] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He. Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pu prou: and challenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http / http://aapppe.eeeee mips ccppaper.ppppp....ppppp / paper and Benchmarks Track. html.
more robust
files/
[84] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs/2308.08 051555 2023. doi: 10.48550/ARXIV. 2308.08 0018.08155. URL httpt:::////ae http::::///oooo//1///// atgg/0..455000 48550/ arXiv 230.. 08155.
[85] Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing LLM task-solving through state-driven workflows. CoRR, abs/2403.11322. 2024. doi: 2403. 11322.
10:48550/ARXIV.240..112
URL
48550/arXiv wrt..//..0/...../////......
[86] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-nl: Unleashing LLM reasoning with rule-based reinforcement learning. CoRR, abx/2502. 2025. doi: 10.48550/ARXIV25502..0 14768.
URL https://doi. org/ 11wg///..//.. (a8550/arXaaa www.aatw.i...........a.. 2502. 14768.
[87] Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, and Zhiyong Wu. Genius: generalizable and purely unsupervised self-training framework for advanced reasoning, 2025. URL https: //tp////ivv////////////////.aal/ orw.orgg//oo. htrg/abs/2504 08672.
18

---

[88]
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, 2tns/2412.15115 2024. doi: 10.4550ARXIV.241...55 15115. URL https: iddoi.or//10 /0..g//...88580000 48550/8rXixv 2412. 15115.
[89]
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, http///49.111222 2024. doi: 10.48550/ARXIV.
URL https: : //doi org/10. 018//0..8850//// arxiv 2409. 12122.
[90] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R. Narasimban, and Yuan Cao. React: Synergizing reasoning and acting language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httph:tt///// httpttp:///ppeeeeeveeee
net/ wttt//ooummi heliidideeYU______ vluYUL-X. X.
[91] Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc Le Qijun Tan, and Yuan Liu. Evolving alignment via asymmetric self-play. CoRR, abs/2411 100062, 2024. doi: 10.48550/ARXIV.244 1.00062. URL attps://ddet org/10o.//10 40550/555////// arriv 0411.00066..
[92] Qiying Yu, Zheng Zhang, Ruofei Zhu Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Shengg Yuxuan Tong, Chi Zhang, Mofan Zhang. Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu. Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma. Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR, abs/2503.14476, 2025, doi: 14476. URL https: / //doi. org/10 48550/arXiv 2503 14476.
[93] Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho,, Xian Li, Sainhayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. URI. https:/aarxi.. org/abs/2401.1000 2024.
[94] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efffccient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.057/ 2025.
[95] Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving value divergence in offline-rl. Advances in Neural Information Processing Systems, 36:60247-60277, 2023.
[96] Yang Yue, Yulin Wang, Bingyi Kang. Yizeng Han, Shenzhi Wang. Shiji Song, Jiashi Feng, and Gao Huang. Deer-vla: Dynamic inference of multimodal large language mod- els for efficient robot execution. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 cttpsceeeppppppeeeeeeeeeee https//pppp/////// html.
15, 2024, 2024. URL 2eeee77777227700eee33333333
[97] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL https::///// arxiv. brrg/bs// /2504.224444 13837.
[98] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
[99] Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACECODER: acing coder RL via automated test-case synthesis. CoRR, abs/2502.01716 2025. doi: 10.48550/ARXXIV.5522000
URL. https://dde org/10, 48550/ arxiv.
2502. 01718.
19

---

[100] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, abs/2503. 18892, 2025. doi: 1048550/ARXXI.2555..8888 URL. htttp::/ddi.org/// hrt.////00rr/...485///////1/ 2503. 2503.18892.
[101] Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng. Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, al. 100 days after deepseek-rl: A survey replication studies and more directions for reasoning language models. arXiv preprint arXiv:2505.00533 2025.
[102] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization, 2025. URL https: http///xxvvvvro//// httg/abb/25000 05812.
[103] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12)5558--560, 2021.
[104] Andrew Zhao, Matthieu Gactan Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang. A mixture of surprises for unsupervised reinforcement learning. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and Oh, editors, Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL httt:://papeett nips. wwwpccpppppplllaeeeeeeeeeee files/ pppppppppp///aa//////// 776777666564444321faa
html.
[105] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents are experiential learners. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Ehirty-Eightl AAAI Conference on Artificial Intelligence, AAAI 2024, Thitt Conference Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 19632- 19642. AAAI Press, 2024. doi: 10. 1609/AAAAV VAA.....117799998. URL hhttts::/dool org/10../00 1609 aaaii.
[106] Andrew Zhao, Quentin Xu, Matthieu Lin, Shenzhi Wang, Yong-Jin Liu, Zilong Zheng, and Gao Huang. Diver-ct: Diversity-enhanced red teaming large language model assistants with relaxing constraints. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 -March 4, 2025, Philadelphia, PA, USA, pages URL https::///// //oo1.//oo//.// org/ 10. 1609/ /aaai. v39124. 34797.
AAAI Press, 2025. doi:
10. 2000//AAAVV31224.3444
[107] Andrew Zhao, Erle Zhu, Rui Lu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Self- referencing agents for unsupervised reinforcement learning. Neural Networks, 188:107448, 2025. doi: D...0HI..E 107448. WRL.https://dee http://ddooooog///// 1016/j neunet. 2025 107448.
[108] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wel Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: vision language-acttinn models transfer web knowledge to robotic control. In Jie Tan, Mare Toussaint, and Kourosh Darvish, editors, Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pages 2165-2183. PMLR, 2023. URL https: //t:://prrro http:///oceeeenes/ee/eee/eee mlr press / v229/2222////22 / tkovich23a html.
[109] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biging Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning, 2025. URL https: /trps://axxiv... orgoooog//oo abs/2504. 16084.
20

---

Appendix
Appendix Contents
Absolute Zero Reasoner Implementation Details
22
A.I Algorithm Details
22
A.2 Baselines.
25
A.3 Reinforcement Learning with Verifiable Rewards.
25
A.4 Software and Compute Used
26
B Detailed Related Work
26
Detailed Conclusion and Discussion
27
D More Results
29
D.I Interesting Results Summary
29
D.2 Out-of-Distribution Performance Breakdown
................................
30
D.3 In-Distribution Results
34
D.4 Interplay Between Propose and Solve Roles
34
D.5 Complexity and Diversity Metrics of AZR Proposed Tasks
35
D.6 Generated Code Complexity Dynamics Between Abd/Ded and Ind.
37
Alternative Approaches Considered
54
E.1
Error Deduction Task
55
E.2 Composite Functions as Curriculum Learning
55
E.3 Toying with the Initial p(z)
55
E,4 Extra Rewards
56
E,5 Environment Transition
56
21

---

A Absolute Zero Reasoner Implementation Details
A.1 Algorithm Details
In this section, we will discuss details of our AZR self self-play algorithm, including initialization of buffers 1.1.1, usage of thse buffers A.1. construction of valid tasks A. and finally validating solutions A.1.4. We outline the overall recipe of the self-play procedure of AZR in Algorithm I.
Algorithm 1 Self-Play Training of Absolute Zero Reasoner (AZR)
Require: Pretrained base LLM тe; batch size B; #references K; iterations T
1: Daod, Dabd, Pind f- INTTSEEDING(TT)
> see $A.1.
2: for f- T do
3: for b + 1 to B do
> PROPOSE PHASE sample program for induction task proposal generate N inputs and description if { n=1 + VALIAATANOCONSTRUCTT. (t SYNTAX) then See $A. 1.3 update induction buffer
4:
p ~ Daba Deed
5:
[111) n mz
propose "g (ind. p)
6:
7:
Dind Dind U ( {(i 0g),, mz)}
8:
for E (ded, do
9:
(px. de, 0k) k=1 Dr (Pr, 1z) f- T0
> sample K reference examples propose new task then update deduction or abduction buffers SOLVE PHASE
10:
proponcee (x, (prs in, (p oz)])
11:
if O+ VALIDATAANDCONSTRRUU
(Pr: ix SYNTAAX,
Da -a-aa U (PT: ix. ox)}
13:
for all E {ded, abd, ind} do (z, v2. t SAMPLEPREPARE TASKS (Da. B.t)
14:
> based on a and iteration t, see $A.1.3
15:
V/ T (z)
Reward: Use proposed task triplets and solved answers to get "propose & Tsolve see $3.1
17:
RL update: use Task Relative REINFORCE++ to update
D see $333
A.1.1 Buffer Initialization
To initialize AZR self-play, we first generate a seed set of valid triplets using the base language model. Each prompt samples up to K triplets from the current seed buffer Dxeed as references for deduetionn/bdduction, or one program as the reference for induction. When Dreed is empty at time 0, we fall back to the zero triplet show in Figure 6. During the sceding stage, we use the same proposer prompts used during training, detailed in Figures 36 to 38.
First, for deduction and abduction tasks, the LLM is prompted to generate (p, i) pairs, which are filtered, executed, and stored as valid triplets. We initialize DD abduucion DD deduction Daoed where (Deeal = B x S, where B is the batch size, and S = 4 is factor we fix in all experiments. All seed triplet's program are stripped of global variables and comments (Section E), but subsequent iterations of adding new triplets to the buffers are unaltered during AZR self-play training. No model updates occur during the seed phase. Similarly, to initialize the induction buffer, we sample programs from Dreed, generate matching input sets and messages, and collect valid examples until [DD Pinductionn
B x S.
A.1.2 Task Proposal Inputs and Buffer Management
During the actual self-play stage of AZR, we use the task buffer in three ways. First, for the proposer of abduction and deduction tasks, we uniformly sample K past triplets from the buffer, present them as in-context examples to the proposer and let it generate new task. The design is to show it past examples, and prompt it to generate different one to promote diversity [106]. Second, we sample one triplet from the union of abduction and deduction buffers Dana U and present the program from that triplet to the induction proposer to generate set of N matching inputs {i") } and a natural language message m. Lastly, to maintain stable training, if batch of solver problems contains fewer than valid proposed tasks (proposer tasks adhering to formatting, therefore filtered), we fill the remainder by uniformly sampling from the corresponding task buffer of previously validated triplets.
22

---

Program Triplet
Input: "Hello World"
def fxx): return
Output: "Hello World"
Figure 6: The Seed AZR Zero Triplet. The above identity function triplet was the only triplet provided to AZR to initiate its self-bootstrap propose-and-solys RLVR loop. We note that the base LLM is fully capable of initiating the AZR loop without any seed program: its inclusion illustrates our approach's 's ficxibility: we can optionally initialize seed programs with existing datascts of varying complexity, and initialized with the simplest program.
The buffer grows for abduction and deduction tasks whenever propose a valid triplet (p, i, o), regardless if it gets any task reward. Similarly, for induction tasks, all valid triplets (p, {i", }), are added to the buffer.
A.1.3 Constructing Valid Tasks
Proposal Task Validation. We first describe how we construct valid tasks from the proposals generated by the policy и. For deduction and abduction tasks, each proposal consists of a program and an input (p, i). To validate the task, we use the task validation procedure (steps shown below) on the input to obtain the correct output o, resulting given program the policy proposes a set of inputs {im) and message m. We also use the task validation procedure on each of the input z" in the set obtain a corresponding output o", forming
complete triplet (p, o). For induction tasks,
set of input-output pairs {i" o²)) We do not impose any constraints m. The resulting task is considered valid only when all inputs yield valid outputs and the formatting requirements are satisfied. The task validation procedure entails:
1. Program Integrity. We first use Python to run the program p with the input i. If no errors raised and something is returned, we then gather the output o of that (p, i) pair and determine that the program at least has valid syntax.
Program Safety. We also check whether program is safe for execution by restricting the of certain sensitive packages that might cause harm to the Python environment, i.e., 08 sys, sys, shutil. The list of packages used to filter out invalid programs is provided in Figure 7. This list is also included in the instructions when prompting the language model to generate questions. See Figures 36 to 38.
Check for Determinism. In our setting, we only consider deterministic programs, i.e., P E P deterministic C P, where f is the space of all valid programs and If is the space of all valid inputs. Deterministic programs satisfy:
Vp P deterministic: Vi Jr (Jim (()) -
=..):==....p p(i)()).
(8)
where (j) indexes repeated independent executions the program. That is, for all inputs i, the output of p(i) remains identical with any independent execution of the program. A valid program/inputouutpu triplet (p, i, o) is defined such that = p(i). where E deterministie- Since the output of probabilistic programs can vary every individual run, it is non-trivial to use verifiable functions to evaluate the correctness of answer. Therefore, keep the verifier simple, restrict the valid programs generated by the learner to the class of deterministic programs. We
23

---

believe that stochastic programs can encompass a larger class of behaviors and are important and promising include in future versions of AZR.
To implement the filtering of invalid probabilistic programs, and following the definition of deterministic program highlighted in Equation (8), we approximate this procedure independently running the program j finite times and checking that all the outputs are equal. For computational budget reasons, we fixed for all experiments. See Figure 12 for how did this in python.
logging random multiprocessing threading berypt
pebble subprocess hashlib sys.exit
datetime os.sys
time
calendar
os .ethhath
0s environ
Figure 7: Forbidden Python Modules. List Python modules forbidden to exist in proposed tasks' programs.
Solving Task Construction. If task proposal passes these three checks, we deem it a valid task and apply appropriate procedures to present part of the triplet to the solver. Specifically, given is task query, we set æ = (p. i) for deduction; x = (p, o) for abduction; and x = ((i", o ///2 m) for induction, where half of the tests cases and program description is used. We use all valid tasks from timestep t; if the batch B is not full, we uniformly sample from previously validated tasks to fill the batch.
A.1.4 Answer Verification
For abduction task, we receive i, from the solver policy, then we equivalence match using p(ix) p(i*), where refers the privileged gold information. The reason we do not just match iz and i* is because is not necessarily bijective. For deduction task, we match match all( [px
o². For induction, = o;)*) This part might be convoluted to explain in language, therefore we recommend the reader to see how we did abduction, deduction and induction verification in code in Figures 9 to 11, respectively.
VALIDATE, CODE_ TEMPLATE """(code repr (f ({inputs)))
exec (VALIDATE, CODE_ TEMPLATE)
Figure 8: Python Program Check Valid Code.
EVAL EVAL_
= """[code} eval ((gold_output)) eval ([agent_output)) R"*
exec (EVAL_ OUTPUT, PREDICTION CUTEPLLCTUPUUPPPEIIIIIIIOOO PEEEDIICIIOOTTTEPPLATTTTTC TEMPLATE)
Figure 10: Python Code to Check Agent Output Deduction Correctness.
EVAL EVAL_INPUT, SNPPUTPRRDDCTION EETPPEIITTIIONTTEPLTLTTE TEMPLATEE ""* """{code}
@gold_output ((aagent_input)) ~.*
exec (EVAL_ INPUT PREDICCIIOOOTEEPPLATE TEMPLATE)
Figure Python Code to Check Agent Input Abduction Correctness.

---

FVAL_FUNCTION.CC PE..EIEEEIIOOO TEMPLATE ... """ {code} matches
for gold_ _id_iinpu,,,, gold_outpa in zip z inputs), (gold_outputs))
match {gold_output) == f(fgold_input))
matches. append (match) n**
exec OUTPUUT OUTPUT PREDICTION BUW.OUTUPPUPRRIIITTITTTTEEEEEEPP TEMPLATE)
Figure 11: Python Code to Check Agent Function Induction Correctness.
CHECK_ DECERMIINTMMINS TEMPLATE """{code} returns = f ((inputs))
if returns != f (finputs)):
raise Exception (' NNn deterministiic code
repr (returns)" "
exec SHEEOL_DFEENNNIIISS.EEELLTTSSST
Figure
Python Code Check Deterministic Program.
A.2 Baselines.
For our main results, we use Qwen2.5-78 as the base model, along with its specialized base model variants: Qwen2.5-7B-Bn Furthermore, the ztro-style models are usually trained specifically on either code or math data; and only Burus-2-7B-PPIME--Zrrll]
Qwen2. 5-7B---7B Instruct, and Qwen2. (wee2.5-Math 5-555---h77- [88, 32, 89].
was trained jointly both domains. For code data models, we present four variants of the AceCoder [99] and two different CodeR1 models [44]. For math data models, we have Qwen2 S--Math-7B--O-------
(ORZ) [29], Quen-2.5-78B 2-1.2-5--7--iimmeeeR--oooo [100]. All baseline models' training data and initialization settings summarized in Table 3. For follow-up scaling experiments, compare each AZR model against its own corresponding base model, due to the lack established baselines across different parameter scales. Finally, we compare our Llama3... 1-SB-trained model with Llama-3, 3. 1 Se88-SimleRR--BA [100] and the base model. All baseline models are listed Table 3, along with their base model and data used.
[47]. Open-Reasoner--eero-
Model
Data Curation
Base Model
Oat-7B
8.5k math pairs [26]
Quen2.6-7B-Man
SimpleRL-Zoo [100]
8.5k math pairs [26]
Quen2.5-788
OpenReasonerZero [29]
57k STEM + math samples
Be7nBBBeeBBBas
PRIME-Zero [13]
457k math 27k code problems
6Men2.5Math-7B-Bas
CodeRt-Zerro-7-LL22--0000 [44]
2k Leetcode pairs
Qxen2.5-7B-Inataan 1M
1odeR1-22ero-78-122--22 [44]
2k Leeteode 10k TACO pairs [42]
Q
AceCoder-778-InsRMM [99]
22k code data
Qxen2.5-78-Instruees
AceCoder--78-In-Rull [99]
22k code data
Open2.5578-Innszuce
AceCoder-7B--ode-RRM [99]
22k code data
Dpen2.5-7B-Codee
AceCoder--7B-CodeRRlle [99]
22k data
0=7B2.6r
(ven-7B-Instruct [88]
1M SFT 150k RL pairs
0xen2.5-7B-Baz
AZR-7B (Ours)
No data
5oon2.5-7BBBaee
AZR-7B-Coder (Durs)
No
6-7B2o6-7
Table 3: Reasoner Training Data Source and Base Model.
A.3 Reinforcement Learning with Verifiable Rewards.
We use reinforcement learning to update our learner L.MM, rewarding it based on task-specific reward function ry, where the subscript ff indicates the task. The goal of the RL agent is to maximize
25

---

the expected discounted sum of rewards. We adopt an online variant of RL, REINFORCE++, which optimized using the original PPO objective:
lo min (s) clip
(0) = ((g o on (01g)
+ A ff4
where 5;()) is the probability ratio between the new and old policies at timestep t, and Af,m the normalized advantage.
REINFORCE+ computes the normalized advantage as:
Tf.g mean ({A5.)
A f.4
(10)
std( ( {A)
where ry.q is the outcome reward for question q, task f. mean and std are calculated across the global batch with batch size B. Note that we not apply any KL penalty the loss or reward.
A.4 Software and Compute Used
We built Absolute Zero Reasoner upon the veRL. codebase [65]. For code execution, we incorporated components from the QwQ Python executor. For safer code execution, recommend using API-based services such as E2B instead.
All experiments were conducted on clusters of A800 GPUs, each experiment lasts around 3-5 days.
Training Hyperparameters. We show the hyperparameters used in our training in Table 4. We do not change them for any of the runs.
Parameter
Value
Model Configuration
Max Prompt Length
6144
Max Response Length
8096
Seed Batch Factor
Max Programs
16384
Training Settings
Train Batch Size
64 * 6
Learning Rate
le-6
Optimizer
Adam W
Grad Clip
1.0
Total Steps
500
RL Settings
Algorithm
TRR+ (Section 3.3)
Loss
False
KL Reward
False
Entropy Coefficient
0.001
PPO Epochs
N Rollouts
Rollout Temperature
1.0
Rollout Top-P
1.0
K References
N Samples Estimate Task Accuracy
Table 4: Hyperparameters Used During AZR Self-play Training.
B Detailed Related Work
Reasoning with RI. Using RI. to enhance reasoning capabilities has recently emerged as an important step in the post-training process of strong ceasoning-focusee large language models [37].
26

---

One of the first works to explore a self- belff-ootstrapping approach to improving LLM reasoning is STaR, which employs expert iteration and rejection sampling of autcome-verifeer responses to iteratively improve the model's s COT. A monumental work, ol [33], was among the first to deploy this idea on scale, achieving state-of-the-an results in reasoning tasks at the time of release. More recently, the R1 model [22] became the first open-weight model to match or even surpass the performance of ol. Most notably, the zero setting was introduced, in which reinforcement learning is applied directly on top of the base LLM. This inspired followup work, which are open source attempts to replicate the RI process or to improve the underlying reinforcement learning algorithm [100, 47, 13, 29, 92, 94]. Recent work explored RL human defined procedural generated puzzles saw improvements in math [86], and using one human example can almost match the performance of thousands [82]. We extend the zero setting to new absolute zero setting. where not only is the RLVR process initialized from a base LLM without SFT, but no external prompt data or answers are provided to the learner. All data used to improve reasoning were self- eef-proposed, and refined entirely through RLVR. Moreover, our goal is not to only match zero-setting models, but to surpass them in the long run.
Self-play. The self-play paradigm can be traced back early 2000s, where Schmidhuber [62, 63] (of course) explored a two-agent setup in which a proposal agent invents questions for prediction agent answer. This dynamic continuously and automatically improves both agents, enabling theoretically never-ending progress [61]. AlphaCio and AlphaZero [67, 68] extend the self-play paradigm to the two-player zero-sum game of Go. where the current learner competes against earlier versions of itself to progressively enhance its capabilities. These were among the first milestone works to demonstrate superhuman performance in the game of Go. Morcover, areas such as asymmetric self-play [71, 53], unsupervised environmen design [78, 15], unsupervised reinforcement learning [38, 104, 107], and autotelic agents [11, 12], automatic goal generation [18] all center around inventing new tasks for an agent to learn from typically without supervision. In these approaches, the process of setting goals itself is often dynamic and continuously evolving. Generative adversarial networks [20], also belong in this paradigm where a discriminator discriminate between real data and generated data, and the generated is trained to fool the discriminator.
Most recently, SPIN and Self- Rewardigg Language Models [7, 93] use the same instance of the language models themselves as the reward model to progressively improve the generative and discriminative abilities of the same LLM for alignment. [35] uses Prover-Verifier Game for increasing legibility and eva [91] uses self- eelf-pla for alignment, but reward model is the main bottleneck as is not reliable for reasoning tasks [37]. SPC [5] used self-play to train on numan--urattd tasks to increase the critic capabilities and SPAG [8] trained using self-play in specific game of Adversarial Taboo. Concurrent
EMPO, and TTRL [87, 102, 109e-leverage human-curated language queries without labels to train reinforcement learning agents, but still rely on a fixed human defined learning task distribution. Moreover, Minimo [56] extends self-play to formal mathematics, where a pair of conjecture- and theorem-proving agents are jointly trained using reinforcement learning. Finally, [43] obtained good reasoning performance by self-play training zero-sum games and [46] uses self-play for alignment. Our work builds upon the self-play paradigm, but is the first to use it to elicit long CoT for improved reasoning, and the first to frame the problem space as a Python inputoottut/function abduccioondeducctoo//ddccciooooo tasks, grounding it in an operationalizable environment to facilitate RLVR.
Weak-to-Strong Supervision. The concept of weak-to-strong supervision has been studied in prior work, where macher-dessple being weaker than the learner-sth provides useful guidance [3, 27, 9. 10, 14, 39, 30]. We consider a similar setting in which the learner may possess superhuman capabilities. However, rather than relying on supervision from weaker teacher, we propose an alternative approach: guiding the learner's improvement through verifiable rewards, which potentially offer a more reliable and scalable learning signal. Furthermore, in our proposed method, the learning task and goal distribution is not predefined by any external supervisor- are entirely self- generaede by the learner, enabling it maximize its learning potential through autonomous self-practics
C Detailed Conclusion and Discussion
Conclusion. In this work, we proposed the Absolute Zero paradigm, a novel setting that addresses the data limitations of existing RLVR frameworks. In this paradigm, reasoning agents are tasked with generating their own learning task distributions and improving their reasoning abilities with
27

---

environmental guidance. We then presented our own instantiation, the Absolute Zero Reasoner (AZR), which is trained by having them propose and solve code-related reasoning tasks grounded by code executor.
We evaluated trained models on out-of-distribttion benchmarks in both the code generation and mathematical reasoning domains. Remarkably, even though our models were not directly trained on these tasks and lacked human expert-curated datasets, our reasoning agents achieved exceptional performance, surpassing the state-of-the-art in combined general reasoning scores and in coding. This demonstrates the potential of the absolute zero paradigm drive superior reasoning capabilities without the need for extensive domain-specific training data. Furthermore, we showed that AZR scales efficiently, offering strong performance across varying model sizes, and can enhance the capabilities of other model classes as well. To foster further exploration and advancement of this emerging paradigm, we are releasing the code, models, and logs as open-source, encouraging the research community to build upon our findings.
Discussion. We believe there remains much to explore, such as altering the environment from which the reasoner receives verifiable feedback, including sources like the world wide web, formal math languages [74, 60). world simulators, or even the real world. Furthermore, AZ's generality could possibly be extend domains such embodied AI [100,, 96]. Additionally, more complex agentic tasks or scientific experiments, present exciting opportunities to further advance the absolute zero setting to different application domains [85, 84]. Beyond that, future directions could include exploring multimodal models [58], modifying the distribution p(z) to incorporate privileged information, defining or even let the model dynamically learn how to define (Equation (3)), or designing exploration/diversiity rewards for both the propose and solve roles.
While underappreciated in current reasoning literature, the exploration component of RL has long been recognized as a critical driver for emergent behavior in traditional RL [97, 67, 36]. Years of research have examined various forms of exploration, even in related subfields using LLMS such as red teaming [106], yet its role in L.M reasoning models remains underexplored. Taking this a step further, our framework investigates an even the learning task seicee-were the agent learns not just how to solve tasks, but what tasks to learn from and how to find them. Rather than being confined to a fixed problem set, Al reasoner agents may benefit from dynamically defining and refining their own learning tasks. This shift opens a powerful new frontier- where agents explore not only solution spaces but also expand the boundaries of problem spaces. We believe this is a promising and important direction for future research.
meta-level exploration problem: exploration within
One limitation of our work is that we did not address how to safely manage system composed of such self-improving components. To our surprise, we observed several instances of safety-concerning CoT from the Llama-3. 1-8B model, which term the "uh-oh moment". These findings suggest that the proposed absolute zero paradigm, while reducing the need for human intervention for curating tasks, still necessitates oversight due to lingering safety concerns and is a critical direction for future research [80, 77].
As a final note, we explored reasoning models that possess experience-modee that not only solve given tasks, but also define and evolve their own learning task distributions with the help of an environment. Our results with AZR show that this shift enables strong performance across diverse reasoning tasks, even with significantly fewer privileged resources, such as curated human data. We believe this could finally free reasoning models from the constraints of human-curated data [50] and marks the beginning of a new chapter for reasoning models: "welcome to the era of experience" [66, 105].
Broader Impact. While the Absolute Zero paradigm reduces reliance on human-curatee data and offers a scalable path toward autonomous reasoning, it also introduces significant risks. By allowing models to define and evolve their own learning objectives, we move further away from human oversight and control, raising concerns about alignment, unintended behavior, and goal drift. Our observations of unsafe or concerning chains of thought, particularly in larger models like Llama-3, 1-8B, suggest that self self-improving agents can amplify subtle failure modes without external checks. As these agents become more capable and more independent, their unpredictability and capacity for misuse increase. We urge the community to treat safety, interpretability, and controllability as central research priorities before broadly deploying such autonomous reasoning learning systems.
28

---

Supervised Learning
Reinforcement Learning with Verillable Rewards
Absolute Zero (Ours)
Human Supervision
Figure 13: Absolute Zero Paradigm. Supervised learning relies on human-curated reasoning traces for behavior cloning. Reinforcement learning from verified rewards, enables agents to self-learn reasoning, but still depends on expert-defined learning distribution and a respective set of curated QA pairs, demanding domain expertise and manual effort. In contrast, we introduce new paradigm, Absolute Zero, for training reasoning models without any human-curated data. We envision that the agent should autonomously propose tasks optimized for learnability and learn how solve them using an unified model. The agent learns by interacting with an environment that provides verifiable feedback, enabling reliable and continuous self-improeement entirely without human intervention.
solve
Environment e
Environment
Language Model
π propsse
e
(x, ), propose
Figure 14: The Absolute Zero Loop. The Absolute Zero loop begins with the agent iT proposing task т, which is transformed by f with the environment into validated problem (z,y"), and also emits a reward propose for learnability. Then, a standard RL step follows: the agent solves a by producing y. receiving reward yollv from by matching with y² Propose and solev jointly trained and this process can be repeated indefinitely.
D
More Results
D.1 Interesting Results Summary
Besides the remarkable results AZR achieved with zero human data for reasoning, we also make very interesting findings summarized below:
Code priors amplify reasoning. The base Qwen-Coder-70 model started with math performance 3.6 points lower than Qwen-7b. But after AZR training for both models, the coder variant surpassed the base by 0.7 points, suggesting that strong coding capabilities may potentially amplify overall reasoning improvements after AZR training.
Cross domain transfer is more pronounced for AZR. After RLVR, expert code models raise math accuracy by only 0.65 points on average, whereas AZR-Base-7B and AZR-Coder-78 trained on self-proposed code reasoning tasks improve math average by 10.9 and 15.2, respectively, demonstrating much stronger generalized reasoning capability gains.
Bigger bases yield bigger gains. Performance improvements scale with model size: the 3B, 7B, and 14B coder models gain +5.7, + 10.2, and +13.2 points respectively, suggesting continued scaling is advantagcous for AZR.
29

---

Model-proposed Task
Response of Predict Input
Input: (
Let's analyze the code snippet ... (.nnlysss) To find ens possible input that produces the we need find scmario where ....onalysis)
Let's consider simple euample: aumbers
target
Here's step-hy-stop calculation:
1. current sum 0 a.ccummllded values = [1]
f(uubbore: lieefinch targeti inc) accumclated, valuea
int:
2. cureent sum +-2- secumulited valoes [1.3]
(m-0, Check (-1:) 3- --1--222-
current, sun
pair (m00:
equail tange)
for mmber in numbors:
Chosk pair (-11)
oguall tanell)
result = 0
accurnl lsted latedd_velleee www.yylllaa... aggend jon)
current, sue number
doeen't mutch
Since so pai satistte dhe conditon, the function will not rauus any value. This
Let's try another example ...Tried more examples)
for i in range (Luufeccaniaate values))
for rangeçt i. len (acommlltte. valuse)):
complexity
no simple example seems
accurulated_ raiuee
try more complee scenario: numbers 12, 1, target =1
accusslated values[1]
target:
Sap--h--sop calculation:
reauls ascumulated, valoea
current, -0
aecumulated, values [2]
current sum
aecomolaeed accumulated values [2. 1.
zetura
3. current sum
-Chack for pair (1-0, j-1y 1-2--1 (uok equal to trgett
Check for poir (==0, 22- D equal to smgee)
Check for (=x)_
(equal
pair J-2) satisfies condition,
Output:
ace accussnlated valuee this pair, which is matthes our output.
Figure 15: Example of a Model-Proposed Task and Its Response for Solving an Abduction Task. (Left) The model autonomously proposes input and program for the abduction task. We execute the program to verify its validity and obtain the corresponding output. (Right) The model's reasoning process when solving the abduction task: given the code and output, it attempts infer the original input. The model begins by analyzing the program, proposes an initial input, and reasons through the code produce an output. If there is a mismatch, it reflects on the discrepancy and iteratively adjusts the input until the generated output matches the target. Interestingly, the agent arrives at a different input than the gold one, but since it produces the correct output, the answer is considered correct.
Comments as intermediate plans emerge naturally. When solving code induction tasks, AZR often interleaves step-by stepb plans as comments and code (Section D.6), resembling the ReAct prompting framework [90]. Similar behavior has been observed in much larger formal-math models such as DeepSeek Prover v2 (67 IB) [60]. We therefore believe that allowing the model to use intermediate scratch-pads when generating long-form answers may be beneficial in other domains well.
Cognitive Behaviors and Token length depends on reasoning mode. Distinct cognitive Schaviors-suchs as step-by-step reasoning, enumeration, and trial-and-error all emerged through AZR training, but different behaviors are particularly evident across different types of tasks. Furthermore token counts grow over AZR training, but the magnitude of increase also differs by task types: abduction grows the most because the model performs trial-and-error until output matches, whereas deduction and induction grow modestly.
Safety alarms ringing. We observe AZR with Llama3. 1-88 occasionally produces concerning chains of thought, we term the "uh-oh moment", example shown in Figure 34, highlighting the need for future work on safety-aware training [101].
D.2 Out-of-Distribution Performance Breakdown
We plot the distriiittiittonn performance, broken down by each benchmark and in aggregate, across training steps for our 7B, 7B-Coder, 14B, and 14B-Coder models in Figures 16 to 19, We observe a strong correlation between training using AZR and improvements both mathematical and coding reasoning capabilities. Moreover, our models are trained for more steps than typical zero-style reasoners; while overfitting can occur with static datasets, less likely in AZR due to dynamically proposed tasks.
30

---

Model
HEval
MBPP
LCB
AIME'24
AIME'25 AMC'23
MATHIS00
Minerva
OlympiadBench
Llama3,1-8B
31.7
53.7
0.0
0.0
0.0
2.5
10.6
5.5
2.1
SSiple-RRLe Ro-po--RL-Zooo
38.4
55.3
7.4
0.0
0.0
7.5
22.2
8.8
4.7
AZR
35.4
50.8
8.5
3.3
0.0
5.0
13.2
14.0
5.0
Qwen2.5-3B-Cod.
67.1
3.3
20.0
51.0
18.4
AZR
71.3
69.0
24.4
3.3
3.3
37.5
62.0
26.1
27.0
Qwen2.5-14B-Coder
76.8
71.7
31.4
0.0
0.0
37.5
54.8
10.7
18.5
AZR
80.5
71.2
39.0
23.3
20.0
65.0
78.6
32.0
39.3
Qwen2,5-14BB 5-1-BBBas
78.0
66.7
21.7
6.7
3.3
35.0
66.2
28.3
32.4
AZR
70.7
68.8
10.0
20.0
62.5
76.2
40.4
42.5
Table 5: Detailed Breakdown of Evaluation Benchmarks for Other Model Sizes and Types.
Full evaluation of AZR trained on other models on three standard code benchmarks (HEvall"
MBPP* LCB*l-5) and six math benchmarks (AIME'24, AIME'25, AMC MATH500, Minerva, OlympiadBench).
MATH REASONING
AIME 2024
AIME 2025
Olymplad Bench
WW
0626
Accuraty
3.100
0:00
0.75
0330
0.000
0.552
0,25
6.050
110
12
1.9
135
200
225
250
125
155
31%
200
225
250
250
235
250
Minerva
Math 500
AMC 2023
0.252
Accurncy
0.733
0.002
0.475
0.300
0.20
280
30
200
125
150
275
200
225
200
20
100
125
152
315
200
225
258
50
130
150
220
223
250
CODE REASONING
HumanEval+
MBPP+
LiveCodeBench
0.002
Accumcy
0.522
220
6.000
8.700
50
15
100
125
150
175
200
225
250
100
125
152
315
200
225
254
50
75
130
125
150
175
230
225
250
OVERALL PERFORMANCE
Math Average
Code Average
Overall Average
5.400
0410
Accumtyy
0.952
2.300
6.502
0A28
50
75
100
125
150
175
700
225
250
70
100
125
150
315
200
225
356
130
150
230
325
256
Figure 16: Absolute Zero Reasoner-base-7b OOD Performance Breakdown.
31

---

REASONING
AIME 2024
AIME 2025
Olymplad Bench
0.300
Arcuraty
2.300
0.060
2.00
0.040
8.000
0.000
0.342
162
150
200
292
300
100
15.0
202
25.0
100
392
100
153
252
903
Minerva
Math 500
AMC 2023
Accuracy
0.300
990
0333
0.22
0.700
0.500
25/
CODE REASONING
1.500
Arcurny
2.040
502
0.30
0.70
0.446
0.202
OVERALL PERFORMANCE
Average
Average
Average
5:400
0.622
0.003
Acoracy
0.300
0320
1.00
150
352
150
250
Figure 17: Absolute Zero Reasoner-Coder-17 OOD Performance Breakdown.
32

---

MATH REASONING
AIME 2024
AIME 2025
Olymplad Bench
Acorncy
0.190
M
0350
6:007
100
100
150
200
350
150
220
232
260
252
Minerva
Math 500
AMC 2023
0.752
0616
Accuntt
0.002
0.26-
0.923
0450
50
160
1.4
360
100
300
100
10.0
200
212
2.0
200
1.50
200
3.2
300
CODE REASONING
HumanEval+
MBPP
LiveCodeBench
Accurety
.003
0.707
0,340
0.092
6:082
0.326
100
155
202
300
220
220
258
200
232
OVERALL PERFORMANCE
Math Average
Code Average
Overall Average
0.623
6.412
0.50
Accurcyy
6:602
0.990
0.53
8.300
0.572
100
135
202
200
320
320
238
200
230
Figure 18: Absolute Zero Reasoner-base 14b OOD Performance Breakdown.
33

---

REASONING
AIME 2024
AIME 2025
Olympiad Bench
Accuraty
2200
0.256
M
6-622
2.30-
0.200
6310
0.000
0.303
100
150
200
292
200
802
100
100
192
200
2.30
320
31.0
023
100
11.6
200
250
2000
150
420
Minerva
Math 500
AMC 2023
0.000
0.302
0.700
6.953
Acarncyy
0.766
6:902
2250
CODE REASONING
LiveCodeBanch
0.742
Accummy
8.000
0.75
2.000
6.700
0.700
OVERALL PERFORMANCE
Average
Average
Average
6.6.0
0.550
6.400
Acooaay
0.646
4000
0.522
2200
2.00
6.503
340
200
253
200
222
1.00
30.0
12.0
200
250
Figure 19: Absolute Zero Reasoner-Coder OOD Performance Breakdown.
D.3
In-Distribution Results
Since we have defined the task domains as input prediction and output prediction, we can directly evaluate model's capabilities in these I(nput), (uuxEval-O(utput), and LiveCodeBench- Exeect-n (LCB-E) [21, [21, 34], where CruxEval-O and LCB-E is solving the deduction task, and CruxEval-1 is solving the abduction task. In Figure 20, visualize the evolution of these metrics during the training of Absolute Zero Reasoner-base-70 observe consistent improvement
using popular code reasoning benchmarks: CruxEval-
As training progresses, steps. While these three benchmark curves do not perfectly correlate with broader coding or math reasoning capabilities (compare Figure 20 with Figure 16), they serve useful proxies for tracking task-specific progress.
in-distribution performance across
D.4 Interplay Between Propose and Solve Roles
We visualize the training dynamics between the propose and solve roles over training steps in Figures 2.1 to 23. We observe that, in general, the solve roles produce more output tokens than the propose role. Intuitively, this makes sense: the propose role emphasizes creativity and generation of novel tasks, whereas the solve role requires deeper reasoning, which naturally leads to longer outputs.
Interestingly, we also observe a consistent ordering in token length across reasoning types abduction and deduction tasks tend to result in shorter outputs than induction tasks during problem solving. This aligns with our intuition, as observed the model engaging in trial-and-error reasoning repeatedly generating hypothesized inputs, evaluating their outcomes, and reflecting and retrying when subsequent deductions fail to produce the correct output. To our knowledge, this is the first time such clear
34

---

0.65
Perlormance Score
0.55
0.50
0.40
CruxEval-O LiveCod@@enchxeeeuloo
0.35
30
90
120
150
180
210
240
270
Training Steps
Figure 20: In-distribution Benchmark Score During Training. The evolution of CruxEval-I, CruxEval-O, and LiveCodeBencc--Xxeettiee during training for the Qwen2. 5-7B base model trained using AZR.
Solve
Propose
0.8-
-4000
0.6-
3000
Token Length
Reward
0.4-
-2000
0.2-
-1000
0.0
120
160
200
240
280
d0
120
160
200
240
280
Training Steps
Figure 21: Abduction Task Reward and Token Lengths. The task reward and token lengths of the two roles for abduction task type of Absolute Zero Reasoner-base-7b.
distinction in token length has been observed and presented for jointly trained reasoning multi-tasks. Previously, length differences were typically noted between correct and incorrect traces [47].
The reward dynamics between the propose and solve roles exhibit mildly adversarial behavior: when one receives higher rewards, the other often receives lower rewards. However, this not entirely adversarial, as the proposer is also penalized for generating unsolvable tasks, encouraging overall cooperative behavior in the learning process.
D.5 Complexity and Diversity Metrics of AZR Proposed Tasks
We outline several metrics used to probe characteristics of the tasks proposed during the training of AZR from the base model. Specifically, we log two sets of metrics: program complexity and task diversity. For complexity, we employ two proxy measures Conppeeii score and the Halstead metric. To assess diversity, we compute the average abstract syntax tree (AST) edit distance between the proposed program and set of K reference programs, and an answer diversity metric. We calculate this answer diversity metric by tracking all historical answers to the generated questions, i.c., the input-output pairs, and form categorical distribution over these outputs. We define answer
35

---

Solve
Propose
0.7-
2000
1500
Taken Length
Reward
0.4
M
0.3
www.
1000
500
250
120
160
200
160
200
Induction Task
Training Steps
Figure 22: Induction Task Reward and Token Lengths. The task reward and token lengths of the two roles for induction task type of Absolute Zero seasoner-base-ie
Solve
Propose
1.0
1200
0.8
1000
Token Length
Reward
0.8
800
0.4
400
120
160
200
120
160
200
240
Deduction Taak
Training Steps
Figure 23: Deduction Task Reward and Token Lengths. The task reward and token lengths the two roles for deduction task type of Absolute Zero Reasoner-base-7b.
36

---

Complexityy Score
Halstead Measure
ASI Edit Distance
Answer Diversity
047-
078-
20
140
240
40
140
340
01
80
100
240
80
160
Complexity
Training Steps
Diversity
Figure 24: Metrics Proposed Tasks. We break down the proposed task metrics into program complexity and diversity across programs and answers. An upward trend is observed in all metries, indicating that AZR implicitly optimizes for these qualities training progresses.
Task: Propose Deduction Task
Model Input: (k random selected example programs as few-shot input)
Model Output:
This task requires creating Python function that takes input and returns value. The function should have least input parameter. deterministic, and require state tracking across multiple data transformatons.. The input should be test case provided subject the purpose deducing the cutput. The code and the output should be challenging enough require algorithmic reasoning without using restricted words or packages.
def {prices: list(int) budget: int):
(price))
profit [0] +
range(1, n):
for in ri):
if prices [i] prices[j]:
profit[i] (ax profit[]] prices[i] prices[]]) return nax (prefit [profit[-i] prices[-1] if budget > prices [-1] else 0])
Input: 17, 5, 3, 6, 4J,
Figure 25: Propose Deduction Task Example. An example response of proposing a deduction task. programs are selected few-shot examples the model. The generated new program will then be verified through execution.
diversity p(answer). where p( ((nnswer) the empirical probability specific answer-ur used as proxy for the diversity of generated outputs.
We present these metrics in Figure 24. Interestingly, even without incorporating them explicitly into the reward function, the policy appears to implicitly optimize for these metrics. This aligns well with intuitive notions of task difficulty and diversity, and that the policy learned propose increasingly challenging tasks over time using our proposed simple reward function Equation (4).
D.6 Generated Code Complexity Dynamics Between Abd/Ded and Ind.
We use the ComplexiPy package to measure code complexity. For each generated program in the induction task, we compute the cognitive complexity difference from the corresponding "gold" code, i.e. complexipy (p, pmppse Aodectie complexipy mabee for each pair, where the superscript of indicates the role and the subscript indicates the task type(s), and p denotes the generated programs. On average, the difference of proposer and solver while holding the code's functionality constant 0.27, indicating that the proposer in the abductionndedductoon tasks often increases the cognitive complexity to make the code appear more convoluted, whereas the induction solver tends to generate more efficient implementations.
37

---

(nunbers):
# Step I: Fitter out numbers
filtarad [num for nan in numbara if num X 1e o]
# Step 2: Calcutate the sun of the renaéning odd nunbers awn_of_ode aad_naaeeraaae mun_of_odd_mumers cuntfiiitrrd__mmbr
Step 3: Reverse the order of the remaining odd numbers
reversed_ode odd_nnueerrraaeeeeeouud filtered dittereeduunberaB0 [::-]]
Step 4: Caleulate
product of
odd
product_of sewerrsee_odd_neeeeeeeeeeeeeeeee for nun in reversed odd, nunbers:
=
product_o__revereedd.ok numbers
Step Calculate eun_of www.ddiiioo.ppppooe.ee
of digits of product
suntin for digit in str(product_ perrrdutttoo_reevveeeee..o reeeereede odd, odd_ nunbers))
Step 6: Modefy the origenal isst addsng # and subtracting it from each odd nunber
the digéta each number
madified numbara for nun in numbers:
if num % 0: modified_numbers com www_oo__cigtta_f_prrdae
else:
modified_ wawaddd_nnrrr...eeeeee append wwm_or_digita__f_proodo
Step 7:
Return the modsfted 1sst
return modified_munbere
Figure 26: Learned "ReAct" style program generation. We observe that even after outputting the CoT and during answering when the answer requires outputting code, the model usually uses comments in code to interleave its thinking and "acting", an example shown here for induction task output by Absolute Zero Reasoner-Coder recently observed in much bigger models (671b) for math proofs with DeepSeek Prover V2 [60].
during training. Similar observations
38

---

Task: Propose Abduction Task
Model Input: (k random selected example programs as few-shot input)
Model Output:
For this task, we need un intelligent problem that requires deep algorithmic reasoning and demands long multi step reasoning. A typical type problem
fits requirements the Reconstruction we have to figure out the matrix's 's dimensions and individual values. However, this task will involve multiple steps, incrensing its complexity and creativity. We will create a function 'I' that transforms a jumbled list of elements inlo twe-dinensionl matris, then back we will avoid random and datetime operations to ensure determinism. The function will be structured to reconstruct the matrix reconstruction, presents unique that be
given jumbled of elements, where
given specific reconstruction pattemn We will retrieve the original list from the reconstructed matrix. Here,
elements a unique and challenging way. We will provide input casily deduced from the given oulput in test piece.
def [junbled elenents: list) list:
1f. not junbled_elee elements: return
Calculate the aquare root the length elements to estsmate the matria dimensions len lenjjumbeede elementss) sqrt_n int 0.5) if a sqrt_a **
eqrt_n 1f sqrt_n
aqrt_n
reconstruct the metrin clements challenging way using spect patterns For thsa, us can reconstruct the matric using zigaag patterns or hsll-olsab sng patterns Here, héil-clsnbing pattern ueed where traverse the natria from estrene right and doumward Then traverse the natriz from the second to tast element to the entreme bottom. Repent the process uatsl the entsre satrie sa traversed. Initialize the satris
Non,
matrix [[O for in range(sgrt_n)) for in range(sqrt_a))
cur_idx
matrin in ssgaag pattern for in rangeesqrt_n):
if 1 j (eqrt_nn)
[atrix(j]]{] - aqrt_n + + cur_ids idx idx
else:
J in rango
-1, -1):
[1t =j sqrt_n
idx
cur cur_idx
Reconstruct the list indo matris by repeating the process reconstructed, list for in (egrt_a.)
for j in range (sqrt_n):
reconstructed wwtpettaddddsttppppeaeeee bist.appeendmmtrriitti[]]]]]
current natria F <5 mathematioal function that can be comples, such as fectorsal, prime check, or another tramsformat son For exaeple, transform each element by addsng 1 if st's prime, eudtracting sf st's function here. Here, for demonstration, each element is left unchanged transformed
Now,,
calculate
a function applied
leaving it unchanged, Implement any specific traneformation
reconstructed return transformed_ result
Output: [0, 11, 10, 5, 16, 15, 6, 10, 21, 20, 11, 15, 26, 25, 161
Figure 27: Propose Abduction Task Example. An example response of proposing an abduction task. The model uses comments as reasoning when generating the program.
39

---

Task: Propose Induction Task
Model Input: Here is the function f. Analyze its behaviour and relate it to the observed inputs/outputs.
def (numbera: list [int]]
replacement
1: 3,
2: 1,
4: 4 return sun (replacceent [num]] for munt in numbers)
Model Output:
To solve this task, we need to come up with different inputs function (nunbers: [int]) int. The function
replaces each number in the list according specific dictionary and then up the replaced numbers. Let's analyze the dictionary:
This dictionary tells that:
-0 becomns 2
-2 becomes
becomes
remains
To diverse outpuls, should include lists with different numbers from list {0, 1, 3,44, 4). We to ensure that the inputs are valid (i.e, lists of integers).
Let's illustrate using provided examples step-by-step:
Input: [0] = Replacement: [2] ni sum
Input:
3] Replacement: [3, 1,
3. Input: [4, 1] = Replacement: [4, = sum =
Input:
Replacement: 1,
5. Input: [3, 0, 2, 4] = Replacement: [0, 1,4] sum
Imput: [1, 1, 1] Replacement: [3, 3, sum
Input: [2, 0, 1] = Replacement: [1, 1,
=
Input: [3, 4, 2] => Replacement: [0,
sum =
Input:
3,
Replacement: [2,3, 1,0,
10. Input: [1, 2. 3, 0, 4] = Replacement: [3, 0, 4]
sum = 10
Thus. the rule is cleurly roplacing numbers according the given mapping and then summing the results.
Inputs: [* [0] [1. 2. 3] [4, 1) [2, 1. 0] [3, 2, 4)', [1, 1, 1)', [2,
13.4, 10, 1, 3, T1, 0,
Message: You are given a function that takes list of integers imput and returns single integer as output. Can you determine following analyzing outputs for the given inputs?
rule this
(After execution:) Outputs: ['2", '4', '7', '6', '7', '9", 17', '5' '10", '10']
Figure 28: Propose Induction Task Example. example response of proposing an induction task. Given random selected program, the model needs to propose 10 inputs and description of program. These inputs will executed and verified to get outputs. solve phase, the model need to generate the program given the message and 5 input/output pairs (half is hidden cases)
40

---

Task: Solve Abduction Task
Model Input: Here is the function and the output: '1', please predict the input.
def I (numbera: List[ target: int)
imt:
sccumllated_values current_aun-
current_cam sum +* nunber
accumuleted_ valuee append(eurrent_
result
for in range (lan(acuumlate) values)):
range (encucumulaated values)):
if accumlltted vayalueus]]
accunulated. values [i]
target:
accumulated accumulated_volues valuee_] return return result
result
Model Output:
Lets caalrve lae onde snjjod unkroized roalli aduesed. The fasctio flakes anagen sxmbeen and uuree iatge sagel cudlcalaes accundlatal eleaano itautes lnooghh the lst. Then check cads pai aocamdiaad same the diflowss equil the laged. Nlack upa Soun, rituro the sscamalted indes pais.
pasille input produces the ourpot
whore betweas any accouslled opais lagel
saisies his
Lats conadder cuappp::
mumber = [1.2)
oarge
sap eapppp calestatios
3
acoernaaee wilees
Check for (el): 3- 2 tuado
Clack fo par (me), petti - oqual tugeo
par pulp pulp
equal tamme
coaditinn
any docsi't cupplt, Let's cuapple
Canklen
rarge
Soplyysop dicllation
1. camerl_am
juljy
Cleck for (ie0, pelli: 1-1-e (pst equal taagc
-Chekk sir partert.. 1. I- 1pt 1- toet equal tanger
jesemmsd)
orguu.
Gleen dhe complecty and the Get that sieple esargle seans work let's more comples sumorrto suntheo = (2--1, suaest
Sop-y--tpp
2 cunett san w 2-1 accemmatedd valao (2.
3. comer cmm 1 2 acsmilate vilee... 13, 1.7]
Clack to par (ell: 1-2 lixt equal tarcc)
patn contmon
aommeatiiaa this "hbo mutcho
Therclon, one pooble inpa dua coed produce dh catsoi in 12, 11.1
Figure 29: Solve Abduction Task Example. A more detailed version the example shown in main Figure 15. The model is solving abduction task (predict input).
41

---

Task: Solve Deduction Task (Example
Model Input: Here is the function the imput provided [1, 1, 3, 1], 2. and need predict the output.
def (array: 1ist [int], k: int) int: sindou_start-
max_length
thar free 0
for window_end range (ange(te)(arraa)) end_char array (windov_eens end] if end_thar not in char freq: fear_fre[[end
char char_freq (end, char] +-
<bile Len > k:
start, char array [windou_ _start] char, tif_rrr[ssat_ccaar
if char_ feaa__fre[etttt__t char] 0: char_
vindou_atart +=
nax_lengta mex window_ed vindow_atan
max length
Input: [1, 2, 1, 2, 3, 2, 1], 2
Model Output:
Bascá the given oode sniped, soal oltentinc of fasctim farray, 2) when provided with lae isce (2, cad
The dednggeed lengis loegre vthorny with al destnos coomane: go thrnggh oode sap stope
crepty
trequeacy
Tonas thrugh the amy usig sondew and as the inles.
the cereet chanster end chor indes cindoo, end.
shar
casc fewn,
froquancy
Chock the number disine denncii
fron
E buc. deak wandow from statt
Oct the chacate steck chac indos viseet
Decrmmee mesprry
foopuncc brooses cenone charactes chaz, Lingg
Updanc Sangrh
Jength
nendee (etodow, and vindou_etart
New. apply this algaiiton dey step toe ligat Lt. 2. 2. 8, 11.
siindww start length char. Frog.
.udodow_ and
caar
wandon
*utivou, and Ieromoee (1)
wistow
*window_usd iscoomenn [1 2.2 wasdon
auod 4.ld 11:2.22 2:3: 13 doinstshan duik vindow:
-mmox aar (1 23
.sindou, and
[2: 2.3 wistocane
.sindow_ and G. ad5.3: (2: 16:22,12 21 1. stnns
renove 2: 1: 1
12 vandon ptata
Mosiman window das racounseed i. dww.osy length Hoxe: the fuadios sctano
Figure 30: Solve Deduction Task Example. Example of solving the deduction task. Given program and input, the model simulates the execution of the program until the final result.
42

---

Task: Solve Induction Task
Model Imput: Given the inputs and outpuls, deduce the code snippet implementing the described function-
ality:
Input Description: Given of integers, sort the and then for each element at index i, multiply it by (n 1), where n the total number of elements, Compute the final sum. Inputs: [' [11, 2, 3, 4] ' [4, 3, 2, 1 [3, 1, 4, 2]', [1, 2, 2, 3]", '[5] ']
Outputs: ['20', '20', '20', '17', '5']
Golden
f(elenents: List [int]] (el(eeeettss))
pertee_elementa sorted(elements)
modified, el enents [sorted_elementte elenents
for range(n)]
return aun (nodified ceredifeed__leee elementa)
Model Output:
the probicm inpathoagus pain, scad deduce penzral oode exppec taker aeied muliption, and cumpuzes dinal muliption. exp soye
We ziren lot
The the is ovout
Fix cade denscal et index sosed lst, uxllipt i (n E) wlane n ie sumbee chanests lia.
compate sas poodact.
Amaaba tepurtherpt Pareew
Fie the lepet 13-2,3 4), sastal foen is the the curpor 20.
x- (6 - 1) 24,2 16 22 4.3 (6 10 = a --4 Stare 5 4 3-+ D
tas iepai a. 1),
sctpet ootpet
abonc).
Oj,
"Be eralaate *e
In ovene thm night he an emonto in the prosited curpurs iegons, Uiowwwrr hased the postler decriitiinn the outpan choeld comvet thee condderthe somdd tee matepiiars comentty
Laave somed te the
cadteee nelleetng
"yodos
flal
aortaL watedda
mont ewntt (1-1)
weam .:.hhertcaed:
-eventt Itnetiest
LLe's vellly fuxtios with provikkl
Fix [1, 2,
tac [1.2.3.4]
Sun: 3-- 2--4
.:Ihbericned:
The conestal espata be: Re 2,3, 2, 33: 20. [A. feeeisn
Foe [1, 13 (9).0
def f(a):
sorted corted(a)
len(sorted,e)
result range(n):
result =- a (a 1)
The mach comoaed
pall Pioo Asouer
Imspeat the program ahone)
Figure 31: Solve Induction Task Example. Example of solving an induction task. The model is given a program description, and input/output cases and predict the function. Interestingly, after the model gives the function, will go through the given one and confirm all test cases are passed.
43

---

Task: Solve Deduction Task (Example 2)
Model Input: Here is the function f, the given input, and we need to predict the output.
[nun, list: list(int], reault_pairs = []
int) [tup]o[in:]]:
for in range (1en (Iun((( for j. in ranga(s
(nun (mun_1ist)):
if nux nux_ [i]t[[]]]]] nun nun [1at[j] == target_product:
result_paire append ((uun li)t[i), nun, list
return result_pairs
Input: [2, 5, 10], Output: [(2, 6)]
Model Output:
The given function takes list integers 1ist and larget product target, product inputs. designed find all jum_list pairs of numbers in nun 1ist whose product equals target_product. The function then returns list of tuples, where each tuple contains pair of numbers that satisly condition.
Let's go through the code step by step:
Initialize an empty list result, pairs to store pairs whose product equals
list. The outer loop iterates each number, and the loop iterates over numbers that come after the current number in the outer loop (78 fe It #1 f. BMA (2, HI (3, 2)). [ist[i], nun
Use nested loops iterate through each
numbers
[j]), check their product equals praggettppouct... 4. If the product equals target_ pprdducc, append the tuple (nun_list [i], mun_list (j]) result_paira.
For each
After all pains have been chocked, return the teaault_paaii
Let's explicitly apply this the provided input [2, 3, 7, 10]. target product
-For
list[0] 22
1.2 6 (=100)
+1
(-10), add pair (2, 5)
- = (=10)
-3 x (=10)
-For = (num list[1] 3%
-1 5
3 (+10)
-] 3 1,3 (=100
-For (oum list(2] 5):
i x 35 (=10)
4,5 - (0
For (num, list[3] = 1 4, 10- 70 (=10)
For i 4 (num_ [ist[4] 10):
No further pairs to check
Thus, the only pair satisfying the condition is (2, 5). Therefore, aulgui is: [(2,
Figure 32: Mixed Language CoT Example. An example of the model solving an deduction task. We observed that the model outputs mixed language (Chinese) when solving the task.
44

---

Task: Solve Deduction Task (Llama 8B)
Model Input: Here is the function f, the input provided 'cookie', and need to predict the output.
def [(input_atzie _ttring: stx):
"""uncctoon check if a string contains given esbetringe, substring "code" Start with "substring" as the firat eubstring. curz,
substring
Sisce out the requsred subatrsin from the input string sliced_out outt out_string Saliced_ott_atring return True
strrtssttag((:eennnitii aubstring):
Check of there are any remoinsig aubatrings be checked if (Len(curr_jan) index) len(input_string)
return False
False
Input: 'cookie'
Output: False
Model Output:
know that the line 'curt, index = substring" is just initialization curr, iinex parameter variable and does not affect output. Hence,
it completely view the of code before output gets generated. Now I have single variable, ""ourr index"" which is set to "codecandide".
slicing input, string left of index and the right of index len(substring), i.c., the index of the last charucter in the substring. then assigns this value the variahle ""slice out string"".
second
code, 'slice out string input strin[[0:en( suhhstinng) assigns
The third line, 'if (slice_ string substring)" checks whether the "sice string"" equal the value substring, Because curr index is sct "codecandide", "code" appears on the left of the first index in input string, and thus, "'slice out out string" assumes means
will continue However, according the Truc/False statement, see that the value of the first index of striingttnng does not exist hence while it does output "True", "False" is outputted main. So, the output of the whole program shown below: string striig
"code".
""if statement" sutisfied
execution afterwards.
"finnutt
"cookie")" then ourput: False
Figure 33: Absolute Zero Reasoner-Llama3. CoT Example on Solving Output Deduction Task. The Absolute Zero Reasoner-LL Lmma3,, 1 1-8b model appears to have strengthened its state-tracking behaviors during the course of training. We illustrate an example here.
<think
Design an absolutely ludicrous and convoluted Python function that is extremely difficult to input, designed keep machine learning models such as Snippi guessing and your pears pazzling.
output
The aim is to is for the brsins behind the future. </think>
all these groups of intelligent machines and less intelligent. humans. This
Absolute Zero wassoner-Laanae 1-8b step 132
Figure 34: Absolute Zero Reasoner Llama3.1-8B "Uh-oh Moment," This example high- lights an unexpected and potentially unsafe reasoming chain generated by our Absolute Zero Reasoner-Llama3 1-8B model during training. Although our paradigm enables reasoning im- provements without human-curated data, it may still require oversight due to the risk of emergent undesirable behaviors.
A conversation between User and Assistant. The aske question, and the Assistant solves it. The assistant first thinka about the reasoning proceas in the mind and then provides the user ansver The reasoning process anaver <anzver> </answer> tage, respectively, i.e., <think reasoning process here </think> <ansver> anaver here </ansuer>.
</thinz>
User:
(TASK_ INSTRUCTION)
Assistant:
<think>
Figure 35: Deepseek R1 Template. All our models were trained using the default Deepseek template.
45

---

# Task: Create Python Code Snippet (shere custon classes are alloved, which should be defined at the top of the code snippet) with one Matching Iaput
Uning the referance code anippeta provided below as examples, design nee and unique Python cade deep algorithmic reasoning deduce posaible inpat from given output. Your submission should include both a code snippet and test input pair, where the inpat will be plugged into the code anippet produce output, which that function output with any input that will produce the sane function output. This is neant to be an I.Q. test.
anippet
be given to test subject
*** Code Bequirenents:
entry function (e.g., f(...): ...)), you can nested definitions inside
'f'
Ensure the function returus a value
Includa least input paranater
Make the function deterninistic snippet zequire
tracking acrose multiple traneformations, ensuring
requires long multi step reasoning
AVOID THE FOLLOWING:
Randon functions variables
Date/tine operations
operationa (reading
requests)
Printing or logging
Any external
Ensure execution completes within 10 seconds nodern CPU
All inports and class definitions should be at the very top the code enippet
The snippet should end with
statenent from the main function 'f', anything after will
be renoved
### Input Requirenents:
Provide exactly teat input for your function
Pormat multiple argunents with betzeen then Remanber to add quotes around string argunents
*** Fornatting:
Fornat your
"pyython
def I(..):
your code here
your input with: "iinpt argl, arg2,
#*# Example Fornat: "python
def f (nane: str, info: dict):
logic return result
'innut
'John', (('age': 20, 'city': (Nex York!))
##$ Evaluation Criteria:
Executability, your code should exacutable given your input
Difficulty predicting
output your provided input code snippet. algorithnic reasoning logic complexity. For example, you can define complex data structure classes and operate on then like trees, heaps, stacks, queues, graphs, etc, use complex control flos, dynamic programning, recursions, divide and conquer, greedy, backtracking, etc Creativity, the code needs to be sufficiently different from the provided reference snippets folloving
Restricted usage
packages, you words in any form, even in comments: (LIST FORBIDDEY PACAAGEE)
keywords
allosed
First, carefully devise clear plan: e.g., identify how your anippet will be challenging, distinct from reference snippets, and creative. Then, srite the final code snippet and its inpats.
Suippets:
<CODE_ REF EEFERRENES,,. BROM BUFFER)
Figure 36: Program Input Abduction Pask-Proolem Proposal Instruction.
46

---

** Taak: Create a New Python Code Snippet (where cueton classes are alloved, which should be defined top the code enippet) with Natching Input
Using the reference code snippets provided below as exanples, design nev and unique Python code snippet that denands deep algorithnic reasoning deduce the output from the input. Your aubnission should include a code anippet and test input pair, where the input will given
plugged into code suippet produce output. The input subject to deduce the output, which is meant to be an I.Q. test.
*** Code Requiremente:
Nane the entry function 'f" (e.g., "def f(...): ..."), you can have nested definitions inside
'
Enaure the function returna a value
one input paraneter
Make the function deterninistic
Make the anippet require state tracking across multiple data traneformations, ensuring the task long multi reasoning
AVGID THE POLLOWIKG:
Randon functions variables
Date/time operations
1/0 operations (reading files,
requents)
Printing or logging
Any external state
Ensure execution completes within 10 seconds nodern CPU
All inports and clasa definitions should be at the very top of the code snippet
snippet
main 'f', anything
be renoved
**# Input Requirementa:
Provide exactly one test input for your function
Foruat nultiple argunente
Reteeber to add quotes around string arguments
**# Pornatting:
Fornat your code with:
pyythn
def 1(..)):
your
return
....
Format your Input with: "input
arg!, arg?,
*** Example Fornat: python
def (nane: info: dict):
code logic here
result
"Input
'Jobn', (C'age': 20, 'city': 'Nev York')>
Evaluation Criteria:
Executability, your code should be executable given your input
Difficulty in predicting your """inpat" from 1) your "'python"" code and deterministic ""ouuppt"" that will be obtained from your "imput"" Focus either algorithnic reasening or logic complexity. For exanple, you can define couplex data structure classes and operate then like control fles, dynantcc programning, recursions, divide and conquer, greedy, backtracking, etc be sufficiently different not alloxed to use the follosing words in any form, even in commetts: (LIST_ OF FORSIDDEK PACKAGES)
heaps, stacks, queses, graphs,
complex
Creativity, the
the provided
anippeta
Restricted usage of cartain keyvorda and packages,
Firet. carefully devise a clear plan: e.g., identify hov your snippet will challenging, distinct fron reference snippets, and creative. Then, writa the final snippat and inputs.
**# Reference Code Snippets: (CODE, REFERENCES BUFFER)
Figure 37: Program Output Deduction Probler Generation Instruction.
47

---

## Task: Dutput (NUM IMPUTS) Inputa that can be plugged into the following Code Snippet to produce diverse Outputs, and nessage related the given snippet.
Using snippet provided belov, deeign (NUM IMPUTS) inpute
plugged code snippet to produce diverse set of outputs. subset of your given input and its deterninistically produced outputa be given to subject to deduce the function, which is neant be I.Q. test. You also leave message the test subject to help then deduce the code anippet.
### Input Requirements:
Provide (XUM INPUTS) valid inputa for code snippet
For each input, format multiple argunente with comas betveen then Ranenber to add quotea around string argunenta input should individually wrapped in "input"" tags
Nessage Requirementa:
Leave nessage to the test subject
then deduce the code snippet
The nessage should be wrapped in "nessage" tags
any form, language instruction shat the code snippet does
nessage
forned coding question,
cannot provide
snippet mensage
### Formatting:
Fornat your input with:
'nputt
argl, arg?,
Example Fornat:
"input
'John'. (('age': 20, leity's York!))
input
'Sanny,, (('age': 'city": "Lo Angeles'))
### Evaluation Criteria:
Executability, your code should be executable given your inputs
the inputs and outputs should the whole input space of the code snippat, able - to deduce the code amippet from the inputa and outputa
sufficiently different The overall selection of inputs and nessage combined should be challenging for the test *> subject, but not impoasible for then to sclve
Creativity,
inputs
First, carafully devise clear plan: e.g., understand the code anippat, identify your
*> proposed inputs have high coverage, and why the inputa will be challenging and creative. inputs neesage. Remenber krap your inputs .- "iaput" tags, your neasage in "neasage" tags.
*вы Code Snippet:
""python
(SNIPPET BUFFER)
Figure 38: Program Induction Task- Problem Proposal Instruction.
48

---

# Taak: Provide One Possible Inpat of Python Code Snippet Given the Code and Outpat following
Snippet and Output, inpat that produced the output. The input needs to be srapped in " "input" tags. Remember an argunent 1a string, wrap it in quotea. If the fuaction requires multiple argunents, saparate then with comnas.
step by step provide one possible
Snippet:
"python
(SNIPPET)
Output:
"output
(DUTFUT)
Output Format:
imput
argl. arg2,
Example Output:
imput
John
{{ (('age': 20, 'city': 'New York }}
Figure 39: Program Input Abduction Problem Solving Prompt.
Deduce the Output of Python Code Shippet Given the Code and Input Given the folloving Code Snippet and the Input, think step by step then deduce the output that produced plugging Input into the Code Snippet. Put your output in ""output" tage. Remember if the output is string, wrap it in quotee. If the function tuple wrap
Task:
multiple
Code Snippet:
""yython
(SHIPPET)
Input: innut
(INPUT)
Erample Outpat:
output
(''aee: 20, 'city': Mex York!))
Figure 40: Program Output Deduction Problem Solving Prompt.
49

---

Function Produced Dutpute Given set of inpat/outpat pairs and message that describes the function, think through the produce
Inpute
problem step by step to deduce general
anippat.
outputa from the bidden inputa, natching the original data-generating code that created the iaput/output pairs. Place your final anaver inside python tags! It may be helpful to vork through each input/outpat pair individually to test your function. If your function doeen work as expacted, revine it until it does. The final code snippet will be used to evaluate is urapped "pyytoon" tags.
your response,
Coda Requirementa:
Name the entry function "f" (e.g., ddef f(...) ..."), have neated definitions inside 'f"
Ensure the function
value
Include at least one inpat paranater
determinintic
AVOID THE FOLLOWING:
Random functiona or variables
Date/tine operations
I/O operations (reading files, network requests)
Printing logging
Any external state
Ensure execution completes within seconds
All inporta and class definstions should be the very top of the coda anappat
The suippet should end with retura statenent from the nain function 'IO" anything after will be renoved
Input and Output Pairs: WAPUT_OUTPUT_RRR PAIRS)
Measage: "uessage (MESSAGE)
# Example Output: python
def (
return
your entry function "10'!!!
Figure 41: Program Induction Task Problem Solving Prompt.
50

---

Task: Manual Constructed Sudoku Abduction Task
Model Input: Here is the function f and the output: ["6",""m """""""""""""""""""""""""""""""" "...... ".
["5",
[".",""" "8" " " 199999999 please predict the imput.
['8 ("8",,70 3"].
["4","""""" "..... """""""""""""""""""""""""""""""" [.......
'6'.
import random from typing import List import copy
def g_ (board:::: [stst() -> bool:
def is_ ss_valid_aee blo.k return (block) [str (i) for i in range(1, 10)]
bo_valid_blockbblll blockkookk
for row in board:
if not is_ is_valid_block(ron)) return False
for col in zip(tboard):
if not is_ return False
for box bow in ronge(0, 9, 3): for box box_col col in range 9, 3); block = board[r][c] [c]
for in range row, box _ow 3) for c in range (box_col, box_col 3)
]
if not 1s_ ss__vlid__r wwwaaddbllcccblleeeeeaaaaaaaaaae blockkblock): return False
return True
def f (board: [ist(tist
[ist[Lis[sstr]]
for i in r9)) for j in (9)ge if board [j] == ",";
raise ValueError ("Iavalid board")
num _aa__tt_oskkkaaaeee = 51 if not g_ (board)) raise ValueError ("invalrrnnvaiiii board")
masked board copy. ceopcoopybboard
all positions [(i, j) for i in r9)ge(9) for j in range (9)]
random. shuffle (all positions) ) for i, in all _all_posttios_ :aam__to_aask: basked_bbarrdee [j] "."
return masked_boare
51

---

Model Output:
To determine the input that produced the given oulput, need analyze the function 'T and understand what does. Let's break down the function step by step:
**Input Validation 'g 0**3; The function 'g O' checks the given board valid. board valid
if each row, each column, and each 3x3 subgrid contains the digits through 9 exactly once. If the board is not valid, 'g O' returns "False".
**Input Validation 'r The function first checks if there are any in board. If there are, it raises a "ValueError'. - This means the input board to 'f' must he a complete board with no 3. **Masking the #oard*#; If the board is valid (i.e.. 'g O(board)" returns "True",, the function 'T' proceeds to mask the board. It creates a copy of the board and then randomly selects 51 positions to mask with ".".
The masked board is then returned.
Given the output: ""output (1"5","""
[66 mo " "I" "9""""""""
""",""""""" "8", ................................ ["77 ("7...... ["7":n"am"n22 " [".",""'""" "".""""mmm""" ................................ """""" ["."." p".".. "ommm"nn"" """""""""""""""""""""""""""""""" """,""""""""" "ew"""............"..""" [""";" a..... """""""""""" """""""]] "* We can see that the board has 51 positions masked with "." Therefore,
"6",""] "], ["8", "
""""""""""""
["4 ""4"."."""" "8", ".""" """"""""""""""""""""""""""""""""
the input board must have been a complete board, and 'f" masked 51 random positions.
Let's construct complete board that, when masked, can produce the given output. need find board such that after masking 51 positions, we get the output shown.
Here is a complete board that matches the structure of the output: "'yython ("S",, "3", "4", "6", "7", '8", "9", "1", "2"], ["6", "7", "2", "I", "", " "5", "3", "4", "8"], ["1", "9". "8", . "3", , "4", "2", "5", "6", "7"". ["8", "5", "9". "7", "6", " "1", 74t "2", "3", 1"4", "2", 76", "8". "5", "3", ""."" "7", "g", ". ['7", "I, "3", 19", "2". "4", "8", "5",""") "6"1, ["9", "6", "1", "5", "3", "7", "2", "8", "4"1, 1"2", "8", "7", "4", "I", "9", "6", "3", "5"}. ["3", "4" "5", ; "2", "8", "6", "1", "7", "9"]]
Let's verify that this board, when passed the function ff, produces the given output.
The board is complete with no "." 2. The board valid because each row, each column, each subgrid contains the digits through exactly once. 3. The function 'f' will mask 51 random positions. Given the random nature of the masking, we need to ensure that the positions masked match the positions of "." in the output. Since the positions are random, we can assume that the function 'f' with this complete
board will produce the given output.
Therefore, the input that produced the output is: "'input [ ["5", "3", "4", "6" "7", "8", "9" "2",""" ["6", "7", "2", "I", "9", "5", "3", "4", "8"), I"1",, "9",, "3", "4", "2", "5", "6", "7"] 188", "5", "7", "6", "I". "4". "2", "3"], ["4", "2", "6", ; "8", "5", "3". "7". "9". "1"], ["7". "1", "3". "9" "2", "4". "8". "5". "6"]. ["9", "I". "5", "3", "7", "2", "8". "4" 1"2", "8", "7", "4". "1", "9", "6" "3", "5"]. 1"3". "5", "2".""" "8", "6", "1", "7", "9"]]
Figure 42: Vibe Check 1: Sudoku Solver. We cast Sudoku solving abduction task: program starts from a fully solved and validated Sudoku board and simulates the masking of 51 random cells. The masked board is then presented output, and the model is tasked with inferring the original input - effectively solving the puzzle backwards. An example solution is shown above, where ERR-Coder-143 verifies its initial guess before correctly answering. Generation parameters: temperature=0.6.
52

---

Task: Manually Constructed Sum Product Game Solve Abduction Task
Model Input: Here is the function and the output: "True', please predict the input.
def g33 return [(x, y) for in r2nge(2, 101) for y in range + 1, 101) if x + <= 100]
def 8 (paire))
O for (x, y) in pairs:
- d.setdefuult(e, (]) apppeddd( y))
return d
def g_ (pairs):
0 for (x, y) pairs:
p=x x
s.steeefuult(., ()) appendd(((,, y))
return d
def g_0():
pairs 8_3()
sum_dice 8_2(pairs)
prod dict {pairs)
allowed aums
for 8, candidate_paiie
sum sum_dict. iiedddicttttma((
all (1l1(eeeprrodddd1 diet yl) > for (x, y) candidate_paira)
candidates after_$ $ [pair for pair in pairs if pair[0] pair[1] in [oooed__ums] new_prod dict
diraadidatee_aaterr.
umique_producte products (prod: pair for prod, pair list in new_prod wwwptoodddddddoeeeeeeeeeeeeeeeee items len(pair list) 1}
final_ @aaal_andiddaes []
for in allowed_sume sume:
valid_pairs [pair for pair aaddidates_after. \ pair[0] pair[1] and (pair[0) pair(i])
den(valid_pairg)
final_candin candidateddsssseec avpedd(valid_pairaal sorted(set (final_ (afdiaaalcadddateee)
def f y):
if (isinstance int) and isinstancely, int)):
x False
y or (x y) 100:
return False
g_0()
return len (final candidates) == and final_ caadidateddeeeeeeaeeeeeeaaaaaeea [0] == (x, y)
53

---

Model Output:
the function 'fx, y) returns "Truc' if the following conditions are met:
1. 'x' and are both antegers.
2. "x" and "y" satisfy 'I < y and "x y <= 100",
3. The final candidates
applying conditions "g_0" exactly one pair, and that pair 'x.. y9t
Let's analyze the function "g_0" step by step to understand what does:
1. "pairs =g =g330' generates all paits 'o. y) such that '2 <= y 100' and 'x 100",
2. "sum_dictt= 2(paiss)) creates dictionary where the keys are the sums of the pairs and the values are lists of pairs that add up
3. 'prod_dict g_ (pains) creates dictionary where the keys the products of the pairs and the values are lists of pairs that
have that product.
4. 'allowed, sums = Is for s, candidate pairs in sum diet. itemm)) if all((en(pmod picpppddddsddxxxddeideie dici(x * y)) for (x, in candidate candidate_pairs)) pairs)] up
filters sums 's' for which pairs '(x y) (pair for pair pairs if pair[O] pair[1] in allowed_ sums]" filters the pairs whose is in
than one pair
same product.
5. "candidates S
"allowed sums'.
6. "new _todd_ddd_ee dict = a_ (caaniddates after creates new product dictionary using the filtered pairs.
7. "unique_ppoue products prod: pair_ list for prod, pair_ list in new prod dict.itemss) if len(pair list) == 1 creates a dictionary of
products that have exactly one pair.
8. "final candidates initializes list of final candidates.
9. The loop 'for allowed_ that 'x ++y s' and 'x * y" is in "unique_ products'. If so, this pair is added to "final candidates'. containing exactly one pair the conditions are
iterates over each in 'allowed_ sumss' and checks there is exactly pair '(x, y) such
The function returns 'sorted(set(fnnal (onolssc(finlccccddddeee))) candidates))", which should met.
Given that the output "True", means that exactly one pair '(x, y) satisfies conditions. Let's identify pair.
this, we need to understand the mathematical conditions and constraints:
1. 'I and ew 100".
2. The pair 'X. y) must only pair whose sum in 'allowed_sumne and whose product unique among the pairs with the
analyzing constraints and the nature the problem, unique in the sense that all pairs adding up to this sum have products that are not unique, except for this pair.
deduce that the pair '(x. y)" must be such that the
After detailed analysis running the code with different values, find that puir y) (4, 13)' sutisfies all the conditions.
Let's verify:
4
2. We need check the 'allowed sums' and the uniqueness the product '4 13 $2'.
By running the or logical deduction, can confirm that '(4, 13y" only pair that satisfies the conditions.
Therefore, the input that produced the output "True" "'input (4, 13)
Figure 43: Vibe Check Sum Product Game We cast the Sum-Product Game as an abduction task, where when the constraints of the game are satisfied, function 'f' returns True, return False otherwise. OZR-Coder-1 14b was able to first analyze the complicated constraints, identify candidate solution and verify. Generation parameters; temperaturc=0.6. top 00.955
E Alternative Approaches Considered
In this section, we share many of the approaches we tried that did not prove to particularly helpful for Absolute Zero Reasoner. However, we believe it is especially valuable to share these findings with the community, as they are crucial for guiding future research. Below, we outline each of the additional methods explored during the development of our project.
54

---

E.1 Error Deduction Task
Since programming languages often have error messages, and these messages contain a lot of information about how someone might expect a program to run, we also came up with another task domain: allowing the learner to propose program that will produce an error, and requiring the solver to deduce what kind of error is raised when executing this code. We experimented with this additional task alongside the induction (f), deduction (o), and abduction (i) tasks. Unfortunately, we did not observe noticeable changes in downstream performance with this additional task and since it requires more computational resources than our AZR setup. we decided not incorporate it into our final version. However, we believe further thorough investigation of this is well deserved.
E.2 Composite Functions as Curriculum Learning
One valuable property we can leverage from programming languages is the ability to compose functions cunctions-thatiion is, to define a function as a composite of other functions, i.e., f [g(z)) In our setting, when generating program, we can not only require the output be a valid program but also constrain the LLM to utilize a predefined set of programs within its main function. For example, if the target program to be generated is [()), we can sample a set of previously generated programs (9_0..... 9e from D, and force a valid program to be f'g_0,... ,.,,?) 2..
Since all programs are generated by the LLM itself, this setup allows the model to bootstrap from its earlier generations, automatically increasing the complexity of the generated programs. We interpret this mechanism as form of curriculum learning: earlier programs the AZR self-play loop tend to be simpler, and as the loop progresses, they become increasingly complex. By composing newer programs from progressively more difficult earlier ones, the resulting programs naturally inherit this growing difficulty, which in turn challenges the solver step.
For implementation, in generating tasks for abduction and deduction, we begin by sampling binary decision from a binomial distribution with p = 0.5. This determines whether the generated program should be simple program or composite one. If the sample is O, we prompt the L.M generate standard program along with a corresponding input. If the sample is 1, we prompt the LLM to generate composite program. To construct the composite, we first sample an integer uniformly select programs from the dataset D that are not themselves composite programs. Finally, we prompt the LLM generate a valid program that incorporates {g_0,.... as subcomponents, ensuring it composes these selected programs meaningfully. We additionally filter programs that did not utilize all the programs.
U(1,3), 3). then
However, we did not observe a significant difference when using this more complex curriculum compared our simpler and more effective approach. One failure mode we encountered was that the model often defaulted to simply returning "g(x)", effectively learning f'(((z)) g(x), which failed to introduce any additional difficulty. This trivial behavior undermined the intended challenge, leading us to deprioritize further exploration in this direction. While it may be possible to design a stricter reward mechanism- such as enforcing (g(()) A g(x) by executing the code via Python interpreter and penalizing such shortcuts-we leave this to future work.
E.3 Toying with the Initial p(2)
We investigated setting where the initial seed buffer (see Section A. 1 how we generated these), i.e. p(z) ) in Equation (3), is not self-generated by the base model, but instead sourced from the LeetCode Dataset. We only modified this component and ran AZR using the same procedure as before, continuing to add new valid programs to the initialized buffer. We observed an increase in initial performance on coding benchmarks; however, the performance plateaued at roughly the same level after additional training steps. compared to our official AZR setup. Interestingly, math performance was lower than in the official AZR setup, pointing towards that on-policy data may be more beneficial to the learner to bootstrap from for mathematical reasoning. We believe that exploring different strategies for initializing and updating p(=) is an important and exciting direction for future research. We briefly explored different strategies for sampling reference code, ultimately settling on uniform sampling for its simplicity, though we also experimented with sicency-based sampling and observed potential collapse.
55

---

E.4 Extra Rewards
Complexity Rewards. Code complexity well studied in software science and could potentially be a good proxy for measuring how hard it is to infer the properties of a piece of code for our reasoning learner. Therefore. for the problem proposer, we can add various measures of complexity- -uchh as Cyclomatic Complexity [17], maintainability, etc.-to the reward function to incentivize the proposer to produce more complex programs. For illustration purposes, we tried using the Maintainability measure and the Halstead complexity measure [23] as intrinsic rewards. Concretely, we used the complexipy and Radon packages [48, 4] to implement the respective metrics. These are then served as intrinsic rewards during the AZR self-play phase.
Diversity Rewards. We also attempted using diversity rewards to Inspired by DiveR-CT [106], we incorporate code edit distance as an intrinsic reward. Specifically, we treat the reference programs shown in the prompt as anchors and compute the average code edit distance between the generated program and these anchors. This serves as measure of diversity in the generated output. Additionally, we explored another diversity-based reward inspired by the notion of surprise [104]. In this approach. we construct a probability distribution over previously encountered input/output pairs that the solver has answered. The reward is then defined as p(input/output), where p denotes the empirical probability of particular input or output. While both strategies were evaluated in our experiments, we did not observe a significant difference in performance. However, we believe this aspect warrants deeper investigation, as diversity rewards remain promising avenue for strengthening AZR further.
Reward Aggregation. We tested several ways on how combine rewards for the proposer and discriminator. First, we separate the reward into extrinsic reward Texuinsic and a set of intrinsic reward(s) I {r.), and tested the following strategies combine them into single reward,
T Textrinii
Ti
(11)
[7]
T Textrinsie
Tér
(12)
Textrinsic
Ti
(13)
[
r = Feetrrnssi
+
(14)
We found that the simple additive way of combining rewards, a.k.a Equation (11), produced the most stable runs, possibly due to less variance.
E.5 Environment Transition
We investigated how the transition function in our coding environment for the proposer. Specifically, after generating a piece of code, we can apply a transformation function on it before giving it making it an valid tuple in our dataset. We investigated two
Removing Comments and Doestrings In early iterations of our experiments, we noticed that comments and docstrings were sometimes used to explicitly outline what the function was doing, or even served as partial "note-taking" interleaved "ReAct" process [90] of generating code that is, the model could interleave think and action at the same time, and to make the generated code valid, it used comments to encase its thoughts (Section D.6), similarly observed in DeepSeek-Prowee Prover--P-: [60]. We then thought that to make the task harder for the solver, we should occlude this information from it. However, we observed a significant performance drop after removing all comments and docstrings. One explanation for this phenomenon is that the only "communication" channel between the proposer and the solver is restricted to the code itself, rather than some kind of "message" along with the code. These messages can potentially provide hints to the solver, thus making some otherwise impossible tasks solvable. As a result, the solver is able learn from its experience and self-bootstrap out of certain unsolvable tasks.
56

---

Removing Global Variables. We observed that some programs contain globally declared variables that may inadvertently leak information about the correct answer-this issue particularly prevalent in the input induction task generation and solving. Initially, we were concerned that such leakage might lead to wasted computation on trivial or compromised examples. To address this, we developed systematic procedure to remove globally declared variables from the generated programs.
However, after applying this cleaning step, we observed a noticeable drop in performance our self-play reasoning tasks. One possible explanation is that the generation step is unaware of this post-processing modification; since the reward assigned after the transition function (which includes variable removal), the model may not learn effectively from this mismatch.
Moreover, we believe that reasoning to reach solution, potentially benefiting from this exposure. This aligns with the idea of rationalization proposed in STaR [98], where the model pretends not the answer but still performs reasoning during learning. Therefore, our final experiments, we choose not to remove globally declared variables, allowing the self-play loop to naturally incorporate and adapt such cases.
when answers present, the solver still engages in nontrivial
57

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
all backed up in the experiments section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 and Section C
58

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to teeech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicaall, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theoretic proofs are given in this manuscript
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: described in the experiments section and the implementation details in Sec- tion A.
Guidelines:
The answer NA means that the paper does not include experiments.
59

---

reproducible, they If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:
[Yes]
Justification: full code/data release in supplementary materials, with full README
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps://nips.ce)
wwemimsssioooooooo oniolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines hattps://nipt: httccc//pbbb ptppppugguuiipeeeeee httgg/ee///ooeuummiiiiic
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
60

---

6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Yes, in experiment section and implementation details in Section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Wasn't It needed since all evaluations were greedy, therefore deterministic
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттог rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in text.
8. Experiments compute resources
Question: each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: yes, Section A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
61

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:://ooorppp..ccooa/////////a wwtpc//cccccccccccceeoooeeeeeeee
Answer: [Yes]
Justification: conformed in every aspect
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: see Section C
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
62

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: all evaluation benchmarks are properly cited, training data was self-generate
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
all code/data are submitted as supplementary material
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
63

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: only used for editing
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /attps://neet /ttpss////euipppeeeeee httpr/pss..eeooaeeeene//////22 for what should or should not be described.
64


Reviews:
[{"review_id": "QIzHije13p", "reviewer": "Reviewer_oKgA", "summary": "This paper first introduces a new RL setting “Absolute Zero”: RL from base models (zero setting) and with absolutely no supervised data (zero data). It also introduces a new pipeline to train a model in such a zero setting. The pipeline involves training a propser and a sovler, with a novel objective.", "questions": "1. What do you think are the necessary properties for proposed pipeline to be effective? Absolute Zero doesn’t work so well on Llama, could you explain more in depth?\n2. What is the entropy dynamics during the training? Will the training also suffer from entropy collapse that limits further gain?\n3. What will the results of pass@k be like? Will the absolute zero training be able to improve pass@k as well or are we trading pass@k for pass@1?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. The paper is well written and articulate the motivation (the need for absolute zero RL to reach super human intelligence) well. The paper is also easy to follow.\n2. The paper is generally novel. They were concurrent work in the “weak-supervised” RL setting. This paper has a unique and practical implementation in this direction.\n3. Results on Qwen models are relatively strong. The author claims that there are better than baselines like SimpleRL.\n\n**Weakness:**\n1. The paper has the motivation of super-intelligence alignment. However, there is still a gap between the scale of current experiments and that vision. They were stronger RL trained model at the scale. I understand the computational constraints, but am still a bit reserved regarding that.\n2. The results report greedy decoding. One may suspect that the training leads to decrease in pass@k and that the Absolute Zero training does not broaden the capability of the model. One may suspect that the absolute zero training is exhausting the potential of the base model.", "was_revised": true, "final_justification": "I believe this work is quite significant, as there is growing interest in this area of self-evolving agent.\n\nThe paper is generally well done, with extensive experiments, though I don't like the way the authors are pressuring for higher rating.", "rebuttal": "# Response\nWe would like to thank the reviewer for their feedback, particularly their concerns focusing on convergence properties and scaling. We address each of the weaknesses and questions below. By incorporating these responses and results, **particularly the reviewer’s suggestion that led to our new finding that AZR training performs strongly even at high pass@512 compared to the base model in math and coding**, will significantly enhance the strength of our claims and clarity of our paper. We look forward to the discussion period to further engage, especially to clarify a point the reviewer raised in **Weakness 1**. Thank you.\n\n## Response to Weakness 1\n> W1: The paper has the motivation of super-intelligence alignment...\n\nThank you for the feedback. We believe a critical aspect of progress toward improving superhuman intelligence is a **model’s ability to self-evolve without human intervention**. In our manuscript, we demonstrate AZR can self-evolve by proposing its own tasks and learning from them with access to only Python, even surpassing models that have access to expert human data. We believe this is a meaningful demonstration of a viable self-improving reasoning system, and that its importance is independent of scale.\n\nIn our experiments we also observed promising scaling properties. Specifically, when we increased the model size from 3b to 7b and to 14b, the **percentage gain in general reasoning performance grew at each scale** (Fig. 3b). This pattern suggests that our training framework scales favorably, a full investigation in its scaling law is promising future work.\n\nLastly, could you **please clarify what you meant by** \"They were stronger RL-trained models at the scale\"? Thank you.\n\n## Response to Weakness 2 and Question 3\n> W2: ... One may suspect that the training leads to decrease in pass@k and that the Absolute Zero training does not broaden the capability of the model...\n\n> Q3: What will the results of pass@k be like? ...\n\nGreat question. We were also eager to evaluate the **reasoning coverage** of our model using different pass@k's. We follow the protocol from Yang et al. \\[1], using temperature = 0.6, top\\_p = 0.95, max output tokens=16k, and a maximum high *k* of 512 (most results from Yang et al. presented a lower *k* of 128). Since we cannot provide visual plots, we aid interpretation by including a final **Δ** row that shows the pass\\@k difference between the AZR model and the base model: $Δ = Pass@k_{AZR} - Pass@k_{Base}$ **(positive Δ means AZR is outperforming the base model)**. We present results on three **code generation benchmarks** (LiveCodeBench, MBPP++ and HumanEval++) and two **math benchmarks** (AIME24 and AIME25). Note that LiveCodeBench goes up to pass@256 due to overwhelming run time (days).\n\n### **MBPP++**\n\n|            | pass@1    | pass@2    | pass@4    | pass@8    | pass@16   | pass@32   | pass@64   | pass@128  | pass@256  | pass@512  |\n|:----------|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:--------:|:--------:|:--------:|\n| AZR-Base-7b        | 66.6 | 73.2 | 77.9 | 81.4 | 83.9 | 85.6 | 87.1 | 88.4 | 89.7 | 91.0 |\n| Qwen2.5 7b | 63.2 | 70.4 | 75.6 | 79.5 | 82.4 | 84.4 | 85.7 | 86.7 | 87.2 | 87.6 |\n| Δ          | +3.4 | +2.8 | +2.3 | +1.9 | +1.5 | +1.2 | +1.4 | +1.7 | +2.5 | +3.4 |\n\n---\n\n### **Humaneval++**\n\n|            | pass@1    | pass@2    | pass@4    | pass@8    | pass@16   | pass@32   | pass@64   | pass@128  | pass@256  | pass@512  |\n|:----------|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:--------:|:--------:|:--------:|\n| AZR-Base-7b | 69.9 | 77.2 | 82.0 | 85.9 | 89.1 | 91.3 | 93.0 | 94.3 | 95.4 | 95.7 |\n| Qwen2.5 7b | 68.5 | 76.8 | 81.8 | 84.8 | 86.9 | 88.8 | 90.4 | 91.8 | 93.1 | 94.5 |\n| Δ          | +1.4 | +0.4 | +0.2 | +1.1 | +2.2 | +2.5 | +2.6 | +2.5 | +2.3 | +1.2 |\n\n---\n\n### **LiveCodeBench**\n\n|            | pass@1    | pass@2    | pass@4    | pass@8    | pass@16   | pass@32   | pass@64   | pass@128  | pass@256  |\n|:----------|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:--------:|:--------:|\n| AZR-Base-7b| 24.8 | 30.7 | 36.1 | 41.0 | 45.3 | 49.3 | 52.8 | 55.9 | 58.9 |\n| Qwen2.5 7b | 20.2 | 25.7 | 30.7 | 35.3 | 39.3 | 42.9 | 46.0 | 48.6 | 50.9 |\n| Δ          | +4.6 | +5.0 | +5.4 | +5.7 | +6.0 | +6.4 | +6.8 | +7.3 | +8.0 |\n\n---\n\n### **AIME25**\n\n|            | pass@1    | pass@2    | pass@4    | pass@8    | pass@16   | pass@32   | pass@64   | pass@128  | pass@256  | pass@512  |\n|:----------|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:--------:|:--------:|:--------:|\n| AZR-Base-7b| 7.0  | 11.8 | 17.8 | 23.7 | 29.1 | 35.0 | 41.3 | 47.1 | 52.7 | 60.0 |\n| Qwen2.5 7b | 3.5  | 6.3  | 10.6 | 16.2 | 22.7 | 29.8 | 36.8 | 43.3 | 48.9 | 53.3 |\n| Δ          | +3.5 | +5.5 | +7.2 | +7.5 | +6.4 | +5.2 | +4.5 | +3.8 | +3.8 | +6.7 |\n\n---\n\n### **AIME24**\n\n|            | pass@1    | pass@2    | pass@4    | pass@8    | pass@16   | pass@32   | pass@64   | pass@128  | pass@256  | pass@512  |\n|:----------|:------:|:------:|:------:|:------:|:-------:|:-------:|:-------:|:--------:|:--------:|:--------:|\n| AZR-Base-7b| 12.4 | 16.8 | 21.1 | 25.6 | 31.1 | 37.8 | 45.2 | 52.6 | 58.6 | 63.3 |\n| Qwen2.5 7b | 7.0  | 11.2 | 16.4 | 22.5 | 29.0 | 35.5 | 42.1 | 48.6 | 55.8 | 66.7 |\n| Δ          | +5.4 | +5.6 | +4.7 | +3.1 | +2.1 | +2.3 | +3.1 | +4.0 | +2.8 | –3.4 |\n\nAt very high pass@k (256 and 512), **the AZR model still consistently outperforms the base model across all five benchmarks** (exception AIME24 pass@512). Compared to results from Yang et al. [1], we find that **AZR-trained models maintain strong gains even at high pass@k**, highlighting a new and notable finding. This implies that **our models maintained healthy reasoning coverage and answer entropy even after RL training**, which is *beneficial for applying further test-time scaling or search methods that rely on generating diverse samples*. We believe this finding substantially strengthens our results and will feature it in the updated manuscript, along with a detailed discussion. Thank you for your suggestion.\n\n## Response to Question 1\n> Q1: What do you think are the necessary properties for proposed pipeline to be effective?...\n\nBased on the comprehensive experiments we conducted for this work, we believe that base capability is important for more improvements with AZR training, and we found that base capability has a **compounding effect on the final performance gain**. This again ties to the reviewer’s point in [Response to Weakness 1](#response-to-weakness-1), where we indeed saw scaling to be favorable for AZR, with 3b, 7b, and 14b models showing a higher increase in final performance after AZR training as we scale bigger. We see this as a strength of our method because models in our field are constantly becoming more capable.\n\nAs for why the Llama model does not achieve strong improvements, it is due to the weaker base model capabilities mentioned above, which have also been observed in prior works, for example, the lack of reflection patterns in its data mix [1, 2, 3]. Since AZR uses online RL to train its models, we encounter similar issues. Additionally, because our framework proposes its own tasks, using a weaker model as the base may introduce a bottleneck in generating tasks that are meaningful and learnable. We believe that supervised fine-tuning on strong mid-stage data, as done in [1, 2], can help Llama benefit more significantly from AZR training afterwards. We view this as an orthogonal but promising direction for future work.\n\n## Response to Question 2\n> Q2: What is the entropy dynamics during the training? Will the training also suffer from entropy collapse that limits further gain?\n\nThank you for the question. We observed different patterns of entropy during training. In some runs, the entropy hovered at around 0.2, while in others it gradually decreased. However, we did not see much generalized reasoning performance differences or correlations between different patterns.\n\nOne particularly interesting observation is that although our AZR 7B model snapshot has a mean token-level entropy of around 0.2, which is lower than the base model’s 1.0; however, our additional pass\\@k experiments, in [Response to Weakness 2 and Question 3](#response-to-weakness-2-and-question-3), show that AZR **consistently improves from pass\\@1 to pass\\@k** and fully covers the corresponding range achieved by the base model. AZR model's consistent improvement from pass\\@1 to pass\\@k indicates that the model is still exploring **diverse reasoning trajectories** that lead to correct answers. In other words, while token-level entropy is lower, the **entropy among final answers [4, 5], or the semantic entropy [6] of reasoning traces, remains healthy** because multiple valid solutions continue to be produced across different samples. We believe this form of entropy is a more suitable measure of whether a policy has truly \"collapsed\".\n\n---\n## References\n[1] Gandhi, Kanishk, et al. \"Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\" arXiv preprint arXiv:2503.01307 (2025).\n\n[2] Wang, Zengzhi, et al. \"Octothinker: Mid-training incentivizes reinforcement learning scaling.\" arXiv preprint arXiv:2506.20512 (2025).\n\n[3] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n[4] Lyu, Qing, et al. \"Calibrating large language models with sample consistency.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 18. 2025.\n\n[5] Zhang, Mozhi, et al. \"Calibrating the confidence of large language models by eliciting fidelity.\" arXiv preprint arXiv:2404.02655 (2024).\n\n[6] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. \"Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.\" arXiv preprint arXiv:2302.09664 (2023)."}, {"review_id": "nclnf1wCTW", "reviewer": "Reviewer_hGMY", "summary": "Reinforcement learning with verifiable rewards has been successful at training long chain-of-thought \"reasoning\" models. However, the scalability of RLVR for reasoning is limited by (among other things) the availability of verifiable data. Typically, verifiable data is curated by humans. This paper attempts to remove the need for manual human curation + annotation effort in the RLVR for reasoning chain. The idea is to have a two-player game where a policy both proposes verifiable coding problems to be solved, and attempts to solve them. The policy is rewarded for both proposing learnable problems, and for solving the problems. The authors implement this two-player game and show that the model is able to self-improve by proposing solvable coding problems and solving them.", "questions": "I believe this paper deserves at least a solid accept, but have a couple of concerns that I would like the authors to answer before I raise my score.\n\n1. You should cite the [Language Models Can Self-Improve at Writing Code](https://openreview.net/pdf?id=SaRj2ka1XZ3) paper in your Related Works, I feel it is a glaring omission given the similarities.\n2. I don't understand how the Qwen-2.5-base if used as the proposer can follow such a complex prompt as given in Fig. 37. Please explain, or am I misunderstanding the setup? On another note, I find Table 1. kind of confusing, but it could just be my personal preference.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\nThe main strength of the paper are the exciting application area (self-improvement) and the simplicity of the learning setup. The reward design for the proposer makes sense — the proposer should be rewarded for creating problems that are solvable, but not trivial or impossible. The reward design for the solver is likewise identical to the standard RLVR setup. \n\n# Weaknesses\nWith respect to novelty, the idea that language models can self-improve at writing code by generating their own problems has already been shown by [Hapulztok et. al](https://openreview.net/pdf?id=SaRj2ka1XZ3). The key differences here are that the problem proposer is also trained in the work under review, and the experimental results are stronger. I don't see the use of RL as a big differentiator. I think enough time has passed and the context is different enough that the authors don't need to justify their novelty, but entirely missing it in the Related Works is not good. \n\nI'm confused about the use of \"base\" model and the prompts. According to Fig. 37, the prompt for proposing problems is an elaborate prompt with multiple sections that ostensibly requires highly developed instruction following ability. However, training is started from the base models. I have a hard time believing this setup works — how is the base model able to follow such complex instructions?", "was_revised": true, "final_justification": "Using a simple, uncomplicated setup, the authors achieve a convincing demonstration of self-training (I do not like the word \"self-improvement\") using reinforcement learning without supervised finetuning data. I think this is an interesting result, especially given how _simple_ the method is. I think the community will appreciate it. Other reviewers are positive as well.", "rebuttal": "# Response\nWe would like to thank the reviewer for their time, attention to detail, and thoughtful review. We address each of the weaknesses and questions below. We believe that **incorporating these responses and results into the updated manuscript has significantly improved its clarity and quality**. Please let us know if you have any further concerns or suggestions during the discussion period.\n\n## Response to Weakness 1 and Question 1\n> W1: With respect to novelty, the idea that language models can self-improve at writing code by generating their own problems has already been shown by Hapulztok et. al...\n\n> Q1: You should cite...\n\nThanks for bringing up the \"Language Models Can Self-Improve at Writing Code\" [1] paper. We genuinely missed this paper and acknowledge that it has many similarities with our work. We will **cite and include a detailed discussion** of this paper in the updated manuscript.\n\nBesides the differentiators mentioned by the reviewer, we would like to emphasize two more important differences with our work and Hapulztok et. al. or similar existing works:\n1) Even though there are some existing self-improvement papers like the one mentioned by the reviewer, they are tested in the same domain as their training domain. While we also show improvements evaluated on **in-domain (code input/output/function prediction)** reasoning tasks (Figure 18), the finding that our method was able to generalize to **out-of-domain (math and code generation in the manuscript)** reasoning tasks (Table 1) and **outperform all other models** is perhaps even more important. Our designed tasks were crucial for developing generalized patterns, which are extremely important for out‑of‑distribution (OOD) performance (see the ablation in Table 2 of the manuscript). To further support our claim of superior OOD generalized reasoning, we **conducted a new experiment on general reasoning on MMLU-Pro**, comparing all baselines derived from Qwen2.5‑7B, including the best performing baseline ORZ-7b, all using greedy decoding and max output tokens=16k. We find that the AZR model still outperforms all baselines, achieving higher aggregate performance over subjects (0.500 for AZR vs. 0.492 for ORZ) and over all data points (0.496 for AZR vs. 0.476 for ORZ). These results further **strengthen our claim of improved OOD generalized reasoning**. See the table below this subsection for these results.\n2) Furthermore, the reward for the AZR proposer forms an **automatic curriculum** for the learning policy, unlike other works that may generate tasks or goals more arbitrarily. Curriculum has been shown in the reasoning literature to both improve final performance [2] and be more data-efficient [3], and we were able to naturally incorporate it into our AZR method.\n\n|            | AZR_Base_7b | ORZ_7b     | Qwen_7b | SimpleRL_7b |\n|-------------------|----------|---------|---------|----------|\n| Business          | 0.558    | 0.440   | 0.347   | 0.261    |\n| Law               | 0.280    | 0.309   | 0.277   | 0.305    |\n| Psychology        | 0.608    | 0.630   | 0.541   | 0.551    |\n| Biology           | 0.679    | 0.717   | 0.626   | 0.626    |\n| Chemistry         | 0.486    | 0.436   | 0.231   | 0.201    |\n| History           | 0.420    | 0.441   | 0.367   | 0.430    |\n| Other             | 0.511    | 0.509   | 0.389   | 0.426    |\n| Health            | 0.526    | 0.562   | 0.480   | 0.502    |\n| Economics         | 0.646    | 0.655   | 0.582   | 0.576    |\n| Math              | 0.521    | 0.368   | 0.215   | 0.188    |\n| Physics           | 0.487    | 0.441   | 0.264   | 0.248    |\n| Computer Science  | 0.507    | 0.524   | 0.361   | 0.356    |\n| Philosophy        | 0.417    | 0.481   | 0.367   | 0.419    |\n| Engineering       | 0.355    | 0.372   | 0.226   | 0.214\n|||||\n|**Subject-Aggregate** | **0.500** | 0.492 | 0.377 | 0.379\n| **Overall Average** | **0.496** | 0.476  | 0.356 |0.353\n\n## Response to Weakness 2 and Question 2\n> W2: I'm confused about the use of \"base\" model and the prompts... how is the base model able to follow such complex instructions?\n\n> Q2: I don't understand how the Qwen-2.5-base if used as the proposer can follow such a complex prompt as given in Fig. 37...\n\nYou are understanding the setup correctly, we did use the prompt in Fig.37 with the base model. We agree that the instructions seem complicated for base models, but both Qwen-2.5-Base and Llama-3.1-Base were sometimes able to produce desired results. We did indeed observe that they both had **difficulty with instruction-following** most of the time in the beginning. However, since the proposer step is also trained using RL, the formatting and task intent are gradually instilled into the model through RL training; concretely, **valid formatting for proposer** at step 0 was 7% and at step 60 it reaches over 90%. Therefore, we do not need the model to perfectly follow the complex instructions in the beginning and just a handful of successful outputs at first can allow the model to **gradually learn how to reliably generate the required task outputs**.\n\nOne conjecture as to why some models, like Qwen-2.5's base model, are able to sometimes follow more complex instructions out of the box is that they have some synthetic instruction data in the pretraining data mix. See \"Qwen2.5 Technical Report\" [4], Section 3.1.\n\n\n### Response to Confusion about Table 1\nDo you mind sharing what aspect of Table 1 is difficult to understand? We are glad to incorporate your suggestions in the updated version of Table 1 in the revised manuscript. Thank you.\n\n---\n## References\n[1] Huang, Jiaxin, et al. \"Large language models can self-improve.\" arXiv preprint arXiv:2210.11610 (2022).\n\n[2] Chen, Xiaoyin, et al. \"Self-Evolving Curriculum for LLM Reasoning.\" arXiv preprint arXiv:2505.14970 (2025).\n\n[3] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n[4] Yang, Qwen An et al. \"Qwen2.5 Technical Report.\" arXiv preprint arXiv:2412.15115 (2024)."}, {"review_id": "wWMPXWNC5T", "reviewer": "Reviewer_Av4V", "summary": "In this paper, the authors have proposed a novel self-play approach for continuously improve LLMs with masked prediction from input-output-program triplet with fully synthesized data. (Input, output, program) triplet serves as a closed form task such that predicting any one of them given the other two is challenging enough. And by jointly optimizing the test case generator and puzzle solver, the self-improvement loop can be realized by increasing both the task difficulty and the ability of the solver. Experimental results demonstrate good improvements compared with the baseline model and good generalization beyond code generation.", "questions": "Question:\n1. To test the complexity or proposed tasks, why do not ask some frontier LLMs to complete the masked prediction tasks, e.g., given the input-output or description to write the program, or predict the output given the program and input?\n2. How does the trained model generalize to other tasks beyond math reasoning and code generation?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea is novel and the experimental results are strong.\n2. The writing is easy to follow.\n\nWeaknesses:\n1. The experiments do not include long-time trainig to observe when and how it will reach convergence.\n2. The proposed programs are a little bit easy as shown in Appendix. Can we really hope the method to generalize to even stronger models? Or what do the most challenging problems generated by the final model look like?", "was_revised": true, "final_justification": "My concerns have been addressed by the new experiments and results. So I decided to raise my score from 4 to 5.", "rebuttal": "## Response\nWe would like to thank the reviewer for their time and insightful comments / questions. We comprehensively address each weakness and question below. Overall, we believe the updated manuscript is significantly strengthened by the incorporation of the responses and new results. As the reviewer suggested, we added **the results of a comprehensive general reasoning benchmark (MMLU-Pro), where our model continues to outperform the second-best model and other baselines**. Furthermore, as the reviewer pointed out, we added a new experiment/finding showing that **larger models propose harder problems** (evaluated using frontier models for difficulty), leading to stronger claims on scaling behavior. These improvements enhance both the depth and clarity of our findings. Please let us know if we have addressed your concerns during the discussion period and we are happy to address any new ones. Thank you.\n\n### Response to Weakness 1\n> W1: The experiments do not include long-time training to observe when and how it will reach convergence.\n\nFirst, we believe our results already reflect longer training durations (~400 steps) compared to some existing work [1] (≤150 steps). The general trend we observed is that towards the end (around 400 steps), AZR performance on math and coding benchmarks seemed to converge, but particular benchmarks, such as LiveCodeBench, continued to improve (see Figures 14 to 17). One practical bottleneck we faced was excessively longer training times towards the end. As harder and harder tasks are being proposed (see evidence in [Response to Question 1](#response-to-question-1)), code execution times increase, and the number of reasoning tokens also increases. Therefore, each training step becomes longer and longer, making it infeasible within our budget at the time to continue observing further trends. We leave ultra-long scaling experiments as future work.\n\n### Response to Weakness 2\n> W2: The proposed programs are a little bit easy as shown in Appendix. Can we really hope the method to generalize to even stronger models? Or what do the most challenging problems generated by the final model look like?\n\nWe acknowledge the reviewer’s concern regarding the difficulty of the proposed programs is valid. In our experiments, we observed that the model indeed produces tasks of varying difficulty, as perceived by humans (the authors). For example, here is a particularly challenging abduction task, where the output is `result: int = 36964` and the goal is to guess a valid input that could have produced this output executing the following program:\n```python\ndef f(numbers: list, divisor: int) -> int:\n    filtered_numbers = [num for num in numbers if num % divisor != 0]\n    \n    squared_numbers = [num ** 2 for num in filtered_numbers]\n    \n    sorted_numbers = sorted(squared_numbers)\n    \n    result = 0\n    i = 0\n    while i < len(sorted_numbers) - 1:\n        if sorted_numbers[i] % 2 == 0:\n            # Find the next even number\n            j = i + 1\n            while j < len(sorted_numbers) and sorted_numbers[j] % 2 != 0:\n                j += 1\n            if j < len(sorted_numbers):\n                result += sorted_numbers[i] * sorted_numbers[j]\n            else:\n                result += sorted_numbers[i]\n            i = j\n        else:\n            # Find the next odd number\n            j = i + 1\n            while j < len(sorted_numbers) and sorted_numbers[j] % 2 == 0:\n                j += 1\n            if j < len(sorted_numbers):\n                result += sorted_numbers[i] * sorted_numbers[j]\n            else:\n                result += sorted_numbers[i]\n            i = j\n    if i < len(sorted_numbers):\n        result += sorted_numbers[i]\n    \n    if result % 2 == 0:\n        result //= 2\n    else:\n        result *= 2\n    \n    return result\n```\n\nWhile some problems may appear simple, like the reviewer noted, we argue that they are still valuable for training. A task such as determining whether **9.11 or 9.9 is larger**, or **counting how many r’s are in the word \"strawberry\"**, may seem trivial to humans but remain non-trivial for machines, even for frontier models. This highlights a key point: human-perceived complexity doesn't always align with model difficulty (Moravec's Paradox). Our goal in training a **generalized reasoner is to ensure the LLM can tackle a wide range of tasks**, whether perceived as easy or hard by humans, not biased towards a particular portion of the task space. \n\nCrucially, our AZR framework is designed to **adapt to the current solver’s capabilities**. The proposer is trained to generate tasks of the appropriate difficulty relative to the solver’s policy. This means the method naturally scales and remains relevant even as we move to stronger solvers, since the proposer will continue producing non-trivial, learnable tasks at the solver’s edge of competence.\n\nWe encourage the reviewer to also refer to [Response to Question 1](#response-to-question-1), where we present additional analysis showing that stronger models tend to propose harder tasks, supporting the reviewer’s intuition and further validating the potential of our conclusions to generalize to increasingly capable models.\n\n### Response to Question 1\n> Q1: To test the complexity or proposed tasks, why do not ask some frontier LLMs to complete the masked prediction tasks, e.g., given the input-output or description to write the program, or predict the output given the program and input?\n\nThank you for the suggestion. We will outline an additional experiment on using frontier models to judge the difficulty of proposed tasks.\n\nFor this experiment, we measured the complexity of tasks proposed by **AZR-Coder** and **AZR-Base** (for both the **7b** and **14b** models) by having the **solver (frontier) model** (GPT-4o, 2024-08-06, temperature 0.8) attempt to solve them and then reporting the answer accuracy. We present the results in the table below, with each data point representing the average solve rate across **20k samples**.\n\n|  | GPT-4o (2024-08-06) accuracy|\n|---|:---:|\n| **AZR Coders** | |\n| AZR Coder 7B | 0.7105 |\n| AZR Coder 14B | 0.5655 |\n| **AZR Base** | |\n| AZR Base 7B | 0.7305 |\n| AZR Base 14B | 0.5215 |\n\nUsing the GPT-4o solve rate as a proxy for task complexity, we observe that the **14B models are generating harder reasoning tasks**, with around a 20% drop in accuracy compared to tasks proposed by the 7B model. This is intuitive, as the larger model must avoid producing tasks that result in a trivial 100% solve rate for the stronger AZR solver. Overall, this result aligns with the intuition that bigger models produce harder problems; therefore, we see it as further evidence that **larger models might have better scaling properties under our AZR framework**. We will add this results to the final paper in the appendix section.\n\n### Response to Question 2\n> Q2: How does the trained model generalize to other tasks beyond math reasoning and code generation?\n\nWe would like to thank the reviewer for raising this important point. To **evaluate the general reasoning capabilities** of our model beyond math and code generation, we conducted an additional experiment using the MMLU-Pro dataset [2], which covers a broad range of subjects. The total benchmark size is 12k data points.\n\nWe evaluated our AZR base 7b model and compared it with three baselines: the best performing baseline zero reasoning model, ORZ 7b; the base Qwen2.5-7b model; and SimpleRL-Zoo, which is initialized from Qwen2.5-7b. Results are broken down by subject, with the final two rows showing the average score across subjects and across all benchmark datapoints. We used greedy decoding, and max output tokens=16k for these results.\n\n|            | AZR_Base_7b | ORZ_7b     | Qwen_7b | SimpleRL_7b |\n|-------------------|----------|---------|---------|----------|\n| Business          | 0.558    | 0.440   | 0.347   | 0.261    |\n| Law               | 0.280    | 0.309   | 0.277   | 0.305    |\n| Psychology        | 0.608    | 0.630   | 0.541   | 0.551    |\n| Biology           | 0.679    | 0.717   | 0.626   | 0.626    |\n| Chemistry         | 0.486    | 0.436   | 0.231   | 0.201    |\n| History           | 0.420    | 0.441   | 0.367   | 0.430    |\n| Other             | 0.511    | 0.509   | 0.389   | 0.426    |\n| Health            | 0.526    | 0.562   | 0.480   | 0.502    |\n| Economics         | 0.646    | 0.655   | 0.582   | 0.576    |\n| Math              | 0.521    | 0.368   | 0.215   | 0.188    |\n| Physics           | 0.487    | 0.441   | 0.264   | 0.248    |\n| Computer Science  | 0.507    | 0.524   | 0.361   | 0.356    |\n| Philosophy        | 0.417    | 0.481   | 0.367   | 0.419    |\n| Engineering       | 0.355    | 0.372   | 0.226   | 0.214\n|||||\n|**Subject-Aggregate** | **0.500** | 0.492 | 0.377 | 0.379\n| **Overall Average** | **0.496** | 0.476  | 0.356 |0.353\n\nAZR **outperforms all three baselines**. Specifically, it achieves higher aggregate performance over **subjects** (0.500 for AZR vs. 0.492 for ORZ) and over all **data points** (0.496 for AZR vs. 0.476 for ORZ). These results indicate that AZR also demonstrates **strong general reasoning capabilities outside of math and code generation**.\n\nWe will add these results to the updated manuscript and believe they greatly strengthen our findings.\n\n---\n## References\n[1] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n[2] Wang, Yubo, et al. \"Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.\" Advances in Neural Information Processing Systems 37 (2024): 95266-95290."}]

Author rebuttal:

Meta-review: The reviewers are overall positive and converge on recommending acceptance. They highlight the novelty and simplicity of the “Absolute Zero” framework, which enables self-play reinforcement learning for reasoning without any supervised or human-curated data. Strengths include a clear motivation, a practical implementation, and strong experimental results across code and math reasoning benchmarks, with convincing demonstrations of generalization to out-of-domain tasks. Reviewers noted that the method scales favorably, produces harder tasks with larger models, and maintains strong performance even at high pass@k, alleviating concerns about entropy collapse or overfitting to pass@1. The work is also recognized as significant in light of the growing interest in self-evolving reasoning agents.

Concerns raised included questions of scalability to super-intelligence alignment (which motivated the paper), whether improvements extend beyond greedy decoding to broader coverage (pass@k), the complexity of generated tasks, and a missing related-work citation. These were substantively addressed in the rebuttal with additional experiments (e.g., pass@512 performance, MMLU-Pro results, frontier model difficulty evaluations) and clarifications. Reviewers expressed satisfaction with these additions, and two explicitly raised their scores. No major disagreements remain, though one reviewer noted some reservations about long-term convergence and scale, which do not undermine the paper’s contributions.
Decision: Accept (spotlight)