Title: Absolute Zero: Reinforced Self-play Reasoning with Zero Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: reasoning

language model

reinforcement learning

self-play

LLM
Abstract: Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from rule-based outcome rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external human or distillation data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability. AZR uses a code executor to both validate self-proposed code reasoning tasks and verify answers, serving as an unified source of verifiable feedback to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.


Peer reviews:
comments: SUMMARY:
In this paper, the authors have proposed a novel self-play approach for continuously improve LLMs with masked prediction from input-output-program triplet with fully synthesized data. (Input, output, program) triplet serves as a closed form task such that predicting any one of them given the other two is challenging enough. And by jointly optimizing the test case generator and puzzle solver, the self-improvement loop can be realized by increasing both the task difficulty and the ability of the solver. Experimental results demonstrate good improvements compared with the baseline model and good generalization beyond code generation.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The idea is novel and the experimental results are strong.
2. The writing is easy to follow.

Weaknesses:
1. The experiments do not include long-time trainig to observe when and how it will reach convergence.
2. The proposed programs are a little bit easy as shown in Appendix. Can we really hope the method to generalize to even stronger models? Or what do the most challenging problems generated by the final model look like?

QUESTIONS:
Question:
1. To test the complexity or proposed tasks, why do not ask some frontier LLMs to complete the masked prediction tasks, e.g., given the input-output or description to write the program, or predict the output given the program and input?
2. How does the trained model generalize to other tasks beyond math reasoning and code generation?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReinforcement learning with verifiable rewards has been successful at training long chain-of-thought \"reasoning\" models. However, the scalability of RLVR for reasoning is limited by (among other things) the availability of verifiable data. Typically, verifiable data is curated by humans. This paper attempts to remove the need for manual human curation + annotation effort in the RLVR for reasoning chain. The idea is to have a two-player game where a policy both proposes verifiable coding problems to be solved, and attempts to solve them. The policy is rewarded for both proposing learnable problems, and for solving the problems. The authors implement this two-player game and show that the model is able to self-improve by proposing solvable coding problems and solving them.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe main strength of the paper are the exciting application area (self-improvement) and the simplicity of the learning setup. The reward design for the proposer makes sense — the proposer should be rewarded for creating problems that are solvable, but not trivial or impossible. The reward design for the solver is likewise identical to the standard RLVR setup. \n\n# Weaknesses\nWith respect to novelty, the idea that language models can self-improve at writing code by generating their own problems has already been shown by [Hapulztok et. al](https://openreview.net/pdf?id=SaRj2ka1XZ3). The key differences here are that the problem proposer is also trained in the work under review, and the experimental results are stronger. I don't see the use of RL as a big differentiator. I think enough time has passed and the context is different enough that the authors don't need to justify their novelty, but entirely missing it in the Related Works is not good. \n\nI'm confused about the use of \"base\" model and the prompts. According to Fig. 37, the prompt for proposing problems is an elaborate prompt with multiple sections that ostensibly requires highly developed instruction following ability. However, training is started from the base models. I have a hard time believing this setup works — how is the base model able to follow such complex instructions?\n\nQUESTIONS:\nI believe this paper deserves at least a solid accept, but have a couple of concerns that I would like the authors to answer before I raise my score.\n\n1. You should cite the [Language Models Can Self-Improve at Writing Code](https://openreview.net/pdf?id=SaRj2ka1XZ3) paper in your Related Works, I feel it is a glaring omission given the similarities.\n2. I don't understand how the Qwen-2.5-base if used as the proposer can follow such a complex prompt as given in Fig. 37. Please explain, or am I misunderstanding the setup? On another note, I find Table 1. kind of confusing, but it could just be my personal preference.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first introduces a new RL setting “Absolute Zero”: RL from base models (zero setting) and with absolutely no supervised data (zero data). It also introduces a new pipeline to train a model in such a zero setting. The pipeline involves training a propser and a sovler, with a novel objective.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. The paper is well written and articulate the motivation (the need for absolute zero RL to reach super human intelligence) well. The paper is also easy to follow.\n2. The paper is generally novel. They were concurrent work in the “weak-supervised” RL setting. This paper has a unique and practical implementation in this direction.\n3. Results on Qwen models are relatively strong. The author claims that there are better than baselines like SimpleRL.\n\n**Weakness:**\n1. The paper has the motivation of super-intelligence alignment. However, there is still a gap between the scale of current experiments and that vision. They were stronger RL trained model at the scale. I understand the computational constraints, but am still a bit reserved regarding that.\n2. The results report greedy decoding. One may suspect that the training leads to decrease in pass@k and that the Absolute Zero training does not broaden the capability of the model. One may suspect that the absolute zero training is exhausting the potential of the base model.\n\nQUESTIONS:\n1. What do you think are the necessary properties for proposed pipeline to be effective? Absolute Zero doesn’t work so well on Llama, could you explain more in depth?\n2. What is the entropy dynamics during the training? Will the training also suffer from entropy collapse that limits further gain?\n3. What will the results of pass@k be like? Will the absolute zero training be able to improve pass@k as well or are we trading pass@k for pass@1?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}