R-Zero: Self-Evolving Reasoning LLM from Zero Data
Authors: Chengsong Huang
Wenhao Yu
Xiaoyang Wang
Hongming Zhang
Zongxia Li
Ruosen Li
Jiaxin Huang
Haitao Mi
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
reinforcement learning
self-evolving
reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose R-zero, a data-free method to improve LLM reasoning ability.

Abstract: Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.

Published as a conference paper at ICLR 2026
R-Zero: SELF-EVOLVING REASONING LLM FROM ZERO DATA
Chengsong Huang!.2 22 Wenhao Yu't Xiaoyang Wang's ,Hongming Zhang', Zongxia Li
Ruosen Li Jiaxin Huang", Haitao Mi', Dong Yu' Tenneet AI Seattle Lab, @Washington University in St. Louis,
University of Maryland, College Park, "The University of Texas Dallas
i Core contributors
chengsong@wustl. edu; wenlaowyylgllcall com
ABSTRACT
Self-evolving Large Language Models (LLMs) offer scalable path toward super- intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heav- ily on vast human curated tasks and labels, typically via fine-tuning or reinforce- ment learning, which poses fundamental bottleneck advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, fully autonomous framework that generates its own training data from scratch. Starting from single base LLM,, R-Zero initializes two inde- pendent models with distinct roles Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is re- warded for proposing tasks near the edge the Solver's ' capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B- 4B-Base by +6.49 on math reasoning benchmarks, and +7. on general-domain reasoning benchmarks.
Code: https: htttps:/githhbb com/ Cttp///ennnnnnneeeeeeeeeeee Huang R- Zero.
Challenger iterative
Challenger iteeative
Challenger Herafiw
RLVR SFT
Task
Solution
Label-Free
R-Zero (ours)
Task
Solutios
Bose Model
Selver iterative
Solver iteretive 2
Solver iterative
Evoless
Prowides Training Data
Provides Reward
Figure 1: (Left): R-Zero employs a co-evolutionary loop between Challenger and Solver. (Right):
R-Zero achieves strong benchmark gains without any pre-existing tasks human labels.
INTRODUCTION
Self-evolving Large Language Models (LLMs) represent promising frontier for advancing lan- guage intelligence. By autonomously generating, refining, and learning from their own experiences, these models provide scalable pathway toward artificial superintelligence (Tao et al., 2024; Tan

---

Published as a conference paper at ICLR 2026
et al., 2024). A critical requirement for training such self-evolving LLMs is access large volumes of expertly curated tasks and labels, which serve as supervision signals for fine-tuning or reinforce- ment learning with verifiable rewards (RLVR) (Shao et al., 2024; DeepSeck-AL Al et al., 2025). How- ever, relying on human annotatoos to create these tasks and labels is not only costly, labor-intensive, and difficult to scale, but also presents fundamental bottleneck to advancing Al toward capabilities that could eventually surpass human intelligence (Su et al., 2025; Zhao et al., 2025a).
To reduce dependence on human-curated data, self-generated and label-free methods have been proposed to eliminate the need for explicit supervision. In particular, label-free RL derives reward signals directly from the model's own outputs, such as sequence-level confidence scores (Li al., 2025a; Prabhudesai et al., 2025; Huang et al., 2025) and output entropy (Agarwal et al., 2025; Cheng et al., 2025). However, despite removing the need for explicit labels, label-free methods still relies on pre-existing corpus of tasks, which limits its scalability in truly self-evolving settings. On the other side, self-challenging approaches train LLMS on tasks generated by the models themselves (Zhou et al., 2025a; Wang et al., 2025a; Zhao et al., 2025a). While promising, many of these methods rely on external code executors to ensure that the synthesized tasks are both feasible and verifiable. However, in domains that lack an explicit verification oracle, such as open-ended reasoning, ensuring the quality and correctness of self- geferaeed data remains a significant challenge.
In this paper, we propose R-Zero, a framework for training reasoning LLMs that can self-evolve from zero external data. In R-Zero, a single base model is initialized with two roles - a Challenger and a Solver that independently optimized but co-evolve throughout the RL process. During co-evolving, the Challenger is rewarded for generating tasks targeted to be at the edge of Solver's current abilities, while the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. Framework details are provided in Section 2, but briefly, in the Challenger training phase, the Challenger is trained via Group Relative Policy Optimization (GRPO) (Shao et al., 2024) to generate difficult questions. The reward signal is derived from the uncertainty for the frozen Solver, which measured by the self-consistency of its multiple generated answers. In the Solver training phase, the Solver is fine-tuned with GRPO on a filtered set of these challenging questions generated by the now-frozen Challenger, using the pscudo-labels voted by itself. This entire process repeats, creating a self-evolving cycle that operates without any human intervention.
Our experiments demonstrate that R-Zero is a model-agnostic framework, consistently and itera- tively improving the reasoning abilities of different backbone LLMs. For example, Qwen3-4B-Ban model's 's average score on math benchmarks increased by a significant +6.49 points after three itera- tions of self-evolution. Moreover, the reasoning skills learned through our math-focused questions can generalize to complex general-domain tasks, with models trained using R-Zero showing signif- icant improvements on general domain reasoning benchmarks like MMLU-Pro (Wang et al., 2024) and SuperGPQA (Du et al., 2025). Our further amalysis finds that R-Zero can act as a mid-training method, as models first improved by our method achieve higher performance after fine-tuned on labeled data. In addition, we provide an in-depth analysis that validates our framework's 's compo- nents, demonstrates its synergy with supervised fine-tuning, and characterizes the co-evolutionary dynamics to identify both strengths and limitations, offering insights for future research.
2
METHOD
2.1 OVERVIEW
We propose R-Zero, a fully automated framework featuring a Challenger and a Solver, both initial- ized from the same base L.M. The framework operates in an iterative loop. We illustrate the main framework in Figure 2. First, the Challenger (Qo) is trained with Group Relative Policy Optimiza- tion (GRPO) to generate synthetic questions that are challenging for the current Solver (Sec. 2.2). A training dataset of question-answed pairs is then constructed from these synthetic questions using a filtering strategy and a majority-vote mechanism (Sec. 2.3). Next, the Solver (Sa) is fine-tuned on this new dataset. also using GRPO (Sec. 2.4). This iterative process allows the Challenger and Solver to co-evolve, leading to a progressively more capable Solver. The entire framework is self- supervised, requiring no human intervention.
2

---

Published as a conference paper at ICLR 2026
Update GRPO with Uncertainty Reward
21
FyL V1 ... vmm
fi
ma majority voting
Challenger
Solver
1 91.2- Vem
y/
Challenger training
Solver training
21
1 V1..3 H1.m
f
ma majority, voting
Challenger
Solver
2/
(31.1 91.2 N.
f
Update by GRPO with Filtered Question
Figure 2: An overview of R-Zero framework, which illustrates the co-evolution of the Challenger and the Solver. Top: In the Challenger training phase, the Challenger is trained via GRPO to generate difficult questions. The reward signal is derived from the uncertainty for the frozen Solver, which is measured by the self-consistency of its multiple generated answers. Bottom: In the Solver training phase, the Solver is fine-tuned with GRPO on a filtered set these challenging questions generated by the now-frozen Challenger, using the pseudo-labels voted by itself.
2.2 CHALLENGER TRAINING
The Challenger, Qe. is an autoregressive language model trained to generate challenging questions. We train Qo using the GRPO algorithm detailed Sec. I. The core of this process lies designing reward function that accurately captures the desired properties of "good" question. This final scalar reward, rú, is then used in the GRPO. We focus on generating questions specifically within domain of mathematics, as it provides a convenient and self-containes setting for our framework; the objective nature of mathematical answers allows for the straightforward generation of pseudo-labels via majority voting, without the need for external verification environments like code executors.
Uncertainty Reward. To guide the Challenger toward producing challenging yet solvable ques- tions, we first define an uncertainty score. For a generated question x, we query the current Solver S4 for m responses {yi,-..: :mm The most frequent response is treated as the pseudo-label j(x:), and compute the Solver's empirical accuracy p(:; Sy) tainty reward is then defined
}.
Σド 1 ū(x)). The uncer-
T'uncertiitty (z: 3)
p(x; St) !
This function incentivizes questions where the Solver is maximally uncertain (accuracy approaches 50%). We provide theoretical motivation for this reward function in Appendix F.
Repetition Penalty. To encourage diversity within training batch X, we introduce repetition penalty. We could use any similarity metrie, but in our case, we specifically use the BLEU score for faster computation, as this calculation must be performed numerous times during the rollout process. We compute pairwise distances using BLEU score similarity, dij = BLEU(r,, and group questions where dı < TBLEU into clusters C {CI, Cr) cluster C₁ proportional to its relative size:
J. The penalty for a question æ; in
Trep (xt) = |Ckl
B
where B is the batch size and À is scaling factor. In our experiments, we set À = 1. The imple- mentation details are shown in Appendix B.4.
Format Check Penalty. critical first step in the reward pipeline a structural format check to verify that each generated question is correctly enclosed within <question> and < s/questions tags. the output does not adhere to this required structure, it is immediately assigned a final reward of 0, and no further reward signals computed.
3

---

Published as a conference paper at ICLR 2026
Composite Reward and Policy Update. For all questions that pass the format check, calculate a composite reward. The final scalar reward ry for each valid question 2; combines signals for uncertainty and repetition:
ri = max 0,
Funcertainsy (3r4; Φ) Trep (2x))
With these rewards (ri,-.. rG) for batch of generated questions, compute the advantage Å, for each question and update the Challenger's policy Qo by minimizing the GRPO loss LARPO(0)
2.3 SOLVER DATASET CONSTRUCTION
After updating the Challenger, we use generate a new, curated dataset to train the Solver. This process acts as a curriculum generator. We first sample large pool of N candidate questions from the Challenger's policy, 2; ~ Qo Po). For each question, we obtain m answers from the current Solver, determine the pseudo-label is via majority vote, and calculate the empirical correctness P;- A question-answer pair (I; M/) is added to the training set S only its correctness falls within an informative band, L/: 1 S 5. This filtering step discards tasks that are either too easy or too hard.
While the primary goal of this filtering is to discard tasks that are too easy or too hard, it also serves implicit quality control mechanism. Since pseudo-labele are derived from majority vote, a very low empirical correctness P; often indicates that the question itself is ambiguous, ill-posed, or that the resulting pseudo-label is unreliable. By filtering out these low-consistency items, method simultaneously improves the quality and the uncertainty calibration of the training data.
2.4 SOLVER TRAINING
The Solver, Sg. is then fine-tuned on the curated dataset of challenging problems S. We also use GRPO for this stage, but with simpler, verifiable reward signal. For given question 2; E S with its pseudo-label V, the Solver generates batch of answers, each assigned binary reward ry:
{o:
1, if Ty matches with the pscudo-label G!. otherwise.
This verifiable reward is used to compute the advantage Å,, and Solver's policy S sub- sequently updated by minimizing the GRPO loss (GRoo(#). This process enhances the Solver's ability to correctly answer the difficult questions generated by its co-evolving Challenger.
EXPERIMENTS
3.1 MODELS AND TRAINING DETAILS
We employ the Qwen3-4B Base (Yang et al., 2025) and Qwen3-8B (aaa--8B-Bacee models to assess the impact of scale within single architectural family. Second, to ensure our approach is effective on a dis- tinct lineage, we utilize the OctoThinkkrr338 and Octo'Tinkker-8B models (Wang et al., 2025b). choice is particularly relevant as Wang et al. (2025b) reported that applying RL. training directly to Llama models yielded suboptimal results. As the Tctokhinker series continually trained from the Llama-3.1 models (Dubey et al., 2024), this comprehensive selection allows us to test our frame- work across different foundational architectures Qwen vs. Llama. We assess our framework on comprehensive suite of benchmarks, with the evaluation benchmarks presented in Appendix C.
Our entire framework implemented based on the EasyRI codebase (Zheng et al., 2025b). In each iteration of the R-Zero co-evolutionary loop, we follow a specific set of hyperparameters. The 8, 000 questions. To construct the train-
Challenger (Qo) first generates a candidate pool of ing dataset for the Solver, these questions are filtered based on consistency. For each candidate question, we sample m = 10 answers from the current Solver (Sa). question retained for the training set only if the number of answers matching the majority-vote pseudo-label is between 3 and 7, inclusive (6 = 0.25). This numerical range is consistent with the methodology used in previous research (Zhang & Zuo, 2025; Li et al., 2025b; Bercovich et al., 2025). When training the Chal- lenger, the uncertainty reward r(x; Φ) is calculated by sampling m = 10 responses from the Solver. For the intra-batch repetition penalty, we the clustering distance threshold to TBLEU 0.5.
4

---

Published as a conference paper at ICLR 2026
Table I: Comprehensive results on mathematical reasoning benchmarks. We compare each base model against R-Zero (& challenger) baseline (where the Solver is trained on questions from an untrained Challenger) and our method, R-Zero. The peak performance is highlighted in bold.
Model Name
Avg.
AMC
Minerva
MATH
GSM8K
Olympiad
AIME25
AIME24
Qwen,3-4B-B 3-----BBsse
Base Model (w/o training)
42.57
45.70
38.24
68.20
87.79
41.04
10.30
6.7
Absolute 7ero
46.42
52.45
41.96
76.20
89.34
42.56
10.20
12.20
R-Zero (& challenger)
45.01
45.00
45.22
72.80
87.87
41.19
10.20
12.80
R-Zero (our method)
49.93
57.27
52.94
79.60
92.12
44.59
9.60
13.40
Qwen3-8B- Been3-8B-Baat (B-Base
Base Model (w/o training)
48.64
51.95
50.00
78.00
89.08
44.74
12.10
14.60
Absolute Zero
52.68
57.89
57.90
76,60
92.00
47.80
18.20
18.40
R-Zer (% challenger)
52.10
60.70
57.72
81.60
92.56
46.44
11.60
14.10
R-Zero (our method)
53.72
61.67
60.66
82.00
94.09
48.89
13.30
15.40
Octo Thinkerrn
Base Model (w/o training)
26.64
17.19
24.26
$5.00
73.69
16.15
0.21
0.00
Absolute Zero
27.23
22.50
22.70
53.20
75.80
13.20
0,50
2,70
R-Zero (% challenger)
27.51
20.19
24.63
54.60
74.98
15.70
R-Zero (our method)
29.32
27.03
27.57
54.20
74.98
OctoThinker-BB
Base Model (w/o training)
36.41
32.11
41.91
65.20
86.96
26.52
1.56
0.62
Ahsolute 7ero
36.60
32.50
44.90
62.80
87.00
25.60
3.30
0.10
Zeroro (% challenger)
36.98
29.30
42.28
88.10
27.56
R-Zero (our method)
34.03
87.19
27.56
3.1.1 EVALUATION SETTING
The evaluation code is adopted from General-Reasoner (Ma et al., 2025). To ensure consistency, we reran the relcased evaluation code and reported the corresponding results. The reproduced results well aligned with General-Reasonee and those in the Qwen-3 technical report (Yang et al., 2025).
The results presented in all experimental tables are obtained after 45 training steps, while in figures we report evaluations performed at checkpoints every 15 steps during solver training. All results are reported based on the held-out test sets, ensuring a fair comparison across baselines and reproduced methods. Further implementation details and prompts be found in Appendix B.
3.2 RESULTS IN MATHEMATICAL REASONING
The comprehensive results of experiments presented in Table 1. The findings confirm that our proposed framework, R-Zero, a highly effective, model-agnostic method for enhancing the performance of language models on mathematical tasks across different architectures and scales.
Our iterative training process consistently and substantially improves upon the performance of the base models. This holds for large models like Owen3-8B-Bass where three iterations of R-Zero raise the average performance from a baseline of 49.18 to 54.69, significant gain of +5.51 points. Similarly, on the smaller OhtoThink,r-3B, our method improves the average score from 26.64 to 29.32 (+2.68 points), demonstrating the broad applicability of our self-supervised training loop.
The critical role of the Challenger's Ra-based training validated by the immediate performance leap from the Base Challenger to the first iteration of R-Zero. On Owen3-8B-Ban this first iteration provides +1.52 point gain over the baseline, and the improvement is even more pronounced on Qwen3-4B-Base at +3.7 points. This confirms that the intelligent curriculum generated by the RL- trained Challenger is significantly more effective than that non-trained generator.
3.3 RESULTS IN GENERAL REASONING
Previous work has demonstrated that training language models on reasoning- intensive domains, such mathematics, can lead to improvements gemeral-domain capabilities (Huan et al., 2025). A key question, however, is whether this generalization effect still holds when the training curriculum is not human-labeled, but entirely self-generated through R-Zero.
5

---

Published as a conference paper at ICLR 2026
Table Results on general-domain reasoning benchmarks. The table compares the Base Model,
R-Zero (# challenger) baseline, and our R-Zero. The peak performance is highlighted in bold.
Model Name
Overall Avg.
SuperGPQA
MMLU-Pro
BBEH
Qwen3-4B-o 4B--ueeB
Base Model (wlo training)
26.34
20.88
50.58
7.57
Absolute Zero
29.33
27.10
52.60
8.30
R-Zero (% challenger)
28.52
24.77
54.20
6,59
R-Zero (our method)
31.15
27.55
55.47
10.42
Qwen3-8B-Bar
Base Model (wlo training)
31.98
28.33
58.97
8.63
Absolute Zero
34.40
31.89
60.50
10.80
R-Zero (% challenger)
31.29
30.12
54.14
9.60
R-Zero (our method)
34.50
31.38
61.53
10.60
OctoThinker-3B
Base Model (w/o training)
7.47
10.09
10.87
1.46
Absolute 7ero
16.03
17:70
24.30
6.10
R-Zero (A challenger)
10.04
11.19
14.53
4,40
R-Zero (our method)
11.12
12.44
16.71
4.20
Octoll hinker-SB
Model (wlo training)
11.70
13.26
20.21
Absolute Zero
18.40
18.80
31.40
R Zerero ( challenger)
21.30
16.99
41.46
R-Zero (our method)
23.00
19.82
40.92
8.25
As shown Table 2, this transfer of skills is evident across all tested models. For instance, three iterations of math-focused training improve the average general-domain score of Qwen3-8B- Base by +5.13 13 points and DetoThinker-3B by +3.65 points. This generalization also extends to the key performance patterns observed in the mathematical results. This confirms that method does not merely teach domain-speciff knowledge, but enhances the model's underlying capabilities in way that successfully generalizes across domains.
4 ANALYSIS
In this section, we conduct series of in-depth analyses to better understand the behavior and effec- tiveness of our R-Zero framework. To ensure consistency, all analytical experiments presented here are conducted on the Qwen3-4B-Bass model, unless explicitly stated otherwise. Some additional analyses are shown in Appendix D.
4.1 ABLATION STUDY
Removing Repetition Penalty and Task Filtering will harm the final performance.
To isolate the contribution of each key com- ponent within our R-Zero framework, con- duct comprehensive ablation study on the Owen3-4B-Bas model. We specifically in- vestigate the importance of two critical mod- ules by disabling them one at a time and ob- serving the impact performance. The results are summarized in Table 3.
Table Ablation results on Qwen3-4B-Ban w/o Filtering: Disables the difficulty-based curricu- lum filtering. w/o Rep. Penalty: Removes the repetition penalty from the Challenger's reward.
Method
Math AVG
General AVG
R-Zero
49.07
31.15
Ablations
w/o Rep. Penalty
45.76 47.35
28.73 26.69
As shown in the table, removing any core components leads to significant degradation in performance. Removing the Repetition Penalty harms performance, indicating that generating a diverse set of questions is crucial for effec- tive Solver training.
w/o Filtering

---

Published as a conference paper at ICLR 2026
Finally, disabling the Task Filtering module results in notable performance drop, particularly the general-domane average, which falls by over 6 points. As discussed in Section 2.3, this filtering serves a dual purpose: it calibrates the curriculum's 's difficulty and acts implicit quality control mechanism by removing questions with low answer consistency. Without this filter, the Solver is trained on noisy data that likely includes ambiguous ill-posed questions, which harms its ability to learn robustly.
4.2 ITERATION SCALING
Model performance eventually converges, with the timing depending model size.
Previous results demonstrate that R-Zero enhances the Solver's capabilities. This raises critical question about the long-term stability of our self-improvement loop: what are the limits of this process, and whether the eventual performance degradate? In this section, we conduct analysis to investigate these iteration scaling dynamics, aiming to diagnose the underlying cause of this instability.
As illustrated in Figure 3, our framework ini- tially delivers its promise, with models of all sizes showing significant performance im- provements in the early stages of co-evolution. Unfortunately, this virtuous cycle does not con- tinue indefinitely. After multiple iterations, observe consistent and concerning trend of performance degradation across all models. In- triguingly, we found direct correlation be- tween model scale and resilience this col-
Diverc- dif-lxxn
Gove 10--
Performance (AVG Score)
.50
43
40.53
60
38.6
W
30.75
130
35.06
20
16:77
Base Model
Step 15
Step 30
Step 45
Step 60
Figure 3: Math performance across different iter- ation times and model scales. The star markers in- dicate the peak performance for each model size.
lapse: the larger the model, the later the onset of performance degradation.
For instance, the smallest 0.6B model reaches its peak performance early as the first iteration (Step 15), after which its capabilities begin to decline. In contrast, the largest 4B model sustains its upward trajectory for three full iterations, only experiencing sharp drop Step 60. This pattern strongly suggests that while larger model capacity can delay the negative effects, it does not pre- vent them. This eventual collapse points to inherent instability or limitation within our current self-improvemmnt framework, highlighting a critical area for future investigation. We present some additional analysis results for this in Appendix E.
4.3 SYNERGY WITH SUPERVISED DATA
Using R-zero as a mid-training method boosts the effect of later training on human data.
To analyze the utility of our framework in scenarios where labeled dataset is available, we measure the synergy between R-Zero and traditional supervised fine-tuning using labeled datasets The GRPO settings for this experiment were kept identical to our main experiments.
We first establish supervised baseline by fine-tuning the base model directly on the labeled data. For this process, we also employ GRPO.
We then apply our R-Zero framework, where at the end of each co-evolutionary iteration, the re- sulting checkpoint is also fine-tuned on the same labeled dataset. The results show that our method provides significant additional gains. As highlighted in Figure 4, this represents gain of +2.35 points over the laman-label-one baseline for 4B model.
This finding confirms that R-Zero is not redundant with labeled data; instead, it acts power- ful performance amplifier. The co-evolutionary process enables the model better leverage the supervised information and achieve performance levels unattainable by standard fine-tuning alone.
7

---

Published as a conference paper at ICLR 2026
B-Wan--B-Base
55
50
40.02
64.92
53
Avg Soore
53:15
06
48 64
Ave
52.30
53:24
64
Only
80
R-Zoro Only
ZeZor + Human
R: Zoro Human
42.5
Step
Step
Figure 4: Performance of R-Zero when combined with supervised fine-tuning. The dashed line rep- resents the baseline of fine-tuning the base model on labelled data alone, showing that our iterative method provides better initialization.
We also studied the scenario where human-labeled datasets are mixed into the R- R-Zero generated dataset for training. We found that this concurrent training approach outperforms using either R- Zero dataset or humanlabel dataset alone, but it does not surpass the sequential strategy of first applying R-Zero and then performing SFT. We hypothesize that mixing the labeled data during R- Zero training may dilute the high-quality signal while partially mitigating noise. Performing R-Zero first allows the model to acquire reasoning ability before leveraging the high-quality human-labeled data, which appears to be the most effective strategy. The precise underlying reasons require further investigation.
AMC
Miserva
MATH
GSMHK
Olyngiad
AIME25
AIME24
Supe:GPQA
MMLUU-Po
BBEH
55.15
80.8
9.58
23.49
$7.03
9.71
57.27
& Zeroelluman $7.9
53.2
81.2
92.2
47.7
10.32
14.67
29.4
58.2
11.5
Table 4: Performance comparison across benchmarks when training with human-labeled data, R- zero self-generated data, and a mixed training strategy.
4.4 EVOLUTION OF QUESTION DIFHCULIYY AND DATA ACCURACY
Question difficulty increases progressively with decreasing pseudo-label accuracy.
Table Performance and data accuracy analysis. The highlighted column represents the true accu- racy of the self-generated pseudo-labels for each question set.
Performance of Evaluated Model (vs. Ground Truth)
Base Model
Solver (step 15)
Solver (step 30)
Solver (step 45)
Pseudo-Label Acc.
Dsuu 15
48.0
59.0
57.0
61.0
79.0%
Diup 30
52.5
53.0
51.5
53.5
69.0%
Dstep
44.0
47.0
45.0
50.5
63.0%
To understand the co-evolutionary dynamic, we analyzed how the Challenger's 's generated questions and their corresponding pseudo-labels change across iterations. We sampled 200 questions from Challenger's policy after each of the first three training iterations. creating three distinct test sets: Dstep 15. Deep and Dsue 45. For this analysis, we assumed the external oracle model, GPT-4o, to be a perfect annotator, providing the ground truth answers for all generated questions.
The evaluation was conducted as follows: the performance of our internal models was measured against these GPT-4o ground truth answers. The reported for GPT-4o itself, however, reflects the true accuracy of our self-generated pseudo-labels by comparing the pseudo label against the ground truth from the oracle (GPT-4o). The results the filtered dataset are summarized in Table 5.
This analysis reveals a falli--fccetdd dynamic. The first finding is that the questions generated by the Challenger become progressively more difficult. This is directly evidenced by evaluating fixed
8

---

Published as a conference paper at ICLR 2026
model against the evolving question sets. For instance, the performance of the static Solver (Step 15), when measured against the consistent GPT-4o ground truth, drops from 59.0% on the questions for the Step 15 training to 47.0% on the questions for the Step 45. This confirms that the Challenger is successfully increasing the intrinsic difficulty of its curriculum.
The second finding, revealed by the highlighted column, pertains to the true accuracy of the self- generated dataset. Unfortunately, while the accuracy of the pseudo-labels initially high at 79.0%, systematically drops to 63.0% by the third iteration. This trend indicates that the system gen- erates more difficult problems, the Solver's majority vote becomes a less reliable source for ground truth. This decline in data quality a critical trade-off and a potential bottleneck for the framework's ultimate performance.
RELATED WORK
5.1 LABEL-FREE REINFORCEMENT LEARNING
significant trend in recent research Label-Free Reinforcement Learning, which aims to improve LLM reasoning without human-annotace data. Many such methods use the model's own outputs as a reward signal. This includes leveraging sequence-level confidence (Li et al., 2025a; Prabhudesai et al., 2025), the consistency of answers derived from varied reasoning paths (Zhang et al., 2025a; Zuo et al., 2025; Zhang et al., 2025b; Zhou et al., 2025b; Prasad et al., 2024), minimizing the output entropy (Agarwal al., 2025; Cheng al., 2025), or even random (Shao al., 2025) negative reward (Zhu et al., 2025). These signals are often used within self-training loops where models fine-tune on their own most plausible solutions (Shafayat al., 2025; Zhao et al., 2025b). While these methods all rely on a pre-existing set of unlabeled problems, R-Zero removes the need for any seed dataset.
5.2 SELF-PLAY IN LARGE LANGUAGE MODELS
The paradigm of self-play, where models take on dual roles to create a self-improvement loop (Chen al., 2024: Zhang et al., 2024), has recently been adapted improve language models without human data. This approach has been particularly fruitful in verifiable domains like code generation, where a "Coder" agent's program is verified by a "Tester" agent's unit tests (Lin et al., 2025; Wang et al., 2025a; Pourcel et al., 2025; Jiang et al., 2025). More advanced frameworks push autonomy further by learning to generate the problems themselves, creating an adaptive curriculum from small seed of examples or from scratch (Zhao al., 2025a; Li et al., 2025c; Zhou et al., 2025a; Fang et al., 2025). Our work distinguishes itself by extending this paradigm to general reasoning domains that lack such verifiable environments, like coding tasks.
5.3 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS (RLVR)
Reinforcement Learning with Verifiable Rewards has been widely adopted as a versatile paradigm for enhancing LLMs across multitude of tasks (Li et al., 2025d; DeepSeck-Al et al., 2025; Shao et al., 2024). Its effectiveness demonstrated in diverse applications such as relation extraction (Dai et al., 2025), interactive GUI navigation (Shi al., 2025b) and search-engine utilization (Jin al., 2025). While early implementations relied on rule- based verifiers, recent work has begun to explore more sophisticated, model- verifiers (Ma et al., 2025; Li et al., 2025b; 2024).
6 LIMITATION
While R-Zero demonstrates strong improvements in reasoning performance across multiple do- mains, several limitations remain. First, the core mechanism of R-Zero relies on domains where correctness can be objectively verified. The Challenger-Solver co-evolution process depends on clear, deterministic evaluation signals to produce reliable training feedback. Consequently, applying R-Zero to open-en ended subjective tasks, such creative writing, dialogue, generation, remains difficult, as these tasks lack unambiguous correctness criteria. In addition, R- Zero currently employs specific labeling and verification strategies that may not generalize all task
dreierenceedriven

---

Published as a conference paper at ICLR 2026
types. Developing more robust and broadly applicable labeling mechanisms would further expand the range domains where R-Zero can be effectively applied.
7 CONCLUSION AND FUTURE WORK
In this paper, we introduced R-Zero, a fully autonomous self-evolving framework that overcomes data dependency by having Challenger and Solver co-evolve to create self- seff-generating curricu- lum. Our experiments demonstrate that R-Zero significantly improves LLM's reasoning capability in multiple domains. Future work could further focus improving efficiency, exploring more bust labeling techniques, and expanding R-Zero to new domains. Extending this self-evolutionary paradigm to open-ended generative tasks, such creative writing dialogue, where evaluation is subjective, remains a significant hurdle for future research.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, we provide detailed information regarding our data and experimental setup. All datasets used in this work publicly available; provide details data sources, any postprocess steps in Appendix B. A comprehensive list of all hyperparameters for each experiment can be found in table in Appendix B and Sec. 3.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in Ilm reasoning. ArXiv preprint, abs/2505. 15134, 2025.
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, Jeong Yeon Nam, et al. Online difficulty filtering for reasoning oriented reinforcement learning. ArXiv preprint, abs/2504.03380 2025.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Yaniv,,, et al. Llama- nemotron: Efficient reasoning models. ArXiv preprint, abs/2505.00944 2025.
Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their output: An analysis of the self-consuming training loop. ArXiv preprint, abs/2311.16822, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 2024.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, et al. Reasoning with exploration: An entropy perspective. ArXiv preprint, abs/2506.14755 2025.
Karl Cobbe, Vincet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/21 10. 14168, 2021.
Rumpeng Dai, Tong Zheng. Run Yang, and Hongtu Zhu. -re Cross-domain relationship extraction with rlvr. ArXiv preprins, abs/2507.0466 2025.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, et al. Deepseck-rl: Incen- tivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501.12948, 2025.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. ArXiv preprint, abs/2410.048400, 2024a.
Elvis Dohmatob, Yunzhen Feng, Pu Yang. François Charton, and Julia Kempe. A tale tails: Model collapse change of scaling laws. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21 21-27, 2024, 2024b.
Xinrun Du Yifan Yao. Kaijing Ma, Bingli Wang, Tianyu Zheng, et al. Supergpqa: Scaling Ilm evaluation across 285 graduate disciplines. ArXiv preprint, abs/2502.14739 2025.
10

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783. 2024.
Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, et al. Serl: Self-play rein- forcement learning for large language models with limited data. ArXiv preprint, abs/2505.20347, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Association for Computational Linguistics, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proc. of ICLR, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, et al. Measuring math- ematical problem solving with the math dataset. ArXiv preprint,
2021
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, et al. Does math reasoning improve general IIm capabilities? understanding transferability of Ilm reasoning. 2025.
Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. ArXiv preprint, abs/2503.0003/ 2025.
Hao Jiang, Qi Liu, Rui Li. Yuze Zhao, Yixiao Ma, Shengyu Ye, Junyu Lu, and Yu Su. Verse: Verification- self-play for code instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24276-24284 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ils to reason and leverage search engines with reinforcement learning. ArXiv preprint, abs/2503.09516. 2025.
Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, et al. Tülu 3: Pushing frontiers in open language model post-traine pest-training. ArXiv preprint, abs/2411.15124, 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022.
Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan V. Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. ArXiv preprint, abs/2506.06395, 2025a.
Ruosen Li, Ziming Luo, and Xinya Du. Fg-prm: Fine-grained hallucination detection and mitigation in language model mathematical reasoning. ArXiv preprinr, abs/2410.06300 2024.
Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, and Jordan Lee Boyd-Graber Semantically-awe aware rewards for open-ended rl training in free-form generation. ArXiv preprint, abs/2506. 15068. 2025b.
Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. Videohallu: Evaluating and mitigating multi-modal hallucinations on synthetic video understanding. ArXiv preprint, abs/2505.0148., 2025c.
Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi. et al. Self-rewarding langunel model via reasoning decomposition. arXiv preprint arXiv:2508. 19652, 2025d.
Zi Lin, Sheng Shen, Jingbo Shang, Jason Weston, and Yixin Nie. Learning to solve and verify: A self-play framework for code and test generation. ArXiv preprint, abs/2502, 14948, 2025.
11

---

Published as a conference paper at ICLR 2026
Xueguang Ma, Qian Liu, Dongfu Jiang. Ge Zhang, Zejun Ma, al. General-reasoner: Advancing Ilm reasoning across all domains. ArXiv preprint, abs/2505. 14652, 2025.
Julien Pourcel, Cédric Colas, and Pierre-Yve Oudeyer. Self-improvin language models for evolu- tionary program synthesis: A case study on arc-agi. 2025.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, et al. Maximizing confidence alone improves reasoning. ArXiv preprini, abs/2505.2266// 2025.
Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mohit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistnncy preference opti- mization. arXiv preprint arXiv:2411.041000 2024.
Mohamed EI Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Mérouane Deb- bah. How bad is training on synthetic data? a statistical analysis of language model collapse. ArXiv preprint, abs/2404.05090, 2024.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? ArXiv preprint, abs/2505.21 21444, 2025.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, et al. Spurious rewards: Rethink- ing training signals in rlvr. ArXiv preprint, abs/2506. 10947, 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Jun-Mei Song, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning. ArXiv preprint, abs/2504.05520, 2025a.
Yucheng Shi, Wenhao Yu, Zaitang Li, Yonglin Wang, Hongming Zhang, et al. Mobilegui-rl: Ad- vancing mobile gui agent through reinforcement learning in online environment. 2025b.
Mehrangiz shoaa kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, et al. Big-bench extra hard. In Annual Meeting of the Association for Computational Linguisties, 2025.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained recursively generated data. Nature, 631, 2024.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, et al. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. ArXiv preprint, abs/2503.23822 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thoughh can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, 2023.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Proc. of EMNLP, 2024.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, et al. A survey on self- evolution of large language models. ArXiv preprint, abs/2404. 143387 2024.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving I1m coder and unit tester via reinforcement learning. ArXiv preprint, abs/2506.03136, 2025a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
12

---

Published as a conference paper at ICLR 2026
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid- Mid-training incentivizes reinforcement learning scaling. ArXiv preprint, abs/2506.20512 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 technical report. ArXiv preprint, Abs/2250509388, 2025.
Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty- reinforcement learning approach for concise mathematical reasoning in language models. ArXiv preprint, abs/2504.0966 2025.
Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang. Baisheng Lai, et al. Consistent paths lead to truth: Self-rewarding reinforcement learning for Ilm reasoning. ArXiv preprint, abs/2506.08745, 2025a.
Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised Ilm reasoning incentivization. ArXiv preprint, abs/2504.05E5 05812, 2025b.
Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Wenhao Tang, Shiyu Huang, Deheng Ye, Wenbo Ding, Yaodong Yang, et al. A survey on self-play methods in reinforcement learning. arXiv preprint arXiv:2408.01077 2024.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, et al. Absolute zero: Reinforced self- play reasoning with data. ArXiv preprint, /0w/25505.0335, 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Xiaodong Song. Learning to reason without external rewards. ArXiv preprint, abs/2505. 19590, 2025b.
Yulai Zhao, Haolin Liu, Dian Yu, S. Y. Kung, Haitao Mi, and Dong Yu. One token to fool llm-as-a- judge. volume abs/2507.08794, 2025c.
Tong Zheng, Lichang Chen, Simeng Han, R Thomas McCoy, and Heng Huang. Learning to reason via mixture-of-thought for logical reasoning. arXiv preprint arXiv:2505. 158/7, 2025a.
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangehi Feng, Dongdong Kuang, et al. Easyrl: An efficient, scalable, multi-modalityy rl training framework. 2025b.
Yifei Zhou, Sergey Levine, Jason E. Weston, Xian Li, and Sainbayar Sukhbaatar. Self-challenging language model agents. ArXiv preprint, abs/2506.01716 2025a.
Yujun Zhou, Zhenwen Liang, Haolin Liu, Wenhao Yu, Kishan Panaganti, Linfeng Song, Dian Yu, Xiangliang Zhang, Haitao Mi, and Dong Yu. Evolving language models without labels: Majority drives selection, novelty promotes variation. arXiv preprint arXiv:2509. 15144, 2025b.
Yujun Zhou, Jiayi Ye, Zipeng Ling, Yufei Han, Yue Huang, Haomin Zhuang, Zhenwen Liang, Kehan Guo, Taicheng Guo, Xiangqi Wang, et al. Dissecting logical reasoning 1lms: A fine-grained evaluation and supervision study. arXiv preprint arXiv:2506.00 04810, 2025c.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, et al. The surprising effective- ness of negative reinforcement in I1m reasoning. ArXiv preprint, abs/2506.01347 2025.
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, et al. Ttrl: Test-time reinforcement learning. ArXiv preprint, abs/2504. 16084, 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
acknowledge the use of large language models (LLMs) as assistive tools in this research, with usage limited to refining grammar and improving language clarity in the manuscript, writing utility scripts for data preprocessing and postprocessing, and debugging; all outputs from these models were meticulously reviewed, revised, and verified by the authors, who retain full responsibility for all content presented in this paper.
B EXPERIMENT DETAILS
B.1 TRAINING HYPERPARAMETER
This section summarizes the most critical algorithmic hyperparameters for the Solver and Chal- lenger training stages. All experiments were conducted using B (BF16) mixed precision and FlashAttention 2.
B.1.1
SOLVER TRAINING
Global Batch Size: 128
Learning Rate: x 10-6
Weight Decay: 1 10-2
KL Penalty Coefficient (AKl): 1 x 10-2
Max Steps: 15
Number of Rollouts: 5
Rollout Temperature: 1.0
Rollout Top-p: 0.99
B.1.2
CHALLENGER TRAINING
Global Batch Size: 128
Learning Rate: 10-6
Weight Decay: 1 10-2
KL Penalty Coeflicient (AKL): 1 x 10
Max Steps: 5
Number of Rollouts: 4
Rollout Temperature: 1.0
Rollout Top-p: 0.99
B.2
PROMPT TEMPLATES
This section presents the exact prompt templates used for the solver and challenger models.
Solver Prompt Template
System Message:
Please reason step by step, and put your final answer within \boxed(].
User Message:
poroblemstatementt
Note: (probiem.stateeeiee is placeholder for the actual math problem.
14

---

Published as a conference paper at ICLR 2026
Challenger Prompt Template
System Message:
You are an expert competition-mate problem setter. FIRST, in your private scratch-pad, think step-by-step to design brand-new, non-trivial problem. The problem could come from any field of mathematics, including but not limited to algebra, geometry, number theory, combinatorics, prealgebra, probability, statistics, and calculus. Aim for a difficulty such that fewer than 30% of advanced high-school students could solve it. Avoid re-using textbook clichés or famous contest problems.
THEN, without revealing any of your private thoughts, output exactly the following two blocks:
<questions
(The full problem statement on one or more lines) </question>
(boxed(final.answer
Do NOT output anything else-no explanations, no extra markup.
User Message:
Generate one new, challenging reasoning question now. Remember to format the output exactly as instructed.
B.3 GPT-4o JUDGE PROMPT
To programmatically evaluate the correctness of answers on mathematical benchmarks where the final answer can be complex (e.g., simplified expressions), use GPT-4o as a judge. The exact prompt and configuration used for this evaluation are detailed below.
Configuration for GPT-40 Judge
Model: gpt-4o 44
Temperature: 0.1
System Message:
You are math answer checker.
User Message Template:
Hi, there is an answer: (answer).
and the ground truth answer is: (response).
please check whether the answer is correct or not, and return the **nlyy* Yes or No.
Note: {answer) is placeholder for the generage solution, and (response) is the ground-truth answer from the benchmark.
B.4 REPETITION PENALTY IMPLEMENTATION
To encourage the Challenger generate diverse set of questions within each batch, we apply repetition penalty, Trep- This penalty is designed to disincentivize the model from producing seman- tically similar questions in same batch. The implementation a multi-step process based on clustering questions by their BLEU score similarity.
Pairwise Distance Calculation via BLEU Score First, we compute pairwise distance matrix for all questions in batch. The distance dij between any two questions, H, and zy, is defined as one minus their BLEU score:
dij =1 1- BLEU 2.))
15

---

Published as a conference paper at ICLR 2026
For this calculation, specifically use the sentence.bleu function from the NLTK li- brary (raatttttttt tea.kk.ransattte bleuusccore) To ensure numerical stability, especially for shorter questions with limited n-gram overlap, Smooth AmmoothingFunct... in () .oeth The questions are tokenized for the BLEU calculation by splitting whitespace; further text normalization, such as lowercasing or punctuation removal, is performed.
employ its first smoothing function,
2. Agglomerative Clustering With the pairwise distance matrix computed, then group similar questions using agglomerative hierarchical clustering. This step is performed using the Clustering implementation from the scikit-leas library. The clustering algorithm is con- figured with the following key parameters:
Metric: Set precomputed' indicating that we provide our custom BLEU-based
distance matrix instead of having the algorithm compute distances.
Linkage: Set to 'average'. This method defines the distance between two clusters as the average of the distances between all pairs of questions across the two clusters.
3. Final Penalty Calculation Once each question the batch assigned to a cluster, the repe- tition penalty (x;) for given question æ; is determined by the relative size of the cluster C₁ to which it belongs. The penalty is calculated as:
|C&b B
Here, CCxl represents the number of questions in cluster Ck, and B is the total number of questions in the batch (i.e., the batch size).
C EVALUATION BENCHMARK
We assess our framework on a comprehensive suite of benchmarks. Although the question-generator prompt for our method primarily focused on mathematical problem-solving. a key objective of our evaluation is to explore whether the resulting improvements in reasoning ability can generalize to other domains. Therefore, our evaluation is divided into two main categories.
generator
Mathematical Reasoning, We use seven challenging benchmarks: AMC, Minerva (Lewkowycz et al., 2022), MATH-500 (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021), Olympiad- Bench (He et al., 2024), AIME-2024, and AIME-2025 For these tasks, where answers can be complex, we employ GPT-4o as programmatic judge to semantically verify the correctness of the final answer against the ground truth (Zhao al., 2025c). For the difficult AMC and AIME bench- marks, we report the mean 32 metric. For all other math benchmarks, we report accuracy based on greedy decoding.
General Domain Reasoning, To test for the generalization of reasoning ability, we evaluate the following challenging benchmarks:
MMLU-Pro (Wang et al., 2024): An enhanced version of the MMLU (Hendrycks et al., 2021a) benchmark, featuring a more challenging suite of multi-task questions designed to provide a stricter evaluation of language model capabilities.
SuperGPQA (Du et al., 2025): A large-scals benchmark focused on graduate-level rea- soning. It comprises questions across 285 distinct disciplines that have been verified as unsearchable on the web, thereby isolating true reasoning ability from simple knowledge recall.
BBEH (shoaa kazemi et al., 2025): This benchmark builds upon the foundation of BIG- Bench Hard (Suzgun et al., 2023) by incorporating new selection of tasks specifically engineered to be more difficult, thus providing a more accurate measure of complex rea- soning skills.
For this category. we follow the experimental setup, prompts, and evaluation codes from (Ma et al., 2025), reporting Exact Match (EM) accuracy obtained via greedy decoding.
16

---

Published as a conference paper at ICLR 2026
D PARAMETER SHARING BETWEEN CHALLENGER AND SOLVER
Separating the Challenger and Solver into two models will be better.
Table 6: Comparison of math performance and pseudo-lahel accuracy between the standard R-Zero
(two-model) and Single- (unified model, shared parameters) frameworks across iterations.
R-Zero (ours)
Single-R-Zero
Iteration
Performance
Pscudo-label Acc (%)
Performance
Pseudo-label Acc (%)
Step 15
48.06
71.0
47.31
63.4
Step 30
48.44
56.2
46.95
46.6
Step 45
49.12
48.8
45.57
32.6
Step 60
46.52
42.2
43.89
33.8
To investigate whether the separation of the Challenger and Solver into two independent models is a necessary component for the success of R-Zero, we conduct an ablation study using a unified model with shared parameters. In this configuration (Single-R-Zern) performing both roles, i.c., generating a challenging curriculum and subsequently learning from it.
single model tasked with
The results, presented in Table clearly indicate that separating the Challenger and Solver into two independent models is crucial for both performance and stability. We observe two key findings. First, our standard two-model R-Zero framework not only achieves a higher peak performance (49.12) but also sustains improvement for more iterations, with its collapse occurring after the third iteration. In contrast, the unified Single-R-Zero model's performance peaks after the very first iteration and degrades immediately thereafter. Second, the Single-R-Zero model, where the agent must generate and solve its own problems, produces pscudo-labels of significantly lower accuracy at every stage. For example, in the first iteration, its pscudo-label accuracy is already substantially lower than the R-Zero's s (63.4% vs. 71.0%). We hypothesize that this is because having the problem-setter and solver originate from the same model leads to form of overconfidence that comes from internal bias.
E BEYOND LABEL NOISE: UNPACKING THE ROOTS OF INSTABILITY
The most immediate hypothesis for this performance col- lapse is the degradation of pseudo-label quality, po- tential failure mode of the self-correction mechanism we discussed in Section 4.4. As the Challenger generates increasingly difficult problems, it is plausible that the Solver's majority vote becomes a less reliable source for ground truth, resulting in noisy training signal that could ultimately harm performance. To empirically test the extent to which this is the primary cause, we sampled 500 questions from a later training iteration to conduct more granular investigation into the relationship between pseudo-label fidelity and the observed performance drop.
Table 7: Accuracy of self-generated pseudo-labels (%), labeled by Gemini. Shaded and bolded values indicate the
best checkpoint for each model size.
Step
Model Size
0.6B
1.7B
4B
Step 15
70.6
69.4
71.0
Step 30
53.4
55.2
56.2
Step 45
50.8
52.2
48.8
Step 60
44.0
45.2
42.2
Although the degradation of pseudo-label accuracy is consistent trend across iterations, our anal- ysis suggests this not the primary, nor even the sole, driver of the eventual performance collapse. Table 7 presents the pseudo-label data quality for each model at the onset of its performance col- lapse. Intriguingly, there appears to be no universal accuracy threshold that triggers this degradation. For instance, the 0.6B model begins to decline when data accuracy is still high as 70.6% (Step 15), whereas the 4B model tolerates an accuracy as low as 48.8% (Step 45) before its performance drops.
This suggests that the absolute percentage of label noise is not the sole determinant of instability. Another potential, and perhaps more fundamental, reason is form of model collapse that can be introduced when training exclusively on self-synthesized data (Tan et al., 2024: Shumailov et al., 2024; Dohmatob et al., 2024b; Zhou al., 2025c; Seddik al., 2024; Dohmatob et al., 2024a;
17

---

Published as a conference paper at ICLR 2026
Briesch et al., 2023; Zheng et al., 2025a). A model can enter degenerative feedback loop, suffering from a loss of diversity or an amplification of its own biases, which presents a significant challenge.
F THEORETICAL ANALYSIS
In this section, we provide a theoretical motivation for our uncertainty reward function, Funcertainty OC - [1; 5²) I. which is maximized when the Solver's success probability, p. is 50%. Our analysis is grounded in recent work that formally establishes that the most efficient training occurs when a learner is exposed to tasks at the frontier of its capabilities (Shi et al., 2025a; Bae et al., 2025).
The core insight from these studies that the learning potential of the current Solver, with policy optimal policy S* This divergence, Dxzz (Spl(S))
S5. can be quantified by the KI. divergeee is lower-bounded by the variance of the Solver's 's reward. For the binary reward signal used in our framework, the success probability is p. This leads the specific lower bound:
(K2(5 ((1-1)
2,32
where / is the temperature parameter that controls entropy regularization. The right-hand side of the inequality, which proportional the reward variance, is maximized precisely when ò 0.5. Therefore, by designing the Challenger's 's reward to incentivize questions that push the current Solver towards this point of maximum uncertainty, our framework is theoretically motivated to generate maximally efficient curriculum in each iteration of the co-evolutionary process.
18

---

end Published as a conference paper at ICLR 2026
G ALGORITHM OF R-Zero
Here, we provide the pseudocode for our method.
Algorithm 1: R-Zero: Self-Evolving Challenger- Framework
Input:Initial models Qo, Sp: Group size G; Solver samples m; Dataset size N; Filtering threshold & Repetition threshold TLEUU- Output:Fvolved models Qo and Sp. for each self-play iteration do
// Phase 1: Challenger Training (Sec 2.2)
for each Challenger training step do
Sample question group {a; Qo(-); for each question z; do
if FormatCeeck(a)) is invalid (e.g., missing tags) then
r4 4 0; end else
Sample m answers {uy fj=1 S- 2;); Get pseudo-label D; Majority Vote {リı 1=1):
Compute correctness P; + شグ 1 (リ = y:
Compute uncertainty reward "uncertainty <- 2 2
Compute repetition penalty Trep (25) via BLEU clustering (Sec 2.2);
Final reward: max l'uncertainty Trep (x:));
end
Update Qo via GRPO using rewards {r:)en end
Phase 2: Solver Dataset Construction (Sec 2.3)
Initialize curated dataset S + 0;
Sample N candidate questions (xx N Qo(:): for each candidate 7x
Sample m answers (u, j=1 So (. IR)::
Get pseudo-label
Voteritt Compute correctness Pa 1. Eド= 1[y ūz}:
pp 1 S then Add (Ik, 5k) to S;
end end
//
Phase 3: Solver Training (Sec 2.4)
for each minibatch (x. 9) S
Sample G answers { ~ S.(. ();
Compute binary rewards r; + [1y; = 7):
Update S& via GRPO using rewards fry 1G
end
end
H GENERALIZING BEYOND THE MATH DOMAIN
To further evaluate the generality of our proposed approach, we conduct an additional experiment in which the model is no longer instructed to generate exclusively mathematical problems during data construction. Instead, we remove the math-specific constraint from the prompting stage, allowing the model to produce broader mixture of questions, including commonsense, logical reasoning, and other non- mathematical categories. This setup enables us to examine whether the improvements ob- served in earlier experiments merely a consequence of alignment with math-heat benchmarks or whether our method can maintain (or even enhance) performance under a more domain- -ennral training regime.
19

---

Published as a conference paper at ICLR 2026
Table Results of different models and prompting strategies on mathematics and reasoning bench- marks.
AMC
Milserva
MATH
GSMBK
Olyuspidd
AIME25
AIME24
SuperGPQAA
MMLU-Pro
BREH
Qwen3-4B
45.7
38.24
68.2
41.04
20.83
37.38
7.57
R-Zamo
S7
52.94
79.6
92.12
44.50
9.6
13.4
27.55
55.47
10.42
Zero tso modh)
$6.27
32.56
79.2
92.36
43.56
9.9
13.2
28.14
54.75
10.31
Qwas3-88
78
R:Zerr
61.67
60.66
82
94.09
48.89
13.3
15.4
31.38
61.53
10.00
Zero Uso mosh)
65.53
61.19
81.5
93.89
89.24
13.2
1522
32.29
61,83
10.38
The motivation behind introducing this experiment stems from concerns that many mathematical benchmarks contain well-structured and heavily studied problem formats. If the training data is stricted to similar mathematical tasks, may artificially inflate performance gains while providing limited evidence regarding the method's real-world applicability. By diversifying the data genera- tion process, we aim to assess whether the model can benefit from richer and more heterogeneous supervision signals.
The experimental results, summarized in Table 8, demonstrate that when the base model has suffi- cient capacity (e.g., the 8B variant), training both math and non-math benchmarks. Notably, the "R-Zero (remove math prompt)" setting exhibits the highest performance on the general benchmarks for the 8B model, suggesting that our approach effectively transfers to broader reasoning tasks beyond pure mathematics.
general data leads to further improvements in
These findings provide additional evidence that the proposed method is not limited to mathematical domains; rather, it scales effectively with model capacity and supports improved reasoning perfor- mance under more diverse training conditions.
PRELIMINARIES
Our work builds upon recent advancements in reinforcement learning for fine-tuning large language models. We briefly review two key methodologies that are relevant to our framework.
I.1 GROUP RELATIVE POLICY OPTIMIZATION
Group Relative Policy Optimization (GRPO) (Shao et al., 2024) reinforcement learning algo- rithm for fine-tuning policy LLMs wo without separately learned value function. Its key idea is to normalize rewards within a group responses sampled from the same prompt, thereby stabilizing optimization.
Setup. Given query the old policy Tew generates G candidate responses (o..... J. Each response o assigned a scalar reward R(q, o;). Group- wise z-score normalization then applied to obtain an advantage shared across the tokens of the response:
R(q. o.) mean (R(q. o1). Ríq. oG)L) Á
std( (R(q, R(q, (G))) + Enorm
where noor is small constant for numerical stability. The same normalized advantage Åq, is applied to all tokens of response o,.
me(ou. lq.o.cc.. Policy Update. Let ro(o),) To 19. [q)) The policy is updated with clipped surrogate objective, similar to PPO, combined with a KL-divergence penalty to constrain policy drift:
(oRro()) E 1 I min(rolood) (ooooo))A... A clip(rolo.)) 1- + A.a) KI(==(q) || (q)
Maximizing the negative of this loss encourages the policy to increase the probability tokens contributing to responses with positive relative advantages, while clipping prevents overly aggressive updates. The KL penalty, weighted by 8, further stabilizes training by preventing the new policy from drifting too far from the old one.
20

---

Published as a conference paper at ICLR 2026
1.2 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS
Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024) is a paradigm for fine-tuning models in domains where response quality can be deterministically verified. RLVR relies on a rule-based verifier v. X + {0, } that assigns a binary reward to each response Ty.-
1, if 2; satisfies task-specific correctness check,
Ti (2z)
0, otherwise.
This reward structure is especially effective for tasks like math, code generation with clear correct- ness criteria, and serves as the foundation for the reward mechanism in Solver training.
21


Reviews:
[{"review_id": "eOstHQ1aiS", "reviewer": "Reviewer_Bdrg", "summary": "This paper proposes R-Zero, a fully autonomous self-evolving reasoning framework that requires no human-annotated data. The framework uses a Challenger model to generate challenging tasks that lie near the capability boundary of a Solver model, which in turn learns to solve them. The system iteratively co-evolves without any external supervision. Experiments show consistent improvements in both mathematical reasoning (e.g., GSM8K, MATH, AIME) and general reasoning benchmarks (e.g., MMLU-Pro, SuperGPQA, BBEH).", "questions": "The authors have already provided extensive and detailed ablation studies, so I do not have additional questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper reports consistent performance improvements across both mathematical and general reasoning benchmarks.\n\nIt conducts rich and detailed ablation studies, revealing several interesting phenomena.\n\n1. Models fine-tuned after R-Zero pretraining perform better than those fine-tuned directly.\n\n2. Both task filtering and repetition penalty are shown to be essential components.\n\n3. The paper identifies a model collapse phenomenon after multiple self-evolution iterations, with larger models showing better resistance.\n\n4. The accuracy of self-generated pseudo-labels gradually degrades (from 79% to 63%), highlighting an important data quality issue.\n\n5. Separating the Challenger and Solver is shown to be necessary to avoid overfitting and instability.", "weaknesses": "Some of the reported improvements in Table 1 appear to be statistically insignificant, weakening the empirical strength of the main claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YK5Jaase4f", "reviewer": "Reviewer_kwkW", "summary": "This work proposes R-Zero, a co-evolving framework that generates new training data from scratch. R-Zero consists of two independent models, the Challenger and the Solver, which work cooperatively and are trained separately within a co-evolving pipeline. The Challenger is trained based on an uncertainty reward from the Solver model and a repetition penalty given the diversity of its generated training batch. The solver is trained with the RLVR objectives. It conducts experiments on Qwen-3 and OctoThinker, where R-Zero shows noticeable improvements over the base model without training and a fixed challenger variant in the math domain and a set of general-domain reasoning benchmarks, including SuperGPQA, MMLU-Pro, and BBEH. This work finally proposes an analysis in terms of the effects of the repetition penalty and the filtering, and eventually shows that R-Zero serves as a good mid-training strategy for enhancing model’s post-finetuning on labelled datasets.", "questions": "1. Could you please clarify and compare more precisely your approach with the Absolute Zero method?\n\n2. In the R-Zero framework, you propose independent roles of the challenger and the solver. What if we train the same model with both roles?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Generating novel data from scratch is a valuable research direction, where the co-evolving framework that includes a dual-agent setup is novel and insightful.\n\n2. The experiments conducted in this work are extensive, and the empirical performance improvement appears to be large.\n\n3. The ablation study is thorough and provides fruitful findings for future research in this direction.", "weaknesses": "1. The baseline studied in this work is relatively weak. There are other data generation approaches, such as Absolute Zero [1], which have been discussed but not directly compared empirically.\n\n2. Meanwhile, as the author also mentions, the RLVR methods with zero-shot training objectives, such as maximizing the model confidence and entropy, also need to be compared against the data generation approaches, given that they are all a form of zero-shot approaches. \n\n[1] Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Yue, Y., Lin, M., Wang, S., Wu, Q., Zheng, Z., & Huang, G. (2025). Absolute Zero: Reinforced Self-play Reasoning with Zero Data (No. arXiv:2505.03335). arXiv. https://doi.org/10.48550/arXiv.2505.03335", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RbHalUb3zp", "reviewer": "Reviewer_Qxwk", "summary": "The paper introduces R-Zero, a fully autonomous framework designed to enhance the reasoning capabilities of Large Language Models (LLMs) without relying on any human-curated data for alignment. The core problem it addresses is the significant bottleneck and cost associated with creating large, high-quality datasets for fine-tuning and reinforcement learning.   \n\nR-Zero's methodology is based on a co-evolutionary loop between two distinct roles, both initialized from the same base LLM :   \n\nThe Challenger: This model's objective is to generate new, challenging problems that lie at the very edge of the Solver's current abilities. It is rewarded based on the Solver's uncertainty, which is measured by the consistency of its answers to a given problem.   \n\nThe Solver: This model's goal is to solve the increasingly difficult tasks presented by the Challenger. It is trained on a filtered set of these problems using \"pseudo-labels\" that are generated by a majority vote of its own answers.   \n\nThis self-contained cycle creates a dynamic, adaptive curriculum that progressively increases in difficulty, allowing the model to improve its reasoning skills without external supervision or pre-existing task datasets.", "questions": "Q1: The paper focuses on a self-evolving framework where the Challenger and Solver are initialized from the same base model. Have you considered an asymmetric or heterogeneous setup—for instance, using a more capable model as the Challenger and a less capable one as the Solver? It would be interesting to know if such a configuration could potentially delay the onset of performance collapse.\n\nQ2: Regarding the training hyperparameters, the number of rollouts for the GRPO algorithm is relatively small. Could you elaborate on the motivation behind this choice? I'm curious if a larger number of rollouts was explored and whether the current setting risks insufficient exploration, which could potentially lead to a less accurate advantage signal.\n\nQ3: The analysis of synergy with supervised data shows that using R-Zero as a pre-alignment step before Supervised Fine-Tuning (SFT) is beneficial. A compelling alternative would be to integrate the labeled data directly into the R-Zero training loop, perhaps by mixing it into the Solver's training dataset at each iteration. Have you experimented with this concurrent training approach, and do you have any insights on whether it might yield superior results compared to the sequential (SFT than R-Zero) method?\n\nQ4: This is a minor suggestion for presentation clarity: in Figure 3, the x-axis is labeled with training steps (e.g., \"Step 15,\" \"Step 30\"). For better consistency with the narrative, it might be more intuitive to label it with the corresponding iteration numbers (e.g., \"Iteration 1,\" \"Iteration 2,\" etc.).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "R-Zero's primary strength is its ability to create a self-improving loop for reasoning tasks without human-labeled alignment data. It successfully adapts the self-play paradigm to a domain that lacks a perfect external verifier (like a code executor or game engine), cleverly using a majority-vote mechanism to create a noisy but effective \"pseudo-ground truth\".\n\nThe framework demonstrates significant and consistent performance gains across different model architectures (Qwen3, OctoThinker) and scales. For instance, the Qwen3-8B-Base model achieved a +5.51 point increase on average across math reasoning benchmarks after three iterations.   \n\nThe paper validates the importance of its key design choices. Ablation studies show that removing components like the repetition penalty (to ensure question diversity) or the task filtering mechanism (to control curriculum difficulty and quality) leads to a significant drop in performance.", "weaknesses": "The most significant limitation is that the self-improvement process is not indefinitely stable. After a few iterations, all tested models experience a \"performance collapse,\" where their scores on benchmarks begin to decline. Larger models are more resilient and collapse later, but the eventual degradation appears inherent to the current framework.\n\nThe performance collapse is directly linked to a decline in the quality of the training data. As the Challenger generates progressively harder problems, the Solver's ability to form a reliable consensus via majority vote diminishes. The paper shows the true accuracy of these pseudo-labels systematically drops with each iteration (e.g., from 79.0% to 63.0%), introducing increasing noise into the training signal.\n\nThe framework's effectiveness has only been demonstrated in the domain of mathematics. This is a carefully chosen domain where answers are objective and can be verified through simple string matching, making the majority-vote mechanism viable. The method's applicability to more subjective, open-ended domains like creative writing, dialogue, or nuanced analysis remains a major, unaddressed challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fknvRYy6ns", "reviewer": "Reviewer_XCxR", "summary": "This paper presents R-Zero, a framework for training reasoning LLMs through co-evolution of two models (Challenger and Solver) without requiring any human-curated data or labels. The approach uses reinforcement learning to create a self-improving curriculum where the Challenger generates increasingly difficult questions and the Solver learns to solve them. While the motivation is strong, there are numerous weaknesses that should be resolved before publication. While there is solid base model coverage (4 in total), the train domain is only 1 (math) and the synergy and use-case when labeled data is available is limited. Also, there is no existing approach baseline other than the base model performance, therefore, the authors should provide a strong rebuttal.", "questions": "1. Are these methods in the baseline, if applicable? one of my main concern is the lack of baselines, hopefully the authors can clearly rebute this. \n\n> self-challenging approaches train LLMs on tasks generated by the models themselves (Zhou\net al., 2025a; Wang et al., 2025a; Zhao et al., 2025a).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Paper is well motivated\n\n2. Presentation is clear\n\n3. Mid-training is an important direction for practitioners, therefore, this part is appreciated\n\n> Our further analysis finds that R-Zero can act as a mid-training method, as models first improved by our method achieve higher performance after fine-tuned on labeled data. \n\n4. Analysis section is appreciated", "weaknesses": "1. It is unclear if sufficient comparisons have been made with existing approaches, especially as there seems to be no baseline other than the base model. There should be significant label-free works that could serve as reasonable baselines. One that immediately comes to mind is Self-Consistency Preference Optimization.\n\n2. a pseudocode to clearly follow this method step by step would be appreciated especially since it is a GAN-style back-and-forth going on. I believe that an intuitive view of how this algorithm works will be helpful. If there is no space it should at least be in the appendix. However, i think there is enough space if sec. 2 is removed. It is unclear why GRPO and RLVR needs to be introduced at this lvl of detail.\n\n3. It is concerning that the training data is only on math domain. While i understand that various benchmarks have been used on the testing side, on the training side there should be more training domains to see generality of this approach. This is especially important since a lot of math benchmarks that you are seeing gains here already has labeled data, therefore, the experimental setup is less convincing from a practical view.\n\n> We focus on generating questions specifically within the domain of mathematics, as it provides a convenient and self-contained setting for our framework; the objective nature of mathematical answers allows for the straightforward generation of pseudo-labels via majority voting, without the need for external verification environments like code executors. \n\n4. There seems to be a lot of newly introduced hyperparameters. This does seem like a weakness for practical applications. Are these set heuristically? Arbitrarily? Are there any sensitivity tests for this?\n\n> The Challenger (Qθ) first generates a candidate pool of N = 8, 000 questions. To construct the training dataset for the Solver, these questions are filtered based on consistency. For each candidate question, we sample m = 10 answers from the current Solver (Sϕ). A question is retained for the training set only if the number of answers matching the majority-vote pseudo-label is between 3 and 7, inclusive (δ = 0.25). This numerical range is consistent with the methodology used in previous research (Zhang & Zuo, 2025; Li et al., 2025b; Bercovich et al., 2025). When training the Challenger, the uncertainty reward r(x; ϕ) is calculated by sampling m = 10 responses from the Solver. For the intra-batch repetition penalty, we set the clustering distance threshold to τBLEU = 0.5\n\n5. This method seems to make most sense for benchmarks that do not have a corresponding train set. In a practical perspective, there is no need to not use the train set of e.g. GSM8K. Either there should be meaningful empirical evidence on benchmarks with no training data, i.e., no labels. Or if you want to show that this approach is orthogonal, you should include regular SFT and GRPO as a baseline for benchmarks that have labeled data. Another approach would be to do SFT/GRPO on the existing train data + use your proposed approach to show that it adds gains. Without this, practical impact is not evidently clear. While i see that Sec. 5.3 exists, this kind of practical use when labeled data is available should have been available in the main experiments, as this is very important for practical use.\n\n6. Contents of Sec. 5.3 should be expanded even at the cost of removing e.g. sec. 2.\n\n7. There should be a clear analysis on the cost structure of this approach. I am guessing that the compute overhead over regular SFT is significantly larger as many samples needs to sampled and there are additional compute overhead for all these metrics, clustering, etc. the method itself is quite complex. \n\n8. The top right part of Fig. 1 is not intuitive. My suggestion is using tick and X mark or something similar instead of color shades. Some readers might read the paper in black-and-white\n\n9. It would be nice to have labels for the light blue and dark blue for the grouped bar chart in Fig. 1\n\n10. It would be nice if there was a clear limitation section for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)