Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR
Authors: Xiao Liang
Zhong-Zhi Li
Yeyun Gong
yelong shen
Ying Nian Wu
Zhijiang Guo
Weizhu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning; Reinforcement Learning; Self-envolving
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose an online Self-play with Variational Problem Synthesis strategy for RLVR training that iteratively leverages model responses to synthesize variational problems for augmentation.

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, vanilla RLVR training has been shown to improve Pass@1 performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass@k performance, which typically represents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's generation diversity from the perspective of training problems and find that augmenting and updating training problems helps mitigate entropy collapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively maintains policy entropy during training and substantially improves Pass@k compared with standard RLVR, sustaining prolonged improvements and achieving absolute gains of 18.3% and 22.8% in Pass@32 performance on the competition-level AIME24 and AIME25 benchmarks. Experiments on 12 reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS.

Published as a conference paper at ICLR 2026
BEYOND PASS@I: SELF-PLAY WITH VARIATIONAL PROBLEM SYNTHESIS SUSTAINS RLVR
Xiao Liang Zhongzhi Li Yeyun Gong?, Yelong Shen?, Ying Nian Wu',
Zhijiang Guo Weizhu Chen
I Unniversity of California, Los Angeles "Microsoft
SShooo of Artificial Intelligence. Chinese Academy of Sciences
'Hong Kong University of Science and Technology
6 HHong Kong University of Science and Technology (Guangzhou)
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a key paradigm for post-training Large Language Models (LLMs), particularly for complex reasoning tasks. However, standard RLVR training has been shown improve Pass@) / performance at the expense of policy entropy, leading to reduced generation diversity and limiting the Pass @k performance, which typically repre- sents the upper bound of LLM reasoning capability. In this paper, we systematically analyze the policy's 's generation diversity from the perspective of training data and find that augmenting and updating training problems helps mitigate entropy col- lapse during training. Based on these observations, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, which uses the policy's s correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This self-improving strategy effectively preserves policy entropy during training and substantially im- proves Pass compared with standard RLVR, sustaining long long-term term improvements and achieving absolute gains of 18.3% % and 22.8 % in Pass @ 32 performance on the competition-Ievel AIME 24 and AIME 25 benchmarks. Experiments reasoning benchmarks across varying model sizes from 3B to 32B consistently demonstrate the generalizability and robustness of SvS. The code is available http://guuub..com/MMattteiiiio/e
12
AIME 24 & 25 Pass@32
AIME 24 & 25 P Pass@1
0.7
RLVR
SvS
SvS
AIME ACC
AIME ACC
0.30
Xxc
0.4
0.20
0.15
0.3
0.10
200
300
500
600
100
200
600
700
Steps
Steps
Figure 1: We train Qwen2.5-322 Innttntut on the DAPO-178 17k dataset using our SvS strategy and standard RLVR. SvS achieves superior efficiency and effectiveness on competition-leve AIME benchmarks, showing significant improvements in Pass @ 32 and Pass@) (average 32 times) scores.
* "quual contribution. Work done during Xiao's and Zhongzhi's internships at Microsoft. !Corresponding authors: Yeyun Gong, Zhijiang Guo, and Weizhu Chen. M yegong@mi crosoft. zhi j angggggggggttt -gz. edu. en; wzchen @mi crasoft.ccmm

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
The reasoning capabilities of Large Language Models (LLMs) have been significantly enhanced by Reinforcement Learning with Verifiable Rewards (RLVR: Guo al. 2025a). However, recent studies (uue et al. 2025; Cui et al. 2025b) have shown that standard RLVR training, such as GRPO (Shao al. 2024) optimization, may diminish the generation diversity of the policy model, enhancing sampling effciincy and Pass@1 / performance at the expense of output richness, thereby failing to improve Puss @ over the base model. RLVR, training entropy used to quantify the
diversity of model outputs (Cui et al. [2025]] Zhu et al.] 2025 Cheng et al.) (025), while improvements in Pass@ k indicate more advanced exploration. Together, these metries reflect the model's ' potential to continue improving in RLVR training. When training entropy collapses to zero, the policy tends to produce homogeneous solutions to training problems, thus losing the opportunity to explore more advanced reasoning trajectories and causing Pass @k performance to plateau. Ultimately, the Pass @ / score also plateaus due to the lack of further exploration opportunities. Therefore, maintaining training entropy and ensuring Pass@k k improvement are both critical factors for sustainable RLVR training. Recent works such as (Cui et al. 2025b Li et al. 2025b) also show that regulating policy entropy can enhance rollout diversity, thereby improving RLVR training.
The primary cause of entropy collapse and plateaued @ass@k k is RLVR training on limited problems, where the policy is easily rewarded for repeatedly generating memorized correct solutions-a behavior akin to "hacking" the RLVR training. Intuitively, maintaining policy entropy and generation diversity requires using a broad and diverse range of problems, or entirely new problems each training step. However, collecting large problem sets with verifiable answers for RLVR is non-trivial. High-quality, human-annotated problem sets are scarce and may not align with the strong reasoning capabilities of modern LLMs (Cobbe et al. 2021 Hendrycks et al. 202I). While synthetic data is common alternative (Yu et al. [2023] Huang et al. 2024 Liang et al. 2025), a critical limitation is the absence of precise reference answers, which are difficult to derive. These challenges naturally raise the question: Can we develop a simple yet effective problem augmentation strategy that maintains sustainable data diversity, aligns with the model's 's capabilities, and ensures accurate labeled answers?
To answer this question, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, where the policy model is prompted to generate variational problems based on its correct solutions to challenging and underperforming training-set problems. The rationale for augmenting only the challenging problems is to efficiently target the policy's weakest capabilities (Liang et al. 2025). Since the correct solutions must capture all essential information from the original problems, the policy is naturally encouraged to produce variational problems with rephrased descriptions and structures while preserving the original semantics. Most importantly, the variational problems should share the same reference answers as the original ones, ensuring precision and eliminating the need for additional labeling computation. After synthesis, the policy model is prompted to solve its self-generated variational problems, and the consistency between its produced answers and the reference answers of the corresponding original problems serves to validate the correctness of the variational problems. Finally, the solutions to original problems, the self-generated variational problems, and the solutions variational problems are gathered for policy updating, enabling it to jointly learn both problem solving and problem synthesis. Notably, the SvS framework relies exclusively on the policy model itself, without any external guidance or distillation, achieving all improvements through end-to-end self-improvement. Moreover, the SVS augmentation is agnostic to RLVR optimization algorithms and can be flexibly incorporated into other methods, such as PPO (Schulman et al., 2017), GSPO (Zheng et al. 2025) and Reinforce++ (Hu et al. 2025a).
To validate the effectiveness and generalizability of SvS, we conduct experiments on LLMs ranging from 3B to 32B and evaluate their performance across 12 widely used reasoning benchmarks. The results show that SvS consistently outperforms standard RLVR across all model sizes and benchmark levels, achieving an average absolute improvement of approximately 3% over the baseline in all experiments. Thanks to the online data updating strategy, SVS training consistently maintains policy entropy within a stable range without noticeable decline or explosion. indicating more sustainable training and prolonged self- inffimmroveeent. Most importantly, SvS achieves substantial gains of 18.3% % and 22.8% in Pass @ 32 on AIME 24 and AIME 25 (MAA b). where the standard RLVR shows little improvement. Experiments in Section 5.2 and results in Table provide a detailed demonstration that SVS achieves scalable Pass @k improvements across four authoritative benchmarks, highlighting that
2

---

Published as a conference paper at ICLR 2026
Train/Test Entropy Comparison
Test Pass 32 on AIME24, 25 and AMC
0.21
Bose Test
0.18
Base Train
Pollyy Entropy
0.14
Tall
0.11
Bave Passdl 32 Aug Passigg 32
0.04
100
Training Steps
Training Steps
Figure 2: Policy entropy and Pass @k during RLVR training under different data strategies. The dashed line indicates policy entropy on evaluated competition- benchmarks in the right figure. The augmented problems in the Aug experiment are updated at the 300th step. All curves smoothed with a window size of 5, and the original data points are marked with faint dots.
our framework can significantly extend the model's 's reasoning boundaries (uue al. 2025 We also provide a comprehensive study of SvS from multiple dimensions in Section 5.3 and Appendix
Our contributions can summarized as:
(i) We propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, where the policy's 's correct solutions for underperforming training samples are used synthe- size variational problems without additional answer labeling, enabling self-improvement without any external guidance distillation. (ii) The variational problems synthesizing in SvS supports online data augmentation, thereby maintaining stable policy entropy and output diversity during training and improving overall performance, particularly in Puss@k k on competition- Extensive experiments across models of varying sizes, together with evaluations on wide range of benchmarks and additional analyses, demonstrate the generalizability of our proposed SvS.
benchmarks. (iii)
RETHINKING THE
TRADE-OFF IN RLVR
Recent study (Cui al. 2025b) demonstrates a trade-off between policy entropy and model perfor- mance, where gains in test accuracy come at the expense of response diversity. Specifically, when using a fixed RL training set without entropy intervention. the policy's 's performance improves over time while its entropy steadily degrades, with the two variables exhibiting a logarithmic relationship: Performance
exp( Entropy b). Meanwhile, Yue et al. (2025) shows that RLVR training improves Pass @k on evaluation benchmarks only when I is small, with no further gains when scales to tens or thousands. This suggests that standard RLVR training narrows the reasoning trajectory toward most reward-prone solutions, reducing exploration capacity without fostering more general or advanced reasoning beyond that of base models.
When the policy is iteratively trained limited problem set, it tends to memorize specific correct solutions and repeatedly produce similar correct trajectories to obtain positive rewards, leaving less and less room for improvement as training progresses. Intuitively, increasing training data diversity and incorporating online updates can help mitigate policy entropy collapse during training. each iteration involves different problems, the policy is foreed continually explore optimal solutions to new challenges rather than repeating high-reward solutions from previously seen problems, which promotes continuous exploration of advanced reasoning strategies and enables sustainable learning.
To explore how data diversity affects policy entropy and performance, we conducted experiments using RLVR train the same policy model with different data strategies. We demonstrate the policy entropy and Pass @ k scores during training in Figure The blue line shows results on the MATH- 12k (Hendrycks al., 2021) dataset throughout training, while the orange line begins with a mixture of MATH-12k and 36k rephrased problems from MetaMath (Yu et al., (023); ; at the 300th step, the rephrased problems are updated with similar ones. Notably, augmented training sets consistently slow the decline of policy entropy for both training and test problems. Furthermore, when the training data is updated at the 300th step, policy entropy stops decreasing and begins to rise, indicating that the policy is re-exploring new reasoning patterns and thereby sustaining learning. Concurrently. evaluation results illustrate that training with an augmented and periodically updated problem set consistently improves Pass@332 performance, particularly the update steps.
3

---

Published as a conference paper at ICLR 2026
Takeaways for Problem Diversity in RLVR
Impact of Problem Diversity on Entropy (Figure 2, left): Adding augmented problems with diverse formulations, even when the knowledge and domains are close to the originals, can effectively counteracts the entropy drop during RLVR training.
Impact of Problem Diversity on Pass@k (Figure 2, right): Diverse problems significantly improve Pass k during RLVR training compared to vanilla problems.
Although effective, rephrasing- based augmentation has notable limitations. Rephrased problems generated by external LLMs may introduce semantic inconsistencies, thereby compromising the accuracy of reference answer annotations and undermining the training stability. Moreover, since rephrasings often use the original problem as context, their diversity cannot be guaranteed. Based on our preliminary experiments, the limitations of rephrasing- based augmentation, and recent stud- ies (Wen et al., 2025 Chen et al. 2025 Liang et al. 2025) advocating the selection of problems appropriate to the model's 's capabilities, we conclude that ideal data augmenlation for RLVR should be iterative, provide precise reference answers, and be aligned with the policy' 's capabilities.
To this end, we propose the Self-play with Variational problem Synthesis (SvS) strategy for RLVR training. which features targeted online problem augmentation and pure seff-improvement paradigm. This strategy augments training problems using the policy's ' correct solutions to underperforming problems, ensuring that the golden answers of synthetic problems precisely match the originals. Sections 3 and present the framework, experiments, and detailed analysis, respectively.
3 METHOD
3.1
OVERVIEW FOR SVS
To achieve the ideal data augmentation for RLVR as discussed Section 2 we propose the SVS frame- work, which uses the policy itself to online augment training problems through self-play, leading to self-improvemment The policy synthesizes variational problems from its correct solutions to un- derperforming training set problems and then attempts to solve these synthetic problems. Ideally, these variational problems preserve the semantics and, crucially, the reference answers of the original ones, while their structures and descriptions may differ significantly, thereby eliciting novel diverse reasoning strategies from the policy.
Specifically, as shown in Figure and Algorithm T the full online augmented training batch each step / comprises three components: (1) Original Problem Solving: The policy generates solutions to training set problems. with the underperforming ones retained for augmentation. (2) Variational Problem Synthesis: The correct responses containing full information of the underperforming problems are used as context to synthesize variation problems for online training data augmentation. (3) Synthetic Problem Solving: The policy is prompted to solve the self-synthesized variational problems, which share the same reference answers as the original ones. Following strategic filtering and reward shaping, the three types of training data are mixed for policy updating.
3.2 SELF-PLAY WITH VARIATIONAT. PROBLEM SYNTHESIS
Fach experience collection step in SVS training alternates between problem solving and problem synthesis, enriching the training data buffer B online throughout the RLVR iterations. Without any external guidance or distillation, the policy independently generates and solves its synthetic variational problems in a self-improving paradigm.
Original Problem Solving. At the beginning of each RLVR iteration, the policy we is prompted to solve problems sampled from the original training set D. For each sampled problem-anse answer pair (a, a) in D. the policy no generates group of G solutions (y, 1 The correctness reward R. for each response y; is determined by its consistency with the ground truth answer a:
R (e a) T(Extract(y:) a)
(1)
4

---

Published as a conference paper at ICLR 2026
Batch Data Workflow with Variational Problem Synthesis
Filtering Data for Policy Updating
Batch Somples
[1 v.) (o(3)) 1}
TT(B.) مسر
Problom Solving
For Problen Synthesis
Variational Problem Synthesis
{(a, su)
{ Be)t
(z,a) E
Z4 E X acty, 2oro
( To
Acste) 00
Art(g) smy
Synthetic Problen Solving
1 (R(m))
(Ho))2
{yr mo(=)
Policy Update
R.
(Re(
Exp Buffer
Qriginal Problem Solving
B
Figure 3: The data workflow of our SvS in a training iteration, comprising original problem solving, variational problem synthesis, synthetic problem solving, and policy update data filtering.
where I(-) is the indicator function, and Extract(-) extracts the final answer from the reasoning trajectories. Since the advantage for groups with all-correct or all-incorrect solutions degrades to zero in GRPO, filter out problems with group accuracy equal to 1 or O. The remaining problems with solution groups (xx 3)){))))) and their corresponding rewards are added to the training buffer B.
Variational Problem Synthesis from Responses. After generating solutions to the original prob- lems, SvS identifies underperforming problems with low solve rates and synthesizes their variants to online augment the training set. Specifically, underperforming problems are defined as those with group average accuracy Acc(x) falling within the range [aco,, acc,] (Line 11 in Algorithm thereby excluding problems that are either too easy or unsolvable. This filtering strategy focuses the augmentation effort on problems that match the current model's 's frontier capabilities.
After identifying underperforming problems, SvS leverages the policy's correct solutions synthe- size corresponding variational problems for augmentation. Since a correct response contains the full informational content of the original problem æ, each solution y/4 serves as context to generate a group of G, variational problems, { enriching the originals with more diverse structures and descriptions. The detailed prompt is present in Figure [8 Because the variational problems are derived from correct responses to the original problems, they are expected to share the same reference answers. This constraint not only serves variational problems, but also bypasses the need for additional answer annotations, which is crucial for RLVR data augmentation, where the reference answers provide the only training signal. Except for problem-solving augmentation, the correctness of generated variational problems is also incorporated into RLVR training, encouraging the policy learn the inverse mapping from solution to its problem statement and fostering a deeper understanding of the problems' semantics and structure. In we provide further analysis of how problem synthesis training helps problem solving.
criterion for validating the correctness of the
Appendix F2
C.. generated from ys, the Synthetic Problem Solving. Once a set of variational problems { 1
policy is tasked with solving them in the same way as solving the original training problems. For each variation problem 2/. the policy produces group of G' solutions (0x G k= and the original ground-trutt answer paired with reused evaluate their correctness. The corresponding correctness reward R, is computed as:
Relie a) = I(Extract(jk) = a)
(2)
Similar the original problem solving filtering for experience buffering, we retain only vari- ational problems for which the policy produces mix of correct and incorrect solutions, i.e.,
0 Ex= Re( a) G, as they provide effective training signals in Eq. of GRPO.
Reward Shaping for Problem Synthesis. Ideally, the correctness reward for variational problem synthesis, R,, is determined by whether the reference answer matches the original answer. Since precise reference answers for synthetic problems are unavailable and they must align with the policy's capabilities, an intuitive implementation, problem ae is considered correct if the policy can produce solutions whose extracted answers match
adopt proxy criterion for validating them: synthetic
5

---

Published as a conference paper at ICLR 2026
Original Challenging Problem:
Synthetic Variational Problems
Lei fe
2+++ 101 27 fine e tet & be fele) 34 P-
pobyanmcal
by Juld)
186
not prsticc
coaaing deccslaad anagenyy past to actrod inages GI Anower 52k
iruaiaay pat cimeeti of the cianana A.
noundd inagu fire, and then compue the thee noad
postior tmugiarr
raugiaay parts
Ace (s n8, o0 on..nnnn no. n2] GT Amnerr 1 Uhi 2.0 X. (Fiiled synthesio)
Lor cuaniee the furetion defiaddly by Fulel
34
500
23'
Sor are M Taakuse al noos
X
Juda) - Desormioo the offl flose nots whese inag part poilin, expresed in fhess and saze finll zurver bothoaa and scacel Prevest aoweris the dectppepria rounted Bexz.
Dove all nateral sanderrs
noadeed the
Ace (PA) 1.0
1.
Amorrr 4 -.
Correct Solution:
innoiigaiin fu(-) 37 b
gudliii polyncsinl fn(z)
Lonalr damily fanciion fald)
whav o laioal fixd. ing saz modts postivve imsaginary part dait baken tacrti fxld) your ancoer after ronnding
whine
satual Equecs imagisry pare nomvaa traggex presentng the catable frmm Toe of ontor of couplec aunder exn Aoroo sauning red and hagglary paw repanas)).
langinay pf Sin 88, 1022
Ace 1.0.
10.16, 10] Arower 2-+ 231 Re -
overiippe
Hoxe th forrs encepelkae
Original Problem Solving
Variational Problem Synthesis & Synihetic Problem Solving
Figure 4: Illustrations of a challenging problem, its correct solution from policy, the synthetic variational problems from the solution, and the reward-shaping strategy for the synthetic problems.
the original answer a, formulated as
(2y) = -ACC(((C > =0)
While straightforward, we find this reward can be easily exploited by the policy, which may embed excessive hints or even directly include the correct answer in the synthetic problems. Since they are generated given the correct responses, they can become trivial to solve, allowing the policy to obtain the reward in Eq. 3 Consequently, such variational problems are over-simplified and fail to encourage advanced reasoning of the policy, making the pipeline unsustainable and convergence suboptimal.
To ensure that variational problems remain diverse and effectively elicit stronger reasoning of the policy, we introduce reward-shaping constraint validate them, requiring that they maintain an appropriate level of difficulty for the policy. Specifically, we assign positive rewards to a synthetic problem only if the policy achieves moderate level of group accuracy-neither too high nor entirely incorrect- than simply rewarding it for which a correct answer is sampled. The reward for each variational problem & is defined as:
(2/) (acoI & Acc( a) < aach))
Notably, shown in Figure if synthetic variational problem can be fully addressed solution aligning with a can be sampled, it receives a negative reward. This discourages the policy from generating overly hint-laden, unverifiable, unsolvable problems, ensuring that synthetic problems remain challenging while providing effective learning signals.
Full Training Data. After experience collection, for each training step, the final training buffer B contains three distinct types of prompt-response-rwwar tuples; (1) Original Problem Solving:
(I, ye, Re a)); (2) Variational Problem Synthesis: (y,, ... Ry (3) Synthetic Problem Solving: (+/, Uc. RoK a)). Utilizing the augmented buffer B, the SvS framework updates the policy we according to the GRPO gradient update objective in Eq. 6 By jointly training on the problem solving and synthesis tasks, the policy learns to solve the given training problems, generate challenging problems for itself, and solve the self-generated problems, forming a powerful self-improving loop.
4 EXPERIMENTS
4.1 SETTINGS
Models and Datasets. We employ models of various sizes (3B to 32B) for validating the effectiveness of our proposed SvS, including Qwen2.5-3B Instruct LLaMA-3.1-88 115.1-88-Insstrrtt (Grattafiori et al. 2024), and Qwen2.5-32B 3-2nstuutt Yang et al.) 2024). All models are trained the MATH- 12k dataset (Hendrycks et al. 2021), with the 32B model additionally trained on the DAPO- 17k dataset to enhance competition-level reasoning capabilities.

---

Published as a conference paper at ICLR 2026
Pass1
Pas
Model
AIME24 AIMEZ5 BAIME
Math2lo OlymE Olymit Aug.
AIME25 BAIME Math2oo OlymE Olymii Avg.
Open-Source Models
Qwen2.5-32B
2.4
1.6
15.6
18.7
34.0
24.6
15.2
24.5
Owen2.5-32B-TT
10.0
13.0
7.4
26.0
8.6
2.0
11.2
40.2
34.6
24.0
67.8
35.2
9.5
35.2
32mpleRL-32B
22.1
13.9
8.3
25.5
9.4
3.7
13.8
62.0
38.5
27.4
69.9
42.5
19.4
43.3
ORZ-32B
24.2
26.3
10.9
16.1
12.2
1.1
15.1
55.7
47.0
29.4
58.0
45.9
12.3
41.4
MATH-12K
RLVR
11.5
29.2
66.0
16.4
38.6
SVS
30.3
21.7
13.8
42.7
20.1
3.3
22.0
63.6
55.1
41.5
79.2
63.6
24.8
54.6
+5.9
+2.3
+16.2
+18.7
+12.3
413.2
+8.4
+16.0
DAPO-17k
RLVR
28.8
30.0
14.0
39.6
17.9
4.8
22.5
52.5
42.4
35.9
71.2
47.1
18.3
44.6
SvS
39.3
40.5
19.2
44.1
21.8
2.7
27.9
70.8
65.2
45.9
76.5
43.4
16.7
$3.1
A
+10.5
+10.5
+5.2
+.5
+3.9
-2.1
+5.4
+18.3
+22.8
+10.0
45.3
-3.7
-1.6
+8.5
Table 1: Comparison of model performance on challenging benchmarks using the Pass @ (average 32 times) and Pass@ 32 metrics. The A row shows the improvement SvS over standard RLVR. The BAIME, Math24o, OlymE, and OlymH benchmarks correspond to BeeyondAMME Math24o and the en-easy and en-hard subsets of OlymMATH respectively.
Evaluation. We evaluated the models on wide range of mathematical reasoning benchmarks, includ-
ing GSM8K (Cobbe et al. 202I). MATH-500 (Lightman et al. 2023). Minerva Math (Lewkowyzz et al. 2022). Olympiad-Benn BennchhhBBhhhhe (He et al. 2024), Gaokao 2023 (Zhang et al. 2023), AMC (MAA a AIME (MAA, b) and Beyond-AIMB (Byte Dance-Seed 2025). To more comprehensively evaluate the models' advanced reasoning capabilities, we also evaluated their Pass @K k and Pass@) / (average 32 times) performance on additional challenging benchmarks, including OlymMATH (Sun et al.) 2025) and Math-24o (CLUEbenchmark] (2024.. Details of the training, evaluation implementation, and baseline settings are provided in Appendix
4.2 MAIN RESULTS
SvS significantly improves both Pass@l and Pass@k. As shown in Figure naive RLVR training plateaus at @ass@ 32 and Pass@1 I on competition-level AIME benchmarks after roughly 450 steps. In contrast, the model trained with the SvS strategy achieves substantial and sustained improvements in both metrics on these challenging benchmarks. ssooss that models trained on the DAPO dataset with the SvS strategy achieve absolute gains of 18.3 and 22.8 points on Pass 32 for AIME 24 and AIME 25, respectively, compared to the standard RLVR baseline. These results not only demonstrate the effectiveness of SvS, but also highlight the potential of self-play-style RLVR training to enhance Pass @ k and expand the model's reasoning capabilities. The rising Pass k during training also facilitates greater exploration, which in turn improves @ass@1. 1.
SVS boosts RLVR across all settings. Table 2 presents experimental results for models ranging from 3B to 32B across all evaluated benchmarks using the Pass / metric. To mitigate high randomness, evaluate models smaller than 8B AIME-level benchmarks using an average of 32 inferences. Notably, the SvS strategy consistently outperforms standard RLVR across all model sizes, yielding overall improvements of 2.9%, 1.7%, and 2.5% for the 3B, 8B, and 32B models when trained on the MATH-12K dataset. Notably. for O.wen2.5-3BBIIntrrue RLVR training on MATH-12K does not improve performance on the MATH-500 benchmark, whereas SvS yields 3.0-point gain, demonstrating its generalizability. Experiments for the 2-e.nn.5-32B-Insiiiii using both the MATH-12k and DAPÓ-17k training sets. When trained on MATH-124 our model demonstrates improved performance across all benchmarks, with an overall gain of 2.5 absolute points. On the DAPO-17k experiments, SvS significantly enhances performance on AIME 24, AIME 25, and Beyond-AIME with improvements of 20.0, 6.7, and 6.0 points, respectively. Nevertheless, it results in reduced performance on benchmarks with open-ended answers, likely because the model overfits to DAPO-17k's inleger-only format during augmentation. By training the model using SvS on DAPO- 17k with 8k open-ended problems from DeepMath (He et al., (0255), the model restores its performance related benchmarks and achieves the best overall results.
model are conducted
7

---

Published as a conference paper at ICLR 2026
Training Data
MATH
Minerva Math
Olympiad Bench
GaoKao
Beyond AIME
Model
GSM8K
AMC23
AIME24
AIME25
Avg.
Qwen2.5-338 Dwen2.5-3B-Imstrruc
Init Model
87.3
67.8
29.4
30.7
59.0
37.5
4,8
1.7
1.7
35.5
L. RIVR
M12K
86.4
67.4
29.4
30.2
57.7
57.5
6.7
2.7
37.9
SvS
M12k
88.9
70.8
31.2
38.4
61.6
55.0
10.5
2.8
40.8
LLaMA-3, -.-alA-----BB-ttsstuut
Init Model I, SvS
85.6
48.2
24.6
18.3
39.7
22.5
2.5
0.3
0.5
27.0
M12K
57.4
33.8
47.8
1.5
M12K
90.3
62.2
32.4
26.4
54.8
45.0
8,5
2.0
35.9
Qwen2.5- -0t.2-5-2-B--Itree
Init Model
95.4
82.6
43.0
49.2
73.2
13.3
13.3
7.0
49.0
RIVR
M12k
95.8
86.4
45.6
52.7
74.5
77.5
26.7
11.0
54.8
L SvS
M12K
96.1
87.2
46.0
56.7
78.7
80.0
30.0
26.7
14.0
57.3
L SyS
D17k
95.9
75.6
42.3
45.9
62.9
82.5
53.1
43.3
19.0
57.9
SyS.
D25k
95.2
88.6
47.8
59.9
79.2
87.5
50.0
40.0
17.0
62.8
Table 2: Performance comparison between the standard RLVR and our SvS strategy on mainstream reasoning benchmarks, using different training sets and models of varying scales and families. The datasets M12k, D17k, and D25k correspond to MATH-12K DAPO-17K, and DAPO-17k augmented with 8k problems with open-ended answers from Maep respectively.
LLaMA-8B MATH--12K
Qwen-328 on MATH-11K
Qwen-320 DAPO-17k
1.5
Baseline
Easeline
0.8
Baseline
1.2
Pollcy Entropy
24
Policy Entropy
0.6
0.9-
Policy
0.16
0.3
0.08
0.0
0.0
200
50
150
280
300
Training Steps
Training Steps
Training Steps
Figure 5: Policy entropy trajectories during training for standard RLVR and the SvS strategy across various models and datasets. Lines are smoothed with window of 5 steps.
5 ANALYSIS
5.1 SvS STABLY MAINTAINS POLICY ENTROPY IN TRAINING
In RLVR training, policy entropy reflects the model's capacity for sustained exploration (Cui et al. 20256 Cheng et al. 2025). Standard RLVR algorithms typically result in a steady decline in entropy. enhancing policy sampling efficiency and Pass performance but reducing generation diversity (Cui ct al. 2025b). To evaluate whether the SvS strategy faces the same limitation, we record the entropy trajectories of both SvS and RLVR (GRPO with Clip-Higher throughout the training in Figure Notably, the RLVR baseline shows continuous decline in entropy, whereas SvS maintains entropy within a relatively stable range, supporting sustained exploration and avoiding training collapse. Such advantages stem from the ever-updating problems in SvS that prevent the policy from memorization. The entropy stability explains the continuous improvements in both Pass@ and Pass 32 achieved by SvS, shown in Figure whereas RLVR saturates after certain number of training steps.
5.2
SvS PUSHES THE REASONING BOUNDARY
Recent study Yue et al., 2025) discusses that standard RLVR often fails to expand the reasoning boundary of the base model, yielding improvements in Pass @ only for small values of k. Since our SvS training achieves a substantial improvement in Pass @ 32, we further evaluate its effectiveness and limits in incentivizing reasoning by scaling Pass @ k from 1 to 1024, testing whether the SvS- trained model can solve problems beyond the capability of the base model. As presented in Figure 6 experiments demonstrate that both standard RLVR and SvS improve Pass@k k scores the competition-lest AIME benchmarks across all k. with SvS significantly outperforming the RLVR baseline. For Pass@i @ scaling MATH-500, standard RLVR outperforms the initial model small
8

---

Published as a conference paper at ICLR 2026
Beyond-AIME
103
60
Rote 19M
-2
30
15
SV5
Sv5
SvS
2- 24
62
Number Samples stale!
kumber Samples
Number Samoles
Figure 6: Evaluating the scaled-up @ass@k performance on the AIME 24 & 25, Beyond-AIME, and MATH-500 benchmarks. The maximum response tokens here is set 24k.
Pass
Pusv@l 32
Model
AIME24
AIME25
BAIME
Math24o
OlymE
OlymH
Avg.
AIME24
AIME25
BAIME
Math24o
OlymE
OlymH
Avg.
RLVR
28.8
30.0
14.0
39.6
17.9
4,8
22,5
52.5
42.4
35,9
71.2
47.1
18.3
44.6
Ext
31.7
31.3
15.8
43.2
21.5
24.6
57.0
33.3
72.6
51.8
15.0
46,3
Eup
28.7
26.8
13.9
40.7
16.0
4.2
21.7
64.2
54.6
40.7
76.2
50.2
19.6
50.9
SyS-Asp
31.6
27.3
13.7
43.4
16.5
3.9
22.8
50.7
48.0
30.4
66.4
44.3
17.0
42,8
Full SVS
30.3
""00.""
192
""11
218
2,77
"23""
"10.8"
"6622"""""""""""
45.9
76.5
""334"""""""""""""""""""""""""
"1677"
"33
Table 3:
Comparison between SvS and alternative augmentation strategies and ablation study, including extending RLVR training (Ext), enhancing underperforming problems in RLVR (Fup), and augmenting simpler problems (SvS-Asp). For benchmark abbreviations, see Table !
values but is surpassed at larger k. In contrast, SvS consistently outperforms both RLVR and the initial model as & increases, demonstrating its strong generalization and robust reasoning diversity. We attribute this enhanced diversity to the diversity maintenance of SvS, which supports exploration of more advanced reasoning strategies for solving complex problems throughout training.
5.3
ALTERNATIVE AUGMENTATION STRATEGIES AND ABLATION STUDIES
24 & Pass@1 (Avg@32)
AIME Pass@32
0.40
0.7
0.35
SrS-Aop
Sr5-Asp
0.6
0.30
5pore
0.25
0.5
0.4
0.3
0.05
Figure 7: The intermediate evaluations the AIME 24 and 25 benchmarks of all alternative augmentation strategies in Section 5.3 as well as the Full SvS setting.
In SvS training, we incorporated additional self-synthesized variational problems into policy opti- mization. this section, we investigate whether the improvements in SvS arise merely from scaling up training samples. To this end, we compare SvS with alternative augmentation strategies and perform an ablation on the augmentation of underperforming problems. Specifically, they are:
1. Extending standard RLVR training: prolong the standard RLVR training until the policy is exposed to the same number of problem-solun soluutoon pairs as SvS.
2. Enhancing underperforming problems in RLVR: second rollout stage is assigned to underper- forming problems, ensuring that the training pairs at each step align with the SvS trajectory.
3. Augmentating simpler problems in SvS: we augment simpler problems (accuracy in 37.5%-75.0%% as an alternative to underperforming ones used the original SVS.
All experiments in Table 3 are conducted using @wwen2.5-32B. Insstruut on the DAPO-17K training set, with checkpoints selected based on the best average scores from AIME 24 and 25. Notably, neither strategy surpasses the full SvS. Extending standard RLVR training yields overall performance improvements, aligning with the results of (Liu et al., 2025a). For Enhancing underperforming problems with additional rollouts, achieves much higher Pass @ 32 but lower Pasy@ I compared to

---

Published as a conference paper at ICLR 2026
standard RLVR, suggesting that it prioritizes reasoning exploration exploiting generated correct responses. This corresponds to the conclusion in (Zhu et al. 2025), as this strategy introduces more negative samples from underperforming problem augmentation, and such exploration effectively improves the model's Pass@ & scores. For Augmenting simpler problems in SvS, it achieves similar Pass @ / standard RLVR but yields lower overall Pass@32, indicating that this augmentation accelerates overfitting to the policy's s already mastered capabilities while limiting exploration.
From these observations, two conclusions emerge: (1) Response-based augmentation in RLVR should focus primarily on underperforming problems (Eup v.s. SvS-Asp); and (2) Maintaining diversity in problem augmentation, rather than fixing the training set, also crucial (Eup v.s. Full SvS). We also provide additional multidimensional analyses of SvS, presented in Appendix
5.4
SVS GENERALIZES BEYOND MATHEMATICS: RESULTS ON CODE GENERATION
TACO
Codeforces
APPS
Mmmm M
Mad w
M mm
Score
Scora (%a
Score Q%4
Hamm
Su5
Su2
200
350
300
620
530
200
0
132
203
300 Slap
405
500
650
300
300
350
100
520
650
Step
Stepp
CadeContests (Avgig 16)
Average
Entropy
Score (36)
Entrapy
5
520
Stop
Stap
Stop
Figure 8: The intermediate evaluation comparing SvS and RLVR baseline on code generation tasks.
We incorporate the SvS strategy into RLVR training for code generation tasks to demonstrate its the (Qwen2.5-BB-nnttruus model
generalizability beyond mathematical reasoning. Specifically, to perform RL on 12k code generation problems from PRIME-RL (Cui et al., 2025a). covering sources such as Apps, CodeContest, Taco, and Codeforces. For evaluation, we sample 100 instances from each validation set. The hyperparameters remain unchanged from our other experiments. The intermediate evaluation presented Figure Notably, SvS training achieves significant improvements over the RLVR baseline and maintains stable policy entropy, demonstrating the strong generalization of this online self-play augmentation strategy in RLVR.
CONCLUSION
In this work, we propose an online Self-play with Variational problem Synthesis (SvS) strategy for RLVR training, in which the policy model independently synthesizes variational problems to improve its performance on underperforming training samples, enabling sustainable self-improvement process. By generating structurally diverse yet semantically aligned problems without requiring additional ground-truth annotations, our method ensures both diversity and verifiability of the training data throughout RLVR iterations, effectively maintaining policy entropy and generation diversity for sustained exploration. Extensive experiments show that SvS consistently outperforms standard RLVR across various model scales and benchmarks, particularly improving Pass @ scores at larger k competition-level benchmarks, where standard RLVR exhibits limited gains.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The authors have read and agree to comply with the ICLR Code of Ethics. In this work, we propose a self-play-based problem synthesis strategy for reinforcement learning in LLM reasoning. This study does not involve human subjects, nor collect any new datasets containing personally identifiable or sensitive information, apart from mathematical reasoning-focused problem synthesis and problem- solving generations derived from open-source LLMs. Both the training and evaluation datasets are publicly available, widely used the research community, and do not raise immediate privacy or data bias concerns. While we ucknowledge that we cannot fully guarantee all responses from the trained models are helpful and harmless, this research project focuses solely on improving the robustness of LLM reasoning. We do not foresee any direct negative societal impacts stemming from this research.
REPRODUCIBILITY STATEMENT
We provide an anonymous code package in the suppiementary materials, which includes source files, experiment configurations, and full algorithm implementations for reproducing all figures and tables. Practical training and evaluation details, along with the experimental settings, are provided in Section E The complete algorithmic workflow is described in Appendix [ while the overall framework illustrated in Figure 3 The prompts used in this work are provided in Section [ The training models, datasets, and evaluation benchmarks are provided in Appendix 4 while descriptions of all baseline methods are included in Appendix E
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning advanced reasoning models, 2025. URL htt.nttps::/hhktt
githuu...o//log/2255/PP.ss.
ByteDance-Seed. Beyondaime: Advancing math reasoning evaluation beyond high school olympiads. Seed/
/ttp://eeeeeee.taaeeeee/eyeeeeee Sat.aaannnneeeeeeeeeeeeeaiaaaaaa BewondAIooodii 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXhv:2107.03377 107.03377 2021.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shocybi, Bryan Catanzaro, and Wei Ping. Acereason nemeooonn- Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505. 2025.
Daixuan Cheng, Shaohan Huang, Xuckai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang. and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.1477 2025.
Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances Ilm reasoning. Advances in Neural Information Processing Systems, 11211125551122662 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1rXiv:1803.05457, 2018.
CLUEbenchmark. Math24o: High school olympiad mathematics chinese benchmark. https 2024. Accessed: 2025-07.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025a.
11

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.226117 2025b.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https:
//github. com http:/cc/w.ugggggg.g..cgggo openn-rl
Zitian Gao, Lynx Chen, Joey Zhou, and Bryan Dai. One-shot entropy minimization. arXiv preprint Sxxv:25005.20282 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.22 27.44077.77... 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501. 2025a.
Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Yu, Huishuai Zhang, and Yikang Shen. Synthetic data rl: Task definition is all you need. arXiv preprint arXiv:2505. 1..505.17000 17063, 2025b.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie- Yan Liu, and Wei-Ying Ma. Dual learning for machine translation. Advances in neural information processing systems, 29, 2016.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang. Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
notion. site/
2025a. Notion Blog.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint wwXiv:2504.11456, 2025b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4): 0-6, 2021.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rihf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03266: 2025a.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-rraasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24223
2025b.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333. 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hclyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1r21v:24/2.16620. 2024.
Eric Hanchen Jiang, Haozheng Luo, Shengyuan Pang, Xiaomin Li, Zhenting Qi, Hengli Li, Cheng-Fu Yang. Zongyu Lin, Xinfeng Li. Hao Xu, et al. Learning to rank chain-of th-uhhttht: An margy--bseed approach with outcome supervision. arXiv preprint arXiv:2505. 14999, 2025.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Dawei Li, Bohan Jiang, Liangje Huang. Alimohammad Beigi. Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:241 16594, 2024.
Hengli Li, Chenxi Li, Tong Wu, Xuekai Zhu, Yuxuan Wang, Zhaoxin Yu, Eric Hanchen Jiang, Song-
Chun Zhu, Zixia Jia, Ying Nian Wu, et al. Seek in the dark: Reasoning via test-time instance-level policy gradient in latent space, arXiv preprint arXiv:2505. 13308, 2025a.
Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang, Miao Liu, Zheming Yang, Minghui Qiu, and Jing Yang. Cure: Critical-token-guided re-concatenation for entropy-collapst prevention, 2025b.
11016
Zhing-Zhi Li, Xiao Liang, Zihao Tang, Lei Ji, Peijie Wang, Haotian Xu, Haizhen Huang, Weiwei Deng, Ying Nian Wu, Yeyun Gong, et al. Tl; dr: Too long, do re-weighting for effcient llm 2025c.
reasoning compression. arXiv preprint arXiv:2506.0260
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. 17419, 2025d.
Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, and Jian Jiao. Task oriented in-domain data augmentation. arXiv preprint arXiv:2406. 16694, 2024.
Xiao Liang. Zhong- Zhi Li, Yeyun Gong, Yang Wang, Hengyuan Zhang, Yelong Shen, Ying Nian Wu, and Weizhu Chen. Sws: Self-aware weakness-driven problem synthesis in reinforcement learning 2025.
for Ilm reasoning. arXiv preprint arXiv:2506.0880
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505. 248644 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1. 1.5b model by scaling rl. DeepScaleR Notion Page 2025. Notion Blog.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08967, 3, 2024.
MAA. American mathematics competitions (AMC 10/12), Mathematics Competition Series, 2023a. WRRLhhtts::////t wwwp:///ogooncoooooootttttttttot
amc
MAA. American invitational ma examiiatio (AIME). Mathematics Competition Series, 2024b.
ions / aime
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (707::07770477 2017.
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback. arXiv preprint arXiv:2503 222330, 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 19256, 2024.
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang. Zheng Liu, Zhongyuan Wang, and Ji- Ri-Rong Wen. Challenging the boundaries of reasoning: An olympiad- level math benchmark for large language models. arXiv preprini urXiv:2503.21380 2025.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 930-957, 2024.
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. International Conference on Machine Learning, pp. 47885- 47900. PMLR, 2024.
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct--I: A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems, 37:34737-34774 2024.
Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu. Explore the reasoning capability of lims in the chess testbed. arXiv preprint arXiv:2411.0666 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95860,95299, 2024b.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and for long cot from scratch and beyond. arXiv preprint 25X3::2503. 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15111:2412.15115. 2024.
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwci Wang. and Jing Tang. Treerpo: Tree relative policy optimization. arXiv preprint 250.::2250000/888.. 2025.
Edward Yeo. Yuxuan Tong, Morry Niu, Graham Neubig. and Xiang Yue. Demystifying long chain-of- thoouutttt reasoning in 1lms. arXiv preprint arXiv:2502.03373. 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2rX3v:2503.14411 2025a.
Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang, Mahmoud Khademi, Hany Awadalla, Junjie Wang, et al. caai--ofrresonnngg Towards unified mathematical reasoning in large language models via a multi martiiga perspective. arXiv preprint arXiv:2501.11 2025b.
14

---

Published as a conference paper at ICLR 2026
Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.008000 2019.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar 250.::2003..82 2025.
Hengyuan Zhang, Xinrong Chen, Yingmin Qiu, Xiao Liang, Ziyue Li, Guanyu Wang, Weiping Li, Tong Mo, Wenyue Li, Hayden Kwok-Hay So, et al. Guilomo: Allocating expert number and rank for lora-moe via bilevel optimization with guidedselection vectors. arXiv preprint arXiv:2506. 14646, 2025.
Xiaotian Zhang, Chunyang Li, Yi Zong. Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint 1rX44.2305.12474, 2023.
Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models. arXiv preprint arXiv:2503.02 023324, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807,, 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in Ilm reasoning. arXiv preprint arXiv:2506.013477 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS FOR SVS
LLM Usage Disclosure
17
Preliminary for GRPO
17
Extended Related Work
17
Reinforcement Learning with Verifiable Rewards
17
C22 Data Construction for LLM Reasoning
18
D Full Algorithm of SyS
18
Implementation Details
18
E. RLVR Training
18
Evaluation
19
F
Addtional Analysis on SvS
20
SvS Elicits Deeper Reasoning
20
F2
How Problem Synthesis Enhances Problem Solving?
20
SvS Generalizes beyond Reasoning Tasks
F4
SvS Outperforms RLVR on Challenging Problems
21
F5
Computation Analysis of SvS Training Compared to RLVR
................................
F66
Analysis of the Correctness of the Synthetic Problems
22
Comparing and Combining SvS and Entropy Regulation Methods
22
H
Initial Unsuccessful Attempts in SvS
24
Intermediate Performance on All Benchmarks
26
Prompts
27
Case Study
29
K.
Response Comparison Between RLVR and SvS Models
29
K.2
Complete Sample Workflow in SyS
................................
32
K.2.1
Case Study for Original Problem Solving
32
K.2.2
Case Study for Variational Problem Synthesis
40
K.2.3
Case Study for Synthetic Problem Solving
16

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
In preparing this paper, we used ChatGPT an OpenAl Large Language Model (LLM), exclusively to polish the manuscript's language for enhancing clarity and readability. The LLM was involved in research ideation, experimental design, data analysis, or drawing conclusions. All substantive content, technical contributions, and interpretations are the authors' own. The authors take full responsibility for the accuracy and integrity of the manuscript.
B PRELIMINARY FOR GRPO
Group Relative Policy Optimization (GRPO; Shao et al. (0224) is an efficient optimization algorithm tailored for reinforcement learning in LLMs, where the advantages for each token a rollout are computed in a group-relative manner without requiring an additional critic model to estimate token values. Specifically, given input prompt x the policy model Te, generates a group of G responses Y (uiir- with acquired rewards R = (ri)e The advantage Ai,t for each token in response yi computed as the group-level normalized rewards:
A mean (1i)e
(5)
std [r,)
To improve the stability of policy optimization, GRPO clips the probability ratio (()
aolve (a).k.e)). within a trust region (Schulman et al. 2017). and constrains the policy distribu- Teaa ( tion from deviating too much from the reference model using KL term. The final optimization objective is defined as follows:
(O)
Eng= D,Y P.Y-Toea
G
Mil min (O),,) clip clip(M..(0)), (O), -,--,,10 1.1 ( BBDx (Tret)
1
19
C EXTENDED RELATED WORK
C.I REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLMs in complex reasoning tasks (Luogg al., 2024 Guo et al. 2025a). Algorithms such as PPO (Schulman
ct al. 2017) and GRPO (Shao et al.] 2024) have shown strong generalization and effectiveness in
LLM post-training Existing efforts scaling up RLVR optimization have focused enhancing exploration (Yu et al. 2025a Yuan et al. 2025 Liu et al. 20256) Yeo et al. 2025) and adapting
RLVR the Long-Coo Co'l conditions Jaech al. 2024 Guo al. [022aa] Li [025d]] Yang et al.)
2025). Yu et al. 2025a) found that removing the KL. constraint and incorporating the Clip-Higher
strategy on top of GRPO facilitates better exploration during training.
However, Yue et al. 2025) raised an insightful question of whether RLVR truly incentivizes capability expansion beyond the base LLM, with experiments showing that it does not enhance Pass @
metric associated with the reasoning boundaries of LLMs. Some studies (Gao et al. 2025 Cui
al. 2025b Zhu et 2025) have also found that the entropy of model outputs declines during RLVR training, especially in the early stages, which hinders sustained exploration in later training. To mitigate entropy decline, Cheng et al. (2025) proposes augmenting the token advantage with an entropy-based term, while et al. (2025) and Chen et al. (2025) find that tuning the temperature appropriately helps maintain rollout diversity during training. Concurrent with our work, CURE et al. (2025b) introduces an effective mechanism for promoting exploration through high-entropy token expansion, which successfully helps prevent entropy collapse in RLVR. In this paper, we analyze policy entropy from the perspective of training data diversity and introduce self-play-style problem augmentation strategy (SvS) for RLVR training, which effectively maintains training entropy within stable range and significantly boosts model Pass@ performance, as k scaled up to 1024.
17

---

Published as a conference paper at ICLR 2026
C.2 DATA CONSTRUCTION FOR LLM REASONING
The construction of training data is crucial for enhancing the model's ' reasoning capabilities Luo
al., [025 Yu et al. 2025a Hu et al. 2025b Zhang et al. |025 He et al. 2025aj [Shen et al. 2025
Li ct al. [025cc Liang et al. 2025). However, high-quality human-labed
mathematical problems are limited and overly simplistic for advanced modern L.LMs (Cobbe et al. 2021] Hendrycks et al.
202T). To augment training data for L.M reasoning, existing data synthesis approaches have explored
2023 Zhao et al. 2025 Liang et al.) [024 Wang et al. [0244a] Li et al. 2024 Tan al. 2024) augmenting responses
generating problem-response pairs (Huang et al. 2024 Tang et al., 2024 Yu et al.
to existing questions (oohhniwal et al. [024 He et al. 2025a Face 2025] Yu et al. 2025b Li et al.
2025c). et al. (2025a): Diang al. (2025) propose incorporating latent representation data model
LLM reasoning. Targeting the training paradigm of RLVR. Guo et al. (2025b) proposes to synthesize
question and answer pairs from the task definition and documents, while SwS (Liang et al., 2025)
generates synthetic problems based on the model's failure cases during RIVR training. Most related to our work, Cheng et al. (2024) introduces utilizing self-play-style instruction data enhance model reasoning through ad versarrial training. In contrast to existing approaches, SvS enables online data augmentation without requiring ground-truth answer annotations. Our strategy effectively maintains training entropy in a stable range throughout RLVR, supports end-to-end training, and performs augmentation using the policy itself without external dependencies, expanding the policy's reasoning boundaries through full self-imprrvement.
D FULL ALGORITHM OF SvS
We present the full algorithm of SvS training in Algorithm [ The inputs to the SvS training framework include the Training set D, the Initial policy Te, the Underperforming accuracy range [acc], acoo] for selecting problems whose correct responses are used to generate variational problems, the Positive synthesis range [ačc,, aêc,,] for defining reward shaping in the response-to coennneett--poobbee to-problem synthesis task, the group sizes G and G. for problem solving and problem synthesis, and the total number of training steps?? T.
Generally, Lines 4-32 describe a complete training step of SvS, with experience collection and policy updating. Lines 5-10 detail the use of the policy to generate solutions for the problem batch sampled from the training set, where only problems with accuracy in (0, 1) are filtered into the buffer for policy updates. Lines 11-12 indicate that problems with accuracy within [acc], acc,, are chosen, and their correct responses serve as context for later variational problem synthesis. Lines 13- describe both the variational problem synthesis process and the utilization of the same policy to solve the synthesized problems. Rewards for the solutions to these synthetic problems are assigned based on whether their extracted answers match the reference answers of the original problems. Similarly, only variational problems with accuracy in (0, 1) are retained for policy updates. Lines 21-26 describe the reward assignment for the variational problem synthesis training pairs. Specifically, only synthetic variational problems whose volicy-sampled solutions achieve accuracies within [ačc], ačcØ] are assigned positive rewards; otherwise, the synthesized problem receives a negative reward. This design compels the policy to generate problems of appropriate difficulty that support effective problem synthesis training. Finally, only correct responses (input of this task) with a mix of positive and negative synthesis rewards are retained in the buffer for policy updates, consistent with the problem-solving task. During policy updates, all three types of training pairs in the buffer are mixed for gradient optimization, then the buffer cleared for the next training step.
E IMPLEMENTATION DETAILS
E. RLVR TRAINING
We choose GRPO (Shao et al. 2024) as our RLVR optimization strategy and incorporate several techniques from (Yu et al. 2025a), including Clip-Higher with set to 0.28, Token-Level Loss, and Dynamic Sampling. We set the learning rate to 1e with a constant schedule. The sampling temperature is fixed to 1.0. The batch sizes for sampled problems and policy updates in each iteration are both set to 256. The group size G of solutions generated from each original and synthetic problem, as well G. for variational problems derived from each response, is set 8. The underperforming
18

---

Published as a conference paper at ICLR 2026
Algorithm 1 Self-play RLVR with Variational Problem Synthesis
1: Input: Training set D, Initial policy тe. Underperforning accuracy range [acct, acca], Positive synthesis range [acc,, J. Group size G and G.. Total training steps T.
2: Initialize: Training experience buffer B Ø
3: for 1,..., do
4: Sample data batch from the training set D
5: for input problem-answer answer pair (ar, a) in the batch do
6: Generate group of solutions {ya) to a using Th
7:
Compute correctness rewards (Re)i= using for each solution .1.-... MG if 0 < Acc(z) < then
8:
9:
B B (Ix,)1) (2, (c))
10:
end if
if acc AAc(()) acca then
12:
Select [(x, Ut) )e) such that == = for accurate solution yi in {(x, ye)
Re(vs, a) = 1}
13:
14:
Synthesize a group of variational problems { |= from using m
for variational problem &/ in { do
16:
Generate group of solutions {ūx for a using Compute correctness rewards [Re using for each generation in,..., . UG end for
17:
19:
Select ( Je such that J1 [j 0 Acc(+)) G} Ji)
20:
B - B o ( ( .i.,,... (4))),..,((),
21:
Select { Jee such that J2 = (j ačc & Acc(t() & aécr}
22:
if [2] > 0 then
23:
for variational problem af in { do Assign Re(2() = if E J2, and Re 0.0 otherwise end for B- B ()), (vi, +9)
24:
26:
27:
end if
28:
end for
29: end if
30: end for
31:
Update the policy mo according to Equation 6 using the experience buffer
32:
Remove collected samples from B: B
33:
end for
problem range face,, acc,] is set to 12.5% 50.0%, while the positive reward range [ačo], ačc;] for variational problem synthesis is defined as 12.5%-62.5% The prompt used to synthesize variational problems from correct responses to underperforming problems is shown in Figure [8] Models trained on the MATH-122 dataset run for 300 steps, while 32B models trained on the DAPO- 17k dataset run for 600 steps for more comprehensive exploration.
E.2 EVALUATION
During evaluation, we use vLLM wKon et al. 2023) with inference hyperparameters set to temperature of 1.0, top-p value of 0.7, and a max response length of 8,192, except Pass @ scaling experiments, where the length is increased to 24,576. For Puss@k evaluation, we employ an unbiased estimation method (Chen et al. 202T) to reduce the high variance from single evaluations. We employ hybrid rulc-based verifier by integrating Math- Verify and the DAPO verifier in veRL (Sheng et al.) 2024). We use the default chat template and enable CoT prompting by appending the instruction: "Let's think step by step and output the final answer within bboxed(] ! after each question.
Baselines. We compare the SvS trained models primarily with the instruction eestruction-tuned initial policy and the standard RLVR models, trained using GRPO with the same techniques described in Section E. We also compare the SvS-trained models with models of the same size from SimpleRL (Zeng et al. 2025) and Open- Open-Reassoee--Zen (Hu et al. 2025b).
19

---

Published as a conference paper at ICLR 2026
F ADDTIONAL ANALYSIS ON SvS
F.1 SVS ELICITS DEEPER REASONING
MATH-12k
Qwen:328
Qwen-328 on
RIVR
3000
RIVR
RLVR
Respanse Lengtn
Response Length
SvS
2500
2000
4000
Respense
13:0
2000
1000
2000
500
50
150
200
250
300
150
200
300
400
500
600
Training Steps
Training Steps
Training Steps
Figure 9: Training response lengths for standard RLVR and SvS across various models and datasets.
DeepSeek-RR Guo et al. (2025a) has demonstrated that RLVR promotes deeper reasoning by encour- aging longer solution paths, where extended reasoning trajectories often involve advanced strategies such as reflection, verification, and exploration. To assess the reasoning patterns induced by SVS, compare its reasoning depth with that of standard RLVR training. We first compare reasoning lengths throughout training, as shown in Figure P We also present an exemplary case in Section [ illustrating problem and the corresponding solutions from RLVR and SvS trained models. No- tably, SvS consistently produces longer reasoning on training batches compared with RLVR. In experiments with LLaMA-8B and Qwen-32B on MATH-12K, standard RLVR consistently failed to generate extended reasoning paths from the initial policy, whereas SvS succeeded. This advantage arises because SvS requires the model to tackle new variational problems at each step, promoting continuous exploration of advanced reasoning strategies, whereas standard RLVR often secures high rewards by reusing memorized correct solutions.
F.2
How PROBLEM SYNTHESIS ENHANCES PROBLEM SOLVING?
Benchmark Evaluation
OOD Benchmark Evaluation
0.55
Less VPS
Accuracy
0.50
Avg. Accuracy
0.60
Full VP5
0.45
M
0.40
0.35
0.51
Less VPS
0.30
100
200
300
400
500
1000
200
300
400
500
Steps
Steps
Figure 10: Evaluating Pass@I accuracy of intermediate checkpoints for the SvS training with full and reduced variational problem synthesis samples on IID and OOD benchmarks.
This section examines how problem-synthesis training improves overall problem-solving performance. To this end, we conduct an experiment where only 20% of the variational problem synthesis pairs are used for policy updating in each RL step, using D0wen..5-32B-Instnee and DAPO-17k. Surprisingly, we find that incorporating problem-synthesis training significantly reduces policy's 's overfitting on the training set. For clarity, we categorize the benchmarks into IID (AIME24, AIME25, and AMC23) and OOD (MATH-500, Minerva Math, and Seempiad-Benchhh groups based on whether their reference answers are integers, given that all answers in DAPO-17K are integers.
The intermediate evaluation results are presented in Figure 10, Notably, SvS with only 20% problem- synthesis training performs comparably to the full SvS on IID benchmarks but is significantly worse on OOD benchmarks, indicating that pure problem-sol ving training is susceptible to overfitting and reduces the model's generalizability. whereas the inclusion of problem synthesis helps mitigate this issue. The effectiveness of problem synthesis in alleviating overfitting its enrichment of the training distribution as well as its regularization of learning through the complementary tasks of problem generation and problem solving (He et al. 2016)).
be attributed to
20

---

Published as a conference paper at ICLR 2026
F.3 SvS GENERALIZES BEYOND REASONING TASKS
Model
MMLU-Pro
ARC-C
ARC-E
HellaSwag
Winogrande
PIQA
BoolQ
HumanEval
AGlEval
Average
Init Model
68.33
58.62
77.31
85.17
73.48
81.01
89.60
56.10
70.54
73.35
4. RLVR
70.25
57.94
76.60
85.28
72.53
80.74
89.36
53.66
70.57
72.99
4 SVS
71.58
58.79
76,98
85,34
73.40
81.34
89.48
56,10
70.89
73.77
Table 4: Evaluation results on general question-annswerin and code benchmarks. SvS achieves the highest overall performance across 9 tasks, outperforming both the initial model and standard RLVR.
Since the SvS training strategy incorporates the variational problem symthesis task, a general question- answering task beyond standard RLVR's problem-solving training, evaluate whether this learning can transfer to improve performance on broader tasks, using the Qwen2.5-32B-Inse Ins-turuuutt model. Ac- cordingly, we evaluate models trained the DAPO-17k dataset using standard RLVR and the and coding benchmarks. The results are presented
SVS strategy across general question-answerte in Table Notably, models trained with standard problem-solving RLVR exhibit decline in per- formance on broad general benchmarks. In contrast, the SvS trained model not only avoids this degradation but also surpasses the initial -estructioo-foolowingg model on several general tasks, includ- ing MMLU-Pro (Wang et al.) 2024b), ARC- -haalhelee (Clark et al. (2018), and HellaSwag (Zellers) jtt al. 2019). These results indicate that the additional problem synthesis task in SVS helps prevent overfitting to mathematical reasoning tasks while effectively preserving or even enhancing the model's general cestruction-folloviii capabilities.
F.4
SvS OUTPERFORMS RLVR ON CHALLENGING PROBLEMS
AIME 24
AIME 25
100
RLVR
RLVR Sv5 > RLVR
5vS RLVR
Accuracy
80
Accuracy
80
RLVR
60
40
40
0
oooooooioooooooo conoooo commmno MANAAN
Problem Instances
Problem Instances
Figure 11: Comparison of instance-level accuracy between standard RLVR and SvS trained model. For each problem, the accuracy is averaged over 1024 generations on both AIME24 and AIME25.
In this section, we present instance-level analysis of SvS and standard RLVR models on the challenging AIME 24 and 25 benchmarks. The comparison shown in Figure bars represent the instance accuracy achieved by both RLVR and SvS, the red bars indicate SvS's s advantage over RLVR, and the blue bars indicate the opposite. Notably, on a large number of problems-such the 15th, 10th in AIME 24 and the 5th, 19th in AIME R-LVR achieves only limited accuracy, whereas SvS attains substantially better performance. reaching up to 80% on these problems. More significantly, SVS able solve problems that standard RLVR consistently fails to answer, such as the 12th in AIME 24 and the 17th in AIME 25, demonstrating its ability to extend the boundaries of model reasoning. In summary, SvS exhibits superior performance in both exploitation and exploration compared with standard RLVR, aligning with the results in Table
The gray
F.5 COMPUTATION ANALYSIS OF SVS TRAINING COMPARED TO RLVR
This section analyzes the computational overhead of SvS compared to the RLVR baseline, The comparison, using our experiments with 32B models trained on both the MATH- 12k and DAPO-17k datasets, is illustrated in Figure [2] All experiments were conducted using 32 H100 GPUs.
Notably, when training DAPO-17k, the initial stages exhibit large number of synthetic variational problems with responses because the policy's 's accuracy on most problems falls within the range [aco], aco.]. As training progresses, the model's performance improves, with accuracies gradually
21

---

Published as a conference paper at ICLR 2026
Var ational Probrema
Responses 2o Vo Vertatiooall Problems
Polkcy Update Time (5)
Step Time (6)
10009
Sud Daro-STe
Sid barolate
Foid NATTHALI
S/5 NATTHE2
1206
7540
Sus DAPO-TTE
2208
24
008
5003
406
mand
468
210
Figure 12: An illustration of the iteration step time and standalone policy update time for SvS and RLVR in the 32B model experiments. The two left panels show the number of variational problems and their corresponding responses in the SvS experiments.
surpassing acch, leading to a rapid decline in synthetic generation. Conversely, when training on the simpler MATH-12H dataset, the Qwen-32B model already achieves an initial accuracy of approximately 80% on the training set. Consequently, the number of synthetic variational problems generated is limited, and the SvS training time is comparable to that of standard RLVR.
F.6 ANALYSIS OF THE CORRECTNESS OF THE SYNTHETIC PROBLEMS
To understand whether the synthetic problems in SvS genuily and logically correct, we employ two state-of-the-an LLMs: Qwen3-235B A22B Instruct-2507 and OpenAI 03 to evaluate 6,000 variational problems (10 problems per step across 600 steps), considering problem correct if at least one of the models deems it valid. We use a prompt (Figure that examines Completeness, Logical Consistency, Solvability, Mathematical Soundness, and Overall Validity, to assess the correctness of the synthetic problems. We report the average validity over a sliding window of 100 steps (1,000 problems) and present the results in Figure 13] We find that more than 80% of them are consistently judged as correct by the models, with a slight downward trend over training steps. This decline may be due to the synthetic problems gradually adopting description style closer to the model's own way of expressing solutions in the later stages of training.
However, we also manually inspected the synthetic problems that both LLMs flagged as invalid and did not find any that were genuinely incorrect. Some problems were labeled as incorrect only because their textual descriptions appeared somewhat unusual, even though the underlying tasks were fully solvable. An example is shown below.
11 101.::01111111110111000011000000 htw.a///ooooiooooooooooooooooooo Jooooo Locooolooo.ooo wolelloooooo 1oo li -oo1"* Each "o" represents 2013 and "1" is 50 Find [rol[] where each string is a 70-character string. Find and express the necessary final evaluation.
Let r = 1100000111001100011111111 htw.::/eoooooooooooooooooooooood
"i 11qloo1oo1o11e
Other examples we evaluated follow a similar pattern: the problems are actually solvable, but both models incorrectly judge them as invalid. For instance:
In an 5 x 5 lattice grid from (0, 0) through (5, 5), there are teleports at coordinates (2, 2) and (3, 3). Whenever Michelle visits either point, she gets instantly teleported to the other if she hasn't visited both. Specifically, if she visits (2, 2) first, she's sent to (3,3), 3): if she visits (3,3) 3) first, she's moved to (2, 2). If she never visits those points, no teleportation happens. Find the number valid ways for Michelle reach (5, 5) from (0, 0). Solve this including all scenarios described. Present your detailed calculation and the final answer.
The Qwen3 Response is shown as follows. Its judgment is incorrect because the problem is fully well- posed under the standard lattice-path assumption that steps only move right or up. The teleportation rule is unambiguous: the first visit to either teleport triggers single one-time jump, after which both points are considered visited and no further teleportation can occur. With this correct interpretation, the problem is entirely solvable and yields unique answer of 84 valid paths.
G COMPARING AND COMBINING SVS AND ENTROPY REGULATION METHODS
In SvS, we maintain the policy's 's generalization diversity by online generating synthetic problems within self- self-play paradigm. Beyond this augmentation strategy, recent work, such as Cui et (2025b) and Li et al. (2025b), shows that regulating policy entropy can also enhance rollout diversity and mitigate entropy collapse. In this section, we comprehensively evaluate Sv$ against representative
22

---

Published as a conference paper at ICLR 2026
Validity: (Noxed(NNVAIII)
Reasoning:
Point 1: The problem is not fully well-posed due to ambiguity in the teleportation rule. The phrase "she gets instandly teleported to the other if she hasn't visited both" is ambiguous in a path-counting context. It is unclear whether "visited" refers prior visits in the path just the current step. Moreover, the rule does not specify what happens if Michelle teleported to point- point-does does arriving at (3. 3) via teleportation count "visiting" it? This affects whether teleportation can trigger again or not, and whether cycles (c.g., going from (2, 2) + (3, 3), then later stepping to (3, 3) again) are possible. These definitions are missing.
- -Point 2: There is potential for logical inconsistency or infinite loops. For example, if Michelle starts
(0, 0), walks to (2, 2), gets teleported to (3,3), then walks back to (3, again later, does this trigger amother teleport? Or if she returns to (2, 2) after being teleported, does trigger again? Without clear rules on state tracking (e.g.. a boolean flag for whether both teleports have been "visited"), ). the behavior is undefined and could lead to infinite paths (e.g.. looping between teleports if rules misinterpreted), making the problem logically inconsistent.
Point The problem lacks sufficient information for solvability. The state space depends on whether
Michelle has "visited" (2, 2) or (3, 3) in her path, but is not specified whether teleportation counts visit, whether the teleportation effect is disabled after the first Without modeling this state (e.g.. using dynamic programming with memory visitation), the number of valid paths cannot uniquely determined. The problem docs not specify if paths are self- self-avoiding or allow revisiting points, further increasing ambiguity.
Fix Suggestions:
Clarify that teleportation only occurs the first time either (2, 2) or (3,3) is physically stepped on,
being teleported to the other point counts as visiting it, after which the teleportation effect is disabled. Specify that paths consist of moving only right or (or define allowed moves), to make the lattice path model standard.
Explicitly state that Michelle cannot revisit teleports a way that reactivates teleportation, or define the state machine governing teleport use.
Define whether arrival via teleportation counts as "visiting" the destination.
Correct/ / Incarrect Ratio by Step Range
$7.3%
$6.00%
83.0%
Coreeit
inceroct
0.6
0.4
185%
23.4%
0.0
100
Figure 13: Illustration of the ratio and trend of synthetic problems deemed correct by SOTA LLMs.
entropy regulation strategy, Clip-CoV Cui et al. (2025b). and investigate the feasibility of combining these methods.
We compare SvS and Clip-Cov using the LLaMA-3, -BB Instruct model, trained on the MATH- 12k dataset for more than 400 steps. Evaluation conducted on GSM8k, MATH-500, Minerva-Math, Olympiad-BB -enhi, Gaokao-2023 and AMC-23, and their average scores. The Clip-Cov parameters follow the default settings the original paper, with a clip ratio = 2 10-4 Whow = 1, and Whigh 5. The hyperparameters for SvS augmentation follow the settings described in Section E As shown in Figure 14 SvS training consistently outperforms the Clip-Cov baseline intermediate evaluations. This improvement stems from SvS's continuous online augmentation of training problems, which promotes consistent exploration, while antropy-collappe mitigation techniques like Clop-Cov fail to stop the policy from memorizing previously correct responses to secure rewards during training.
23

---

Published as a conference paper at ICLR 2026
MATH-500
Minerva Math 272
Olympiad Bench-675
60
Score (86)
Score (S))
35-
250
Gaokao-2023-En-365
ge
Score 136)
50-
Scare 196)
Cito-
CBD
315
50
150
150
150
250
Step
Step
Stop
Figure 14: Comparison between SvS and the Clip-Cov strategy under mathematical RLVR training.
We
also investigate the potential of combining entropy-regulation methods with SvS. Explicit entropy regularization may enable the policy to explore richer and more diverse solution strategies for each augmented problem. From training perspective, SvS generates challenging problems across epochs based on the model's current responses, preventing memorization of previous correct rollouts and promoting sustained exploration. Consequently, SvS mitigate memorization within the entropy-regularized methods, allowing more effective exploration.
augmented variants of
We the same experimental settings in the previous comparison, and the results are shown in Figure 13] The figure indicates that integrating SvS augmentation with Clip-Cov consistently improves policy performance over the Clip-Cov baseline, demonstrating that entropy- methods can be further strengthened when combined with SvS.
MATH-500
Minerva-Math-272
Olympiad-Bendh--77
25
08
Score (K6)
(%5
15
RLR Cov
Cito
40-
200
250
250
250
Stop
Step
Step
Gaokao 2023 385
AMC 23 23-40
Average (%)
50.
Sooree (56
Score (9)
500te
p
Step
Step
Figure 15: Illustration of how effectively SvS be incorporated into the Clip-Cov strategy, and comparison of Clip-Cov with and without SvS augmentation.
H INITIAL UNSUCCESSFUL ATTEMPTS IN SvS
In this section, we present an initial unsuccessful attempt made during the development of SvS, about the reward assignment strategies for synthetic problems. Intuitively, if a synthetic problem is derived from a correct solution to the original problem, it should share the same reference answer as the original problem. Thus, if the policy model generates solution the synthetic problem whose final answer matches the original reference answer, the synthetic problem may appear valid, as it seems to yield the same final answer the original problem.
Therefore, to ensure the validity of synthetic problems, we initially assigned a positive reward to any synthetic problem for which the policy produced at least one solution whose final answer matched
24

---

Published as a conference paper at ICLR 2026
Average Pass@1
Responses Synthetic Problems
Policy Entropy Training
3.60
Namverr of Reseunzss
26
1.50
pace
1.22
1220
am
120
Step
Step
Step
Figure 16: Comparison of the Pass@1 1 performance, the number of valid training responses to synthetic problems, and the policy entropy between SvS and its initial failed version, which assigns positive rewards to synthetic problems whenever the policy produces single solution whose answer matches the original reference.
the original reference. However, this setting led to early failure. The policy quickly exploited this reward scheme by injecting explicit hints about the final answer into the synthetic problems, allowing it to obtain high accuracy simply by copying those hints. As all such hint-laden synthetic problems could easily yield correct solutions, they were consistently rewarded. As shown in the middle panel of Figure [6] the strong hints embedded in the synthetic problems consistently elicit fully correct responses. Concequently, the responses' advantages in GRPO collapse to zero, leaving no meaningful training signal for RLVR. This degradation ultimately impairs model exploration and downstream performance, as illustrated in the right and left panels of Figure I6
To address this failure mode, we propose maintaining the difficulty of synthetic problems throughout training by restricting positive rewards to those whose policy accuracy falls within [1/8,5/8], as adopted in most of our experiments. This adjustment encourages the generation synthetic problems that meaningfully contribute to policy improvement while suppressing overly easy ones, With this modification, SvS training succeeds, shown by the pink curves Figure
25

---

Published as a conference paper at ICLR 2026
I INTERMEDIATE PERFORMANCE ON ALL BENCHMARKS
In this section, we evaluate all benchmarks across intermediate checkpoints for standard RLVR and SvS, using the Qwen2.5-32B-Insteee dataset. All results are shown in Figure[T7 [7 with the final subfigure presenting the average intermediate performance across all competition-les benchmarks, including AMC 23, AIME 24 25, Math24o, and OlymMATH.
model as the initial policy and training on the MATH-12K
Notably, SvS achieves both higher peak performance and faster improvements than standard RLVR on nearly all evaluated benchmarks, including the homologous benchmark MATH-500 and the heterologous AIME, demonstrating clear superiority and strong generalizability. Although trained on MATH-12K, a mredium-difficule dataset for the initial policy, SvS still achieves substantial gains on competition-leve benchmarks such as AIME and Cympiadd-Bench as well as on competition- level averages, indicating that it elicits more advanced reasoning capabilities than standard RLVR. Moreover, unlike RLVR, which often reaches an early performance plateau, SvS continues to improve across multiple tasks, such as AIME 25, demonstrating stronger long-term learning potential.
GSMBK
MATH-500
Minerva-Mathh
20.0
06.0
V.A.
1922
46.3
00.0
Accuracy
65.0
18.6
$4.0
64.3
00.4
43.
42.3
43.4
50
3.0
1.50
200
250
300
52
100
150
200
250
200
30
300
320
200
230
300
Steps
Steps
Steps
Olympiad Bench
Gaokao 2023
Math24a
00.5
10.0
35.2
16.2
77.0
20.0
75.
Acturayy
75.0
28.2
12.0
74.0
20.2
7300
25.0
40.0
720
20.2
Stops
Steps
Staps
AMC 23
AIME 24
AIME 25
27.5
(76)
175)
Arturacy 180
20.3
20.0
20.0
17.5
02.0
15.0
10.2
50
1.50
1.50
2.30
25.0
300
100
15.0
200
250
100
55
30.0
15:0
200
25th
300
Stop
Steps
Steps
OlyMMATH-EEaS
OlymMATH-HHard
Competition Average
7.0-
10.0
Accuracy (56)
Accuracy
12.0
3.0
26.2
2.6-
24.2
6.
22.3
250
300
250
230
300
Stops
Steps
Steps
Baseline
SvS
Figure 17: Visualization of benchmark-level metrics for standard RLVR (blue) and SvS (red), with results smoothed using 5-step window to better highlight underlying trends. The actual data points are marked with faint dots.
26

---

Published as a conference paper at ICLR 2026
J PROMPTS
We provide both prompts used in this work, namely the Problem-Solving Instruction and the Varia- tional Problem Synthesis Prompt. During use, the (REPLACE) string substituted with the input problem and response for problem synthesis, respectively.
Variational Problem Synthesis Prompt
As an expert educational assessment and mathematical problem synthesis, carefully examine following model-genen senee-ggeneraed response:
<response>
[REPLACE)
</response>
The solution is assured to be correct. Your goal is to generate variants for the original problem that would most plausibly elicit such response. To achieve this, carefully follow these steps:
Identify the topic and context indicated by the response.
2. Infer the type of reasoning or calculation involved (e.g.. numerical calculation, conceptual explanation, comparison, opinion).
Determine the most likely educational purpose learning objective behind the problem.
Based on your analysis, write clear, concise, and natural-sounding original problem in English that satisfies the following criteria:
Precisely aligns with provided response.
Reflects realistic problem that could appear in an educational context or standard curriculum. Is explicit, measurable, and unambiguous.
Provide your final synthetic problem formatted strictly as:
"ttet
[Your synthetic problem here]
Figure 18: Prompt for variational problem synthesis from the policy's 's correct responses to underper- forming problems.
Problem Solving Instruction
(REPLACE) Please reason step step, and put your final answer within ()oxed(
Figure 19; Instruction prompt for reasoning problem solving appended to the end of each problem.
27

---

Published as a conference paper at ICLR 2026
Synthetic Problem Validation Prompt
You are mathematical problem validity auditor. Your task is to determine whether the given math problem is *valid and solvable", and provide structured analysis from multiple perspectives. valid problem is that mathematically consistent, unambiguous, and solvable based on the given information.
Evaluation Checklist
Check the problem using the following independent criteria:
** & Completeness**
Are all required quantities, constraints, definitions provided?
Are there missing variables, missing ranges, missing definitions, or unelear terms?
2. **Logical Consistency
the constraints contradict other?
Are there impossible numerical requirements or inconsistent modular/ccongruence constraints?
3. **Solvability**
Does the problem contain enough information unique or well-defined solution set?
Are there infinite solutions without further constraints?
Are the conditions sufficient to perform a calculation?
4. *Mathematical Soundness**
Are operations such as modular arithmetic, combinatories, geometry, algebra applied correctly?
Check for hidden contradictions (e.g.. impossible sums, negative quantities, undefined expressions). 5. **Overall Validity *
Decide:
+*VALID** conditions consistent, solvable, and unambiguous.
#INAALID** - Contradictions, missing data, undefined terms, or impossible constraints.
Final Output Format (STRICT)
Respond using the following EXACT format:
Validity: VALID INVALID. Place your final judgement inside \boxed ).
Reasoning:
Point 1:
Point
Point
(Explain using the checklist above. Be concise but precise.) If INVALID, also provide:
Suggestions:
(List the minimal changes needed to make the problem valid.)
<QUESTION> (REPLACE) <QUESTION>
Figure 20: The prompt used evaluate the correctness of the synthetie problems SvS.
28

---

Published as a conference paper at ICLR 2026
K CASE STUDY
K.1 RESPONSE COMPARISON BETWEEN RLVR AND SVS MODELS
We provide illustration of the responses generated from standard RLVR and SvS trained models to an AlME-level problem in Figure 21 Notably, the SvS model provides more comprehensive analysis of the input problem, with systematic reasoning (c.g., progressively narrowing possibilities rather than relying trial and error). In contrast, the RLVR model fails elicit advanced reasoning abilities and produces only concise step-by-step answers when addressing complex problems.
rigorous logical chain, clearer expression, and more
Question
The 9 members of baseball team went iec cream partor after their game. Each player had singlescoop cone of chocolate, vanilla, or strawberry ice cream. least one player chose each flavor, and number of players who chose chocolate was greater than number players who chose vanilla, which greater than number players who chose strawberry. Let be number of different assignments of flavors to players that meet these conditions. Find the remainder when N is divided by 1000.
RLVR Model Response
solve the problem, we need to determine the number of ways to assign ice cream llavors the players such that the condi- tions
least player chaoses each flavor, the number of players who choose chozolate is greater than the number of players who choose vanilla, and the number players choose vanilla greater
each player
scorp chocolate vanilla (V), strawberry (5)
number players choose strawberry. We will who choose chocolate, vanilla, and strawbery c, and respectively. Therefore, we have the following conditions:
number players
1. + - 2.
3. c, c,0,311
First, we list the possible values for v, and that satisly these conditions. Since or & and c + 9, we start by finding the possible values for s.
If ===
Then
and
The possible pairs (c, (6, and (5,3).
For (c, v. 26 1):
We need choose players of have chocolate, then choose the remaining vanilla, and last will have strawberry.
The number of ways to do this is
81 -3 252.
For (c, v. s) (5,3, 3,
We need to choose players out of have chocolate, then choose 3 out of the remaining to have vanilla, and the last one
will have strawberry.
number ways do
If
Then c++ and
The possible pairs (c, are (4, (5, 2).
For (c, v. (4,3,2) We need to choose players out of 9 have chocolate, then chaose out of the remaining to have vanilla, and the last 2 will strawberry.
number ways do
126
For (c, s) 25, 2):
We choose players of
chocolate, then choose
remaining vanilla, last will
have strawberry.
The numher of ways to do this is (2) (2) 126 6 756.
If greater, then would have be least 4, and would have to least 5, but then would be at least 5 + +-3 - 12, which is greater than 9. Therefore, cannot be 3 or greater.
Now. we sum up all the valid cases:
For (c, v,
1).
ways 252.
For (c, v.
(5. 1). the numher of ways is 504.
For (c, v. s) (4,3, 2). the number ways 1260.
For (c, v. (5, 2, 2), the number ways 756.
The number of different assignments dlavors to players is:
252 504 1260 756 2772
Finally, we need to find the remainder when 2772 is divided by 1000:
2772 mod 1000 - 772
Thus, remainder when N is divided by 1000 is
772
29

---

Published as a conference paper at ICLR 2026
SvS Model Response (Partl)
We have tocal 9 players, each choosing an ice cream flavor among chocolate, vanilla, and strawberry, Let's denote the number of players who chose chocolate by C, vanilla by and strawberyy by S. The conditions given arc:
1. C V + 9,
2. C >>V>> S.
3. Additionally. since at leust one player chose each flavor, means C 3 1. 1. S
Given the inequalities C V. S. and S know V because and V > 2, it follows C > 3.
S implies V 2. Similarly, since C7 V,
Since C, V. and S must be positive integers with C V + S 9 and must satisfy C > V S. let us find the possible values
for (C, V. 9) that satisfy these constraints, Specifically, > 1. > 2, and C a 3, with C
S. We will first identify
the valid triples (C, V, such that C' C S 9 and C > V S
Let enumerate valid (C, V. 8) combinations that satisfy C V C e 3, we also aeed C > V > S. We will check cach possibility methodically by first fixing the value of S. then finding viable and
and C V a 1. Since e 1, > 2, and
Recall S 3 1. Because V > and V. must be an integer. the smallest S can be is 1. Since V > and V must be at least S S+1 3 2. And C > V. No C. must be least V 1. Moreuver, since C. V S 9. let's explore the possible values for S, starting from the smallest.
I = 1, then C + V +1=99 V +
C + Vi = 8. We also require C > V> 9 = So C > V > 1. Since V can gu from up. We need to check which values of and corresponding C
because C
saristy C V= and C V1
I 1, and C V 8 and C
V 1.
-FV -2 2, then C - 8- V
8 2
6. The triple is (C, V) - (6, 1). Check: 6 > 222 2 1. This satisfies C > V > S.
V then 8-
S. V = 4. then C- = 8 V = 4 4. The triple is (C. V. = (4 1). Check: 4 = 4 > 1. This does not satisty
5.
The triplc (C. V. (5,3. 1). Cheek: 5 > 3 This satistics C
>
because is not strictly greater
Thus, (4, 4, invalid.
V 5. since C' = 8 8- V. when V
5 8- 3. The potential triple would be (C, V. 9) (3,5, However.
3.4 5. Since C' - 8 - V and C' must be greater than V. C
- 8 V
8> 2V - V 4. Thus any V < 3. For V 5, 3.
V= 4 will not work hecause
already checked V
invalid, and for V 5, C
which gives C = 3, V = 5, violates C > V. For V > 5, C' = 8 V will be C' & 2. Since V > S = 1, and V > means
8- & Because
+ Larger will make even smaller. Thus V
not yield
C> V > S.
The valid solutions when
are (C, V) (6, 1) and (5, 3, 1).
2. S - 2. then C + C + 7. We need C V > 8 - 2. Thus C' > V 2. Because V> S it means V
-11 and + 7. need C >
VFV 3, then
V 7 4. triple (C, V, - (4, Check >
satisfies C > S.
-IFV = 4. then C- 7 =7=7-- 7- 3. The triple (C, V, (3) Check 3 5 4. This does not satisfy C > V.
VFV 3 5. then C 7 V. When V - 5, C 7 5 2. The triple (C, V. 8) - (2, 2) will have 2 4 5. Since
C and
V
V V
2V
V 7/2 3.5. Thus V
most be to potentially
satisfy C >> V. We already checked V = which gives C' = 4 and is valid but V 3 4 does not woek because C' = 7 V & V.
and not sutisly
Thus, when 2, the only valid solution is (C, V) (4. 3, 2).
3. If 5 3, then C C++++ 5 need V T
_ V+ - - C+V We need C > V - 3. Thus C > V 3.
Becmuse V S
IIS = 3. C + V = 6, and we need C > V / 3.
VV 4. then
6-
2. trille (C, V) - (2, 3). Check 2 2 4. Su
satisly > V.
VFF > 5. then C - 6 V. If
> 5. C 6 V 1. Since V > 5, C - 5 1. The triple (C, V. 5, Check
14 5. In general for
3, and
V =
2V = V < But we need
V 3 here when S 3. Since V < 3 contradicts V 3 4. There are V 3 that saisfy C 6
V V. Thus there
is no valid C, V pair for 5 3 hecause V > and C - 6 - V & 2, will never have C > V.
Therefore, there are no valid triples (C, V. 8) when 5 =3
4. IF 4. Since C C+V+ + S 9 and C V > S S 4. I 4, then C - V 4
C V - 5. We need C > V > 5 = Since S 4. we need + V 5 and C > V 1. Since V 4 and is an integer, V > If 5,
C= = 5 V & This means there
positive integer solutions for C, V since C = 3- V. When V 3 C & O.
and C' must be positive integer greater than V. which is impossible here because C = 5 & when V 5.
So S 2 4 will never work becuuse C + V - 9- S. When $ VV - 5. V $
V. C When
$> C+ V S. If
4, C+V
For S 5, C+V
For $ 2 4, V 2 _ V3 When
V 5. C -==00 5 - VV & For S 5, C V 4. 8 5, not possible
V > S
CV
C= 1-11- V & -2 As increases, C+V = -V/==95: 9- V> = V 3 S+1 C =99-5-
When 8 = 4. C
2(4) O. When 8 9= 5, C g 2(5) -2. As S incrases. C - S with 8
C= 9- 5 5- (S + = 8 25, which is pon-positive when S 3 1. Since C' must be a positive integer and C V 9 V. with V
is increasingly restrictive such
3 (8 1)
S 4.
does not provide any valid triples because C, V, S must be positive integers with > >
making
C+ V = S. and V 3 S 1. When S = 4, C - V = 5, V 3 5. C & O. When 8 5, 4. V 2,666 C Thus, S 3 4 gives no valid solutions.
30

---

Published as a conference paper at ICLR 2026
SvS Model Response (Part2)
Summarizing, the valid triples (C, V,8) that satisfy C++ V and V are:
1. 1)
2. (5.3. (4,3,2).
Next, each these triples (C, V, 9). we will count the number of ways to assign players into groups sizes C, V. and respectively, where C' players chocolate. V players vanilla, and S players get strawberry. The number of ways to assign 9 distinct players into three groups of C, V, respectively (where C + S 9) given by the multinomial coefficient (o,v.s) ØाVi
calculate this each valid (C, V,
(C, V.8) (6, 20,2,13
9!
9X8x7x6
9x8x7 2
504
=252.
2,
6/2!1!
x
2. For (C, V, (5, 1}:
5,3,1 )
9
9!
9x8x7x6x3
9x8x7X6 9x8x7x6 -98877- 504.
5!3!!!
xl
3. For (C, V.
(4, 3, 2):
9x8x7x6x0
9x8x7x6x5
9x8x7x0x5
4,3,2
11322
1IX3X22
31
6
12
simplify 9x8*7x6X5
First calculate the numerator: 9 x 72.
72 x 504,
504 3024,
3024 x 15120.
Now calculate 10190
We can simplify 15120 Note that 12 3.
15120
First, 15120 12. We break down:
3) (15120 4) 3.
Find 15120
15120 4 = 3780.
Next, divide 3780 +
3780 3 1260.
So. (4,3,2) 1260. Summing possible ways from each valid triple:
+ +
add values:
252 504 756,
756 1260 2016.
Thus, 2016. The problem asks for remainder when is divided by 1000.
So we compute 2016 mod 1000.
2016 mod 1000 2016 2000 16 (since 2016 2 1000 16).
Thus, the remainder when N = 2016 divided by 1000 is 16.
Therefore, the final answer
Figure 21: A visualization of mathematical problem from AIME, along with responses generated by RLVR and the Sv S-trained models.
31

---

Published as a conference paper at ICLR 2026
K.2 A COMPLETE SAMPLE WORKFLOW SvS
In this section, present detailed sample workflow (all types of training data from the sample) from an SvS training batch, including original problem solving in Case variational problem synthesis in Case 2 and synthetic problem solving in Case For the report the reward for each response and include only responses (instead of 8 in training) to avoid excessive length. For the problem-synthesis tasks, present all 8 responses generated from the response input and report both accuracy and rewards for the synthetic problems.
tasks,
K.2.1 CASE STUDY FOR ORIGINAL PROBLEM SOLVING
Input Original Problem
(pix) second degree pa ononnniaal that 1,8 3, (p1 2. Then 9p (p (x)] x8 has four zeal selutions, Pind the only such soluzion which not an integer The answer is in the form rac(m) (a), where god(m, n) Flease provide the value of
Responses 1 (Roward: 1.0)
Our objective is to determine the polynemisi A( p(x) M, which is given by M p(x) ax bx and satisfies the concicions:
p(1) 1,, \quad (2)
\guad (3)
Substituting the respective values into the polynomial, get the rollowing system of equations:
A. p(1) = =11) + b(1) + to=a
aoc 1, A)
Al p(2) -2 -2 + b(2)
2b
p(3) (3) b(3)
Tetra write cown the system again for elaritys V.
V da
- V
A 90
=.=. A
To solve this system, we start with eliminating A o V. Subtract the firat equation from the second equation:
Al (4a 25 simpl ries to V Sa bbb -b- (tag(l) VJ
tabb 3
Naxt, aubtract the second equation
equation:
V 19a 30 c) (4a 2b t of = 3, V
which simplifies A/
-1. \tag(2) VI
Now need to solve the system of the resuiting two equations: Af \begin|casea]
3a +
5a + -1. tend(cases)
To eliminate b W, subtract the first equation from the second:
V Za -3, V]
hance,
M (1rrac33 AJ
using this value of A V in the first equation V 3a 2 M): \right) +bb 2,
M Afrac(9:12) M b 2 (2))
A b (2) (Ezac(9H12 \Ezac(13:12). VI
Substitute M W and \t A) back into the firat original equation M + o = M V Vieftf (2ra \right) \left( (Trac(13) \right) e
(frac(3)
\frac[13]
M (2r3)(33-13)21 + 1, A) e-
N \1cac(13:12)
M
(iac -0=--5
M.
Thus, polynomial
Al p(x) -VEra=(3112) x*2 \frac[131:2) x V]
need find the
of (()) V. Ne caloulating p(p(8) M,,
rs start with subatituting VI p6x) MI into itself:
! p(x) - -VEra=(3H12) x*2 (frac[13]:2) 4. 4.
denate p(x A) by M
Al y -Xrrc(331 x"2 (2ra0113)221 4 MI
+
We need to find A ply) V):
\ ply) Mefte -3Frac(3) x*2 \3rae:13) (2) \right!. MI
32

---

Published as a conference paper at ICLR 2026
substitute M < -\frac(3)
\frac(_3)1)
into Af (p)()) -\frae(3) y*2
(2) y -4 W):
M P (left (-
x²2 (13))(133 4 \right) = -\ \left -) 2 (rra (1et((-traa
22 (rra
(trac(13)2
:ight) 4. V]
Let's denote yy -1Frac(3H12) x*2 + \frac/13):21 4 V. Therefore we need to evaluate M p(D1)) plyl -V-rac(3H12) y*2 Mfrac113):22 4.
=
First, expand and simplify 1 y V:
M y ( -V-raer13) [2] x²2 12) *right) Meett (3 (2) x [rae(13) x Aright) (eftt -Mraa(33 x (Meac(13H2)
\right), V
M y22 - Wleft
x (frac(13:112) x- \right) []
M (-fra(111) x*2 \right)
\right) + (-4)-2 - (-e Azight) 1 Meft AMrret33 x Vodot (-4) Arighti
(rae(3:1) x Aedot (rrae(33]
2 Weettr (2xc]13)112 x ledot (-1) Aright) V
At - Veeft (2rae(3)2] x \rigat)
\right) 16 MeefL( \csac
(39)4; x13 \zight) +2 Meett( x²2 \right) + Meftt( -25 Arightt M
M \frra(() x*4 \1froo(169]/41 x-2 + 16 -1froc (2) x"3 + 12 x²2 x2
(frac(9) x-4 -Mfrac139)22 x"3 (ettt (1aa) x 12 x Aright) 52 x
M
(4) x14 ((rro(39) x*3 + [L x²2 + 48 x42)14) \right) 52 x 16
U
M
(9)
(2xac(39) x \frac(217 x22111 x
Thus, we have,
| p(o1x)) (2rac(3)22 y*2 1 [L322 y
M
V -3 \ (3rac(3114) x14 \fraz;39) x (217)(217) x²2 52x 16
right)
(1ra Aleft( -\érae(3) x (frac[13) Aright)
Expanding terms:
N( 32 ( \frac(9) x²4 \right) + -1 Alett -Vfrae(39) xn3 Aright)
(Mrac(3:12) \lefL MCzac(217):44 x*2 \iight)
-Vfracc(3):2) ( -52 \right)
-Afrae(3] [2] (
\right)
Mrrac133::21 ( (M x \right)
VErac[13):2] (ieft( (2)) (right)
\frac|13] Aleft \right)
-AA.
10 (1=AArec(2)))00 xMd (4)))( x²3 (Erac(651) x 78
\rrac1391:4) x" (Crac(169]
Combine like terns:
M = \frac127] xnd [4r xh3
Mettt -Vfrac(651) x-2 (3944139)44) Aright: Aleft (Trac(169:44) x Aright)
+ Meett -24 26
\right) A)
simplity further:
NI - (Erac1651) x*2 -Mrra/331::4 x*2 (E2ac(651) x \Exaac(78 x Mire
(729) x 1]
18 (1))
\rrao(312)(4) (rrac(169)44) (frac(481)
So we get, NI p(pix) -Vfre x²4 \frac(117]:44) xn3 \xxac(7291188 x²2 \frac[d81][4; x
Me want V p(p(()) M/; V -Xfrac(27) [8] x44 \frac(117) x \rrao (8) x (rrac[48]|14 [4] x 54 VJ
Rearranging terms:
M (1 x-4 \érac(117) [4) x-1 \éraz:729) 7291 (8) x-2 \frac[481H(4) x
M -\frac(27) x44 \frac(117] x 129a x
Aleft \rrra]4811/4) x x
eight) 54 - \]
M (Vxacc(27
\frac(117)
\Eras;7291
Mleft \Exac(4611:44 \frac
(4) x \right) = =
At Afrra(22) x"4 \Craa(117):4) x \Cxao;7291(8) x A1rao(47744 54 AJ
Multiply through by to clear fractions:
V -27x"4 + (adot (froc(117) [4] x 729x*2 +8 \odat \frac[477) x -432 = M.
A -27x"4 234x"3 729x"2 + 934x 432 -
Now, we need to find the roots of this polynemisi. Knowing that 11 p(p(x)) = V has four real roots and three of them are integers, we start by testing potenzial integez solutions 1) and p(x) f(x) where \ is another fixed point ir At I \neq x WI, We start by cnecking whether Af p(x) - V has integer solutions. N -VErac(3:11) x²2 + \frsc[13] (2; 4 V]
uning the fact that p((
Multiply through by
clear the fractions:
A -3x"2 13x
2x
N 3x"2 11x 8 -00
the quadretic formula:
Nl \fracl-b Apm (bart (2al
where a V, A 11 M, o B Mr
1( \fract-11 Apm Vagrt(ll<2 11 4 (-3) (-1)))22222233) VI
A \fracl-11 \pm \agrt(121 961][-61 X.
31 x \fract-11 \pm (29rt(25)11-6)
A1 \fract-1l Apm 5)
This gives Lwo solutions:
N x \fraci-1l 5)(-6) \érac]-6) = v
A( x \fracl-11- - (-]-)) = \frac(-161[-6) = (3) A]
So. M x - 1 V and A (31 V.. Clearly M x A) is an integer solution.
33

---

Published as a conference paper at ICLR 2026
Now, confirm, overall scots -27x*4 234x*3 729x^2 determining that it factorizes zo have integer solutions, let's use the fact that M p(x) x V solutions are known, we can confirm those.
To find the factorizetion of the polynomial sligns, let's denote the roots V x x.2, x x_4 V, and given that (I p(x] V'ss soots include integers and VI x=1 V. is one. Hence invest gate polynomtal via rational root theorem notioe that other integers might the rools.
since p(x V provides roots M \frac[8) and three
integers, let's assume other rational and integer roote by Lesting the inceger possiollities, PLag in some integer values directly:
Checking V pIx) V and other incegers:
M
912) - Wighharzow p(p(2))- p(3) 2 \RighLerrow p(p(2)) \neg 2,
Also tasting p(3)) = \Rightarrow p(p(3)1 = p() = \Rightarrow p(p(31) Aneg 3.1) Teating ( p(0))):
(Rightarrow p(-4) (xxxac(33 Vtimes (-4)*2 (Erac(13) Atimes (-4) -(frae(3112; Atizes 16
p(3)
-24 26 -54. \]
Testing higher integer inputs and discover integer values: Let test known roots: TF V x V solves ( ((1()) x W, check A p(x) 1,2,31) as initial feasible values, Also, verify underlying
p(2) p(3) 21) parallels 2,33 differing.
Rowever, the non-integer (and rational) solution must be validazed by: Polynominl p(p(x)) Given we know Vf x (rrao(8)13 Mr we can now cross-reference:
V):
Then for remaining equation roots: simplify to find and verify: Non-integer root is
Given the polynemiel
\neg 2,3,1N):
Conclusion: Af ( m=3,a=3 \rightarrow bbox
att Responses 2 (Reward: 0.0)
start by letting (pIx) ax42 + chl. Uning the given conditions (p1 1V, (2)
33,, and (3) 23,, we can form a ayatem of equations:
(beginicaess]
p(1) a(1) b(1) + =emath
(12) = a2)) + b(2) +0
a(3)
cend(cases)
We can simplify this system into a matrix form: \/
(heginicases)
da + 2b + + 3 11
9s
sendicases)
eliminate (eM) second from the third:
equatiens, subtract first equation
Subtract A
equation second:
(4a 2b la bubbb
Minplies Sa \tag(4)
Subtract the second equation from the third:
(9a 36 + ct (4z 2b
Vimplies Sa Stag(5)
now have the system:
(begin(cases)
3a a
5a =
case(s
Subtract the first equatien (3a 2N)) frem the second equation (5 -1V)
(5a (3s.
Vimplies 2a -3 Vimplies -\Ezac(3)123
Now substitute Ma -)frac(3) (21%) into the equation V + = 2N)
3 Veett -1frzc(3) \right) b Vimplies -VEran(9112)
Vimplies Miraz
(2) |frac/4)12 ((rrac(91 (2)
34

---

Published as a conference paper at ICLR 2026
WEth Va (2) and (b (21a we use the First equazion Ma IA to find VeM!:
AAzzec(HH2 VZzac(13H2) + Fic - Vinplies \Erac(-3 13) Vimplies \Exee;10) limplies 5 + Vimplies = - Aimplies =4-4
Thus, the polynemiel Mp(xI is:
91x) (3220(3122 x (2a6113) x
Be want solve (X(p(x)) XM, First, we will look for the integer solutions and then find the non integge oze.
Consider (p(x) yV and (p xV-
Let's check the polynomial VP(X0V the known pointé:
Nheginjalign=1
(2) - -4- -Xtrac(3) \Crao(13) Atrac p11) -3trac(3:1231))2
(1012]
(2) (2) - (3) \odoz 4 \frac(13)22 Aodot 2 p12) 5= (2
-a -6 -6+ 13 3,
913) Miac(3:122 (2))
(2x ledet (Ixa Vodot \fracl12) (2)
-Vfracl271:21 (frac(39)
2.
land(align+)
So, Mpl) evalusted at 1, 2, and 3 matches given velues and these are consistent. New let/s find the integer solutions for Mp x)):
pix) - Mappless Mraae33)2)x i (frac(13) 4 = Vimplies Mfrac(3H2)x*22 Mfraz (13 -x 4 Vimplies -3 \frec(11H22x 4 Vimplies -3x"2 11x
miltiplying by x 11
3x"2 11x 8
To check for intager solutions, we can use the quadratic formula:
x Vfraci-b Apm Ysgrt 4acl:12a Vfrac112 Apm \sgrt (121 96) \/frec/11 Apm lagrt
(2) = \frac(11 \pm 5/(6).
This sesults in:
\frac(11 5)16) (1ra \rrao(8113
and
Mrrao11 5:16) \[zac(6)(8; 1.
integer, non-integer
\xx \
Next, to confirm these values solutions of (x(p(x1) x5) and identily any other integer
solutions, alresdy
(Pp(1) p12) p13) considared for symmetry as sell since (p((())) = x)) implien fixed points and switched points
2, 2V1 must
We look at fixed points and knew p(1)
points:
V, thus ((V) solves (1)) 1M
Next check ()(22) and (31(3)))
Al P(2) 3, p13)
M
\I (2()2) p31 2; (()(3)) = p() 31)
30 12W and (3V)) are also solutiona of (21) 20, (p)) 3V..
verify the remoining
need solve (x)((x))) For Voy (3)
1/
Vleft \right) - (21x72 \raa|1131/21x
(3).
This yfelds:
(fzaa(3H12 x (frac(13) \
Clear the fractions. multiply everything by M1611
39
\Rightarzow
\Rightarzow x
Solving the quadratíc equation:
x \3rac[39 \pm /sqrt/ Vedot Vedoz (2 Nedot 9) \frec[33 Apm \sgre
0321-14401110 \fraa/39 \pm Sagrt (88) = \frac139 Apm (18).
35

---

Published as a conference paper at ICLR 2026
This yields:
VI
x= \frac148) (18) (81 Aquad (text(and) Aquad x= \+rac130) (18) \frac(() (31-
Thus, the non integer solutions (p(p(x)) xV) are (3 and (5) Letts
verify (
g Vleftr Mrac(55133 \right) Mfrec(3H2: Aieftt (3zec(3) \zight) |rrac|131:2) Aieft
(Frac(5) \right) - 4 (3) ledot \fraz/25) (9) (2xa Nedot \Erac(SH \froc 118] \frac[651(6)
We simplify:
-11rrc17591188 Mracf651(6) 4 11rac(75:1181 + (1a8 (1) Afrac
[-75 + 195 121(18) (233 - (3).
11
And thus M Maftt (3ra(99113) \right) V): A(
VMeft( \3 \right) -3frac(3)
[Nffc/ \=igha) 13r
edot
\frzc/8)
-(frac(3) Vodot (frza(64) (13) \edot \frac|88
\frac|192)1188 (trac[104)16]
11) (trac(104]
-(rrac(96:19) (112c(3121118) -4 -Arrac(192) (1rac(3122 (frac[120)188]
\Ezac(60)19; \frac|201:33 Mrac[12]:33 (Exac(8)
Continuing caloolatiion:
pip (3frac18](311) -p (Mrra(55]33) (frac18) \Rightarrow \text \text/sclve (solve quadratiol.
Rightarrow \text;check (5rac
AI
Varify forther:
N
\left (813))righh))
(1) \Erac(5H33 Aquad \left
given \quad
Rightarrow 'quad p()frac(5) (3)
So this solution point Loo is suitable:
Finally, the nen-integes solution (let's say non-integer root must be (VEzac
M
- \boxed/8)
Responsea 3 (Reward:
Given a polynomial M pix) () of second degree, we assume V p(x) ax bx M. We know
the following:
p[1) = a(1) + b(1) +
212) = (2) + b(2) +c- da 2b 3,
913) a a3)) + (3) 98 3b
11
Me now have system of 1inear equationn:
2. 11 4a 3
3. 11 2
Me solve this system by climination or substitution mathod. Start by subtracting the first.
equation from the second:
V.
(4a 2b 113--a40 33 Vimplies Sa 2. \quad Iquad (4)
Next, subtract the second equation from the third:
(9a 36 ct (1e. 2b 1 c) Aimplies Sa - 1. Aquad Vquad (5)
Me now have equations (4) and (
A 3a
5. VE 5a -1
Subtract fourth equation fifth equation:
V
Mimplies Minplies AErac(31122
Substitute -Mfrac(3112 back into equation (4):
36

---

Published as a conference paper at ICLR 2026
V
Bilet (31 +bb = 2. Vimplies -\frac(9)121 + =.2 Vimplies b = (Exac(9112) immpltea \rrac(4)(2 (Frac19) Mroaiea (frac(13H2),
a -(2rac(3H22 and ACb Afraot (112) M the Firat equation to rind V:
A2zaci3H2) Mraac13H22) f. Minpies MCrao(-3 13) Mmmllea \Izao;10) " limplies 5 + Vimplies -4.
Thus, the polynemiel is:
pix) = (31 x*2 \rra][33) x 4. 1
We now need to find the solutions of the equation ( ((p((()) x V) Start by dencting A 9Ix) W). Rence, need M ply) V. This means V and A are roots of the start by computing p M)
quadratic equation pit)
A
pix) (2acl3))2) x^2 (Ecac(1312) Azightt) x -\frac(3) x²2 \frec([1H2) x (ra (2))
Moltiply through by (-22) to clear the fractiona:
M
3x22 11x 11
solve this quadratic equation using the quadratic formula Mx \frac(-R \pm \agrt(552 {2 where ( A - V,, V B -11 Mr and A.C. - V):
X- Mfrac/11 Apm Neget 1121 96)1(6) Afrac/11 Apm (255) Mrraff1 Apm 51/63 V
Thus, the solutions are:
x \frac(11 5:16)
(6)) \frac:3)33) Aquad \text (and) \guad x \frac 5)
\érac(6](8) =
So the roets of Al p()) = x V are AO x 1 V) and A x \fraz(8)13) V. sence M p(t) = =xx V Mirraa(88133 Vr need
and plyl t-yy M ahould imply y
1 V) or ( ((p)) M[rac(8)33 M..
Solve (Ipi)) 1\
Mrite the equation M pix) 1 V
(Fraa(3112) x*2 (31 1,
- Arrao:3H2) x (2rac
6,
Ezac(3112 x12 (fxac[13)
Multiply through (
11
3x-2 13x
solve this quadratic equation using the quadratic formula where A -13, M
11
\frac(13 \pm \sget/169 1201:16) \6rac(13 Apm (d9)t( \frac(13 Vpm (6).
Thus, solutions
M
x \frac/13 + 7:16) \frac(20) \frae;10) (3), Aquad \frac(13 71(6) = 1.
So VI (X) 1 V has solutions \/ x 1 V and V x \frac(10) (3) V
37

---

Published as a conference paper at ICLR 2026
Solve (x) (2)
Write the equction V pix) (frac(8) M,
(fraa(2112) x²2 (2r 4 \Erac(81:3),,
multiply through by 6 clear the fractions:
( (3) x12 Neighe) + 6 ( (1 \right) 6(4; 611eft \fraz (8) \right),
-9x"2 16.
have:
11
-9x"2
the quadeatic formula again,
Mr V 39 M.C. M
\( \frac(-38 (3992 ((-8) (-40) \éran/-39 Apm \agrt(152) 14401)
Afract-39 \pm 18qrt/81//-184 Mrrac--39 \pm 91:-181.
Thus solutions are:
A \frac/-39 + ()(-18) \rrad-330 (3),
\frac/-39 ()(188) \frac[-481 [-18] (fr
1/
VI p(x) \[rac(3)(3)
solutions
(3) V and Mrrat:HH3; M..
Ne list all potential solutions sa fort X = 1, \1rac(10) (3), (8) (5) At. Since (f Mraac(::13 V one
original solutions of A p(t) if any other solutiens are not integers and distinct.
need
distinct solutions: X (31,, \frac(8)13), \frac15:13) V. non integes solutions are (1 (Ere \frac(5H3) Mr among those V Mirec (3) V) aiready satisfies ( pix) = \) and isz't t additional distinct solution of V p that isn't At p(x.
identify
(x))
So the solutions to A pp((x) = =x Mr, other than iteration M pix) Ar are M \frac (101
(iffrac15 We by ( p(x) =x Wi:
non-irtegar solution that not simply repeated
The integer solutions from above clearly M Mrra(11H33) and (3))), and (3)
verified:
(33 \neg IM and it's correct distinct.
Hance, the only auch solution which not on integer is \( \fraa(5)13) Mr. Bare, A m and (< n 3 A with (5 19.. 30,
M m n+n=5+35
Thus, the finel answer is (1)
#+ Responses 4 (Reward: 0.0) solve the problom, general form:
setting the second dagree polynemial 6p (x) in the
V. (()) ax12 bx
Me use the given conditiens (p(1) - 15, (p(2) 38, and (3) - 28 Lo set up system of equations.
First, aubstituting $x 15 in 5pix) 5, we have
Vi p(1) a1) + b(1)
+
Second, substituting 8x
Al p(2) = 12*2) + B(2)
4a 2b 3. V.
Third, substitut ng Sx 35, we have
V p(3) (3"2) b(3)
Thus, we have the following systen of oguations:
38

---

Published as a conference paper at ICLR 2026
\begin(cases)
4 c 1, Iuuad (1) 11
4a + 2h + =cc = 3, squad (2) 11 qqua
lend(cases)
To solve for Sa5, SES, and $c8, we fi:st eliminate $e8 by taking the diffezences between
equations (2) and {1), and (3) and (2):
Aimplies + b Aquad (4) Vf (3) (21: (9a 3b et 14a 20 c) - Nimplies 5a b -1. Iquad (5). M
(2) (11:
2b
Now, subtract equation from equation (51:
VI (5) (411 (5a of (3a bj
Vimplies za Aimplies (3]r AJ
Next, subatituting Sa \Crac[3):218 into equation (4);
Vf (-Vraa(333 (2) = \froc(4) (2) \frac(:3) X]
Vimplies (1)
Vimplies Mfrac
Then substitute sas 8b8 back into equation (2) solve $c9: M a + b b = Vimplies -3frac|3)
(taac113) [2) + Vimplies Vfracl-3 131/2) a.c.
I Vimplies (2) + + Vimplies 5 1 Vimplies -4.
Thus, the polynomicl (pix) is: Al p(x) ((rra0(3122) x*2 (frac(13) x 4. VI
Now we need to find the solutions to (p(p(x)) xt. First, we start by computing 8p (x)) 18.
Tat
M y p(x) (3) x*2 (2ro
So,
Al p(y) P ( (3) x \frao;13) (2) Aright!. M
To avold overly complicated computation, we denote (9 ys and need to solve By xf So we looking
that the function 9p8 is an involution, i.e., (p(p(x) x6. Hence (pi =x8 or Sp (x) c_18 roots $c_18
sply) x$ for sy p(x)s. therefore, 8p (x)
Sp(x) -x-Us, Letrs rizst rind if there ace solutions to 6p (x) xt since these will be
straightfosward.
M p(x) = x]
M AAtrac (2)) x Mrrac113112 x 4 AJ
M Erac|3:12) x Aleft (1frac(3333) Arigght) -
Afract3 (: (1) x²2 Mrac(11H22
-xx22 11x
-
V 3x"2 - + 8
Now, factorize the quadratic equation 83x*2 11x + - 0s. Ne use the quadratic formuls $x
Afracc-b 1pm (E 122--400 4ac] [2a]3 with Sa = 39, = -118, and sc = 88:
'ooot 3 \odot 81) 12 Aodot 1 x Mrac|1l Apm \sgrt(121 96) A]
\fracl-(11)) \pm \egrt
N x \fracill Apm Nagrt (25))
\frac111 Apm 5)16). V]
So the solutions are:
\fracl11
(rra(16) \frac(8) 13),
10 x Mraaill 5)(5) (1ac(66
VJ
Now have two possible solutions $x-18 and Sx= Let's verify what ether roots from condition Sp my) where sys must a rest of sply) x8. general, sys salves Ep(y) = x5, than for each reot sx$ of the involution condition we muat engure the map is symmetric, which checks (x)) x8.
To find other roats for (p(pix) we invastigate furthar. By the аyrmatey of the involution and leveraging the form of polysomials and solutions already Tound, the factored setup of likely zeveal angles where symmetzy diecates further integer or rational soluticns. Ke racogniza that any $ys that satisfias (p( itself must lead back 6x5 directly in our quartic aystem setup or
fixed points from involution
Ne know already that any fixed point solutions must be rational or integer from the nature of arithmstic. We look further:
Setting SE(K) = p(x) x5, we get VI f(x) - -VEzac(3H22xx12 \Erac(11) 4. V:
Me look at rixed points and other incersectionn onn.. 5o we need to remember roots of Sf to map back, Besides the fixed points $1fraci [3:8 and $1s, the zegs:ive root from composition must check (x) (frec(3) 18 mappings symmetrically, which depends polynomial behavior. Ma solve (p(x) 18, and (p(x) (fraa(8)
Firstly, solve (p(x) 18:
A -Arrac 12))x=2 \frac(13) 44
N
+ (Crac(13) - -
N -3x"2 13x
10 - V
3x"2
13x
0. AI
Using the qu formula $x = \fraa/-b Apm \egrt - 42a]] with $a 35, $0 -133,
8c - 108:
\ x \frac/13 Apm \agrt (1grt(169 (169 (2) A]
A x \frac|13 \pm Vagrt (6) A]
\fraol13 Apm 7)16] VJ
39

---

Published as a conference paper at ICLR 2026
\frac|20](18 [6) VErac(L0) (3), \text M (fzac16)(6) - 1.
VE x
So solutions Spix)
(315, and 6x 16.
Next, solve (x) (2]
-Xfraaa x +fraac13)
11)3 AJ
Multiply everything by to cleaz the denominatore:
NI -9x02 33x 24 16 A]
V -9x02 38x
1/ 9x"2 39x 40 0.
No the quzdrotic formula again, with Sa
-396, and 408:
1[ \Irac|39 Apm 139r2 Vodot Vodot 401 Vodot 9) N
N1 x \frac|39 Apm hagrt(1521 14401:118) VI
5( !Iraci39 Apm (81)
\frac|39 Apm 9)113] VI
Thus,
A \rrac148):18) (Crac(24) Mrrao:8H3:,
1 x= \frac/30) (16) (9)) = \frae/5) M]
Thus, four zoots total are $18, (VEzac(8):3]88, (VEzac(1)) () Since
need the only non-integer solution, it is 10) siEraci88 (318, and (]frac|5::33 (3]5. Since S)fracl88 1a already a rixed point solution, the non-intege- solutions check to a% [3a and #MCcac(5)
only non integer solution which
repeated fixed
point through other direct substitutions naturally is (3
Check form: The reduced fractien H3 in simplest form with 5m=56, Sn=35, Smen 5+3 85.
Therefore, the
(\boxed
Listing 1: An illustration of original problem solving in SvS training.
K.2.2 CASE STUDY FOR VARIATIONAL. PROBLEM SYNTHESTS
#++ Input Correct Response
(Please zefer to Response 1 in Listing 1)
Response (Ace: 0.5, Reward: 1.0):
polynemial (()) V. is given conditions:
p(x) ax"2 + bx such that satisfies the VI p(1) - 1, Aquad p(2) 3, \quaa (3) - 2. A]
First,
polynomial \I (() ax*2 Then, deternine the roots of the equation A (x)) - VI and Find the aun. A mtn Al for the nen-integer root.
that moets thase conditions.
# kesponse 2 (Ace: 0. 625, Reward: 1.0):
polynomial (p(x) 28. Given this pelynomial (p)x) 8, find the non integer root of the equation (p)) (x) x8.
satisCies
conditions
38, Sp(3)
rinally, if this non-integer root is of the form (m)rac(m] [n]$ where $m8 and Snş are coprime integers, determine
sum ns.
#+ Response 3 (Ace: 0.75, seward: 0.01:
Consider pelynomial (p(xi ax bx that satiefies the conditions Sp(1) 15, (2) 38, and (p) = 28. Veternine the coefficients of the polynomial and find its explisit form. Naxt, solve the aquation (x)) the rools are integers. Provide the aun of the numerator and denominator for the noa noa- integer root of the equation.
givan that it four real
that threa of
Response (Ace: 0.375, Reward: 1.0):
à polynemial $p (x) ax"2 28, (3) 25. Vetermine polynomial Sp (x) solve the zeal roots equation (pipix) xs. civen that this equation has four real rosts, with three of them seing Integers, Lind the frast root the form SVIraclm) (nIS and compute the aus Smrne.
c$ such that it satisfies the equations (1) 15, (2)
#+ Response 5 (Ace: 0.875, Reward: 0.0):
that satisfies conditions M P(]) (2) 3, V) and F(3) = 2 V,, determine the roots of the equacion V p(p(x)) = =x V, knowing that there are four real roots and three of then are integers. Calculate \i + }) where ( (frac(m)
polynomial M p(x) ax"2 bx
the nen-integer
in simpleat form.
#+ Response 6 (Ace: 0.75, Reward: 0.0):
polynomial p(x) ax"2 bx + a A) sattaties the following conditiona: V p(1) V,, A P(2) - Vr, and V P(3) = 2 Mr. Cecermine the polynomiel VI p(x) Mr Additionelly, solve for the roots of the equatien p(p(x) rational number in the form of fraction ((rrrac(m) (n) find the value of M M..
AL, Among the roots, chree integers and
40

---

Published as a conference paper at ICLR 2026
*+ Response 7 (Ace: 0.875, Reward: 0.0):
polynomial p()) ax"2 V/ p(1) 1, Aquad p(2) 3, Viued (3) 2.
satiaties the following conditiona:
Determm
tic form
polynontal M pix)
additionslly, find all the rocts of the equation VI ppix)1 x M,, given that this equatior
four real roats, of which three integers. Caloslate provide the \ for the non integer zoot the Zorm 1I ACzac(mFict Mr wheze A M and VI n are coprime positive integers.
Response (Aoos 0373, Reward: 0.01:
à polynomial ( p(x) ax"2 bx Petermine polyncmial A p(x) Then, solve for rocts equation ( (1)) =xx V) given that this equation has four real roots, of which three are integera. Find the value of VCrao(m) V for the non-integer root, and compute \ atn W.
satlefies the conditions M p(t) = Mr, A 2
V, and (3)
Listing 2: An illustration of variational problem synthesis in SVS training.
K.2.3 CASE STUDY FOR SYNTHETIC PROBLEM SOLVING
#4+ Input Synthetic Problem (Reaponse in Listing
A polynomial p(() M is given by M pix) ax?2 + bx + V auch that it sattafies the conditions:
M p(1) Aquad
Aquad (3)
girst, find the polynomial \ (() ax"2 + bx V that meets these conditions. deternine the roots of equatioo ((())) non integer 20ot.
sind the ( A for
Response (Reward: 0.0)
Given the quadratic polynamial A p(x) - ax"2, bx + Mh have three condisions:
\l p()) I, Aquad F(2) 3, Aguad (3) =.
Let's express these conditions in terms of ((h) \/b),, and MeW::
1. Al p(1) sy-2 by \higgl_iye1 [y=1)
V
2. V p(2) - ey-2 by Abiggl 1y=2 1a 26
V
3. p())) ey"2 by \biggliy=33
3b
Me now have the system of iouations: Shegin(casea)
+ + 6 1, \quad (11] V lquaad \text (2)) A Aquad (eext (
da 2b 1 (end(cases)
Let's solve these equatiens step by step. First, we eliminate (cl) from these equetions. Ke subtrast equation (1) from equacion (2):
20
V 3a b \quad (text((()) V]
Next, subtract equstion (21 from equatiion (3) V! (3a 35 e) (da
el M
V. 5a + b -1 lquad \zext (5) VI
Now, we have another system of equatione: M1
(beginicases)
\quad (sext( (d)) 5a à -1. Aquad \text( (5))
lendicsses)
Subtract equation (4) from equation (5): (Sa +b
M.
V 2a = A]
Ala -a- -x-rarcc33122 AJ
Next, substitute 11 a (3) V) back into equation (4):
M 3V1eft (22 +b =
N -\rrac 19:
N 2 (2ra0(9H2)
10 (2) (Ezac(9H1) A]
b (ftac|13)(2)
Fimally, substitute a -Afrec(3H2 V and b \Erac[13H2) M back into equation (1)
V[ -\frac(3:12)
\Eroc(13) =
V/ Ifract-3 13)12) + 1
41

---

Published as a conference paper at ICLR 2026
M 1 AOC -4
Thus, the polynomiel is VI p(x) = (3 x*2 \frac[131 x A]
The general form of ( p(x) A) is:
VI pix) (VEra=(33 x²2 \frac[131:2) x A]
equivalently,
VI p(x) Mfraelf 3x 13x 81:12) AJ
nead find the rosts of the equation p(p())) M. expresaing P (p(x)) V):
First, let's compute V ply) Wr where y y pix) - \Erac|-3x12 + 13x 812 M.
M p(x) M. Then
M
pty) ( \Crao(-3x"2 + 13x 81:21 Arighti
-3Irao(3H22 Alecur Arzac(-3x"2 13x (2)
8:12) \right/"2 (Erac(13)
(VErac(-3x")
Me' start by expanding ( \trael-3xC2 :3x 8112) \right) Vr
(ieft( \fracl-3x'2 + 13x 9H22 (right) = \érac( (-3x*2 + 13x (4)
AI
Expanding
81'2 M
(-3x-2 13x - 81 = (-3x02 + 13x 91 (-3x02 + -3x - 9xh4 2 Vodot 13 Vodet 3 xn3
(169x'2 + Vodot Vedot 8 x²2 641 Aedot 8 Voddt 13
9x"4 78x"3 (169 + 481 x 208x 64 9x²4 78x"3 + 217x"2 208x 64
Thus,
Nieft (NEzac) 3x'2 13x 82 \Cxa0;8x11 78x"3 1 217x'2 208x 64H43
Now, hava
11
-/zacc31122 [left///raci--
13x 3/ -VErac(31 Aedoz \Erao;3x'4 78x
211xx* 208x 641 - (9x^4 78x-3
217x*2 208x 64)
-VErac(2/x<4] [8) VCrac(236x33 (8) xrrac(65"x"2118
(24ec1624x1881
\frac1192)18]
-VEzac(27)18) x \frac(117) (4) x (frec(6511 x (AErac(7B)
-VCcac(27:18) xnd (raac117) x"3 Afrac(651)
AI
Thon
(rrcc(33)22 \ieft! Vfrec(-3x12 13x 6)) \right) Vfrec[13];4] 1-3x"2 + 13x B1 \frec
1-39x"2 + 168x 104] -3frac (4) x (119) x 26
Thus,
NI
(eft p()) \right) -Afrac x-4 \frac1117]14) x-3 -Frac x*2
|frac[391::4 x*2 (rac((69) 26
Mrac127HEB 1
M Mrac{[17HHd x Meeft(
\ffraci511::8 [8) \frac|39 ledet 2: Aright) x²2 + (left 78 (frac(169)) (16 \right) - 54
- x"4 (frao(117]14) x (froa x Aleft (frac[312][4] Afran
(169)14) \eight) x
(Frac[129) x*2 \rrac[481) x -Xfrac(27:1BB) x"4 (frac(117)14) x*3
VI
So, set A ((p())
(8) x (41))( x"3
(T23 x \1rac(181H44 54
A(rac:27) x"4 (41)) x"3
Atzac (729) x Aleft Acrac[481) (4) \eight)
=ffrocc(27:11) x"4 (frac x*3 \froc( (8) x*2 Alett( \frac(481) (4) \fraz
[4) \right) - 54 a
--NE:ac((7::1) x'd (Erac(117){4) x²3 (729) x
(477) 54
AI
This is quartic equatien
-27 x^4 + 234 x33 729 x + 954 x 432
VI
simplify dividing through M--3)
Al
9 9x"4 - 18 x/3 243 x-2 318 x 141 -
11
Next, we must find the rcots of the quartic equation \ 9x"1 78x"3 243x*2 318x 144 -
M. The
include any solusions to the originel polynemial equazions such A p(x) Solving M pix) A):
(fzac:3H2) x (2)
Multiply through by 2:
42

---

Published as a conference paper at ICLR 2026
We solve this quadratic equation using the quedzatic formule, V x VEraei-b Apm Veget (b12 (2)) V, where a -3 Vr ( = 11 Mr, and c = =8 V).
Viraof-11 Yon (2grt - 4(-3) ( (2 \frect Apm \eqet/121 (6;) "free [-11 ipm (5grt/2511/-1 = \ Apm (-
S0, \[ \fraa/-11 (-(6) \frac(-6) =1
11
\frae/-11 - () (frsc(-16) (frsc(8)
Thus, the solutions to A (x)) x V are V x - V. and M \Erac(8H33 M.
Next, se
other roots other roots by V Wr if 3I ((pI))] x A where V pix) \neg x V.. Assume A p()] = = Mr. Mr Therefore
consider
might not satisfy P(x)) =x Lee's dencte
(( pty) WI, Bence, we have V x t
pipix)) ply) x. 11
know the check if there are any further roots. No substitute ( p(x = = Mr then M ply) = x V.
already include x
\ (Erac(8H33 V,
9x*9 78x"3 243x*2 318x + 144 0 V. Ne already have two roots: M x=1 VI and A xeXerac(0) V.
quartic
four
total.
roots
polynomial
Jaing the polynomicl division,
factor ((x11) ( x-1érao(8113) \right/A.
Let's compute the product Vix 1) ix V - x (1 right)x + (frze(0) x*2 (froc(1l))3) (frec(0)) 243x22 318x + 1441) by V(x22 (rrae(11) (frac188
divide \3xx4 78x
sirst, write \(xx2 (3xx \Fxa0:8H33) (Erac(1H33)
11x ВУУ)
So we need to percorm polynomial division \/Irac19x*6 (3) 11x B1))):
78x"3 243x12 318x + 144
AArro19x" 78x"3 243x"2 318x + 14412(3x"2 - 11x + 83 Aquad \text (by cancelling out constant factor):
No perform long division:
Divide (3xx44M) \13x*23) get \13x2XM).
9x"4 - 78x"3 + 243x*2 - 318x 144 (3x'2 - 11x + 8)1 9x'4 38xh3 243x"2 318x
19x04 33x"3 24x*2)
-45xx3 219x"2 318x 144
VI
2. Divide (-45xx33) by (3x*2X) and get :-15xA):
-45x"3 219x'2 318x 144 ((-15x) 13x72 11x 81) -45x"3 219x"2 318x 144
(-45x
*3 + 166x"2 120x)
54x*2 - 198x 144
11
Divide (54x)22) by (3xx2V) and get
54x"2 198% 144 (1x 11x 81) 54x"2 198x 144 154x'2 198x + 144)
Theretore
9x'4 - 76x"3 243x*2 318x 144 (3x12 11x 8) (3x12 13x + 13)
11
Next, solve 133x72 13x + 18 UM VI
\fran/13 \pm Asgrt 1169 Aedet 3 Nodot 181:16) \fraa(13 \pm \oget 216)116)
frac(13 \pm 1-97) VI
Mrraci13 Apm \4q-t1471)161
Thus, the roota are nonreal A Vrra013 (49rt1471)16) V) and M Mfraa133 (491 M.
alsa hava (3x*2 -LCrac(8113; AD).
0V) (far which we previcusly had roota V x=t A
Finally, we
sum min V Since the non-integer rocts are (3) M[frrac133 (leget and Mifraec11 (17
integer root:
Frem the roots, (3 is not complex and another non-integez root's real part leads un the next likely "nominteger" condidate. For the complex roct \ (Vfrael13 + (41) only consider part V Mracc[331() which rational integer.
43

---

Published as a conference paper at ICLR 2026
Therefore V (1rac(13)16) V. leads to M m=23 V and V am5 M. Then, M mtn 19 M.
The theretore (19)
#++ Response 2 (Reward: 1.0)
Given Lhe polynonisl \ p(x) ax"2 1 bx
that meets the condicions 1f p(1 -1 Mr M p
(2) 3,, and P(3) = 2.2) W), we start by forming system of equazions based on those
conditions:
| p(1) =a1)) 1 b(1) 1c-at
M
M p(2) - =a b(2) =- 2b 2 2 M
M p(3) =2 +b(3) =o=
Ne need to solve the following system of 1inear equationa:
1. M + c=1 ==INttag()
2. 4a 1 12b 1-- 3333\agg2)
50c=2 \tag(3)
To elimirats a Mr we subtract equation (1) from equations (2) and (3):
\ (4a 22 e)
b.b
-3-1
Atag((]
similarly,
Al 19a 30
b - -
Ne now solve the system formed by equations (4) and =
A 3a +h = \tag(4]
Multiply equation 14) by to slign confficients of A b A with equation (5):
VI 6a + 26 = (tag(6) V
Suberast equation 15) from equation (6):
Al (6a 2a1 18a 26) -4-11
Ni -
Al - V)
Now subatilule \ (2) A into equation (4):
V. 3 (= \right) +bb A[ -1frac(9:12)
\l
Nt b - ((Iaa(3H1) M t (2] - (1) M -1frac(3):2) and 1 (frac(13) V,, substitute these values Now that have a
into equation (1):
N -\frac(3:12) Atrac(133111 + + 1 Nlett (Aftac//c)) \ -1 Aleft (xxrac(10012//eiiia -
\J
We found a (frrcc33333 \ (f=ac(13) V, and V - -4 V. then,
A[ p(x) = =- x-2 \frac|13] x- 4%
get cleaner polynonial, multiply through by
! p(x) -XEra=(3H2) x²2 Mrac[13]:2) x VErac(1H2) (-3x12 13x 8) M
So we can sawrita original polynomial
p(x) --rracl-xx22 13x 8)12)
Next, we nend to solve p(p(()) = V. First, express 'I (p(x)) V):
lat My p(x) Afrac(-3x*2 13x 81121 V.. Then, compute p(yl Ve
Al ply] -p " Mfacc 3x"2 i 13x 8H2: \right) MMract 3 Viefat Miraet 3x"2 13x
BH12) "tight;"2 +3 Weftt Vfraci-3x'2 pirat, simplify [( Ai Aleft 3x"2 13x 8H42Arighh) Mixacl (-aol(380 3x"2 13x 81 M
13x - () \right) - -812)
13x (2:Arrghht) -2))
Next, expand (1-3x=2 + 13x 81"2)::
V0 3x 13x 6)-2
3x 13x
814 3x 13x 81 AI
9x-4 39x-3 24x"2
39x03 + 169x*2 - 104x + 24x02 104x + 64
9x"1 78x"3
217x"2 208x 64
A)
So,
AL \ieft (Mrract-3333
13x 3HZ:Aright) '2 Mrrac1904 78x"3 2178"2 208X 64314) AJ
Then compute A (x)(()) V,,
Ni p(p(x) = -p (left( y \right) =p Vlfft \fron/-3x12 13x (1 Aright) A)
Vi Alett ( \eight)
Arrrall3 Alett 1
Aright) 13 Alett
\fracl -x"22l3x"2200 13x 8H22 \right) 8) VI
Me now simplify expressiens inside:
NC -3 "left \ 78x"3 217x"2 208x - 64114) \right; = \E=ac(-27x'6 234x'3 651
x 624x 192
NI 13 A Arrac(-3x"2 13x (1(2) \right)
Mracl13113x'2 13x 811 Mrrac139xx2
+ 169x 124)
Then combing overything:
Al p(11))) >rrrae 234x"3 651x*2 624x 192114) Arrac1-39x*2 + 169x
104H12 8:12) \)
Al = (
234x"3
651x'2 624x 1921 (4) 11200101-99222-10x10000)
(rae(32)14))12] 12]
< \fzacl\fraet 27x*1 234x*3
631x^2 i 624x
192
78x-2 1 338x
208
(2) N]
234x"3
729x"2 +
962x 432) (41) (2) N]
A(
-\frac-29x*4 234x*3
129x*2
962x
432: (8)
So,
\C (x((x) -ffxa 234x"3 729x"2
962x (32) VI
( 1,,
N. \frac(-27x"4 234x"3
729x"2 + 362x 432) (8) x N]
multiplying both sidea by 8,
21x"4 234x"3 729x"2
952x
8x A]
V -27x"4 234x"3 729x"2 954x 432 N)
Dividing by -3 all terms,
V[ 9x"4 18x*3 243x*2
44

---

Published as a conference paper at ICLR 2026
Thus solved quartic equation: VI 9x-4 78x"3 243x"2 318x 144 V.
To find the roots, we use the Raticnal Root Theorem to test possible gational roote, which are 2, Apm
A Von gen 36, ypm 48, Apm 54, Ipm 72, loo 108, pp 144, Mfrac 113,, (éract \pm ()(3), (prac))pmm;:133 \frac/ (pacc))m2)19))) (91, \Eract (9)] strac/Apm8) (0m8)(9); (91, \Frac() (916) V.
4, \pm
vpn 9, Apm \pm 16, Apm 18, \Eraci \Eraci>pml) 1pm1; Viracl Apm2
Testing V xx=== x 1)
9(1) 2881113 243(1)-2 318(1) 144 78 243 318 144
So, x 1M
root, Using synchetic division Lo divide A 9x74 78x"3 243x"2 318x
+ 144 1 by 1 -1 -.
M Wbogin (agginaarryyl (arrryl(rirree (F(etrer)
(arrzyl
1
78 243 -318 144
& 9 -69 174 & -144 11
Whline
6
-144 6 WA
lend(arcsyl VI
Reaulting quetient 1s:
VI 9x 69x*2
M
Now, teat x
V):
9(2) (2(2) (24
918) (9 348
144
V 12 275 + 346 144
A 12 + 348 - 276 144
Sp, M x 2 is factor.
Using synthetic division again divide 9xh3 69x-2 174x At bbgg
2
2 9 9 -69 174 -144 11
-102 144 W
thiie
4 a -51 4.72 72 0
Vendtarrayl Resulting quotient is:
VI 9x"2 - 5ix + 72 V
So the originsl polynomial
be factored
Af (x-1) (x-2) 19x-2 51x+72) 0.
For solving the quadratic V 9x"2 51x + 12 0 M):
At (3x"2 17x 24)
Jsing quadratie formula Wiighserrow 3x"2 -17x 24 N.
03),
M x = \fraci-b hpm lagrtib"2 [2c)) = \frac117 Apm (1722
Vodat Yodot 24]112 \
edet
\frae(17 \om 129rt(283 288; Mrra[17 Apm (agr:
\frae(27 Apn 1)(5)
A x (frac118) = 3, Aquad x (froc(16)16) = \3rac(8133) A) Thus, the roots
x 1, \qued
'quad x Auuad \Erac(B) VI
The non integer root is (frac(8) V), and the sum V
V) where V. (frac(m) (n)
frac(B: VI with V iged nl = 1 V is A 8 -
Ar.
Thus, the sum of the non integer root components is: VI
(1oxed(11)
#t# Response 3. (Reward: 1.0)
Given the polynemiel M p(x) ax"2 + bx A) that meets the condicions AI p(1) - 1 V,, M
(2)
and (()
W, start by forming
of equazions based chose
conditiors:
AI p(1) - =1) b(1) rc-albl baaibiell -1
Al p(2 =2 1 b(2) -
p(3) =a bb3)
Ma need to solve the following system of lineer aquations:
1. \ + =1 11
V
3. \ 9a 3b +c=2 Stag(3)
To eliminate a W, subtract equation (1) from equations (2) and (3)
25
1
\tag(d] V. Sa +b
simi
!. ta 2b = itagi5 \]
Na now solve the system formad by equations (4) and (
\( 3n
\tag(4)
Ba
Atag(5) V
Mulligly equation 14) by to align coefficients of At At with equation (5):
Al 6a 2b (tagie) V
Subtrast equation 15) from equation (6):
Ni (6a 201 (8a + 2b1 - ---1 1
V. -2a 3
Sa, Al -Xrrac(3112 V]
Now subetitute Al Afrac(3) AI into equation :4):
(- \right)
N -(rrac(9)12)
1 b - (frac(3H2) \frec(41:2) (2) (1ra)(13) M
45

---

Published as a conference paper at ICLR 2026
we 11 (3) and b (frac(13) Mr substitute values
into equation (1):
\ -) (frac(13) + 1-M
Alett (1rrac(10) (2Prightt)
-4 AA
we found - V, 1 (2rac(13: M, and V -4 V. then,
\( p(x) -3)
Mfrac]131/21
4.
To get a cleaner polynomial, multiply through by 2:
VC p(x] = -VErac(3 x²2 \frac[13]122
4 \Erac(1H22 (-3x12 13x 83 M
rewrita original polynomial V[ p(x) Mfraet
13x 8)12) V
Naxt, nead solve p(p(())
V. First, expresa A (p())) V::
Let M - p(x) Arrac(-3x22 13x 81121 V). Then, compute p(y) Vi:
VI ply) - -p Weett \fracf-3x22 2 + 13x - 2212: \right) - Eraccf-3 Viefz Vfracl -xx22 13x
BH2) \eighti"2 13 Weft Afrac: 3x"2 3x"2 13x 8](2) \right)
8312) M
first, simplify /
+1x 222 -2
13x (//2//rggh)
(fracl (-3x*2 + 13x (4) MI
Next, expand ((-3x"2 + 13x 81"2)1;
M 1-3x22 13x 6)*2 (-3x12
(-3xx2
VI
9x"4 39x-3 24x"2 - 39x03 + 169x22 104x 24x"2 104x + 64
3x*1 78x 217x*2 208x
A)
so,
Vieft (Mrrac 3x
8112Azight) Mrac|9x'1 76x-3 217x'2 208x 64:14)
Then compute (p((xx) V,,
A p(p[x)] = ( \right) p Alaft \fras/-3x12 13x
(2) \right) A)
M <left( \ \right) \fracl-3 Nifft 4xx 24tp://000000111100000000001-200 24211xx22208xx6641 \right) 13 Aleft
13x 8112) Arigat) 81121 M
We now simplify expreseiens inside:
NO -3 Aleft \ 78x 217x"2 208x 64114)
\right! \Erac(-27x'4 234x"3 651
x22 624x 192:14) [J.
13 13 [ Afract 3x"2 1 13x 8)(2) \right) - Mraa||3| 3x"2 13x
81 - Mfracl 39x"2
169% (04)
Then combina everything:
VI p(pix))
234x"3
651x"2 624x 192 + MErac[339x* + 169x
10411; - 8:12)
Al (
234x*3
651x'2
624x
192) + \
frre 14) )
! \frecl - 234x"3
651x"2 1 624x
-78x"2 + 338x 208 (2):44) AI 4321 (4)) X] 432: V
192
AIracl\fras 27x*4
234x"3
129x"2
562x
V \fracl-2Tx*4 234x*3 723x"2 962x
So,
A p(s1x)) - 234x"3
125x"2
962x (82) MI
We need ( Ee
p(p((x)) =x M,,
NC \Eraci-27x"4
234x
128x"2 + 962x 432) (8) =x A)
Multiplying both sidea by
A -27x* 234x*3 729x"2 952x 132 8x M
Nl -27x*4 234x*1 7 + 954x 432 = N
bividing by -3 terms,
A 9x"4 78x"3 243x"2 318x 144 V.
Thus we solved quartic equation:
9x"4 18x"3 243x"2 318x 144
To find the roots, we use the Rational Root Theorem to Lest possible :ational roots, which are Aon Von 1, Apm 2, Apm 3, \pm 4, \pm 5, Apm 8, Von 9, Apm 12, \pm 16, Apm 18, \pm 24, Apm 27, V on 36, Apm 48, Apm 54, Spm 72, Von 108, Apm 144, Afrac/\pal)(33 (3), Afract /fraci\pml)/9 19), \pma HO1, VéraciApm Npm (3), \Erac() (3), (frac//pm2)111 (91, (Erac(\pmd] pm16) (9) V.
Vfree/ (pm8) (9), \Erac()
Testing \ x 1)
M 9(11"4 - 78(11"3 243 318(1) 144 = 9 78 243 318 + 144 A)
So, Mx= 11) la root. Using synchetic division to divide A 9x"4 78x"3 243x"2 318x
144 1 by (r/ A (bogin
1 1 78 243 -318
144 13
1144 M
Whline
6 69 -69 174 -144 6 D DA
lend(arrayl A. Resulting quctient is: \( 9x*3 - 69x"2 + 174x 144 A)
Now, test M1x-2 \):
NI 92))"3 "2(2)'2 1 (74
144 - 918) (9 348 144 AJ
A 12 276 + 346
144 A]
A[ + 348 226
A
V 120 420 00 -0 V]
x 222 is factor.
Using synthetic division again to divide M 9x"3 69x'2 + 174x 144 A by M x-2 V
(C (begin 8 -69
-144
4 5 18 -102 & 144 W
thiine
-51 72 0
Vandtarrayl A)
Resulting quetient 1s:
Al 9x"2 six
46

---

Published as a conference paper at ICLR 2026
originel polynomial can factored VI (x-1) (x-21 19x"2 51x+72) 0.
For solwing the quadraric V 9x 51x 12 = 1):
3
\Rightarrow
-17x
Jsing the quadratic formula for VI 3x 17x 24 01),
V x \fraci-b Apm \agrt (b*2 (2a) = \frac/17 Ape (agrt(1702
Aedat Aedet 24] (2
edot
Afrac(1? Apm (2g8 (61 \frac|17 Apm (8gro(1)16) \frral"7 Apm (](6)
VI x== [frc/|88] = 3, Aquad x \Erac(16:16) = \Erac(0)3) V] the
A x Aquad x - 2, Aquad x Aquad x Mracc[8133 VJ
The non-integer root is V (frac(0) V, and the sum \ m n. where V \frec/m) (n) frac(8) with iged
3 3 11
Thus, the sum of the non integer zeor cemponents is: (bcxed(ll)
Response (Reward: 0.0)
To find the polynomial A p(x) ax bx A that satistiea A p(t). =1 Mr M p12)
and ((P(
start by weiting equations condicions give us:
N
p11) - a(1))2 b(1)
912) = a2) + b(2) 4a 2h
013) - 2 ((3) 98 3b 2.
These equat! ong form the following systim::
11
(beginicsses)
a
6 1, [text]
4a
1 2b 1 + -3 3, 5 \LetL (2) (2)
9a + 2. \text|
(ane(s
1.
begin by eliminating
Subiract (21:
1.
(4a +b la = = \Rightarrow 3a + b 2. Aquad Stext | (4).
Next, subtract (2) from (31:
(9a 3b Me c) \Rightarrow Sa -1, Iiuad Stex: (())
VI
We now solve the system of lineaz equatioas (4) and (
(hegin/cssea)
3a à 2, (4)
5a + -1.
(end(cases)
Subtract equation (4) from equation (5): 11
\Rightarzow
\Rightarzow ()
\)
Substituting \( A(raa(3HH2 A back into equation 14):
Jileft (-Vercc1) (2) (right) b \Rightarrow -\rras(91 \rrac(4) (trac(3) Mfraa(:3H2)
\Rightarrow = Vfrac
[9)
Next, substitute 11- (f V and b \frac[13] () into equation (1):
11 -[frae(31 (2r
\Righearrow \fras/-3 (3) \Rightarrow Afrac VRightarrow o
[101(2 t (Rightarrow 5 81
Therefore, the polynomial Al
pix) - -3fccc(33)2 x*2 (2za x 11
Next, we need to find the roots of equation M p(ptx)1 V. Pirst, note:
pix)
\froc(13) (2112
Be want \( ((p(p()) V. Suppose \( M. Then p(p(()) ply) M. need:
ply) =x lAightarrow (3) y*2 (1:oc(13) y
11
replace A p(x) 1)::
V.
(// (33 x*2 \frao/13) (2) x \right) + (21 AAfrae (2) x*2 Afrac(13)12) Aright)
47

---

Published as a conference paper at ICLR 2026
Let V p(x) - V. Thus A w (2) x12 \Exac(13) x 4.M) and A p(r) =x Rightarrow -]frac13) w<2 (133 4 M)-
Thus,
V1
(2 (eftt x02 + \Frae:13) (2) x "right) + (13)11 (-efte \right)
(3) x*2 (Frac(13)12)
First, consider aimpler casea by checking if the rocta of M p(x) = solve (x)) V) M
Wightarror -3fracl33 (21 x-2 (1rrac(33
(Iac:3H22 x (Ira)[13)
\Righaarzow (Erac13) x A1rac(11:12)
Multiply by \1-20):
x 11
11
Solve this quadratic equation using quadratic formula V \Ezact Apm \aget
1:12al 1):
11
\frac[11 \pm \sqet/121 96)1:6) MIrac|11 Apm (25)) \frae(11 Apm 5116;
VI
5o the solutions are:
[rrac[16] (rrac(8)13) Aquad stext land) 'quaad (6)
11
Thus, (x=MM and (xe3fracc(8) (313) solutions.
need check whether there any other solutions by considaring the pol ynom' form of N ((P( V.
Let's compute Pip(x) step-by-step:
A(
pix) - -122ac(3) MEcac(33]12 -
= W. Then
ply) (3)
(2xa
and substituting 1 y -V22ac \Erac(23)12xx 1)
bluectt-NNrree(3) (1rac(33:22xx \right) -Mracl33 Mett 3
(rac(13) x (right) "2 (frac(13:12) ( -VErac(3) x²2 Mfrac(13H2 x \right
-.
Let's denote ( p(x) (2)x (2ra as: N!
(2)
\frao(13)12)
Now plug this into M ply) V:
ply) -X2acc33112 / -Xcrae:3H2ixx2 \Czac(13)12) *2ight) Mrac[13):22 Merr
(-Irrac(3) x*2 Mraa x \right) -,
11
The expresston for M Weftt -3 + (1rac(13)12) x \right) V):
11
lleft (V]
\1ra Aright) { \Erac(-1x'2 13x 8:
right) (fraa]((3xx 13x ()) Aérao 78x"3 + 233x"2 208x 64) A/
get:
Arracr3112 Atimes \rrac19*<4 18x"3 + 233x"2 208x 64) -traac33 (9x"4 78x"3 61) (11x (Bx) \ (624)
\frac|192:16).
(88):27) X"4 (1rac(117)44] x"3 \txac1699] x (1ra6(78) x 24.
for the term (1 y/:: V
(frac(13)22 (2) Weftt H22xa \éxac(13) \eight) \frac
(131(2) llettl -V(rae(3x(2)12) + \frac[13x] \right) -3rrac x*2 199a 26.
Summarizing everything,
\c p(p(x)) (2) xMd (Ecac x"3 (E:ac(693) x*2 NExac(788)
(4) x*2 (érac(169) x 26 \I
simplitying above, VI
pip()) = (f x"4 \frac(117) x²3 Meft (frac(599) (81 \frac/39 \tines 2) 18) \rightix"2 ( 78 Arrac(199 Aright)
54.
48

---

Published as a conference paper at ICLR 2026
simplifying further,
(33c Vtimes (3 \fxec(78):88 \
(82a x-4 (217) x [312 + 1691:4) \right) x - 54,
\ Mrrec(599 78)(8) \right)x<2
Weftt Miza
Erac(27) (8) x (frac(117) xh3 (Ezac(777) x12
(11ac 54.
Thus, we need:
(2)) x (4re) \[rac(777:18) x
(Erac1481) 54
(xa0:27) x (41)) x (777) x Mrrac(88188
(8) x74 (117) x*3 Atxac(777) x \traa(473)44)
Multiply by to clear denominstors:
27x"4 \edot
177x*2 Yedot
Rodot D,
-27x*4 234x*3 777x*2 946x 432
Already found roots ( - V) and V x \3 V from V p(x) A): VI and ( \frac(8) (3) satisfy this polynomial. Since already determined its polynonial roots, we use polynomial division root local ization.
other roots must be from:
27y-4 234y13 777y12 946y
11
Finding remaining rcots: Using cemmon numerics:. selver factoring tools, solve for
21y"4 234y*3 197y"2 946 432 U.
VI
Vapprox 0,65, \Cxac(8H3:, M, veritying recognize rational rocts /1 Mr A (3) M..
Given non- inteeer root: (20)
(Zzac(2m)
\1zac(13)1201 Mtext
Showing \[
Continual polynomial reduction:
27 \ | y-1 \right) \left/ y-Vfrac(8:13) \right) polynemial \left ( yo12) ax 4 \right) reatored reduction.
VI
Ract V (20) \approx x (alpha V).
From known previews we get contia polynomial VI 0(1) solving for integer parts solving overlap ron-integer:
\bcxed/23).
Listing 3: An illustration of synthetic problem solving in SvS training.
49


Reviews:
[{"review_id": "fodhQ70Mo0", "reviewer": "Reviewer_Snxh", "summary": "This paper proposes a method called Self-play with Variational problem Synthesis (SVS) to address the issue of policy entropy collapse in Reinforcement Learning with Verifiable Rewards (RLVR) for LLMs. The core idea is a self-play strategy where the model synthesizes new, variational problems derived from its own correct solutions. This creates a self-improving loop that maintains policy diversity during training, leading to sustained and significant improvements in Pass@k performance, which is a key limitation of standard RLVR approaches.", "questions": "1. How do you think SVS would perform if combined with other methods that explicitly regulate entropy, such as those proposed in [3, 4]? Could these approaches be complementary?\n2. Could you elaborate on the advantages of SVS over other data augmentation methods for preventing entropy collapse, like SwS [1] and CURE [2]? Since experimental comparisons are absent, a conceptual discussion would be valuable. Is it feasible to combine SVS with these approaches?\n\n[1] Liang, Xiao, et al. \"SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning.\" arXiv preprint arXiv:2506.08989 (2025).\n\n[2] Li, Qingbin, et al. \"Cure: Critical-token-guided re-concatenation for entropy-collapse prevention.\" arXiv preprint arXiv:2508.11016 (2025).\n\n[3] Cui, Ganqu, et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" arXiv preprint arXiv:2505.22617 (2025).\n\n[4] Cheng, Daixuan, et al. \"Reasoning with exploration: An entropy perspective.\" arXiv preprint arXiv:2506.14758 (2025).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple yet Effective: The proposed method is intuitive, easy to understand, and demonstrates significant performance gains.\n2. Self-Contained: The approach does not rely on external models or datasets for data augmentation. It cleverly leverages the capabilities of the policy model itself to generate new training instances, which is an elegant design.", "weaknesses": "1. The synthesis of variational problems introduces additional computational overhead. The paper does not provide a clear analysis of this cost in terms of training time or resource usage.\n2. Despite mentioning SwS [1] in the related work, the paper lacks a direct experimental comparison with this very similar method. Additionally, other relevant works using data augmentation to mitigate entropy collapse, like CURE [2], are not discussed or compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oMK7VwOjmN", "reviewer": "Reviewer_yXhe", "summary": "The paper introduces a procedure to iteratively synthesize problem that a RLVR pipeline solves, and leads to boosted performance on various test set. The synthetic process consists of a few steps, such as conditioning on failed problems to synthesize new tasks, filtering and resolving those problem variations. The methods generally sound intuitive and improve the baseline significantly.", "questions": "=== *main takeaway* ===\n\nI think a main feeling after reading the paper is that it introduces yet another way to synthesize and augment dataset in RLVR process, and despite the performance improvements, I do not feel having learned too much insight from the paper, in additional to knowing yet another set of process to augment data for RLVR.\n\n=== *why certain filtering steps* ===\n\nI think a useful question worth answering is why we'd filter in one way vs another and it's useful to show why filtering the synthetic data according to the current criterion lead to performance vs. a more naive approach. I can see that we iterate a few times on the method while developing the paper and it's useful to understand and present initial failure cases and specific training comparison.\n\n=== *baseline* ===\n\nI think the paper lacks comparison to a few baselines. One is the prior work in the space on synthetic data, e.g., a few ideas related to synthesizing data from the initial training set has been proposed in [1], but maybe we should cite this paper + make proper comparisons, otherwise we do not see additional improvements over existing idea. In flagship results in table 2, such comparisons are warranted.\n\nAnother point is that plots like Fig 6, it seems that the x-axis measures the raw training samples experienced in the original dataset - however, the baseline leverages much more additional compute to arrive at a better performance, due to extra filtering etc. It will be useful to account for such additional compute when making such comparisons otherwise it is unfair to the baseline, one way is to compare against e.g. test time compute + baseline method to see if it's possible to make up for the performance gains, or increasing the number of parallel generation (group size) for baseline GRPO algorithm.\n\n[1] https://arxiv.org/abs/2505.03335\n\n=== *pass@1* ===\n\nWhile the paper motivates the writing with pass@1 metric, I don't see the connection between extra data augmentation with pass@1, after all we seek to augment data for which the model finds hard and try to patch such loopholes. It is not immediately clear to me why data augmentation addresses pass@1 issue at test time, it feels more like a generalization issue rather than an issue with metric (pass@1 vs. pass@32)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is interesting in that it provides another way to augment data for RLVR, leading to performance improvements. This affirms the knowledge that RLVR might benefit from synthetic data generation, and this helps push up even eval time performance.", "weaknesses": "I think overall the idea of synthetic data generation is not novel for RLVR, the idea of self-play and synthesizing new problems with filtering and verifications have been proposed and discussed in the literature. All in all, it feels that the idea is less novel and insights limited from the current paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7pFpP2g4T", "reviewer": "Reviewer_rsiR", "summary": "This paper proposes a self-play style strategy SvS to generate problem variants, which are not too easy or too hard (by model performance) and have the same answer to the original problem, to keep the diversity of prompts in RL training. It can keep better policy entropy during training compared with vanilla RLVR algorithms, and get better pass@k performance", "questions": "1. What's the cost of generating problem variants? What's the filtering rate of generated problem variants? Would the cost increase as model performance increase because model learn more patterns from self-generated variants?\n2. Is it easy to generalize to other domains?\n3. Have you tried training for more steps? Would the acc have similar curve as original RLVR in fig 1 when training step is very large?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and easy to follow\n2. Experiment results look solid, try different 8B/32B models, and see consistent improvement on policy entropy and pass@k/final performance on a lot of common benchmarks.\n3. This method does not need additional cost in labeling new data, and the generated problem can be prevented to degenerate to too simple cases (adding hints), result in better continue training performance.\n4. The method itself is RL-agnostic, so it's a relative general method", "weaknesses": "1. when using DAPO-17k, the training may overfit to int-style answer, and SvS perform worse on open-ended problems, like OlymE and OlymH. This shows that SvS may be limited to the model capability, and the distribution of provided seed data. So it's hard for some difficult problems or problems out of the distribution of model performance.\n2. Maybe need the base model to be strong enough to give some proper variants of the original problem, so more difficult for RLVR on small models.\n\n\nHowever, I think these problems are not that critical, and the results are still impressive to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsIsK5oAUy", "reviewer": "Reviewer_kCjs", "summary": "The paper aims to mitigate the problem of entropy collapse in RL and proposes a solution from the training data perspective. Specifically, the authors use problems with correct solutions under the current policy as a leverage, and diversify the distribution of the training prompts via variational synthesis, which they call Self-play with Variational problem Synthesis (SvS). During training, the current policy first performs rollout (like in any RLVR-flavored algorithm), and selectively augments underperforming problems given the correct responses as context (still under the current policy), given the same-answer constraint. The pool of synthesized prompts are then included for RL training. Experimental results demonstrate stronger performance at Pass@1 and Pass@N, effective prevention of entropy collapse, at the scale of 3B to 32B models.", "questions": "1. In Table 2, we seem to miss numbers from the three AIME columns. Why?\n2. How sensitive is the proposed method to accuracy thresholds (during problem selection for augmentation) and how should they be tuned? (And the same question on how to consistently decide which problems to keep.)\n3. In Figure 7, is the y-axis incorrectly labeled as \"Policy Entropy\" while it actually represents response length?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed idea seems intuitive: variational problem synthesis widens the distribution of the training prompts, or can be considered as a way to \"perturb\" the inputs, hence improving both performance and robustness (i.e., Pass and Average metrics).\n2. The proposed method is simple to implement and can be scalable given sufficient rollout budget.\n3. The proposed approach addresses entropy collapse by design, in an implicit way (by diversifying prompt distribution rather than directly interfering training dynamics, which is cool). This particular way of producing synthetic problems also does not suffer from the \"common pitfalls\" of synthetic data given the strict problem selection mechanism.", "weaknesses": "1. While the authors claim to demonstrate the effectiveness of the method on 12 reasoning benchmarks, they are mostly math benchmarks. I also noticed Appendix F.3 does include results from other reasoning domains (e.g., code, knowledge, etc.), but the improvements are marginal compared with those in the main paper, especially given that on some domains it becomes slightly worse. I do realize this might be ood w.r.t. training prompts, but a different question here is that: if we have training prompts from a different domain (or even mixed domains) like code, does it still work well?\n2. The authors only have vanilla RLVR as baseline (correct me if I'm missing something), even though there are already many other algorithm-based entropy collapse prevention method as of paper submission deadline. There is also one simple baseline missing: what would happen if we simply increase the rollout budget to be consistent with the proposed method?\n3. The proposed method further increase the inference overhead of RL, while missing details on how much this overhead is and if this overhead is worth paying considering that there are algorithms which claims to mitigate entropy collapse without additional inference cost.\n4. The method introduces additional hyperparameters like the difficulty window and decisions on which synthetic problems are rewarded and kept. To me this does not seem like an easy choice to make, or has to be made separately given domains of varying difficulty to the current policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cF8wlYbkFT", "reviewer": "Reviewer_zAc9", "summary": "This paper proposes SVS (self-play with variational problem synthesis) to address the entropy–performance trade-off observed in the RLVR training of LLMs. SVS utilizes the policy’s correct solutions to synthesize variational problems while ensuring their reference answers remain identical to the originals. This mechanism effectively mitigates the significant decline in policy entropy during the training phase and achieves performance gains on both challenging and common reasoning benchmarks.", "questions": "1. Can the authors provide some analysis on the synthetic problems? \n2. Can SVS be applied to domains aside from math?  To be clear, I am not referring to generalization experiments, but rather to the direct application of SVS-based RLVR training within those domains.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper deals with an important and relevant problem: the degradation of policy entropy during the RLVR training of LLMs. The proposed SVS method effectively mitigates this issue in the domain of mathematical reasoning. The experiments provide robust validation of the method's effectiveness, demonstrating clear performance gains.\n2. The experimental evaluation is thorough. The authors present extensive comparisons on both challenging and mainstream reasoning benchmarks. Notably, Table 3 provides a fair and direct comparison between the standard RLVR approach and the proposed SVS, clearly highlighting the SVS's advantages.\n3. The paper is well-written and easy to read. The proposed method is described in sufficient detail to facilitate reproducibility. This work represents a valuable contribution to the research community focused on RLVR.", "weaknesses": "1. First, the validation criterion for synthetic problems in SVS is based on whether their average accuracy falls within a predefined range. While this heuristic is sensible, its correlation with the actual logical correctness of the synthesized problems needs further discussion. It is unclear whether these \"valid\" problems are genuinely logically sound or if they merely steer the model toward reproducing the original answer. This ambiguity requires further investigation. I believe introducing a component of human evaluation or manual review for these synthetic problems would significantly strengthen the persuasiveness of the experimental validation.\n\n2. Following the previous point, it is not surprising that training on synthetic problems mitigates entropy degradation. What is intriguing, however, is that SVS significantly boosts performance on challenging benchmarks. This raises the question of whether this performance gain is primarily an artifact of the experimental setup, which focuses on only math-related benchmarks. In mathematical reasoning, the solutions (e.g., step-by-step derivations) often contain sufficient, or even complete, information to reconstruct the original problem. This characteristic may not hold true for more general domains. It is important to investigate whether SVS can deliver consistent performance improvements in broader settings, such as code generation, puzzles, or other complex RL scenarios.\n\n3. The problem synthesis task in SVS introduces non-trivial computational overhead. The paper currently lacks a discussion of this additional cost. An analysis of this computational cost(e.g., increased training time or resource utilization) would be helpful for the evaluation of the application value of SVS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)