SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
Authors: Xiao Liang
Zhong-Zhi Li
Yeyun Gong
Yang Wang
Hengyuan Zhang
yelong shen
Ying Nian Wu
Weizhu Chen
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning; LLM Reasoning; Data Synthesis
Primary area: deep_learning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model’s capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization by empowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks. Our code and data are available at https://anonymous.4open.science/r/SwS-E6F5/

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
Xiao Liang'; Zhong-Zhi Li Yeyun Gong"t, Yang Wang", Hengyuan Zhang',
Yelong Shen?, Ying Nian Wu', Weizhu Chen
University of California, Los Angeles, 2 Microsoft
3 School of Artificial Intelligence, Chinese Academy of Sciences, Tsinghua University
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and -erifdd-veiiiiiaaiion answers in existing distilltion-orrented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model's capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure and synthesize problems strengthen the model's 's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization by empowering the model to self-identify and address weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks. Our code is available https::/giiuub..om//aaaa Vito/SwS
Qwan Quem22.3 5 32B- IT
(e) Performance actuss Benchunarks
(b) Performance across Domains Frwadgaies
ORZ-128
863
SimpletL-
SwS-32B
MATH
Number Thesey
Intermediste
Algabon
500
Minerva Mate
Precalculus
AB4C23
Algebra
Olymptad
Bench
Frobability
Gacmetry
Figure I: 32B model performance across mainstream reasoning benchmarks and different domains.
"Equal contribution. Work done during Xiao's and Zhongzhi's 's internships at Microsoft.
'Correspondigg authors: Yeyun Gong and Weizhu Chen (yegong, wwwchenmmi.cooooo..oooo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
"Give me six hours to chop down a tree and will spend the first four sharpening the axe." "
-Abraham Lincoln
Large-scale Reinforcement Learning with Verifiable Rewards (RLVR) has substantially advanced the reasoning capabilities of large language models (LLMs)(aaech et al.) 2024, Guo et al.) [025
Team et al. 2025 where simple rule-based rewards can effectively induce complex reasoning skills.
The success of RLVR for eliciting models' reasoning capabilities heavily depends on a well-curated
problem set with proper difficulty levels Yu et al. [025bb) Liu et al. [2025b). Xiong et al. (2025).
where each problem is paired with an precise and verifiable reference answer (Hu et al.) 2025 Luo
et al. 2025 Yu et al. 2025b Guo et al. 2025]. However, existing reasoning focusee datasets for RLVR suffer from three main issues: (1) High-quality, human-labeled mathematical problems are scarce, and collecting large-scale, well-annotated datasets with precise reference answers is cost- intensive. (2) Most reasoning-focue focused synthetic datasets are created for SFT distillation, where reference answers are rarely rigorously verified, making them suboptimal for RLVR, which relies heavily on the correctness of the final answer as the training signal. (3) Existing problem augmentation strategies typically in rephrasing or generating variants of human--ritten questions (Yu et al..
2023 Luo et al.] 2023 Pei et al.] 2025 Liu et al.] [022a]] or sampling concepts from existing
datasets [Huang et al. 2024 Tang al. [2024] Li al. 2024a|| Zhao et al. 2025b), without explicitly
considering the model's reasoning capabilities. Consequently, the synthetic problems may be either too trivial or overly challenging, limiting their utility for model improvement RL.
More specifically, in RL, is essential to align the difficulty of training tasks with the model's 's current capabilities. When using group-level RL algorithms such as GPRO (Shao al. 2024), the advantage of each response is calculated based on its comparison with other responses in the same group. If all responses are either entirely correct entirely incorrect, the token-level advantages within each rollout collapse to 0, leading to gradient vanishing and degraded training efficiency Liu et al. [2025b), Yu et al. 20255b, and potentially harming model performance (Xiong et al. 2025). Therefore, training on problems that the model has fully mastered or consistently fails to solve does not provide useful learning signals for improvement. However, a key advantage of the failure cases is that, unlike the overly simple questions with little opportunity for improvement, persistently failed problems reveal specific areas of weakness in the model and indicate directions for further enhancement. This raises the following research question: How can we effectively utilize these consistently failed cases to address the model's s reasoning deficiencies? Could they be systematically leveraged for data synthesis that targets the enhancement of the model's 's weakest capabilities?
To answer these questions, we propose a Self-aware Weakness-driven Problem Synthesis (SwS) framework, which leverages the model's self-identified weaknesses in RL to generate synthetic problems for training augmentation. Specifically, we record problems that the model consistently struggles to solve or learns inefficiently through iterative sampling during a preliminary RL training phase. These failed problems, which reflect the model's 's weakest areas, are grouped by categories, leveraged to extract common concepts, and to synthesize new problems with difficulty levels tailored to the model's capabilities. To further improve weakness mitigation efficiency during training. the augmentation budget for each category is allocated based on the model's ' relative performance across them. Compared with existing problem synthesis strategies for LLM reasoning [Zhao et al.. 2025b Tang et al. 2024), our framework explicitly targets the model's capabilities and self-identified weaknesses, enabling more focused and efficient improvement in RL training.
To validate the effectiveness of SwS, we conducted experiments across model sizes ranging from 3B 32B and comprchensively evaluated performance on eight popular mathematical reasoning benchmarks, showing that its reakness-driven augmentation strategy benefits models across all levels of reasoning capability. Notably, our models trained on the augmented problem set consistently surpass both the base models and those trained on the original dataset across all benchmarks, achieving a substantial average absolute improvement of 10.0% for the 7B model and 7.7% for the 32B model. even surpassing their counterparts trained on carefully curated 2025, Cui et al., 2025). We also analyze the model's performance on previously failed problems and find that, after training on the augmented problem set, it is able to solve up to 20.0% more problems had consistently failed in its weak domain when trained only on the original dataset. To further demonstrate the robustness and adaptability of the proposed SwS pipeline, we extend it to explore the
labeled problem sets (Hu et al.)
2

---

Question-1
Tiffony is constructing a fence around rectorgular Tennis court. She must use exactly 300 feet of fencing. The fence must enclose four sides the court. Regulation states that the length of the fence enclosure must be least 80 feet and the width must at least 40 fest. Tiffany wonts the area enclosed by the Fence To be as large as possible in order to accommodate benches and storage space. What the optimal
feet?
1nswer1,1
0.5
X
Policye
Verifier
12
Tr
Epach
Failed set
0.9
Answerk,1
RL Training for Tı Epochs
1
Figure 2: Illustration of the self-aware weakness identification during preliminary RL training.
potential of Weak-to ttrrong Generalization, Self-evolving, and Weakness-driven Selection settings, with detailed experimental results and analysis presented in Section 4
Contributions. (i) We propose a Self-aware Weakness-driven Problem Synthesis (SwS) framework that utilizes the model's s self-identifee weaknesses to generate synthetic problems for enhanced RLVR training, paving the way for utilizing high-quality and targeted synthetic data for RL training. (ii) We comprehensively evaluate the SwS framework across diverse model sizes on eight mainstream reasoning benchmarks, demonstrating its effectiveness and generalizability. (iii) We explore potential of extending our SwS framework to Weak-to-Strong Generulization, Self-evolving, and Weakness-driven Selection settings, highlighting its adaptability through detailed analysis.
2 Method
2.1
Preliminary
Group Relative Policy Optimization (GRPO). GRPO Shao et al. 2024] is an efficient optimization algorithm tailored for RL in LLMs, where the advantages for each token are computed in a group- relative manner without requiring given an input prompt æ, the policy model nhw generates a group of G responses Y = { with acquired rewards R = {r]) The advantage As, for each token in response yi is computed as the normalized rewards:
additional critic model to estimate token values. Specifically,
A T; mean((r
(dd(())))))
To improve the stability of policy optimization, GRPO clips the probability ratio (0,) (ol cc), Tew (20, (x) within a trust region Schulman et al. 2017). and constrains the policy distribu- tion from deviating too much from the reference model using a KL term. The optimization objective is defined as follows:
(O)
E Y-
G
min (k,, clip clip(k,.(0), - -,, + - 3DL (DDK((Te) (Frel (Tet))))))))))))
C
MA
Inspired by DAPO Yu et al. optimization, while incorporating the clip-higher, token-level loss and dynamic sampling strategies to enhance the training efficiency of RLVR. Our RLVR training objective is defined as follows:
2025b). in all experiments of this work, we omit the KI term during
G
lud
I J(0) E.DDD Y (x1z))
min (l) clip(ks, -,,
11
1=1
s.t. aCClower < Ki E Y lissssaaaaa jis_accurate(r, y:)l < acCupper-
(3)
where oh denotes the upper clipping threshold importance sampling ratio (0), and acClower and aCCupper are thresholds used to filter target prompts for subsequent policy optimization.

---

Step1: Weokness Identific-
Step2: Extracting and Recompining the concepts from the fallure cases to synthetic new questions.
Step3: Augmenting synthe- tic set into RL training.
ation in initial training steps.
Synthetic Set
Failed Set
Split by Categories
Initial Set
Voka
Ficlds
Clei Mfrert
foodd
Cont- dime
Difficulty Filtering
+
Vanui
Concepts Extraction Recombinbation
Filtered Set + Initial Set
f
AnB (x)
O
(ta) loga (A)
de
liu
A
Solutions
Po.
Po.
Problem Generation and Verification
Sumpled Concepts
(Ploneing) To create challonging
Synthetic Set
question withis the preceloukal (Generated Probien) Ceraider the
Training & Acc Recording
fuection ((x) anch astieties
Consistency Filfering
Training
Quality Verification
X
Problem Genoration Model
Ardwer Generation Model
Figure 3: An overview of our proposed weakness-driven problem synthesis framework that targets mitigating the model's reasoning limitations within the RLVR paradigm.
2.2 Overview
Figure presents an overview of our SwS framework, which generates targeted training samples to enhance the model's reasoning capabilities RLVR. The framework initiates with a Self-aware Weakness Identification stage, where the model undergoes preliminary RL training on an initial problem set covering diverse categories. During this stage, the model's weaknesses are identified as problems it consistently fails to solve or learns ineffectively. Based on failure cases that reflect the model's weakest capabilities, the subsequent Targeted Problem Synthesis stage, we group them by category, extract their underlying concepts, and recombine these concepts to synthesize new problems that target the model's learning and mitigation of its weaknesses. In the final Augmented Training with Synthetic Problems stage, the model receives continuous training with the augmented high-quality synthetic problems, thereby enhancing its general reasoning abilities through more targeted training.
2.3 Self-aware Weakness Identification
Utilizing the policy model itself to identify its weakest capabilities, we begin by training it in preliminary RL phase using an initial problem Xs, which consists of mathematical problems from n diverse categories {D)" each paired with ground-truth answer a. As illustrated in Figure 2 we record the average accuracy as, of the model's responses to each prompt x; at each epoch t E 1........ Ti), where Ti is the number of training epochs in this phase. We track the Failure Rate for each problem the training set identify those that the model consistently struggles to learn, which are considered its weaknesses. Specifically, such problems are defined as those the model consistently struggles to solve during RL training, which meet two criteria: (1) The model never reaches response accuracy of 50% at any training epoch, and (2) The accuracy trend decreases over time, indicated by a negative slope:
F(x:) =T max art < 0.5 A slope ((a1,)==1) <0] [E[LT]]
This metric captures both problems the model consistently fails to solve and those showing no improvement during sampling-based RL training, making them appropriate targets for training augmentation. After the weakness identification phase via the preliminary training on the initial training set Xs, we employ the collected problems X (a, Xss Fr(2a) = 1) as seed problems for subsequent mankness-driven problem synthesis.

---

2.4 Targeted Problem Synthesis
Concept Extraction and Recombination. We synthesize new problems by extracting the underlying concepts Cp from the collected seed questions X5 and strategically recombining them to generate questions that target similar capabilities. Specifically, the extracted concepts are first categorized into their respective categories D; (e.g., mathematical topics such as Algebra or Geometry) based on the corresponding seed problem Ij, and are subsequently sampled and recombined to generate problems within the same category. Inspired by (Huang et al. [024] /Zhao et al.] 2025b], we enhance the coherence and semantic fluency of synthetic problems by computing co-occurrence probabilities and embedding similarities among concepts within each category, enabling more appropriate sampling and recombination of relevant concepts. This targeted sampling approach ensures that the synthesized problems remain semantically coherent and avoids combining concepts from unrelated sub-topics or irrelevant knowledge points, which could otherwise result in invalid or confusing questions. Further details on the co-occurrence calculation and sampling algorithm are provided in Appendix(F]
Intuitively, categories exhibiting more pronounced weaknesses demand additional learning support. To optimize the efficiency of targeted problem synthesis and weakness mitigation in subsequent RL training, we allocate the augmentation budget, i.e., the concept combinations used as inputs for problem synthesis, across categories based on the model's category-specinf failure rates FD from the preliminary training phase. Specifically, we normalize these failure rates ED across categories to determine the allocation weights for problem synthesis. Given a total augmentation budget |Xr), the number of concept combinations allocated to domain D; is computed as:
FD.
XT.D,I = Xrl lb. = |Xr| Σj
(5)
where FD, is the failure rate of problems in category D; within the initial training set. The sampled and recombined concepts then serve as inputs for subsequent problem generation.
Problem Generation and Quality Verification. After extracting and recombining the concepts associated with the model's 's weakest capabilities, we employ strong instruction model, which does not perform deep reasoning, generate new problems based on the category label and the recombined concepts. We instruct the model to first generate rationales that explore how the concept combinations can be integrated to produce well-formed problem. To ensure the synthetic problems align with the RLVR setting, the model is also instructed to avoid generating multiple-choice, multi-part -artt or proof-based questions [Albalak et al., 2025). Detailed prompt used for the concept-basee problem generation please refer to the Appendix K For quality verification of the synthetic problems, we prompt general instruction LLMS multiple times to evaluate each problem and its rationale across multiple dimensions, including concept coverage, factual accuracy, and solvability, assigning an overall rating of bad, acceptable, or persect. Only problems receiving 'perfect' ratings above a predefined threshold and no 'bad' ratings are retained for subsequent utilization.
Reference Answer Generation. Since alignment between the model's final answer and the reference answer is the primary training signal in RLVR, a rigorous verification of the reference answers for synthetic problems essential ensure training stability and effectiveness. To this end, we employ a strong reasoning model (e.g.. QwQ-32B [eam, 2025 )) to label reference answers for synthetic problems through a self-consistency paradigm. Specifically, we prompt to generate multiple responses for each problem and use Math- Verify to assess answer equivalence, which ensures that consistent answers of different forms (e.g., fractions and decimals) are correctly recognized as equal. Only problems with at least 50% consistent answers are retained, as highly inconsistent answers are unreliable as ground truth and may indicate that the problems are excessively complex or unsolvable.
Difficulty Filtering. The most prevalently used RLVR algorithms, such as GRPO, compute the advantage of each token in a response by comparing its reward to those of other responses for the same prompt. When all responses yield identical accuracy-either all correct or all incorrect- advantages uniformly degrade to zero, leading gradient vanishing for policy updates and resulting in training inefficiency [Shao et al. 2024 Yu et al., 2025b). Recent study [Wen et al.) 2025) further shows that RLVR training can be more efficient with problems of appropriate difficulty. Considering this, we select synthetic problems of appropriate difficulty based on the initially trained model's s accuracy on them. Specifically, we sample multiple responses per synthetic problem using the initially trained model and retain only those whose accuracy falls within target range [acc aCChigh] (e.g.. [25%, 75%)). This strategy ensures that the model engages with learnable problems, enhancing both the stability and efficiency of RLVR training.
5

---

Model
GSMBK
MATH 500
Minerva Math
Olympiud Bench
GaoKao 2023
AMC23
AIME24 (Avg 1/32)
AIME25 (Avg 1/32)
Avg.
Quen. 2.5 3B Baae
Qwen2.5.3B
Qwen2.5-3B-11
84.2
62.2
26.5
27.9
53.5
32.5
6.715.00
0.0/2.3
36.7
BascRL-3B
86.3
25.4
31.3
57.9
10.0/9.9
6.7
40.4
SwS-315
87.0
69.6
27.9
34.8
59.7
47.5
10.0/8.4
6.7
42.9
+0.07-1.5
+0.07+3.6
11aMMA
3.1 8B Instruct
85.6
48.2
24.6
18.8
39.7
22.5
6.7/3.1
3,3/2.2
31.1
Baseline RL
88.3
58.4
31.2
23.4
49.6
30.0
16.7 /9.8
6.7/5.0
38.0
SwS-LLaMA-88
90.5
60.2
25.8
40.0
16.7/11.2
6.7.6.88
40.3
A
+2.2
+1.8
+2.3
+2.4
-0.5
+10.0
+0.0741A
+0.07
+2.3
Quan 2.6 7B Base
Owen2.5-7B
88.1
63.0
30.5
6.715.4
0.0112
38.3
Qwen2.5-7B-IT
91.7
75.6
38.2
40.6
63.9
50.0
16.7 10.5
13:3/6.7
48.8
0pen-Reaonee-78
45.6
72.0
72.5
10.07
13.3/17.9
53.3
SimpleRI-BBi BBsse-7B
90.8
77.2
35.7
41.0
66.2
62.5
13.37 14.8
6.7/6.7
49.2
B.seRI.7B
92.0
78.4
36.4
41.6
63.4
45.0
10.07 14.5
6.7/6.55
46.7
SwS-7B
93.9
82.6
41.9
49.6
71.7
67.5
26.71 18.3
20.0/ 18.5
56.7
A
+1.9
+4.2
+5.5
+8.0
+8.3
+22.5
+16.7/43.8
+13.37+12.0
+10.0
Quen
2.5 TB Math
Qwen2.5 Maan2..5-ahhh71
0.0/2.9
32.2
Qwen2.5.Main Math-7--
93.3
80.6
36.8
36.6
64.9
45.0
6.717.2
13.3/6.2
47.2
SimpleRL-Matthh1
89.8
78.0
27.9
43.4
64.2
62.5
23.3
20.0/ 13.6
51.1
Oat-Zero-7B
90.1
79.4
38.2
42.4
67.8
70.0
43,3/29.3
23.3/118
56.8
BascRL-Math-78
90.2
78.8
37.9
43.6
64.4
57.5
26.7/
20.0/14. 14.0
51,9
SwS-Mathi-BB
91.9
83.8
47.7
71.4
70.0
33,3/25.9
26.7 18.2
58.3
+1.7
+5.0
#3.6
+4.1
+7.0
+12.5
+6.7/
/6.7/
+6.4
Qwen
Qwen2,5-328
90.1
66.8
34,9
29.8
55.3
50.0
10.0/4.2
6.7
42.9
Qwen2.5-32B-IT
95.6
72.5
25.3/1500
20.0/
Open Reasoner 32B
95.5
82.2
46.3
54.4
75.6
57.5
23.3/
33.3/
58.5
SimpleRI-Base-
95.2
33.31
10.0/
BaseRL- 32B
96.1
85.6
43.4
54.7
73.8
85.0
40.0 30.7
6.7/24.6
60.7
A
+0.2
+3.8
+3.7
+5.8
+6.5
+5.0
43.3142.3
+33.3147.2
+7.7
Table 1: We report the detailed performance of our SwS implementation across various base models and multiple benchmarks. AIME is evaluated using two metrics: Avg@) @ (single-run performance) and Avg@32 32 (average over 32 runs).
2.5 Augmented Training with Synthetic Problems
After the rigorous problem generation, answer generation, and verification, the allocation budget of synthetic problems in each category is further adjusted using the weights in Eq. 5 to ensure their comprehensive and efficient utilization, resulting in X We incorporate the retained synthetic problems X into the initial training set Xs, forming the augmented training set X [Xs: X We then continue training the initially trained model on X second stage of augmented RLVR. targeting to mitigate the model's weaknesses through exploration of the synthetic problems.
3 Experiments
3.1 Experimental Setup
Models and Datasets. We employ the Qwen2.5-base scries Yang et al. [0224[bb with model sizes from 3B to 32B in our experiments. To further demonstrate the generalizability of our method, we also adopt the LLaMA-3.1-SB-Instee tation. For concept extraction and problem generation, we employ the LLaMA-3.3-70EB - model [Grattafiori al. 2024). , and for concept embedding, we use the L..MAA3.1-88 --se---se model. To verify the quality of the synthetic questions, we use both the LLaMA-3.3: 3-0-----3--BB..... Instrutt and addi- tionally Qwen-2.5-722 2-2---2---2BI YYang al., 2024a to evaluate them and filter out the low-quality samples. For answer generation, we use Skywork-OO ORR 1-Math-7BB (He et al. 2025 for training models with sizes up 7B, and[QnQ QwQQ33B Team 2025) for the 32B model experiments. We employ the SwS pipeline to generate 40k synthetic problems for each base model. All the prompts for each procedure
Grattatiori et al. 2024 model for SwS data augmen-

---

Consistently
Categories
14
13A
last BL Synt RL.
Zore Ratto (18)
1.0
9.3
8.8
0.3
4.2 3.5
3.8
1.9
1.8
0.9
Algebra
Counting & Probubility
Geometry
Intermedinte Algehrs
Number" Theery
Prealgebro
Preceloslus
Figure 4: The ratios of consistently failed problems from different categories in the MATH-124 training set under different training configurations. (Base model: Qwen2.5-7B).
Weakness Identification
Concepts Extraction
Problem Generation
Quality Verification
Answer Generation
Difficulty Filtering
Augmented Training
GPU Hours (b)
2,075
1.6
1,940
975
4,608
768
9,543
Data Quantity
12k
1,339
1,000k
842k
302k
253k
52k
/
836.9
515.5
863.6
329.4
Table 2: Stage-wise GPU Hours for SwS Experiments with Qwen2.5-7B.
in SwS can be found in Appendix K We adopt GRPO (Shao et al. 2024 as the RI. algorithm, and
full implementation details are in Appendix C
For the initial training set used in the preliminary RL training for weaknesses identification, we employ the MATH-12K [Hendrycks al. [021]] for models with sizes up 7B. As the 14B and 32B models show early saturation on MATH- 12k, we instead use a combined dataset 17.5k samples from the DAPO Yu et al. 2025b English set and the LightR [Wen al.
2025
Stage-2 set.
Evaluation. We evaluated the models on wide range of mathematical reasoning benchmarks. includ- ing GSM8K [Cobbe et al. 2021]. MATH-500
Lightman et al. [2023]. Minerva Math [Lewkowycz al. 2022 Olympiad-Bence [He al. 2024 maokao-20233 [Zhang al. 2023), AMC [MC]] aa.
and AIME [MAA] b). We report Pass @ (Avg@) 1) accuracy across all benchmarks and addition- ally include Avg @ metric for the competition-lest AIME benchmark enhance evaluation robustness. For detailed descriptions of the evaluation benchmarks, see Appendix]] J]
Baseline Setting. Our baselines include the base model, its post-trained Instruct version (e.g.. Qwenn.5-77BIInstruce and the initial trained model further trained on the initial dataset for the same number of steps as our augmented RL training the baselines. To further highlight the effectiveness of the SwS framework, we compare the model trained on the augmented problem set against recent advanced RL-based models, including SimpleRL Zeng et al. 2002). Open Reasoner (Hu al.
2005). PRIME (Cui et al. 2025). and Oat-Zero Liu et al 2025b
3.2
Main Results
The overall experimental results are presented in Table Our SwS framework enables consistent performance improvements across benchmarks of varying difficulty and model scales, with the most significant gains observed in models greater than 7B parameters. Specifically, SwS-enhanced versions of the 7B and 32B models show absolute improvements of +10.0% and +7.7%, respectively, underscoring the effectiveness and scalability of the framework. When initialized with MATH- 12 SwS yields strong gains on competition-levs benchmarks, achieving +16.7% and +13.3% on AIME24 and AIME25 with Qwen2.5-7B These results highlight the quality and difficulty of the synthesized samples compared to well-crafted human-written ones, demonstrating the effectiveness of generating synthetic data based on model capabilities to enhance training.
3.3 Weakness Mitigation from Augmented Training
The motivation behind SwS is to mitigate model weaknesses by explicitly targeting failure cases during training. To demonstrate its effectiveness, we use Qwen2.5-7BB to analyze the ratios of consistently failed problems in the initial training set (MATH- 12k) across three models: the initially

---

AIME24 (Pass @32
Intermediate Algebra
Number Counting & Geometry
Model
GSMSK
Prealgebra
Algebra Precalculus
Theory
Probability
Strong Student
92.0
13.8
87.7
58.7
93.8
63.2
86.4
71.2
66.8
Weak Teucher
93.3
88.2
64.3
71.2
93.0
81.4
63.0
Trained Student
93.6
17.5
90.5
64.4
97.7
74.6
95.1
80.4
67.5
Table 3:
Performance on two representative benchmarks and category-specifi results on MATH-500 of the weak teacher model and the strong student model.
MATH
Mínerva Math
Olympiad Bench
GaoKao 2023
AIME24
AIME25
Model
GSMBK
500
AMC23
(Avg@ 1/ 32)
(Avg@ 1/32)
Avg.
Swen2.5-14BB IT
94.7
79.6
41.9
45.6
68.6
57.5
16.7/11.6
6.7/ 10.9
51.4
BaseRI.
94.5
85.4
44.1
52.1
20.0/21.6
20.0
56.6
SwS SE
95.6
85.0
46.0
53.5
74.8
67.5
20.0/19 19.8
20.0/ 17.8
57.8
A
+1.1
-0.4
+1.9
+1.4
+3.1
+2.5
+0.07-18
+0.07 -4.
+1.2
Table 4: Experimental results of extending the SwS framework to the Self-evolving paradigm on the Owen2.5-144BIIntrrac model.
trained model, the model continued trained on the initial training set, and the model trained on the augmented set with synthetic problems from the SwS pipeline. As shown in Figure 4 continued training on the augmented set enables the model to solve greater proportion of previously failed problems across most domains compared to training on the initial set alone, with the greatest gains observed in Intermediate Algebra (20%). Geometry (5%), and Precalculus (5%) its weakest areas. Notably, these improvements are achieved even though each original problem is sampled four times less frequently in the augmented set than in training the original dataset alone, highlighting the efficiency of SwS-generated synthetic problems in RI. training.
3.4 GPU Hours Analysis for SwS
For the specific GPU hours each stage, we use the Qwen2.5 7B experiment example and report the GPU hours for each SwS stage in the Table 2 All time measurements are based on NVIDIA A100 40G GPUs. Notably, the total time spent all problem synthesis stages (8,292.6 GPU hours) is actually less than that required by the final augmented training via RI. (9,543 GPU hours). This comparison highlights the rationale and necessity for allocating computational resources to data augmentation prior to RL. Within the problem synthesis pipeline, the most time-consuming component is Answer Generation, as requires powerful reasoning model to ensure answer correctness. In contrast, other stages mainly involve shorter inference, thus consumes less time.
4 Extensions and Analysis
4.1 Weak-to-Strong Generalization for SwS
helps ensure answer quality. However, when training the top- pe-pefforming reasoning model, no stronger model exists to produce reference an- swers for problems identified as its weaknesses. To explore the potential of applying our SwS pipeline to enhancing state-of-the-aat models, we extend to the Weak-lo-Strome Generalization [Burns et al., 2023 setting by using generally weaker teacher that may outperform the stronger model in specific domains to label reference answers for the synthetic problems.
Employing a powerful frontier model like QwQ Team 2025
Intuitively, using weaker teacher may result in mislabeled answers, which could significantly impair subsequent RL training. However, during the difficulty filtering stage, this risk mitigated by using the initially trained policy to assess the difficulty of synthetic problems, as it rarely reproduces the same incorrect answers provided by the weaker teacher. As a byproduct, mislabeled cases are naturally filtered out alongside overly complex samples through accuracy- based screening. The experimental analysis on the validity of difficulty-level filtering in ensuring label correctness is presented in Table 6
We use the initially trained Qwen2.5-7B-BBB Base as the student and (0wen225-Math--BB-nnntt as the teacher. Table BubleBprra presents their performance on popular benchmarks and MATH- 12k categories, where
8

---

15.
(a) Overall Accuncy
160
(b) Competition Level Accurcy
33.
[cl Training Banch Accumsc (S)
$5.3
Soyp 68 Roog
Seppt Tain
25.
120 342
330 120
110 322
Figure 5: Comparison of accuracy improvements using (a) Pass@1 1 on full benchmarks evaluated in Table I and (b) Avg @32 on the competition-level benchmarks. (c) illustrates the proportion of prompts within batch that achieved 100% correctness across multiple rollouts during training.
Overral
Compettition Level
Training Ratch (S4)
123
95.3
26.3
12:3
Modan
Training Steps
Training Staps
Training Steps
Figure 6: Comparison of incorporating synthetic problems of varying difficulty levels during the augmented RL training. For a detailed description of accuracy trends on evaluation benchmarks and the training set, refer the caption in Figure 5
the student model generally outperforms the teacher. However, as shown Table 3 the student policy further improves after training on weak teacher-labeled problems. This improvement stems from the difficulty filtering process, which removes problems with consistent student-teacher disagreement and retains those where the teacher is reliable but the student struggles, enabling targeted training on weaknesses. Detailed analysis can be found in Appendix
4.2 Self-evolving Targeted Problem Synthesis
In this section, we explore the potential of utilizing the Self-evolving paradigm to address model weaknesses by executing the full SwS pipeline using the policy itself. This self-evolving paradigm for identifying and mitigating weaknesses leverages self-consistency to guide itself to generate effective trajectories toward accurate answers (Zuo et al., [025], while also integrating general instruction-folloiing capabilities from question generation and quality filtering to enhance reasoning.
We use Qwwn2.55-1BB 144--Inssructtt as the base policy due to its balance between computational efficiency and cestruction-folloiiii performance. The results are shown in Table where the self-evolving SwS pipeline improves the baseline performance by 1.2% across all benchmarks, especially on the middle-level benchmarks like Gaokao and AMC. Although performance declines on AIME, we attribute this to the initial training data from DAPO and LightR I already being specifically tailored to that benchmark. For further discussion of the Self-evolve SwS framework, refer to Appendix
4.3 Weakness-driven Selection
In this section, we explore an alternative extension that augments the initial training set using identified weaknesses and a larger mathematical reasoning dataset. Specifically, we use the Qwen2.5 7B model, identify its weaknesses on the MATH- 12k training set, and retrieve augmented problems from Big-Math [Albalak et al. 2025) that align with its failure cases, incorporating them into the initial training set for augmentation. We employ a category-specific selection strategy similar to the budget allocation in Eq. 5 using KNN [Cover and Hart 1967 to identify the most relevant problems within each category. The total augmentation budget is also set to 40k. We compare this approach augmented set incorporated with randomly selected problems from Big-Math. Details of the selection procedure are provided in Appendix
to baseline where the model is trained
As shown in Figure the model trained with weakness-driven augmentation outperforms the random augmentation strategy terms of accuracy on both the whole evaluated benchmarks (Figure and the competition-leve subset (Figure 5.b). demonstrating the effectiveness of the weakness-driven

---

Original Problem
Synthetic Problems of Diverse Difficulty levels
Equilateral ABC side length 600. Q ousside the plane of AARC and are on opposiic sides of the plane. Furthemore.
cone being langent a sphere. The radius of the sphere is and the height of cone A is h. If the ratio the height of cone B to the beight of cone A is k. find the ralio of the surface arca of cone 8 to the surface arca of cone A. Answer: A-, Model Accuracy: 100%
Simple: Two ouncs, and
similar,
PA = PB PC, and QA Q8 QC, and the planes
PAR and QAR form
120" dihodral angle (the angle belween the two planes). There is point O whose distance from ench of A,8,C, and is d. Find d.
Medium: In circle with radius r. two langents are drawn from a point such that the angle between them is 60". If the length of each tangent is ry3 find the distance from P to the center.
Model Accurncy: 50%
Extracted Concepts
Hard: In triangle ABC, let be the incentor and E the excenter opposite A. AE 5, Al
Geometric shapes and their properties
3. and EJ tangent to the incircle at D. find the radias.
Propertis of equilaterul triangles
Answer: 2, Model Accuracy: 6.25%
Understanding of points and planes in 3D space
Unsolvable: In triangle ABC, with AB 7, = 9, and L.A = 60", let D be the midpoint BC. Given BD is more than DC, find A.D.
Distance and midpoint formulas in 3D space
Progerties of perpendicular lines and planes
15/2, Model Aceuracy: 19%
Illustration of a geometry problem from the MATH-12K failed set, with extracted concepts Figure 7:
and conceptually linked synthetic problems across different difficulty levels.
selection strategy. In Figure it is worth noting that the model quickly fits the randomly selected problems in training, which then cease to provide meaningful training signals in the GRPO algorithm. In contrast, since the failure cases highlight specific weaknesses of the model's 's capabilities, the problems selected based on them remain more challenging and more aligned with its deficiencies, providing richer learning signals and promoting continued development of reasoning skills.
4.4 Impact of Question Difficulty
We ablate the impact of the difficulty levels of synthetic problems used in the augmented RL training. In this section, we define the difficulty of a synthetic problem based on the accuracy of multiple rollouts generated by the initially trained model, base from Qwen2.5-7B We incorporate synthetic problems of three predefined difficulty levels-simple, medium, and hard -into the augmented RL. training. These levels correspond to accuracy ranges of [5, 7]. [3, 5]. and [1, 4] out of 8 sampled responses, respectively, For each level, we sample 40k examples and combine them with the initial training set for a second training stage lasting 200 steps.
The experimental results are shown in Figure 6 Similar to the findings in Section 4.3 the model fits more quickly on the simple augmented set and initially achieves the best performance across all evaluation benchmarks, including competition llevel tasks, but then saturates with no further improvement. In contrast, the medium and hard augmented sets lead to slower convergence on the training set but result more sustained performance gains on the evaluation set, with the hardest. problems providing the longest-lasting training benefits.
4.5
Case Study
Figure 7 presents an illustration of geometry failure case from the MATH-12K training set, ac- companied by extracted concepts and our weakness-driven synthetic questions of varying difficulty levels, all closely aligned with the original question. The question focuses on three-dimensional distance and triangle understanding, with key concepts such as "Properties of equilateral triangles" and "Distance and midpoint formulas in 3D space" representing essential knowledge required to solve the problem. Notably, the corresponding synthetic questions exhibit similar semantics-suee as "finding distance" in Medium and "understanding triangles" in Hard. Practicing on such targeted problems helps mitigate weaknesses and enhances reasoning capabilities within the relevant domain.
5 Conclusion
In this work, we introduce a Self-aware Weakness-driven Problem Synthesis (SwS) framework (SwS) in reinforcement learning for LLM reasoning, which synthesizes problems based on weaknesses identified from the model's failure cases during a preliminary training phase and includes them into subsequent augmented training. We conduct a detailed analysis of incorporating such synthetic problems into training and find that focusing on the model's failures can enhance reasoning gener- alization and mitigate its weaknesses, resulting in overall performance improvements. Furthermore, we extend the framework to the paradigms of toouk-tt-Strng Generalization, Self-evolving, and toakknes-driver Selection, demonstrating its comprchensiveness and robustness.
10

---

References
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09330 2023.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le. Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.1716. 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-22) 1967.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025.
Hugging Face. Open fully open reproduction of deepseek-rl, January 2025. URL. https //github. www.ooogggooo.....coocoooooooooe ngf open-r1
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, er al. The Ilama 3 herd of models. arXiv preprint arXhv:2407.2178. 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Iims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 250.::::5501.9999 /2948, 2025.
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. http:s//cccpriiioooss-yatorgeee notion. witsssttttsseeee Siitte//kkwkkkkko hrtaaeoeooo.onooooooooooooooooea w0w.ca0626800099066400000111.0..
Reaonser-
2025. Notion Blog.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4): 0-6, 2021.
Jingcheng Hu, Yinmin Zhang, Qi Han. Daxin Jiang, Xiangyu Zhang, and Heung Yeung Shum. Open-reasoner-7eeoo An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.24290. 2025.
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.02333, 2024.

---

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, al. Openai ol system card. arXiv preprint 1672200, 2024.
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in knowledge-inteniiee tasks, Advances in Neural Information Processing Systems, (55-48573-48600 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E, Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk k Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 05:3843-38575 2022.
Chen Li, Weiqi Wang. Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint ar MrXiv2243.007006 2024a.
Dawei Li, Bohan Jiang, Liangji Huang. Alimohammad Beigi. Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of Ilm-as-a-judge. arXiv preprint 1rXiv:2411.16594 2024b.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502. 1886, 2025a.
Zhong-Zhi Li, Xiao Liang, Zihao Tang, Lei Ji, Peijie Wang. Haotian Xu, Haizhen Huang, Weiwei Deng, Ying Nian Wu, Yeyun Gong, et al. Tl; dr: Too long, do re-weighting for effcient Ilm reasoning compression. arXiv preprint 25xv:2506.00678, 2025b.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system system 2: A survey of reasoning large language models. arXiv preprint urXiv:2502.17419 2025c.
Xiao Liang, Xinyu Hu, Simiao Zuo, Yeyun Gong, Qiang Lou, Yi Liu, Shao-Lun Huang, and Jian Jiao. Task oriented in-domain data augmentation. arXiv preprint arXiv:2406. 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao. Augmenting math word problems via iterative question composing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages -26133, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -zeero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Dakuan Lu, Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. Sep- 116k: A high-quality problem-solution dataset and generalized pipeline for automated extraction in the higher education science domain, 2025. URL htttpp:////// org/ abs/2501... 15587
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao. Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruet. arXiv preprint 2xXxv:2308.0958: 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1. 1.5b model by scaling rl. DeepScaleR Notion Page 2025. Notion Blog.
12

---

Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint #rXiv:240100866, 3, 2024.
MAA. American mathematics competitiions (AMC 10/12). Mathematics Competition Series, 2023a.
URL http:p///maaa..gg///mmaahcooooop
MAA. American intiiattinnniitti exaammmatticeeaamaaattii (AIME). Mathematics Competition Series,
2024b. URL
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19333 2025.
Hicu Nguyen, Zihao He, Shoumik Atul Gandre, Ujjwal Pasupulety, Sharanya Kumari Shivakumar, and Kristina Lerman. Smoothing out hallucinations: Mitigating Ilm hallucination with smoothed knowledge distillation. arXiv preprint arXiv:2502. 11306, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and Rui Yan. Mathfusion: Enhancing mathematic problem-solving of Ilm through instruction fusion. arXiv preprint arXiv:2503. 16212, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint wwxiv:2402.03300 2024.
Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, and Lin Yan. Exploring data scaling trends and effects in reinforcement learning from human feedback. arXiv preprint arXiv:2503.22 22230, 2025.
Guangming Sheng, Chi Zhang, Zilingfeng Yc, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhí framework. arXiv preprint vrXiv:2409.19255 2024.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation and synthesis: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 930-957, 2024.
Zhengyang Tang, Xingxing Zhang, Benyou Wang. and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In International Conference Machine Learning, pages 47885-47900. PMLR, 2024.
Kimi Team, Angang Du Bofci Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with llms. arXiv preprint 2.X.i::501.12599) 2025.
Qwen Team. Qwq-32h: Embracing the power of reinforcement learning, March 2025. URL htttps://vennln http://///geenn.niit...........
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty rejection tuning for mathematical problem-solving. Advances in Neural Information Processing Systems, 37:7821 2024.
13

---

Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-I1 1: A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems, 37:34737-34774, 2024.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:23/2.08935, 2023.
Shu Wang, Lei Ji, Renxi Wang, Wenxiao Zhao, Haokun Liu, Yifan Hou, and Ying Nian Wu, Explore the reasoning capability of lims in the chess testbed. arXiv preprint wrXiv:241..06653 2024.
Yu Wang, Nan Yang, Liang Wang. and Furu Wei. Examining false positives under inference scaling for mathematical reasoning. arXiv preprint (iXixi552220621,,, 2025.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint 2rX3v:2503. 10460, 2025.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008-59033, 2023.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint Arviv:2504. 1343, 2025.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 122444 2023.
An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15115:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint rxXiv:2409.12122 2024b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 2025.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03377 2025.
Bin Yu, Hang Yuan, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. Long-short chain- of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models. arXiv preprint arXiv:2505.0334m 2025a.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint 230.::2309.12284, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 13300.1444. 14476, 2025b.
Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Ziyi Yang. Mahmoud Khademi, Hany Awadalla, Junjic Wang, et al. Chain-of-reasoning:: Towards unified mathematical reasoning in large language models via a multi-paradigm perspective. arXiv preprint arXiv:2501.
2025c.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint 2rXiv:554.005111 2025.
14

---

Yang Yue. Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv:2503.1888 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772 2024a.
Hengyuan Zhang, Yanru Wu, Dawei Li, Sak Yang, Rui Zhao, Yong Jiang, and Fei Tan. Balancing speciality and versatility: coarse to fine framework for supervised fine-tuning large language model. In Findings of the Association for Computational Linguistics ACL 2024, pages 7467-7509, 2024b.
Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. Process-based self-rewarding language models. arXiv preprint arXiv:2503.03744 2025.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models gaokao benchmark. arXiv preprint arXiv:2305.1244 /24744 2023.
Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training. arXiv preprint arXiv:2503. 19633, 2025a.
Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models. arXiv preprint arXiv:2503.023D4 2025b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1tt.:::999000599.. 2019.
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning, 2025. URL https 7/arxiv org/ abs///4...222... 16084
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and instruction in the paper clearly reflect its contribution and scope, with the key contributions summarized in the abstract's final paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper has discussed the limitations in the "Limitations" section of the Appendix.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper has includes the steps to implement all the experiments, with more detailed guidance in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: Since we are doing synthetic data in this paper, and the release of such data along with the code will be under review by the affiliations. However, code is attached in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
hhollidc/gaidee/dddessiniissssss Pilicc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//apps.cc/pullic/uuiiee
swwsssssiioyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper has clearly described the all the datasets, benchmarks and imple- mention details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since data synthesis expensive to reproduce for measuring error bars, do not include it in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: discussion of computational resources provided in the Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research in this paper well aligned with NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: This paper does not state the risks for the used models, since such information is available the technical reports of the used models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We've cited clearly for the assets used in this paper and followed their licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the code and the data introduced in this paper would be open-sourced, but they need time to be reviewed by the affiliation administrators.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We' ve've declared the LLM usage in the Openreview submission.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:s://neurps.... http://urrirssccooooofeeeess//// cc/Coonferencee//2255LLL. for what should or should be described.
22

---

Appendix Contents for SwS
Discussions, Limitations and Future Work
24
Related Work
24
Implementation Details
25
Training
25
Evaluation
26
Motivation for Using RL Weakness Identification
26
E
Data Analysis of the SwS Framework
27
Detailed Data Workflow
27
Difficulty Distribution Synthetic Problems
27
F
Co-occurrence Based Concept Sampling
28
Details for Weak-to-Strong Generalization SwS
29
H Details for Self-Evolving in SwS
30
Details for Weakness-driven Selection
Evaluation Benchmark Demonstrations
32
Prompts
34
K.1 Prompt for Category Labeling
34
K.2 Prompt for Concepts Extraction
37
K.3 Prompt for Problem Synthesis
37
K.4 ..4 Prompt for Quality Evaluation
38
23

---

A Discussions, Limitations and Future Work
This paper presents a comprehensive Self-aware Weakness-drives Problem Synthesis (SwS) frame- work to address the model's reasoning deficiencies through reinforcement learning (RI.) training. Although the SwS framework is effective across wide range of model sizes, there are still several limitations to it: (I) Employing both a strong instruction model and answer-labeling reasoning model may lead to computation and time costs. (2) Our framework mainly focuses on the RL setting, our primary goal is to mitigate the model's weaknesses by fully activating its inherent reasoning abilities without distilling external knowledge. Exploring how to leverage a similar pipeline for enhancing model capabilities through fine-tuning distillation remains open direction for future research. (3) The synthetic problems generated by open open-source instruction models in the SwS framework may still lack sufficient complexity to elicit the deeper reasoning capabilities of the model, especially on more challenging problems. This limitation is pronounced in the Self-evolving 14B model for problem generation, with performance improvements limited to only moderate or simple benchmarks. This raises questions about the ii the main experiments on lop-challenging benchmarks like AIME. One potential strategy is to use Evolve-Insruett Xu et al. [2023] Luo et al. [2023) to further refine the generated problems the desired level of difficulty. However, how to effectively raise the upper bound of difficulty in synthetic problems generated by instruction models remains an open problem and warrants further exploration.
setting in Section 4.2 which relies solely
actual utility of problems generated from the LLLMA--33-70B-Innsnne
In the future, we aim to identify model weaknesses from multiple perspectives beyond simple answer accuracy, with the goal of synthesizing more targeted problems to improve sample efficiency. Addi- tionally. we plan to extend the SwS framework to more general tasks beyond reasoning, incorporating an off-the-shel reward model provide feedback instead of verifiable answers. Lastly, we also seek to implement the SwS pipeline in more advanced reasoning models equipped with Long-CoT capabilities, further pushing the boundaries of open-source large reasoning models.
B Related Work
Recent advancements have significantly enhanced the integration of reinforcement learning (RL) with large language models (LLMs) (Ziggle et al. [2019] Ouyang et al., 2022] particularly in the domains of complex reasoning and code generation Guo et al. (225). J. A Algorithms such as Proximal Policy Optimization (PPO) [Schulman al., 2017] and Generalized Reinforcement Preference Optimization (GRPO) et al.) [024] have demonstrated strong generalization and effectiveness in these applications. In contrast to supervised fine-tuning (SFT) via knowledge distillation Kang et al. 2023). Zhang et al. [024bb). Yu et al. [2025a], RL optimizes a model's reason capabilities on its own generated outputs through reward-driven feedback, thereby prompting stronger generalization. In contrast, SFT models often depend rote memorization of reasoning patterns and solutions (Chu ctt al. 2025), and may produce correct answers with flawed rationales (Wang et al. 2025). In LLM reasoning, RL strengthens policy exploration and improves reasoning performance by using the verified correctness of the final answer in the responses as reward signals for training [Luong et al.
2024 which is commonly referred to as reinforcement learning with verifiable rewards (RLVR) Yue
ct al. 2025].
Robust RLVR for LLM Reasoning. Scaling up reinforcement learning for LLMs poses significant challenges terms of training stability and efficiency. Designing stable and efficient supervision algorithms and frameworks for L.MSs has attracted widespread attention from the research community.
To address the challenge of reward sparsity in reinforcement learning, recent studies have explored not only answer-based rewards but also process-leve reward modeling [Cobbe et al.. 2021 Lightman
cet al. 2023 Wang et al. 2023 Zhhang et al., 2025). enabling the provision of more fine-grained reward
signals throughout the entire solution process Wu et al. 2023). Wang et al. [2023] successfully incorporated process reward model (PRM), trained on process-level labels generated via Monte
Carlo sampling at each step. into RL training and demonstrated its effectiveness. Beyond RL training.
PRM can also be used to guide inference (Cobbe et al. [021] and provide value estimates incorporated with search algorithms [Zhang et al.] 2024a| Guan et al. 2025 However, Guo et al. (2025) found that the scalability of process-level RI. is limited by the ambiguous definition of "step" and the high cost of process-level labeling. How to effectively scale process-level RL remains open question.
24

---

Recent efforts in scaling up RLVR optimization have focused on enhancing exploration Yuu et al.
2025b Yuan al., 2025 Liu et
20256 Yeo al., 2025] and adapting RL to the Long-Cor CoTT
Li et al., 2025c). Yu et al. (025b found that the KI. constraint may limit exploration under RLVR, while Liu et al. [2025b] proposed removing variance normalization in GRPO to prevent length bias. Building on PPO. Yuan et al. (2025 found
conditions [Jacch et al. 2024 Guo et al. 2025
that pre-training the value function prior to RL training and employing a length-adaptive GAE can improve training stability and efficiency in RLVR, preventing it from degrading to constant baseline in value estimation.
Data Construction in RLVR. Although RL training on simpler mathematical questions can partially
elicit model's reasoning ability [Zeng et al. 2025], , the composition of RL training data critical
for enbancing the model's ss reasoning capabilinees Luo et al. [225 Yu et al.) [025bb Li et al. 2222a
Hu et al. 2025 He et al., 2025 Shen et al. 2025 Carefully designing a problem set with difficulty
levels matched to the model's 's abilities and sufficient diversity can significantly improve performance.
In addition, the use of curriculum learning has been shown to improve the efficiency of reinforcement learning (Shi et al., [025]. In this work, we propose generating synthetic problems based on the model's 's weaknesses for RL training, where the synthetic problems are tailored to align with the model's capabilities and target its areas of weakness, fostering its exploration and improving performance.
Data Synthesis for LLM Reasoning Existing data synthesis strategies for enhancing LLM reasoning
primarily concentrate on generating problem-response pairs (Huang et al.
[024]] Tang et al. 2024
Yu et al. 2023 Zhao et al.
2025b
Liang et al., 2024 Luo et al. 2023 Liu et al., 2025a Wang
et al. 2024 Li al. 2024b Tan et al.) 2024 Pei et al. 2025 or augmenting responses to existing
questions Toshniwal et al. 2024] Tong et al. 2024 He et al. 2025 Face, 2025 Wen et al.) 2025 Yu
(ct al. 2025e] [i et al. al.] 2025b). typically by leveraging advanced LLMs to produce these synthetic examples. A prominent line of work focuses on extracting and recombining key concepts from seed problems. KP-Math [Huang et al. 2024] and MathScale [Tang et al. 2024) decompose seed problems into underlying concepts and recombine them to create new problems, leveraging advanced models to generate corresponding solutions. PromptCoT [Zhao et al., 2025b) also leverages underlying concepts, but focuses generating competition-lewe problems. DART-Math (Tong al. 2024) introduces a difficulty-aww aware framework that prioritizes the diversity and richness of synthetic responses to challenging problems.
Recently, several studies have emerged aiming to construct distilled datasets to better elicit the reasoning capabilities of LLM. [Guo et al. [025]. Several works (Face) 2025 Ye et al. 2025 Muennighoff et al. 2025 Lu et al.) 2025| Zhao et al. 2025a] employ advanced Long- models to generate responses for distilling knowledge into smaller models. However, a significant disparity in capabilities between the teacher and student models can lead to hallucinations
the student's outputs [guyen et al. 2025] and hinder generalization to -fitiriibittttonnnnn scenarios [Chu et al.) 2025). In contrast, our framework under the RL setting enables the model to identify and mitigate its own weaknesses by generating targeted synthetic problems from failure cases, thereby encouraging more effective self-improvement based on its specific weaknesses.
C Implementation Details
C.1 Training
We conduct our experiments using the verl [Sheng et al.] 2024] framework and adopt GRPO (Shao
et al. 2024] as the optimization algorithm. For all RL training experiments, we sample 8 rollouts per problem and use batch size of 1024, with the policy update batch size set to 256. We employ a constant learning rate of 5 x 10-7 with 20-step warm-up. and set the maximum prompt and response lengths to 1.024 and 8, tokens, respectively. We do not apply a KL penalty, as recent studies have shown it may hinder ex exploration and potentially cause training collapse (Yuan et al. 2025 Liu et al. 20256 Yu et al.) 2025b). In the initial training stage, we train the model for 200 steps. During augmented RL training, we continually train the initially trained model for 600 steps on the augmented dataset incorporated with synthetic problems, using only prompts with an accuracy between acClower 10% and acCupper 90% as determined by the online policy model for updates.
The probability ratio clipping ranges in Eq. is set to 0.20 and eh 0.28.
25

---

Weakness Identification of Failure Cases, 1000000
Remove Problems with Inconsistent Labeled Answers.
813639
Generating Synthetic Problems.
36 problems Difficulty
Filter out Undesirable Problem Types RL.
Filter out Problems with Low Quality.
175140
137447
41726
17545
1905
Initial training dato
Foiled Problems
All Synthetic
RL-style Problems
High-quality Probiems
Answer-verified Problems
Difficuly-ffittreed Problems
Problems
Figure 8: Demonstration of the SwS data workflow by tracing the process from initial training data to the final selection of synthetic problems in the 32B model experiments. For better visualization, the bar heights scaled using the cube root of the raw data.
Since the training data for the 32B and 14B models (a combination of DAPO (Yu et al. 2025b and LightR1 (Wen et al., 2025] subsets) lack human-annotated category information, we leverage the 1--MA--3.-70B-Instnee model to label their categories. This ensures consistency with our SwS pipeline, which combines concepts within the same category. The prompt is presented in Prompt
C.2 Evaluation
For evaluation, we utilize the vLLM framework [woon et al. 2023 and allow for responses up to 8, 3,192 tokens. For all the benchmarks, Pass @ is computed using greedy decoding for baseline models and sampling (temperature 1.0, top-p 0.95) for RL-trained models. For Avg@32 32 on competition-leve benchmarks, we sample 32 responses per model with the same sampling configuration as used in RL training. We adopt hybrid rule-based verifier by integrating Math- Math-Verify and the PRIME-RL verifier [Cui et al. 2025). as their complementary strengths lead to higher recall. For all the inference, the default chat template and enable CoT prompting by appending the instruction: "Let's think step by step and output the final answer within "i boxed()" after each question.
D Motivation for Using RL in Weakness Identification
In our SwS framework, we propose utilizing an initial RL training phase for weakness identification. However, one might argue that there are simpler alternatives for weakness identification, such as directly sampling training problems from the base model or applying supervised fine-tuning before prompting the model to answer questions. In this section, provide an in-depth discussion on the validity using problems with low training efficiency during the initial RL phase as model's weaknesses.
We first compare the performance of the Base model, SFT model, and Initial RL model by sampling on the training set, where the SFT model is obtained by fine-tuning the Base model for epoch on human-written solutions. For each question, we prompt the model generate responses and report the proportion of problems for which none of the responses are correct in Figure For the Base model, failures may be attributed to its insufficient alignment with sesoonnnn-speciiie tasks. Results from the initial RL model show that the Base model can quickly master such questions through RL, indicating that they do not represent challenging weaknesses. Furthermore, the heavy reliance on the prompt template of the Base model Liu et al. [2025b] reduces its robustness of weakness identification. For the SFT model, there are three main drawbacks regarding weakness identification: (I) The dilemma of training epochs-tor too many epochs leads to memorizing labeled solutions, while too few epochs fails to align the model with the target problem distribution; (2) SFT is prone to hallucination [Chu et al. 2025 Wang al. 2025]: ; and (3) Ensuring the quality of labeled solutions difficult, as human-written solutions may not always be the best for models Guo et al. [2025). For yielding worse results than the Base model, let alone in utilizing its failed problems to identify model weaknesses.
these reasons, the SFT model performs poorly on the initial training
In contrast to the Base and SFT models, the Initial RI. model exhibits the most robust performance on the initial training set, indicating that the failed problems expose the model's most critical weaknesses. Additionally, the training efficiency all problems during initial RL can also be recorded for further
26

---

Ratios Failed Problems Model, SFT Model, and Initial Model MATH-12K
60
Dase Model
Model
45.1
48.4
Initial RI. Model
41.3
37.0
30
20
16.5
17.1
14.8
10.9
0.5
3.8
8.8
6.7
6,2
2.8
0.9
Algebaa
Geospetry
Interrtediate Algebos
Number Tiwary
Presigebre
Preculeuhus
Counting
Figure 9: An visualization of utilizing the base model (Qwen2.5-7B) SFT model and the initial RL model on weakness identification the original training set (MATH-I2K)
analysis of model weaknesses. Meanwhile, the initially trained model can also serve as the starting point for augmented RL training. Therefore, in our SwS framework, we ultimately choose to employ an initial RL phase for robust weakness identification.
E
Data Analysis of the SwS Framework
E.1 Detailed Data Workflow
Taking the 32B model experiments as an example, Figure shows the comprehensive data workflow of the SwS framework, from identifying model weaknesses in the initial training data to the processing of synthetic problems. The initial training set, consisting the DAPO and Light-R1 subsets for the Qwen2.5-32B model, contains 17,545 problem-answer pairs. During the weakness identification stage, 1,905 problems are identified as failure cases according to Eq. 4 These failure cases are subsequently used for concept extraction and targeted problem synthesis.
For problem synthesis, we set an initial budget of million synthetic problems in all experiments. with allocations for each category determined as in Eq. [5] These problems then undergo several filtering stages: (1) removing multiple-choice, multi-part, or proof-required problems; (2) discarding problems evaluated as low quality: (3) filtering out problems where the answer generation model yields inconsistent answers, specifically when the most frequent answer among all generations appears less than 50%; and (4) removing problems whose difficulty levels are unsuitable for the current model in RL training. Among these, the quality-based filtering is the strictest, with a filtering rate of 78.35%, indicating that the SwS pipeline maintains rigorous quality control over the generated problems. This ensures both stability and effectiveness utilizing synthetic problems subsequent training.
We present case study of the quality-based filtering results in Table As illustrated, the positive case that passed the model- quality evaluation features a concise and precise problem description. In contrast, most synthetic problems identified as low-quality exhibit redundant and overly elaborate descriptions, sometimes including lengthy hints for solving the problem, as seen in the first negative case. Additionally, some low-quality problems incorporate excessive mantmaattmaaiaal knowledge, such as Physics, as illustrated the second negative case. The informal LaTeX formatting also contributes to their lower quality. Furthermore, problems with multiple question components, such as the third negative case, are also considered as low quality for RL training.
E.2 Difficulty Distribution of Synthetic Problems
In this section, we study the difficulty distribution of the synthetic problems generated for base models ranging from 3B 32B, as shown in Figure T0, The red outlines in the pie plots highlight the subset of synthetic problems selected for subsequent augmented RL training, with accuracy falling within the [25%. 75% range. These samples account for nearly 35% of all generated problems across the four models. The two largest wedges in the pie chart represent problems that the models answered either completely correctly or completely incorrectly. These cases not provide effective training signals in GRPO (Shao et al. 2024 Yu al. 2025b). and are thus excluded from the later augmented RI. training stage. To further enhance stability and efficiency, we also exclude problems where the model produces only correct or one incorrect response.
27

---

|2g] and 21 22 + 23 O. Using the symmetric polynomial $2 21 + 21 + 2223, find the value of sy
Positive Case # 1: Let 21, 22, and 23 be complex numbers such that |=1l = |22
Negative Case # 1: In village, there are 10 houses, each of which can be painted one of three colors: red, blue, or green. Two houses cannot have the same color if they are directly adjacent to each other. Using combinatorial analysis and considering the constraints, find the total number of distinct ways to paint the houses, taking into account the possibility of having sequence where the same color repeats after two different colors (e.g., red, blue, red), and assuming that the color of one of the end houses is already determined to be red, and the colors of the houses are considered different based on their positions (i.e., the configuration red, blue, green considered different from green, blue, red).
Negative Case # 2: A metal's surface requires minimum energy of 2.5 eV to remove electron via the photoelectric effect. If light with a wavelength of 480 nm is shone on the metal, and mole electrons is ejected, what is the total energy, in kilojoules, transferred to the electrons, given that the energy of a photon related its wavelength by the formula F. he/ X, where 6.626xr 10-34 and 3.00 1000010mm// s, and Avogadro's number is 6 6.0 particles per mole?
Negative Case 3: In triangle ABC, with LA = 60%, LB = 90", AB 4, and BC = 7, use the Law of Sines to find ZC and calculate the triangle's ' area.
Table 5: Case study of quality filtering results in SwS, featuring one high-quality positive case and three low-quality negative cases. The low-quality segments are marked in pink.
Since all synthetic problems generated using the same instruction model (LLaMA-3.3-701 Instruct) with similar competition-level difficulty levels (as illustrated in Prompt 33. and are based on concepts derived from their respective weaknesses, the resulting difficulty distribution of the synthetic problems exhibits only minor differences across all models. Consistent with intuition, the initially trained 3B model achieved the lowest performance on the synthetic questions, with the highest ratio of all-incorrect and the lowest ratio of all-correct responses, while the 32B model showed the opposite trend, achieving the best performance.
F
Co-occurrence Based Concept Sampling
Following Huang et al. 2024 Zhao et al. 2025b]. we enhance the coherence and semantic fluency of synthetic problems by sampling concepts within the same category based on their co-occurrence probabilities and embedding similarities. Specifically, for each candidate concept C from category D, we define its score based on both co-occurrence statistics and embedding similarity as:
Score(c) = CC((c + Sim(c), if e & (21,02....,,C3 c -00,
otherwise.
The co-occurrence term Co(c) is computed by summing the co-occurrence counts from sparse matrix built over the entire corpus, generated by iterating through all available concept lists in the pool. For each list, we increment CooccurMatrix(ce, c by one for every unordered pair where * d', yielding sparse, symmetric matrix in which each entry CooccurMatrixjc, c'] records the total number of times concepts and d' co-occur across all sampled lists:
Co(c) = Σ CooccurMalrix(ee,e]
(6)
while the semantic similarity is given by the cosine similarity between the candidate's embedding and the mean embedding of the currently selected concepts:
Sim(c) = cos
EE₁ k
E
To efficiently support large-scale and high-dimensional concept spaces, we construct a sparse co- occurrence matrix over all unique concepts, where each entry represents the frequency with which
28

---

Synthatic Problems Dirficulty for Sw5-38
Synthatic Problems Difficultyy for SwS-78
122.0%%
(23.38)
(32.22))
17.19%
(9.9%)
64.9%)
(11.0%)
(7.1%%)
16.156)
4 15 15.0%)
3 (5.6 6%
(8.9%)
(5.6%)
15.01)
(6.00%
Synthetic Problems Difficulty for SwS-7B-Meth
Synthetic Problems Difficulty for SwS-326
(28.8%)
(25.7%))
(30.25))
(333.885)
(7.7%%
2 (6.9%)
(9.7%)
(5.3%)
710.2%)
(5.8%)
14.8%)
(4.995)
(5.3%)
2 (6.49%)
(6.2%%)
(5.5%%
4 (5.5566
Figure 10: Difficulty distributions of synthetic problems for models from 3B to 32B in work.
pair of concepts co-occurs within sampled concept lists. Simultaneously. concept embeddings are normalized and indexed via FAISS to facilitate fast similarity computation. During sampling, initial seed concept drawn in proportion to its empirical frequency. For each subsequent concept, scores are computed by efficiently summing its co-occurrence with the current and its embedding similarity to the group mean, while previously selected concepts are masked out. The probability of sampling each candidate is determined via softmax over these scores with temperature
exp (Score(c)/7)
P(c) of (eg.....)) exp (Score(c)/7)
(8)
This process iteratively constructs coherent, semantically related concept sets serve the inputs for synthetic problem generation, ensuring both diversity and fluency.
G Details for Weak-to-Strong Generalization in SwS
To understand the capabilities of the weak teacher and the strong student model, evaluated both of them on the MATH-500 test set by prompting them on each question for eight times. Although the teacher model generally exhibits weaker performance, we found that in 16.4% of problems, the weaker teacher outperforms the otherwise stronger student model. This highlights the potential for leveraging weak teacher to distill its strengths into the student model. A case where the weaker teacher model outperforms the stronger student model is shown in Figure
From the analysis of the SwS framework, well its Weak-to-Strong Generulization extension, we assert that the upper bound for answer labeling is revised form of self-consistency score of the teacher model, where (1) the consistent answer must achieve accuracy greater than 50% across all
29

---

Case Study for Weak-to-strene Generalization
Question:
Compute 99º 99 + in your head. Please reason step by step, and put your final answer within
\boxed(), Golden Answer: 9901
>>> Teacher Response:
To compute 99² + 99 1, compute each part directly:
992 9801
99 + 99 = 9801 + 99 = 9900
99 + 99 + 9900 + 9901
Final answer:
9901
Student Response:
Rewrite the expression using binomial identity:
992 + 99 + = (99 + 1)² 1002 10000
Final answer:
10000
Figure 11: Case study in which weak teacher provides the correct solution, while strong student incorrectly applies binomial identity and derives an incorrect answer.
Intermediate Algebra
Number Counting & Theory
Setting
Size
Prealgebra
Algebra Precalculus
Geometry
All
Probability
Pass@1
500
88.2
64.3
95.5
71.2
93.0
81.4
63.0
80.6
SC
500
96.9
96.0
84.4
84.1
96.2
87.5
67.8
85.4
+ SC>50%
444
97.3
93.2
94.7
98.0
89.6
94.4
+ SC>50% Stu-Con
407
96.8
97.2
97.7
100.0
100.0
96.8
94.9
97.5
Table 6: The performance of the weak teacher model used for answer generation the MATH-500 test set under different strategies and their corresponding revisions. "Stu-Con" refers to filtering out problems where the student model's accuracy falls below the defined threshold of 25%.
responses, and (2) the student model must provide the same as the teacher model's consistent answer in at least 25% of responses. These revision procedures help ensure the correctness of the synthetic problem answers labeled by the teacher model.
In Table 6 we demonstrate the robustness utilizing weaker teacher for answer labeling, as- suming that the MATH-500 test set serves as our synthetic problems. As in the second line, even under the self-consistency setting, the teacher model only achieves an improvement of 4.8 points. However, when we exclude problems for which self-consistency does not provide sufficient con- fidence- specicc--ecii,ccllly those where the most consistent answer accounts for less than 50% of all thsponses-the self-consistency setting yields an additional 9.0-point improvement questions. Furthermore, in our SwS pipeline, we retain only problems where the student model achieves over 25% accuracy ensure appropriate level of difficulty. After filtering out prob- lems where the student falls below this threshold, some mislabeled problems are also automatically removed, resulting in the weak teacher achieving a performance of 97.5% questions. The increase in labeling accuracy from 80.6% to 97.5% shows the potential of utilizing the weaker teacher model for answer labeling as well as the robustness of the SwS framework itself.
the remaining
the final remaining
H Details for Self-Evolving in SwS
As mentioned in Section 4.2 the Self-evolving SwS extension enables the policy to achieve better performance on simple to medium-level mathematical reasoning benchmarks but remains suboptimal on AIME-level competition benchmarks. In this section, we further analyze the reasons behind this phenomenon. Figure 12 visualizes the model's self-quality assessment and difficulty evaluation within the SwS framework. Notably, the model assigns a much higher proportion of "perfect" and "acceptable" labels, and fewer "bad" labels, to its self seff-generated problems compared to the standard framework shown in Figure This observation is consistent with findings from L.MM-as-a-Jude [Li et al. 2024b). which indicate that models tend to be more favorable toward and assign higher scores to their own generations. Such behavior may result in overlooking low-quality problems or mis- classifying problems that are too complex for the model's reasoning abilities unsolvable or of poor
30

---

Algorithm Weakness-Drivel Selection Pipeline
Require: Failed Problems Xs; Total Budget [T]: Target Set Tx; Domains {D:
Ensure: Selected problems Ts
1: Embed all failed problems in Xs and all questions in Tx
2: for each domain D, in {D.}i= do
3: Compute selection budget [T]] for D; according to Eq. 2
4: Extract failed problems Xs, belonging to D;
5: for each q T do
Domain-level KNN
6: Compute d:(4) = mex,,xs, distance( Eg)
7: end for
8: Select top [7: questions from Tx with the smallest d:(g) as St
9:
end for
10: return Selected problems Ts Ui S
Final Selected Set
quality. Beyond the risk of filtering out over-complex problems, the model may also have difficulty in accurately labeling answers through self-consistency for over-challenging problems, thereby limiting the potential of incorporating complex problems through the Self-evolving SwS framework.
Additionally, in Figure T2 it is noteworthy that the initial RL-trained model achieves nearly 50% all- correct responses on its generated problems, whereas only 31 % of problems with appropriate difficulty remain for augmentation after SwS difficulty filtering. This suggests that the self-generated problems may be significantly simpler than those produced using stronger instruction model [Grattafiori et al. 2024), thus it could lead data inefficiency and limit the model's performance on more complex problems during RL training.
Details for Wrivness-driven Selection
As described Section 4.3 we utilize the failed problems identified by Qwen2.5-7B Yang et al. 2024a] the MATH- 12k (Hendrycks et al. 2021 training set, which comprises 915 problems, to select additional data from Big-Math [Albalak et al. 2025 to mitigate the model's weaknesses through the augmented RL training. The complete Weakness-driven Selection extension of SwS is presented in Algorithm] For embedding the problems, we utilize LLaMA-3.1 1-8--B [Grattafiori al. 2024] to encode both the collected failure cases and the problems from the target dataset. The failure cases are then grouped by categories, following the concept sampling strategy in standard SwS. We employ a binary K-Nearest Neighbors [Cover and Hart [967] algorithm select weakness-drive problems from the target set, where the augmented problems are chosen by their embedding distances
Self.judgement for Owen2.5-148. 1nnttuutttt
Deffiiifty Evaluation for Qwen2, 102255544-Inntruce
(6.9%)
(5.1%)
perfect (35.3%)
(5.4%)
(44.8%)
(5.0%)
5 (6.5%)
bad (0.6%)
acceptable (64.1%)
(8.2%)
(12.0%
Figure 12: Illustration of the quality assessment and difficulty evaluation for Qwen2.5-14B, IBsssrruutt under the Self-evolving SwS framework.
31

---

to the failure cases within each category. The selection budget for each category is also determined according to Eq We then aggregate the retrieved problems from all categories, forming selected set of 40k problems, which are then incorporated with the initial set for the subsequent RI. training.
J
Evaluation Benchmark Demonstrations
Dataset
Size
Category
Example Problem
Answer
The ice cream parlor was offering deal, buy 2 scoops of ice cream, get scoop free. Each scoop cost $1.50. If Erin had $6.00, how many scoops of cream should she buy?
GSM8k
1319 Prealgebra
For a constant c, in cylindrical coordinates (r, 2),, find the shape described by the equation
MATH-500
500
Geometry
2 C.
(C) Planc
(A) Line (B) Circle (C) Plane (D) Sphere (E) Cylinder (F) Cone. Enter the letter of the correct option.
If the Bohr energy levels scale as Z², where Z is the atomic number of the atom (i... the churge on the nucleus). estimate the wavelength of photon that results from a transition from n = to n 2 in Fe, which has Z = 26. Assume that the Fe atom is completely stripped of all electrons except for one. Give your answer Angstroms, to two significant figures.
9.6
Minerva Math
272 Precalculus
Given a positive integer n, determine the largest real number satisfying the following condition: for every 4n--pontt configura- tion C' in an open unit square U, there exists an open rectangle in U, whose sides are parallel to those of U. which contains exactly one point of C, and has an area greater than or equal to J..
Olympiad-Rena
675
Geometry
There are three points A, B, C. in space such that AB BC C'A = 1. If 2 distinct points chosen in space such that they, together with A, B, C. form the five vertices of regulur square pyramid, how many different ways are there to choose these points?
Gaokao2023
385
Geometry
How many complex numbers satisfy equation 2. E, where is the conjugate of the complex number z?
AMC23
40
Algebra
Let N be the greatest four-digit positive integer with the property that whenever one of its digits is changed to 1. the resulting number divisible by 7. Let Q and R be the quotient and remainder, respectively, when N is divided by 1000. Find Q+ R.
Number Theory
AIME24
30
609
On AABC points A, D, E, and B lie that order side AB with AD
DE 16, and EB 8. Points A, F, G, and C' lie in that order side AC with AF 13, PG = 52, and GC 26. Let M be the reflection of 1 through F, and let N be the reflection of G through E. Quadrilateral DEGF has area 288. Find the area of heptagon AFNBCEM.
AIME25
30
Geometry
588
Table 7: Statistics and examples of the eight evaluation benchmarks utilized in the paper.
We present the statistics and examples of the eight evaluation benchmarks used in our work in Table 7 Among these, GSM8K [Cobbe et al., [021] the simplest, comprising grade school math word problems. The MATH-500 Hendryykk et al. 2021], Gaokao-2023 [Zhang et al., 2023). Olympiad- Bench (He et al. 2024], and AMC23 [MAA] a benchmarks consist of high school mathematics problems spanning a wide range of topics and difficulty levels, while Minerva Math (Lewkowycz) al., [022] may also include problems from other subjects. The AIME [MAA] bb) benchmark is a prestigious high school mathematics competition that requires deep mathematical insight and precise problem-solving skills. An overview of benchmarks is provided as follows.
GSM8K: A high-quality benchmark comprising 8,500 human-written grade school math word problems that require multi-step reasoning and basic arithmetic, each labeled with natural language solution and verified answer. The 1,319 1.319-question test set emphasizes sequential reasoning and primarily solvable by upper-grade elementary school students.
32

---

MATH-500: A challenging benchmark of 500 high school competition- problems spanning seven subjects, including Algebra, Geometry, Number Theory, and Precalculus. Each problem is presented in natural language with X-fommmmeeee notation, offering strong measure of mathematical reasoning and generalization across diverse topies.
Minerva-Math:A high-difficulty math problem dataset consisting of 272 challenging problems. Some problems are also relevant to scientific topics in other subjects, such as physics.
Cysmpiad-Bench An Olympiad-leved English and Chinese multimodal scientific benchmark featuring 8,476 problems from mathematics and physics competitions. In this work, we use only the pure language problems described in English, totaling 675 problems.
Gaokao-2023: A dataset consists of 385 mathematics problems from the 2023 Chinese higher education entrance examination, professionally translated into English.
AMC23: The AMC dataset consists all 83 problems from AMC12 2022 and AMC12 2023, extracted from the AoPS wiki page. We used subset of this data containing 40 problems.
AIME24 & 25: Each set comprises 30 problems from the 2024 and 2025 American Invitational Mathematics Examination (AIME), prestigious high school mathematics competition for top- performing students, which are the most challenging benchmarks used in our study. Each problem is designed to require deep mathematical insight, multi-step reasoning, and problem-solving skills.
33

---

K Prompts
K.1 Prompt for Category Labeling
Listing I: The prompt for labeling the categories for mathematical problems, utilizing few-shot strategy which each category is represented by labeled demonstration.
CONTEXT
I am teacher, and I have some high-level mathematical problems. I want to categorize the domain of these math problems.
OBJECTIVE
A. Provide concise summary of the math problem, clearly identifying the key concepts or techniques involved.
B. Assign the problem
and only specific mathematical domain
The following is the list domains to choose from: <math domains>
["Intermediate Algebra", "Geometry", "Precalculus", "Number Theory", Counting Probability", "Algebra", "Prealgebra"] </math domains>
STYLE #
Data report.
TONE Professional, acientific.
AUDIENCE
Students. Enable them better understand domain of the problems.
RESPONSE: MARKDOWN REPORT ## Summarization
[Summarize the math problem a brief paragraph.] Math domains
[Select one domain from the list above that best fits the problem.]
ATTENTION
You must assign each problem to exactly one of the domains listed above.
If you genuinely uncertain and applies, you may use "Other", but this should be last resort. Be thoughtful and accurate in your classification. Default to the listed categories whenever possible.
of the listed categories
Add "=== report over ===" at the end of the report.
<example math problem> **Qustiinn*:
Let $ n(gge2) $ be a positive integer. Find the minimum m S, so that there exists (x] ,ille n)$ satisfying: Fo) every $1lle i .jlle n, (fii))=aaaft.d or
x
For every $1l1e i \le n$, there are at most $m$ indices 8k$ with
$x For every $1V1e j \le n$, there are most $m$ indices $k$ with (t_(k))=axx(xx11,xxx.... (k]]).
</example math problem>
## Summarization
The problem involves \( \times V) matrix where each element A x_(ij) V) constrained by the maximum values in respective row or column. The goal is to determine the minimum possible value of \ m \) such that, for each row and column, the number of indices attaining the maximum value is limited to at most A( m V.. This
34

---

problem requires understanding matrix properties, maximum functions, and combinatorial constraints structured numerical arrangements
## Math domains Algebra
report
</ exaample math problem>
** Quesstion**:
In acute scalene triangle $ABCS, points $D lie sides $BC, CA, AB$ respectively, such that $AD \perp BC, BE \perp CA, Aperp ABS. Altitudes SAD, BE, CFS meet at orthocenter sHs. Points $P$ and $Qs lie on segment $EF$ such that SAP \perp EF$ and $EQ \perp EF$. Lines $DP$ and SQH$ intersect at point $R$. Compute $HQ/HR$. </example math problem>
## Summarization
The problem involves an acute scalene triangle with three perpendicular cevians intersecting at the orthocenter. Additional perpendicular constructions are made from specific points on segment \( V, leading to an intersection at point \( V). The goal is determine the ratio \( HQ/ER V, requiring knowledge of triangle geometry, perpendicularity, segment ratios, and properties of the orthocenter.
## Math domains Geometry
report over
</example math problem> ** Questioon*::
Three cards are dealt random from standard deck of 52 cards. 4, the second card
What the probability that the first card a $cclubsuttS, and the third card is a 2? </ s/example math problem>
Summarization
This problem involves calculating the probability a specific sequence of events when drawing three cards from a standard 52-card deck without replacement. It requires understanding conditional probability, the basic rules of counting, and how probabilities change as cards are removed from the deck.
## Math domains Counting & Probability
report over
</example math problem>
** **Qustioon**:
Let 8x8 and $y$ real numbers such that $3x 2y \le and $2x 4y Ale 8.$ Find the largest possible value of Sx + y. </ </example math problem>
## Summarization
This problem involves optimizing linear expression \( V) subject to a system of linear inequalities. It requires understanding of linear programming concepts, such as identifying feasible regions, analyzing boundary points, and determining the maximun value of an objective function within that region.
# Math domaing Internediate Algebra
35

---

report over
</example math problem>
**Qusstion*::
Solve
M[aaccos \arccos \frac()pi]<)) A Enter all the solutions,
separated by commas.
</example math problem>
## Summarization
This problem requires solving a trigononetric equation involving inverse cosine functions. The equation relates two expressions with \ \arccos (2x) \) and \( (arccos(x)) V, and asks for all real solutions satisfying the given identity. It involves knouledge of inverse trigonometric functions, their domains, and properties, well as algebraic manipulation.
# Math domains Precalculus
report over
</example math problem> *uuustion**:
What perfect-square integer closest 273?
</ s/example math problem>
## Summarization
The problem asks for perfect square integer closest
This involves understanding the distribution and properties perfect squares, and comparing them with given integer. It relies on number- theoretic reasoning related to squares of integers and their proximity to target number.
## Math domains Number Theory
report over
</example math problem>
Voldemort bought $6 (6.1overliner6)5 ounces of ice cream at an ice cream shop. Each ounce cost $1$0.60. 60. How much money, in dollars, did he have pay? </example math problem>
## Summarization
The problem involves multiplying repeating decimal, A 6. Noverrine {6} W), by fixed unit price, 10,,60, to find the total cost dollars. This requires converting a repeating decimal into a fraction or using decimal multiplication, both of which are foundational
arithmetic skills.
Math domains Prealgebra
report
<math problem> (problem) </math problem>
36

---

K.2
Prompt for Concepts Extraction
Listing 2: Prompt template for extracting internal concepts from mathematical question.
As an expert in educational assessment, analyze this problem: <problem> (problem)
</problem>
Break down and identify concppp))) foundational concepts being tested. List these knowledge points that:
-Ar core curriculum concepts typically taught in standard courses,
Are precise and measurable (not vague like "understanding math"),
Are essential building blocks needed solve this problem,
Represent fundamental principles rather than problem problem-specisn specific techniques.
Think through your analysis step by step, then format your reaponse Python code snippet containing list of (nun_concepts) strings, where each string clearly describes one fundamental knovledge point.
K.3 Prompt for Problem Synthesis
Listing 3: Prompt template for synthesizing math problems from specified concepts, difficulty levels, and pre-defined mathematical categories. Following [Zhao et al. 2025b], the difficulty levels are consistently set to the competition level to prevent the generation of overly simple questions.
### Given set domain, and a specified difficulty level, generate vell-constructed question that meaningfully integrates multiple listed concepts and reflects the stated level of complexity.
foundational mathematical concepts, a mathematical
### Foundational Concepts: (concepts)
### Target Difficulty Level: (level)
### Mathematical Domain: [domain}
### Instructions:
1. Begin by outlining which concepts you will combine and how you plan to structure the question.
2. Ensure that the question is coherent, relevant, and appropriately challenging for the specified level.
3. The question must be a single standalone problem, not split into true/false
multiple sub-questions.
4. Do not generate proof based, miltiple-choics
questions.
6. The answer to the question should be expressible using numbers mathenatical symbols.
6. Provide final version of the question that is polished and ready for use.
### Output Format:
First, provide your brief outline and planning for the question design.
Then, present only the final version the question in the
following format:
[Your developed question here]
37

---

Do not include any placeholder, explanatory text, hints, or solutions to the question in the output block
K.4 Prompt for Quality Evaluation
Listing The quality evaluation prompt utilized to filter out low-quality math problems. Following prior work (Zhao et al. [0255]], we assess synthetic problems based on five criteria: format, factual accuracy, difficulty alignment, concept coverage, and solvability. Each problem is then assigned one of three quality levels: 'bad', 'acceptable', or 'perfect'.
As critical expert educational problem design, evaluate the following problem components:
==- GIVEN MATERIALS -.-
1. Problem Design Rationale:
bettionale_andpproble
(The rationale describes the author's thinking process justification in designing this problem)
2. Foundational Concepts:
(concepts)
3. Target Difficulty Level: [level]
=== EVALUATION CRITERIA ===
Rate each criterion as: [Perfect Acceptable Bad]
1. FORMAT
Verify correct implementation markup tags:
<! BEGIN RATIONALE [design thinking process] <! END RATIONALE
<!- BEGIN PROBLEM -> [problem] <! END PROBLEM ->
2. FACTUAL ACCURACY
Check for any incorrect or misleading information in both problem and rationale Verify mathematical, scientific, or logical consistency
3. DIFFICULTY ALIGNMENT
Assess problem complexity matches the specified difficulty level Evaluate if cognitive demands align with target level
4. CONCEPT COVERAGE
Evaluate how well problem incorporates the given foundational concepts
Check for missing concept applications
5. SOLVABILITY
Verify the problem has at least one valid solution
Check if all necessary information for solving is provided
==- RESPONSE FORMAT
For each criterion, provide:
1. Rating: [Perfect I Acceptable Bad]
2. Justification: Clear explanation for the rating
FINAL VERDICT
After providing criterion evaluations, conclude your response
'Final Judgement: [verdict]
where verdict must be of:
38

---

'perfect' ? (if both FACTUAL ACCURACY and SOLVABILITY are Perfect, at least two other
criteria are Perfect, and no Bad ratings)
'acceptable" (if no Bad ratings and doesn't qualify for perfect) 'bad' (if ANY Bad ratings)
Note: The 'Final Judgement: [verdict]' line muat be the final line your response.
39


Reviews:
[{"review_id": "72YsROwPrW", "reviewer": "Reviewer_iWhd", "summary": "The paper proposes a method for improving LLMs on mathematical reasoning by synthesizing problems based on specific topics where they fail the most (their \"weaknesses\"). The method first runs RL training on an initial set of mathematical problems. Then, it computes a failure rate for each problem based on both the success rate on that problem being always low (< 0.5) and moreover decreasing during RL training (negative slope in success rate). The LLM then extracts topics from those hard problems, synthesizes new problems from those topics, and then is trained via RL on a combination of the original and synthetic problems. Experiments training Qwen models from 3B to 32B show improvements on a range of mathematical reasoning datasets.", "questions": "* How exactly do you compute the slope in Eq 1 (L106-107)? Have the authors tried using only the first component of accuracy < 50%? What fails if the slope is indeed important here?\n* How do you compute Avg@32? Intuitively, Pass@32 should be generally higher than Pass@1, but this does not seem to be the case in AIME. L192 makes it sound like you're using Avg and Pass as synonyms, but this seems inconsistent with the results.\n* Have the authors tried iterating the procedure, i.e., repeating SwS after one cycle of weakness detection, problem generation and training? If so, do you get further gains?\n* Why are the other datasets besides AIME and GSM8k missing from Table 2? The reference to Appendix C in L242 is also incorrect - I assume the authors meant Appendix E, which says more about the generalization experiment.", "limitations": "The paper doesn't mention limitations very explicitly - for instance, it doesn't mention that all experiments are done by training models of the same family.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper tackles an important problem of improving LLM reasoning abilities without the need to collect further human data. The idea of focusing on specific categories of model failures is interesting. The authors also took care of not generating unreasonable combinations of topics by leveraging co-occurrence statistics of topics in existing problems.\n\nMy main high-level concerns about the paper are (1) using a single training dataset, MATH, and (2) using of a single model family during training (Qwen models).\n\nThe training procedure seems to rely heavily on the initial training problems already being labeled with categories. The categories in MATH, like \"Algebra\", are also fairly broad. This seems to really limit the current method to the coverage of the initial training dataset. While the method synthesizes new problems, the authors also seem to focus on problems that the initial model already has *some* chance of solving (e.g., by the Difficulty Filtering, which ignores problems with a smaller than .25 success rate). Thus, it seems like the method can improve a model's chance of solving those problems where it is uncertain, but not teach it anything completely new, or really address its worst weaknesses.\n\nMoreover, using a single model family for training limits the extent to which the paper can claim to address specific model weaknesses during training. For this, you would need to take different models with very different failure profiles, and show that SwS adapts what it does to those particular model failures. Otherwise, it might just be that SwS turns out to be an effective data synthesis procedure to address the needs of Qwen models, but it might be that (a) the same dataset would also similarly improve other models, like DeepSeek or LLaMA, which would weaken the claim that SwS is addressing model failures that are specific to Qwen, or that (b) SwS would fail to help other model families. Thus, the experiments can't yet support the main claims of the paper.", "was_revised": true, "final_justification": "The authors pointed to existing and new results that address some of my concerns regarding the limitations and use of a single training dataset. I think the paper provides extensive experiments and pushes on an interesting direction, so I'm slightly recommending its acceptance. That said, the most interesting results still consist of marginal improvements (e.g., on the weak-to-strong and self-training settings, where they do not assume a stronger teacher model). For that reason I am not assigning a higher score.", "rebuttal": "We appreciate that Reviewer iWhd finds our topic interesting and experiments concrete. We find the concerns are mostly due to misunderstandings. We have addressed all concerns below and thank you very much for your time and careful review.\n\n## 1. Response to single training dataset and model family (W1):\n\nClarifications ✏️: Regarding using only MATH for training, **we have also used the DAPO [1] and LightR1 [2] datasets for larger models,** as detailed in L186–188. **The category labels in them were generated using the prompt in Appendix I.** Generally, we select training datasets based on models' reasoning capabilities. Specifically, we found that Qwen2.5-7B and smaller models show consistent improvements when trained on MATH-12k. However, the 14B and 32B models quickly saturate on MATH-12k in early training. Therefore, we chose more challenging datasets, including DAPO and LightR1, that better stimulate the reasoning abilities of stronger models.\n\nClarifications ✏️: Regarding the concern about using only the Qwen model family, we primarily chose Qwen due to their strong reasoning capabilities compared to other open-source base models [3]. Some recent studies have discussed the failure cases when applying RLVR training to other base models. **In [4], the authors find that the LLaMA base model lacks significant reasoning patterns like verification and backtracking, thus their performance improvement from RLVR significantly lagged behind Qwen. Besides, recent mid-training work [5,6] also reports that LLaMA bases are ill-suited to zero-RL, performing far below the Qwen family, where the RLVR trainining fails to eliciting the LLaMA base model's reasoning capabilities over its few-shot settings [6].**\n\nExperiments 📊: To demonstrate the comprehensiveness of our SwS framework on models beyond Qwen family, we additionally conduct experiments using the LLaMA-3.1-8B-Instruct model. Specifically, we use MATH-12k for 200 steps RL in Weaknesses Identification, then apply the SwS pipeline to synthesize 18k problems for an additional 300 steps of augmented RL training, while continuing the baseline model for the same 300 steps. The following table shows that SwS augmentation leads to a 2.3% overall improvement.\n\n|  | GSM8k | MATH | Minerva | Olympiad | Gaokao | AMC23 | AIME24 | AIME25 | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| LLaMA-3.1-8B-IT | 85.6 | 48.2 | 24.6 | 18.8 | 39.7 | 22.5 | 6.7 | 3.3 | 31.1 |\n| Initial RL | 88.9 | 56.0 | 29.2 | 21.7 | 48.6 | 30.0 | 16.7 | 6.7 | 37.2 |\n| Baseline RL | 88.3 | 58.4 | 31.2 | 23.4 | 49.6 | 30.0 | 16.7 | 6.7 | 38.0 |\n| SwS-LLaMA | 90.5 | 60.2 | 33.5 | 25.8 | 49.1 | 40.0 | 16.7 | 6.7 | 40.3 |\n\n## 2. Response to SwS augmentation could not learn new capabilities (W2):\n\nClarifications ✏️: Regarding the concern about learning new capablities, we have discussed it with experiments in Section 3.3 (L212–L222, Figure 4). The y-axis of Figure 4 represents the percentage of problems in the original training set **where the model never generates an accurate solution throughout the entire training process, which reflect the base model’s worst weaknesses. The experimental results show that the SwS augmentation enables the model to solve certain problems that cannot be learned through training on the original set alone,** particularly in its weakest areas such as Intermediate Algebra and Geometry, where the SwS-augmented model achieves 20% and 5% relative higher solve rates, respectively. This demonstrates that our SwS augmentation is effective not only in improving performance on uncertain cases but also in eliciting the model’s deeper reasoning capabilities in its most challenging domains.\n\n## 3. Response to the slope conputation and its necessity (Q1):\n\nClarifications ✏️: We sincerely thank you for pointing out the clarification needed regarding the slope calculation in Eq.1. For computing the slope in Eq.1 (L106-107), we compute the slope using a **least-squares linear regression over the sequence of accuracy recorded across training epochs**, implemented using `np.linalg`. This metric reflects the model’s learning progress on a problem throughout the training process. We will include this clarification in Appendix B in the updated version.\n\nExperiments 📊: To assess the impact of the negative slope condition in Eq.1, we study on Qwen2.5-7B's weakness identification results using MATH-12k. **We examine problems with best accuracy ≤ 50% and a positive slope** (in contrast to negative slope in Eq.1). These problems account for one-third of all problems (660/1997) with best accuracy ≤ 50%. We find that over half (350/660) of these problems start with 0% accuracy in the first epoch. However, their performance improves during the initial RL stage, suggesting that learning such problems may not require additional augmentation. We further track the accuracy of these 1997 problems with best accuracy ≤ 50% in later stage of the baseline RL. **We find that only 4.24% of problems with a positive initial slope were never answered correctly by the model during later RL training (step 200 to 800), while 32% of problems with a negative initial slope were consistently failed.** This result indicates that problems with a best accuracy of ≤ 50% and a positive initial slope can be solved through extended training, **whereas those with a negative slope more accurately reflect the model's true weaknesses that naive RL can not learn.** Therefore, we set the negative slope constraint into the failure definition in Eq.1.\n\n## 4. Responses to \"Avg@k\" and \"Pass@k\" metric in the paper (Q2):\n\nRevision 📄: We admit that the metric labeled as Pass@32 in Table 2 should actually be Avg@32, and we will correct this in the updated version. Regarding L192, when k=1, Avg@1 and Pass@1 are equivalent because only a single answer is considered. We will clarify our notation by consistently using Avg@k to avoid any potential confusion in the updated version.\n\n## 5. Responses to repeating SwS after one cycle (Q3):\n\nExperiments 📊: Thank you for the valuable suggestion regarding the iterative SwS setting. This approach could significantly enhance the extensibility of SwS. To understand the potential improvements from iterating the SwS pipeline, **we select failure cases using Eq.1 during the first augmented RL stage and use them to synthesize 18k new problems for a second augmentation stage,** following the SwS pipeline. We run the second augmentation stage for 300 steps, while extend the first augmentation for the same steps. The comparison results are in the following table.\n\nNotably, the second SwS iteration yields a 1.5% absolute improvement in overall performance. Meanwhile, **we also observed that updating training sets during RL helps the model maintain an appropriate level of training entropy,** which is linked to sustainable exploration and potential gains in RLVR training [7]. Specifically, the training entropy dropped to 0.02 by the end of the first augmentation stage. However, during the second augmentation stage, the entropy initially increased from 0.02 to 0.7 within 200 steps before starting to decline. This observation not only shows strong potential that a new round of SwS augmentation can yield performance gains, but also offers insights into when such augmentation need to be updated and are most effective. We plan to further explore it in future work.\n\n|  | GSM8k | MATH | Minerva | Olympiad | Gaokao | AMC23 | AIME24 | AIME25 | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Stage-1 Aug | 93.9 | 82.6 | 41.9 | 49.6 | 71.7 | 67.5 | 26.7 | 20.0 | 56.7 |\n| Stage-1 Aug-Extend | 94.1 | 83.4 | 42.3 | 52.4 | 72.2 | 70.0 | 26.7 | 20.0 | 57.6 |\n| Stage-2 Aug | 94.3 | 84.4 | 44.5 | 50.1 | 73.7 | 72.5 | 30.0 | 23.3 | 59.1 |\n\n## 6. Response to benchmarks in Table 2 and appendix misreference (Q4):\n\nRevision 📄: We thank you for pointing out the misreference in L242 and apologize for the confusion. The correct reference should be Appendix E, and we will revise it in the updated version.\n\nClarifications ✏️: For other benchmarks not included in Table 2, we did not intend to present them here. **Our intention in constructing this table was showing that a teacher model, though weaker in certain areas (lower AIME24), with strengths in specific domains (multiple MATH-500 categories), can effectively guide the student model to achieve overall better performance—both in AIME24 and multiple MATH-500 subcategories.** This illustrates that a teacher model does not need to be generally stronger than the student in SwS, demonstrating its robustness. For clarification, we present the full benchmark results of the student model, teacher model, and trained student model in the following table. **The results align with Table 2 in the paper, demonstrating that a teacher model with weaker performance on AMC and AIME can still effectively enhance the student model’s performance on such competition-level benchmarks.**\n\n|  | GSM8k | MATH | Minerva | Olympiad | Gaokao | AMC23 | AIME24 | AIME25 | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Strong Student | 92.0 | 77.7 | 36.3 | 38.4 | 62.9 | 50.0 | 10.2 | 8.3 | 47.0 |\n| Weak Teacher | 93.3 | 80.6 | 36.8 | 36.6 | 64.9 | 45.0 | 7.2 | 6.2 | 46.3 |\n| Trained Student | 93.6 | 83.3 | 39.7 | 47.6 | 68.3 | 65.0 | 17.5 | 12.5 | 53.4 |\n\n## References\n\n[1] Dapo: An open-source llm reinforcement learning system at scale.\n\n[2] Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.\n\n[3] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild.\n\n[4] Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\n\n[5] Behavior Injection: Preparing Language Models for Reinforcement Learning.\n\n[6] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling.\n\n[7] The entropy mechanism of reinforcement learning for reasoning language models"}, {"review_id": "biUq7GQ9JI", "reviewer": "Reviewer_1yRo", "summary": "This paper introduces the Self-aware Weakness-driven problem Synthesis (SwS) framework, a novel approach for improving the mathematical reasoning capabilities of Large Language Models (LLMs) through Reinforcement Learning with Verifiable Rewards (RLVR). The core contribution is a method that moves beyond generic data augmentation by first identifying a model's specific \"weaknesses\"—problems it consistently fails to solve during an initial RL training phase. The framework then extracts the core mathematical concepts from these failure cases, recombines them to synthesize a new, targeted set of problems, and uses this augmented dataset for subsequent training. The authors demonstrate that this weakness-driven approach allows the model to focus on and overcome its deficiencies more efficiently. Through extensive experiments on models ranging from 3B to 32B parameters across eight reasoning benchmarks, the SwS framework shows significant performance gains, improving average scores by up to 10.0% for 7B models and 7.7% for 32B models over strong baselines. The paper also explores extensions of the framework to weak-to-strong generalization and self-evolving paradigms, showcasing its robustness and adaptability.", "questions": "This is generally strong work, in my opinion. However, I have a few points that need clarification from the authors, and I would appreciate the inclusion of some ablation experimental results.\n\n\n1. The criteria for identifying a \"failed problem\" in Equation 1 are defined by two specific thresholds: an accuracy below 50% and a negative slope in performance over time. How sensitive is the framework's overall performance to these specific values? Have you experimented with alternative definitions, such as a static low-accuracy threshold or different regression criteria, and how did they compare?\n\n2. The SwS pipeline appears to be computationally intensive, involving an initial RL run, concept extraction, problem generation with a 70B model, quality verification, and answer generation with a 32B model. Could you provide a more detailed breakdown of the computational cost (e.g., GPU hours) for each stage of the pipeline? This would be very helpful for reproduction and follow-up work. \n\n3. The weak-to-strong experiment is a promising direction. However, the \"weak teacher\" (Qwen2.5-Math-7B-Instruct) is still a somewhat powerful model. How does the framework's difficulty filtering and overall performance degrade as the capability gap between the teacher and the student widens significantly? Is there a point where an unreliable teacher leads to the filtering of nearly all synthetic problems, or worse, the introduction of noise that harms the student model?", "limitations": "It seems that there is a lack of discussion regarding the paper's limitations. A \"Limitation\" section would be appreciated.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- Clarity: The paper is well-written, clearly structured, and easy to follow. Figures and diagrams are highly effective at illustrating the complex methodology.\n\n- Performance: The reported improvements are substantial and consistent across different model scales, providing strong support for the paper's central claims.\n\n- Analysis: The paper includes valuable analyses on weakness mitigation, the impact of question difficulty, and extensions to weak-to-strong learning, adding considerable depth to the work.\n\n## Weaknesses\n\n- Complexity and Cost: The SwS pipeline is computationally expensive and complex, requiring multiple powerful LLMs for its various stages. This presents a potential hurdle for reproducibility and practical deployment for teams with fewer resources.", "was_revised": true, "final_justification": "I am keeping my initial score, same as Reviewer faYg.", "rebuttal": "We sincerely thank Reviewer 1yRo for considering our paper well-written, with solid experiments and comprehensive analysis. We greatly appreciate your time and thoughtful review, and we address your concerns as follows, particularly regarding the computational cost.\n\n## 1. Response for the potential complexity of the SwS framework (W1):\n\nClarifications ✏️: In the SwS pipeline, we leverage SOTA open-source instruction and reasoning models to generate high-quality problems along with corresponding answers. To reduce computational overhead, we propose two alternative strategies for augmentation based on model weaknesses under RLVR. First, **since all models in SwS are modular and interchangeable, in resource-constrained settings, users can utilize smaller yet powerful models equipped with thinking capabilities**, such as Qwen3-4B [1], which can still generate high-quality questions using our prompts. Below, we provide an example of using Qwen3-4B to synthesize problems from specific concepts and categories. **Secondly, users can adopt our proposed weakness-driven selection strategy in Section 4.3** to augment the training set, thereby avoiding the heavy costs associated with full data synthesis pipeline.\n\n```\n### Input \nConcepts: \n['Knowledge of strategies for solving systems of equations, including substitution and elimination methods',\n 'Direction vectors of lines',\n 'Understanding of the definitions and properties of trigonometric functions, including sine, cosine, and tangent',\n 'Understanding of ratios and proportions, including the ability to simplify and manipulate ratios',\n 'Knowledge of basic trigonometric identities, including sum-to-product formulas and Pythagorean identities']\n \nCategory: Precalculus\n\n### Output:\n<think>\nOkay, let's tackle this problem. The user wants a question that integrates several foundational concepts from precalculus at the AMC12 level. The concepts listed are solving systems of equations (substitution and elimination), direction vectors of lines, trigonometric functions (sine, cosine, tangent), ratios and proportions, and trigonometric identities like sum-to-product and Pythagorean.\n\n......\n\nThis problem requires solving the system of equations (from the intersection and perpendicular condition), using trigonometric identities, and applying the tangent addition formula. It integrates all the required concepts.\n</think>\n\n'''text\nTwo lines intersect at a point (x, y) where x/y = 2. One line has direction vector (cos θ, sin θ) and passes through the origin, while the other has direction vector (cos φ, sin φ) and passes through (1, 0). The lines are perpendicular. Find the value of tan(θ + φ).\n'''\n```\n\n\n## 2. Reponses for the criteria of failed problems (Q1): \n\nClarifications ✏️: We sincerely thank you for the suggestion to more comprehensive analysis of Eq.1. For the accuracy threshold, intuitively, **increasing the threshold in Eq.1 may introduce additional unnecessary problems,** potentially reducing the efficiency of leveraging concepts truly related to model weaknesses in data synthesis. Conversely, **lowering the threshold could lead to missed model weaknesses and a reduction in the diversity of synthesized problems.**\n\nExperiments 📊: To better understand the sensitivity of the failure criteria, we further analyze **the negative slope constraint.** We study the weakness identification experiments of Qwen2.5-7B model on MATH-12k, focusing on problems with best accuracy ≤ 50% and a positive slope (in contrast to negative slope used in Eq.1). **These problems represent one-third of all problems with best accuracy ≤ 50% (660 out of 1997). We find that over half (350 of 660) start with 0% accuracy in the first epoch, but their performance improves during early RL training, suggesting that additional augmentation may not be necessary for them.** Tracking the accuray of these problems with best accuracy ≤ 50% under baseline RL, **we observe that only 4.24% of problems with a positive slope are never correctly answered during later RL training (steps 200–800), whereas 32% of problems with a negative slope consistently fail. This indicates that problems with a positive initial slope can be solved through extended training, while those with a negative slope more accurately represent true model weaknesses that naive RL cannot overcome.** Therefore, we incorporate the negative slope constraint into the failure definition in Eq. 1. However, due to computational resource limits (as discussed in the next response) and time constraints, we regret that we could not comprehensively evaluate alternative failure definitions during rebuttal.\n\n\n## 3. Reponses for the detailed computational cost (Q2):\n\nClarifications ✏️: For the specific GPU hours at each stage, we use the Qwen2.5 7B experiment as an example and report the GPU hours for each SwS stage in the table below. All time measurements are based on NVIDIA A100 40G GPUs. Notably, **the total time spent on all problem synthesis stages (8,292.6 GPU hours) is actually less than that required by the final augmented training via RL (9,543 GPU hours). This comparison highlights the rationale for allocating computational resources to data augmentation prior to RL.** Within the problem synthesis pipeline, the most time-consuming component is Answer Generation, as it requires a powerful reasoning model to ensure answer correctness. In contrast, other stages mainly involve shorter inference, thus require less inference time.\n\nClarifications ✏️: Importantly, unlike prior works that rely on proprietary APIs [2, 3], **all models used in our experiments are open-source, making the pipeline more cost-effective.** Futhermore, **users can freely substitute them with smaller models or follow [4] to train lightweight models for problem generation.** Alternatively, to avoid computation-intensive problem generation, users may adopt our proposed weakness-driven selection strategy in Section 4.3 to augment the training set.\n\n### GPU Hours for SwS Experiments with Qwen2.5-7B\n|  | Weakness Identification | Concepts Extraction | Problem Generation | Quality Verification | Answer Generation | Difficulty Filtering | Augmented Training |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| GPU Hours (h) | 2,075 | 1.6 | 1,940 | 975  | 4,608 | 768 | 9,543 |\n| Data Quantity  | 12k | 1339 | 1000k | 842k | 302k | 253k | 52k |\n| Data / Hour  | - | 836.9 | 515.5 | 863.6 | 65.5 | 329.4 | - |\n\n\n## 4. Reponses regarding the weak teacher model in Section 4.1 (Q3):\n\nExperiments 📊: To analyze the case where the student model differs more significantly from the teacher model, **we use Qwen2.5-3B-Instruct to guide the same student model as in Section 4.1.** The answer labeling statistics are presented in the following table, where the first four rows correspond to the Qwen2.5-7B-Math-Instruct experiments in Table 5, Appendix E. **SC** refers to the teacher model’s self-consistency evaluation, while **Stu-Con** indicates whether the student model produces a sufficient proportion of answers consistent with the teacher model in difficulty filtering. Similarly, by applying self-consistency and filtering out low-consistency answers, the accuracy of the weak teacher-labeled answers increases from 62.4% to 83.6%. \n\nHowever, it is still 10.8% lower than the results of Qwen2.5-7B-Math-Instruct, introducing a 10.8% labeling error rate (i.e., noise). Meanwhile, the number of problems with consistency above 50% drops from 444 (Qwen2.5-7B-Math-Instruct) to 354, indicating that using a weaker model may lose more difficult yet key problems.\n\nExperiments 📊: In addition, we observe that **using a weaker model can lead to an imbalanced distribution in the labeled dataset.** As shown in the last row of the table below, which reports the proportion of problems retained after teacher self-consistency and student difficulty filtering, the 3B teacher model performs relatively well on Algebra, retaining 91% of the problems. However, due to its limited capabilities, it fails to retain many problems in categories such as Precalculus (50%) and Geometry (58%). **One potential strategy is to utilize multiple weaker teacher models—each with different strengths but generally smaller and less capable than the student model—to enable weak-to-strong generalization,** similar to the strategy in [5]. That is, a weaker teacher model should possess some strengths over the student model; if so, it can effectively guide the student from certain perspectives. In contrast, a teacher model that is uniformly weaker across all dimensions may be ineffective.\n\nClarifications ✏️: In addition to using an external teacher model, **our experiments in Section 4.2 (Self-Evolving) demonstrate that the base model itself can serve as the answer-labeling teacher through self-consistency.** Along with Section 4.1, these results show that the teacher model used for answer labeling does not necessarily need to outperform the student model, highlighting the robustness of the SwS pipeline.\n\n### Experiments using Qwen2.5-3B-Instruct as Teacher Model\n\n| Setting | Prealgebra | Intermediate Algebra | Algebra | Precalculus | Number Theory | Counting & Probability | Geometry | All |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pass@1  | 73.2 | 35.1 | 85.5 | 48.2 | 69.4 | 55.3 | 51.2 | 62.4 |\n| SC | 79.3 | 53.6 | 91.9 | 51.8 | 87.1 | 71.1 | 61.0 | 73.2 |\n| SC>50% | 90.6 | 60.8 | 91.2 | 82.1 | 83.7 | 80.0 | 83.3 | 83.6 |\n| SC>50% & Stu-Con | 91.9 | 60.8 | 91.2 | 82.1 | 84.8 | 80.0 | 83.3 | 84.0 |\n| Retained Ratio  | 76% | 53% | 91% | 50% | 74% | 66% | 58% | 69.8% |\n\n\n## References\n\n[1] Qwen3 technical report\n\n[2] Key-point-driven data synthesis with its enhancement on mathematical reasoning.\n\n[3] Common 7b language models already possess strong math capabilities.\n\n[4] Promptcot: Synthesizing olympiad-level problems for mathematical reasoning in large language models.\n\n[5] Language Model Preference Evaluation with Multiple Weak Evaluators"}, {"review_id": "oF68LDAGV3", "reviewer": "Reviewer_faYg", "summary": "The proposed Self-aware Weakness-driven Problem Synthesis (SwS) framework demonstrates innovation and strength. By leveraging the model's self-identified weaknesses and generating synthetic problems for training augmentation, it enhances the model's reasoning capabilities. Compared to existing problem synthesis strategies for LLM reasoning, the SwS targets the model’s capabilities and self-identified weaknesses, enabling improvement in RL training.", "questions": "1. Could the weakness-driven augmentation strategy (Section 4.3) lead to over-specialization by heavily sampling problems conceptually aligned with past failures, potentially causing the model to overfit to specific problem archetypes and hindering broader generalization?\n\n2. Does the weakness criterion defined in Equation (1) (failure rate > 50% and declining accuracy trend) fail to capture critical scenarios? For instance, if the accuracy of a problem drops significantly from 80% to 30%, wouldn't this be excluded from the weakness set due to the initial high accuracy? Recommend adding parameter analysis experiments.\n\n3. In the case presented in Figure 7, does the reorganization of \"properties of equilateral triangles\" and \"points and planes in 3D space\" to generate the simple problem stray too far from the core of the original problem? Could this lead to synthetic problems diverging from the model's true weaknesses? How can we ensure that invalid problems are avoided during the generation process?\n\n4. In the geometric synthesis (Figure 7), the reorganization of \"equilateral triangle properties\" and \"3D point-plane relations\" from the original problem leads to a loss of the core equidistant condition in the simple problem and reduces spatial relationships to 2D in the medium problem, which seems to deviate from the original reasoning structure. Can the current reorganization mechanism fully capture the mathematical essence of the original problem, and how can invalid problems be avoided in the process?\n\n5. The experiment only categorizes problems by accuracy (simple/medium/hard) but does not address their true cognitive difficulty in human evaluation. Could the low accuracy (6.25%) on \"hard\" problems be due to design flaws rather than inherent difficulty? It is recommended to include more examples for illustration.", "limitations": "Several aspects require further clarification. Please reply to my concerns raised in the ``Questions'' section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The SwS framework tackles model reasoning weaknesses by using self-identified gaps and targeted training to generate synthetic problems.\n\n- The framework is tested across various mathematical reasoning benchmarks, demonstrating performance gains across different model sizes.\n\nWeaknesses can be found in the question section. Some aspects require further discussion and clarification.", "was_revised": true, "final_justification": "Thanks to the authors for the rebuttal. The responses are beneficial to facilitating understanding of this paper. However, the authors are still suggested to incorporate the new discussions/experiments/clarifications mentioned in the rebuttal to the paper.\n\nTherefore, I maintain my borderline accept rating.", "rebuttal": "We sincerely appreciate Reviewer faYg for recognizing our method as innovative and our experiments and analysis as comprehensive. Thanks for your time and thoughtful review, and we provide the following clarifications to address your concerns.\n\n## 1. Response to overfitting to past failures (Q1）:\n\nClarifications ✏️: We believe that **the weakness-driven augmentation is unlikely to cause overfitting** to past failures and forgetting. **Since the augmented training set includes the original dataset, such as MATH-12k for smaller models and DAPO + LightR1 for larger models, the model continues to be exposed to the broad distribution of their problems.** If the model forgets how to solve certain problems from the original dataset, during the subsequent augmented RL phase, it will still be required to solve those types of problems, leading to recovery and improvement in those areas. Moreover, if the model fails to learn certain problems from the original set during both the initial and augmented RL phases, it is likely due to the inherent difficulty of those problems rather than overfitting on weakness-driven augmented problems.\n\nClarifications ✏️: The experiments in Section 3.3 and Figure 4 also show that **SwS-augmented training enables the model to solve more problems across all categories of the original dataset compared to training on the original dataset alone.** Notably, the model achieves 20% and 5% higher solve rates on previously unsolved problems in its weakest areas—Intermediate Algebra and Geometry, respectively. Together with the performance improvements on evaluation benchmarks in Table 1, these results demonstrate that the weakness-driven augmentation strategy in SwS promotes better generalization rather than overfitting to past failure cases.\n\n## 2. Response to weakness criterion (Q2):\n\nExperiments 📊: We sincerely thank the reviewer for pointing out the potentially overlooked cases in weakness identification. To examine the mentioned scenario and evaluate the robustness of our failure definition in Equation 1, we conduct a statistical analysis of problems where accuracy drops from over 80% to below 30%, based on Qwen2.5-7B training on the MATH-12k dataset. **We find that only 0.5% of the problems in MATH-12k exhibit such performance drop.** Empirically, these drops are largely due to random fluctuations during training, and performance often recovers in subsequent epochs. Among them, **56.1% of the problems reach a minimum accuracy of at least 50% during augmented RL training epochs, and 84.8% achieve an average accuracy above 50% during those epochs.**\n\nClarifications ✏️: **These statistics suggest that, for such edge cases not captured by our weakness criteria, the RLVR process during augmented training can effectively improve performance without requiring additional data augmentation.** While treating these cases as weaknesses might further enhance learning, it could also lead to concept-ineffectiveness in problem synthesis, which reduces the sampling probability of concepts associated with more severe failure cases. To ensure more effective targeting of true weaknesses, we therefore choose not to include such cases as weaknesses. \n\n\n## 3. Response to potential divergence between synthetic and failure problems (Q3):\n\nClarifications ✏️: We believe that the reorganization of the \"properties of equilateral triangles\" and \"points and planes in 3D space\" is necessary, as the original problem requires knowledge and skills related to the properties of equilateral triangles and dihedral angles. The problems we constructed from sampled concepts—-such as the “Simple”, \"Medium\" and “Hard” examples shown in Figure 7—can help us better understand the underlying fundamental concepts. This case study demonstrates that **it is possible to generate geometry-related questions of varying difficulty based on specific concepts.** Furthermore, by combining resampled concepts drawn from the model’s weaknesses, we can generate a large number of such questions. **Statistically, since the sampled concepts for problem synthesis originate from the base model's weaknesses, training on the synthesized problems comprehensively addresses the weaknesses exposed in the original training set.** \n\nClarifications ✏️: Admittedly, some synthetic questions may be unsolvable or ineffective for RLVR training. To mitigate this issue in augmentation, we **first employ an powerful reasoning model to label the answers using self-consistency and filter out questions with low consistency, which are likely either too difficult or unanswerable. Additionally, we use the initially trained policy model to assess whether the difficulty of the synthetic problems is appropriate for RLVR training, ensuring that all remaining synthetic problems provide effective learning signals during augmented training.** You may refer to lines 448–470 and Table 4 in our appendix for our whole data workflow, where we describe how we filtered out basic formatting issues as well as problems that were too simple or too difficult:\n```\n(1) Removing multiple-choice, multi-part, or proof-required problems;\n(2) Discarding problems evaluated as low quality;\n(3) Filtering out problems where the answer generation model yields inconsistent answers, specifically when the most frequent answer among all generations appears less than 50%; and\n(4) Removing problems whose difficulty levels are unsuitable for the current model in RL training.\n```\n\n## 4. Response to concept capture and potential invalid questions (Q4):\n\nClarifications ✏️: Actually, it is challenging to strictly guarantee that the knowledge structure of synthetic problems is fully aligned with that of the failure cases, **as the concepts extracted from failure cases are shuffled within their respective categories and resampled for problem synthesis.** However, we perform large-scale sampling to ensure that the number of synthetic problems is sufficient to comprehensively cover the identified weaknesses. For example, **as shown in Figure 8, starting from a relatively small set of around 1,900 seed problems, we were able to generate a large number of synthetic problems—up to 1,000,000 in total—resulting in over 40,000 problems after filtering.** Some of these may exhibit high similarity to the failure cases. For categories with a higher proportion of failure cases, we allocated more sampling capacity to increase the likelihood that the synthetic problems share similar knowledge structures with the original ones. \n\n## 5. Response to human evaluation and potential design flaws (Q5):\n\nClarifications ✏️: Regarding the alignment between human-evaluated and model sampling-based difficulty, **since the SwS pipeline is designed to mitigate the weaknesses of a specific model, the assessment of problem difficulty should be based on the model’s current reasoning capabilities.** Given that each model has different strengths and weaknesses in reasoning, a problem may vary in difficulty across models—just as humans have varying proficiencies with certain problems—making human-evaluated difficulty inherently subjective and difficult to standardize. Consequently, **relying on other models or human evaluations, rather than the model itself, could result in inconsistency between problem augmentation and the goal of mitigating model weaknesses.**\n\nFuther more, from the perspective of the GRPO used in our paper, its main training strategy is to reinforce the model’s correct reasoning paths while suppressing incorrect ones. The difficulty filtering stage aims to ensure that problem difficulty matches the current model’s capability, allowing the model to generate both correct and incorrect answers, thus providing effective learning signals. Human-evaluated difficulty may not align with this objective, as the model might produce zero accurate rollouts on problems deemed difficult by humans.\n\nExperiments 📊: To better illustrate the discrepancy between the current model’s difficulty evaluation and external assessments, we conduct a simulated difficulty evaluation using the problems shown in Figure 7. Mimicking human difficulty evaluation, we prompt GPT-4.5 to assess these problems using the following criteria and assign difficulty scores from 1 to 10.\n```\n1. **Conceptual Complexity**: What level of math is needed to understand the problem (e.g., arithmetic, algebra, geometry, combinatorics, calculus)?\n2. **Number of Steps / Depth of Reasoning**: How many steps does it take to solve the problem? Are they straightforward or require abstraction?\n3. **Cognitive Load / Trickiness**: Are there traps, misconceptions, or misleading structures? Is it easy to make a mistake?\n4. **Calculation Complexity**: Does it require long or error-prone computations?\n```\nThe GPT-4.5 evaluated difficulty scores for these problems from Simple to Unsolvable are [5, 6, 8, 5]. The first three evaluations align with sampling-based difficulty, whereas the evaluation of the Unsolvable problem does not. **If augmentation relies on such evaluation scores, the Unsolvable problem—where the model fails to sample an accurate answer—would provide no useful training signal during GRPO training, leading to data inefficiency.** Therefore, using the model’s own sampling-based difficulty evaluation would be more targeted on its weaknesses and data-efficient.\n\nFor better clarity, we will include additional case studies across all categories of synthetic problems (Simple/Medium/Hard) in appendix."}]

Author rebuttal:

Meta-review: This paper introduces targeted data augmentation by identifying a model's specific weaknesses during RL and then synthesize new problems that address these deficiencies. 

All reviewers found the paper's core contribution to be innovative and well-motivated, the experiments to be extensive, and the results to be significant across multiple benchmarks and model scales. The clarity of the presentation and the in-depth analyses were also highlighted as major strengths.

Reviewers' initial concerns, including the computational cost, the reliance on a single model family (Qwen), and the specific criteria used to define model "weaknesses," were generally addressed during author rebuttal. The authors presented new experimental results with LLaMA models, demonstrating the framework's generalizability. They also provided a detailed breakdown of the computational costs and offered analysis for their weakness identification method.

Overall, the paper proposes an effective solution for solving a highly relevant problem, making valuable contributions to the venue.
Decision: Accept (poster)