Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
Authors: Jiayu Wang
Yifei Ming
Zixuan Ke
Caiming Xiong
Shafiq Joty
Aws Albarghouthi
Frederic Sala
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Mathematical Reasoning
Reinforcement Learning
Evaluation
Primary area: evaluation

Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RL enhances performance is still lacking. To bridge this gap, we introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models to formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RL enhances models' ability to integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems---those yielding no RL signals and mixed-quality reasoning traces---can still be effectively used for training. We introduce SparkleRL-PSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide a principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: https://sparkle-reasoning.github.io/.

Beyond Aceuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
Jiayu Wangi", Yifei Ming"", Zixuan Ke Caiming Xiong Shafiq Joty',
Aws Albarghouthil, Frederic Sala" University of Wisconsin-Madison *Salesforce Al Research
Abstract
Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RI. enhances performance is still lacking. To bridge this gap, introduce 3%2 SPARKLE, fine-grained analytic framework to dissect the effects of RI. across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RI. enhances models' ability integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems- -thse yielding no RL. signals and mixed-quality reasoning traces-can still be effectively used for training. We introduce % SparkleRL-PSSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: bittps://sparit sesarkkl-reasooingg reasoning. www.innggggttutttti.e github.io/
Introduction
Reasoning models are among the most exciting recent developments in the large language model space. These models are able to perform mathematical and other forms of reasoning and achieve excellent performance
number of benchmarks [2] [5] 37 47). Reinforcement learning- based training appears to be crucial to achieving these reasoning capabilities, leading to a proliferation of papers proposing new Ra-based training techniques, reasoning models, and evaluation benchmarks.
Despite this explosion of interest, precisely what capahilities are gained during RI. training is not clear. Most works studying RI. for reasoning use set of standardized benchmarks. While convenient, tracking the gain in accuracy for a method on, for example, the AIME 2024 contest [32], ultimately provides limited signal into what behaviors are enabled by RL. To make further progress, argue that it is necessary to develop a a.fine- grained understanding of the benefits of RL.
Coalead. Correspondence to: milawang@@ss wdu.ee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SparkleRL-PSS
Auxiliary Information (Hints)
@ Medium Easy
Stage 1:
Partial Step Scaffolding
Highh-evel Plan
SPARKLE R 1
Performance Gain (A)
Fast If prasetien ain vie Feaily sunt candiditoo that osseryy the aduzton
Bose model
SporkieRL (Stage 1)
Fuil
Stage 2
External Knowledge
Chain of Subproblems
Sparkielt (Stmae
SparitaRt (Stage 2-pos)
mam I Hardest data with Partial Step Scaffoldling
Base +SFT
+RL
With vs. without
Figure 1: Left: SparkleRL-PSS, a two-stage curriculum-style RL training with partial step scaffolding the hardest problems are revisited with auxiliary step-level hints guide multi-stage learning. Middle: Four types of auxiliary information (hints) used in this work: (1)Partial Step Scaffolding (used in Stage 2 RL). (2) High-level Plan, (3) External Knowledge, and (4) Chain of Subproblems (2-4 used in the SPARKLE analysis framework). Right: Net performance gains/losses when models are evaluated with vs. without hints (2-4). While all models struggle on compositional subproblems. RL-tuned models exhibit the greatest flexibility in leveraging auxiliary information whereas base performance drops sharply and SFT models show limited benefit.
To enable such analysis, we introduce SPARKLE, tine-grained analysis framework that examines key elements hypothesized benefit from RL training. Specifically, we evaluate pre- and post-RL RL tuned models along three dimensions: (1) plan-following and execution: (2) knowledge use; and (3) problem decomposition. Our framework enables fine-gratd une-grained assessment of reasoning behaviors, revealing not just where RL enhances performance but also its limitations across different reasoning components.
Existing benchmarks, which consist of problems, ground-truth answers, and occasionally reasoning traces, are insufficient for the fine-grained analysis we perform. For example, testing a model's ability to follow a given plan requires access to planning annotations. To instantiate SPARKLE on mathematical reasoning, we augment math datasets with planning skeletons, requisite knowledge annotations, and candidate problem decompositions, A concrete example is shown in Figure 2 This approach produces novel insights: for example, we observe that giving base and SFT models access to human-cra human-crafted, crafted, correct plans can surprisingly degrade performance, while RL-tuned models are more robust to these, but the plan is better provided as high-level. This suggests that part of the benefits of RL are the flexibility to use multiple plausible plans.
We also investigate a second form of fine-grained analysis, related to problem difficulty. Prior work has observed that RL often fails to exploit hard problems effectively, as these examples rarely yield positive reward signals 49 This has motivated filtering strategies that remove such problems-but this wastes valuable training signal. Instead, we analyze problem difficulty and use the resulting distinctions develop a new multi-stage RL Sipeline((parkleRR--PSS
that exploits data of varied difficulties. In the first stage, we perform RL on a broad set of diverse math problems, analogous to Guo et al. [1]. In the second, we fine-tune the model further on identified hard problems. To further help guide the model on these challenging cases, we give it access to partial solution augmentations or hints without further data generation. This two-stage setup is designed to first give the model a strong general reasoning boost, then hone its skills on the trickiest examples.
Together, SPARKLE and SparkleRL-PSS offer a comprehensive view of how RL shapes model behavior both analytically and algorithmically. They show that RL enhances flexibility in plan following and knowledge integration; however, performance degrades when models are forced to follow concrete, human- crafted plans. Instead, RL-tuned models perform best when guided by high-level plans that align with their internal reasoning dynamics, although robustness in solving chained subproblems remains limited. These findings highlight concrete directions for developing RL pipelines that are more adaptive, data-efficient, and interpretable for reasoning tasks. Our contributions include:
We introduce SPARKLE, novel analysis framework to systematically evaluate plan following, knowledge utilization, and subproblem solving in LLM reasoning.
2

---

Problem:
One Euler's confectures was disproved in the 196as by American mathematiclans when they showed there postrive integer such that
133 + 110" + 84* 27
Find the value of n.
Planning:
Knowledge:
Subproblems:
Step Analyze the modular properties of both sides to determine possible congruences D modulo small primes.
Fact: Buler's of powers conjecture posited that at least nth powers are needed to sum to another nth power. Definitions: 1) Modular arithmetic involves integers wrapping around upen reaching certain value, the modulus; 2) The Chinese Remainder Theorem provides a way to solve systems simultameuus congruences with pairwise coprime moduli.
Q:: What remainder patterns emerge when n² is divided by the primes and 57
Qa: Based on the result from Qt. what is modulo 30?
Step Apply Fermat's Little Theorem to deduce the congruences of n modulo 2 3, and 5
Qs: What are values for n greater
than 133 that satisfy 24 med 3o?
Step 3: Use the Chinese Remainder Theorem combine these congruences and find specific congruence modulo
Q4: What upper bound for by comparing the sum 133" 110? 84" 27 to n²?
Theorema: Fermat's Little Theorem: if is prime number and is an integer not divisible by p. then at" (mod p); 2) Chinese Remainder Theorem: If one knows the remainders of the division of an integer by several patrwise coprime integers, then one can determine uniquely the remainder of the division of by the product of these integers.
Qs: What is the value of n that satisfies all previous conditions and the original equation?
Step 4: Generate candidate values for a that satisfy this congruence.
Step 5: Estimate upper bounds by comparing the sum the left-hand side potential values of xf, using inquuaittes.
Step 6: Test specific values of n within the narrowed range to Idencify the value of satisfies the original equation.
Figure 2: Illustration of the SPARKLE framework's 's three-dimensional analysis approach. For each problem (top), we construct three complementary components: high-level planning skeleton (left) capturing the overall solution strategy, relevant knowledge (middle) required for reasoning, and sequence of interconnected subproblems (right) that decompose the solution process. The augmented benchmark enables a fine-grained understanding of reasoning capabilities and failure modes in LLMs.
We construct SPARKLE benchmark, augmented with planning skeletons, knowledge information and subproblem chains to support comprehensive reasoning analysis.
A simple yet effective multi-stage RL training approach with partial step scaffolding (SparkleRL-PSS) that reuses existing hard problems without additional data generation.
We present comprehensive empirical findings that reveal which aspects of reasoning most enhanced by RI. (e.g., flexibility in plan following and integrating knowledge into its reasoning processes), which remain challenging (e-g., robustness in solving subproblems), and the conditions under which RL. provides the greatest benefits.
2 SPARKLE: A Three-Axis Framework for LLM Reasoning Evaluation
To precisely analyze LLM reasoning, we introduce SPARKLE, framework that decomposes reasoning along three axes: plan-following and execution, knowledge utilization, and subproblem decomposition (Section 2.1). inspired by classic research cognitive science on human reasoning and
problem solving 7 30 35 We also present dataset constructed to instantiate this framework and support systematic evaluation (Section 2.2 Together, the framework and dataset enable fine- grained, interpretable analysis of key reasoning competencies in LLMs (Figure
2.1 SPARKLE Analysis Framework Overview
Axis 1: planning and execution. When LLMs fail to solve challenging mathematical problems, is the cause not knowing what to do-or or an inability to carry out the steps? We evaluate models on problems both 1) with and 2) without accompanying planning skeletons. The plan outlines the major steps needed but omits the details that must be carried out by the model. In the former case, the planning sketch alleviates the planning burden, allowing us isolate and assess the model's execution capabilities (see full example in Appendix B). By comparing performance under these conditions, we can better understand whether RL fine-tuning primarily enhances strategic planning, step-by-step execution, both components of the reasoning process.
3

---

Axis 2: knowledge utilization. Mathematical reasoning relies on access to factual knowledge including definitions, theorems, and formulas- and the ability to apply this knowledge: knowledge defines the premises, while reasoning describes the logical operations performed on those premises. Performance improvements from RI. fine-tuning may stem from enhanced deductive reasoning abilities, improved knowledge utilization,, or a combination of both.
Inspired by prior works on transparent logical reasoning [6] our second evaluation axis addresses this ambiguity by separating between knowledge retrieval and reasoning processes. Concretely, knowledge in our setting refres to the collection of relevant facts, definitions, theorems, and lemmas necessary for solving the problem, while reasoning encompasses the logical operations that manipulate and apply this knowledge toward the answer. We systematically vary knowledge availability to separate these roles. In one condition, the model must retrieve all relevant concepts itself. In the other, we explicitly provide the necessary knowledge (e.g., statements of Fermat's Little Theorem and the Chinese Remainder Theorem, as in the example in Figure 2. This design allows us to identify knowledge- related bottlenecks. A model that succeeds only when given knowledge has intact reasoning but incomplete recall because the information lies outside its training data. A model that still fails despite having all the facts reveals limits in deductive ability.
Axis 3: chain of subproblems. Even when an LLM gives a correct final answer, it may contain flawed intermediate steps [5]]. To uncover where reasoning breaks down, we decompose problems into a chain of subproblems and assess model performance incrementally. At each stage, the model is shown the original problem, the subproblems solved so far, and the current subproblem. For example. when answering Q3 (Figure the model is shown the full prompt and the answers to Q1 and Q2. Crucially, unlike the structured planning sketch in Axis 1, these subproblems are not prescriptive instructions. Instead, they act as checkpoints- smaller, self-contained problems that are individually solvable but collectively build toward the full solution. They provide no hints about what method to use, only what question to answer. This framing allows us to identify the precise step where the model's 's reasoning fails, offering a fine-grained error analysis.
Remarks. We focus on these three axes as they capture core aspects of reasoning that can be systematically and quantitatively evaluated (Section(5). These dimensions are not strictly orthogonal- for example, retrieved knowledge can inform planning-bil together they offer a practical and interpretable framework for analyzing the impact of RL. on reasoning behavior.
2.2 SPARKLE Benchmark Construction and Validation
Extant benchmarks- typically consisting of problems, answers, and reasoning traces llak the components needed to study RL along the three axes we have proposed. To address this, we augment popular reasoning evaluation sets to produce the SPARKLE benchmark.
Pipeline overview. We construct our benchmark through a unified annotation pipeline that supports all three evaluation axes. For each problem, we begin with its ground-truth solution and prompt a high-capacity agent with access to the Internet (e.g., GPT-4.1 (36]) to: (I) extract a planning skeleton summarizing key reasoning steps, (2) decompose the problem into a sequence of well-defined subproblems with answers, and (3) identify relevant knowledge components (facts, definitions, theorems, lemmas). The model is instructed to retrieve knowledge from reliable sources on the Internet when necessary. To ensure annotation quality across all three dimensions, we employ second verification agent (e.g., GPT-4.1) that checks the outputs for correctness, coherence, completeness, and pedagogical soundness. any aspect fails, the annotations are regenerated. Finally, expert validation is conducted by graduate students with advanced mathematics background to ensure that the annotations faithfully capture the underlying reasoning and required knowledge.
SPARKLE benchmark statistics. SPARKLE is created from diverse mathematical problem benchmarks including AIME24 [2). AMC23 (3I, MATH500 [17), GSM8K [. and
OlympiadBench [16] (test splits). Each problem is augmented with planning information derived from ground-truit reasoning traces, relevant knowledge components, and a sequence of subproblems curated via the pipeline described above. An example of the augmented problem is illustrated in Figure We also annotate the difficulty level using AoPS Competition Ratings I and mathematical domain (e.g., linear algebra, geometry, number theory). The resulting SPARKLE benchmark contains 2,564 open-ended questions spanning 10 difficulty levels and 9 domains. More details are provided in Appendix B

---

3 Problem Difficulty and Its Implications on Reinforcement Learning
The SPARKLE framework enabled us to conduct a fine-grained evaluation of how reinforcement learning affects the reasoning capabilities of LLMs. This evaluation, however, did not examine problem difficulty. We now tackle this axis, again seeking insights into RL behavior.
prominent belief that performing RL on problems that are too challenging (e.g., beyond current frontier models, too complex for low-capacity models) is not useful because models are unlikely to obtain any reward. Such samples are filtered out at the cost of reducing our dataset size. We study this phenomenon, asking: Can difficult problems still contribute meaningfully learning? To perform this study, we use two training setups:
RL from base LLMs. In the first setup, we fine-tune base LLM using RL on mathematical problems. This setup mirrors standard approaches used in recent RLHF-style training pipelines where the reward is derived from correctness or other problem-specifii heuristics. This setup serves as our baseline, offering insights into how general-purpose (single-stage) RL affects reasoning across adopt Group Relative Policy Optimization (GRPO) [3), a variant of Proximal Policy Optimization (PPO) 38). has demonstrated remarkable performance common benchmarks [I5 29] and is more computationally efficient than PPO. More details about GRPO are provided in Appendix
wide range of problem difficulties. In particular,
Multi-stage RL from base LLMs. To further probe how RI. shapes reasoning under different conditions, focusing on varying difficulty, we introduce a second, more structured setup that aligns with curriculum learning principles [34]] In this multi-stage variant, we continue RL fine-tuning from the first-stage model checkpoint, on a subset difficult problems selected from the full training set. This stage is designed to further enhance the model's ability by learning from challenging samples. Within this setup, we explore three curriculum variants: (1) Mixed Difficulty uses a random mixture of easy and hard problems to maintain exposure diversity; (2) Hard-Only restricts training to difficult problems, concentrating learning on high- cosplcompyyxtty cases; and (3) Hard-Augmented (ours; SparkleRI.-PSS introduces partial solution scaffolding- -uuidiinngchhhhhhh as intermediate steps hints- to help the model navigate complex reasoning paths more effectively. These controlled variants allow us to assess how RL interacts with problem difficulty and solution augmentation. Additionally, we study how difficulty interacts with the planning, execution, and knowledge axes defined by our SPARKLE framework.
4 Experimental Setup
Next we provide the detail for the experiments we perform using the evaluation principles and approaches from Sections 2 and First, to remove the potentially confounding effects of supervised fine-tuning (SFT), we apply RL directly to base pretrained LLMs. This complements prior studies on 52).
the impacts of SFT and the interplay of SFT and RL
Reward design. We use a rule-based reward following Guo et al. [15]. which also mitigates the reward hacking problem of using reward model 9 10 12 We evaluate both answer correctness and solution format using the formula below:
2, if answer_ corser_correct(j. y) A format_correctt()
R(D.v)
1, if ansver_correct((8 v) A -fo.mattccoreccttt
11 otherwise
where ý represents the model's generated answer, y is the reference answer, answer correct evaluates numerical equivalence, and format_ pormat_correc((( assesses adherence expected answer format. This encourages the model to answer correctly with encouraged format correctness.
Training details. For Stage 1, we use the training set from DeepScaleR -Prevleww [29]. which contains 40K math questions spanning a wide range from AIME (1984-2023), AMC (pre-2023), MATH [17]. Still [22. and Omni-MATE [IT]. Our SparkleRL-Staan Stage 1 model is trained on these 40K problems using GRPO. To curate the training set for Stage 2, we first identify 6.5K most challenging problems
WWille difficulty- based sampling can be integrated into single-stage GRPO training by rejecting samples that are too easy or difficult based on their estimated advantages as done in DAPO [53 we opt for two-stage setup disentangle general RL effects from curriculum-dee driven improvements.
5

---

Model
A1ME24
AMC23
MATH500
GSM8K
OlympladBench
Avg.
Qwen-2.5 Swenn225-Math-- 1othh7B--i Base
16.67
42.50
44.03
42.53
28.65
35.23
SparkleRL-Stage
46.67
67.50
80.00 977
91.77
39. 39.11-10.48
65.01
SparkleRL-Stage 2-hard
41.67
65.94
80.50 p36. 47
92.45 82.
37 18.74
63.59
SparkleRL-Stuge 2-mix
40.00
233.33
63.44 :20.04
80.78
(36.6) 75
92.52 (40.90
38.85 18.25
63.12
SparkleRL-Stage 2-pss
50.42 133.76
71.25 128177
81.00 936.97
92.38 (403884 (49.55
40.11 40.11-11.48
67.03
Table I: Performance comparison of Qwen-2.5-Mathh
and tuned models from multi-stage RL. We report results of metric Avg@8. We bold the best results.
that the best Stage I model fails to solve after 20 attempts. We then validate this subset using a EPT--411-bsee Web Agent [36), followed by human verification to further filter out items with flawed solutions. This results a curated set of 5.7K difficult problems. For problems lacking reasoning traces, adopt reference solutions from NuminaMath (27].
For Stage 2, we initialize from SparkleRL-Stage and explore three fine-tuning variants. SparkleRL.- Stage 2-mix is trained on mixture of easy and hard problems. SparkleRL-SStage 2-hard is trained on the 5.7K most difficult problems identified from Stage 1. SparkleRL--Stage 2-pss trained on the same set of difficult problems, but with partial solution augmentation: each reasoning trace is divided into four semantic chunks, and for each problem, we construct multiple examples by providing between O and 4 chunks additional input context (Figure I left) The model is then prompted to complete the reasoning and arrive at the final answer.
5 Results
We present our main findings. Our evaluation focuses on both high-level performance outcomes and fine- fine-grained analysis of reasoning capabilities, guided by the following key questions:
Multi-Stage RL and Role of Problem Difficulty (Section T.T:: How effective multi-stage RL. at improving reasoning performance? How do problem difficulties impact RI.? We show that appropriately structured hard problems can provide additional benefits.
Sample Efficiency (Section 5.2h: Does RL improve the model's ability to solve problems with fewer samples? We show that RL-tuned models achieve higher performance at lower attempts compared to base models.
Plan Following and Execution (Section 5.3): How does RL impact the ability to follow externally provided plans? We find that RL-tuned models demonstrate improved flexibility in plan following yet often perform better with self- generaeeddteed planning strategies.
Knowledge Integration (Section 5.4): Can RL enhance model's ability use external knowledge? Our results reveal that R[-tuned models show significant improvements when provided with supplementary knowledge, while base models struggle.
Subproblem Resolution (Section (.5):: Does RL improve the model's ability to systematically solve decomposed problems? We observe that while RL substantially improves overall performance, all models still struggle with detailed subproblem resolution. Scaling Reasoning Benefits with Task Difficulty (Section 5.6): How do the benefits of knowledge and planning guidance vary with problem difficulty? We demonstrate that knowledge integration becomes increasingly valuable as problem difficulty rises.
5.1 Are Difficult Problems Still Valuable for RL training?
Table presents performance comparison between 0wen-2.5-Math-77B and several RL-tuned variants across five benchmarks, reporting Avg@8 scores. Stage-1 training on the full dataset establishes strong reasoning, achieving substantial gains across all benchmarks (an average of 29.78% improvement). For the second stage, we systematically investigate the impact of problem difficulty on RL training through three variants: training exclusively hard problems, using a mixture of easy and hard problems, and employing hard problems augmented with partial solutions (as detailed in Sec. 3 The results reveal that while further training harder
'The CoT traces used for partial-step scaffolding are of mixed quality and come directly from the existing datasets, no additional data generation external distillation is performed.

---

problems or mixedddifficul problems improves performance on simpler tasks like GSM8K (difficulty level 1/10) and MATH500 (difficulty level 1.5/10), performance decreases on more challenging benchmarks. Contrary to prior work suggesting that GRPO cannot benefit from the hardest problems due to absent positive reward signals [49], we demonstrate that hard problems remain valuable when appropriately structured. Specifically, while training solely on hard problems yields modest additional gains, augmenting them with partial solution guidance proves most effective, consistently improving performance across all benchmarks by enabling models to navigate complex reasoning paths more systematically"" This approach yields an average improvement of 2.02% over Stage 1 and AIME24- 4LME24-pprffrmmncc comparable to SoTA pure RL-tuned 32B models (50% on AIME24 [53]). Statistical significance tests confirming these improvements provided in Appendix
peak of 50.42% on the most challenging task
AIME24
5.2 Does RL Improve Sample Efficiency?
0.8
Figure presents pass@k results across models. In the following sections, we use SparkleRI-Stage 2-pss as the representative model for Stage since demonstrates the best performance among the three variants. We
0.7
0.6
0.5
observe two key patterns: (1) multi-stage RL with partial solution augmentation (SparkleRL-Stage 2-pss) consistently outperforms single-stage training, achieving higher performance lower k values compared to both Stage 1 and the base model; and (2) as k increases, the performance gap between all three models gradually narrows. However, improved sampling efficiency alone cannot fully explain the observed gains. These gains may reflect fundamental shifts in model behavior. What is really happening under the hood? Next, dig deeper to
0.4
0.3
Qwen2 5-73 SparkleRL-Stage Stage
SparkleRL
1
2
4
8
16
32
64
128
Number of Samples k
Figure 3: Pass@K comparison between Dwee-225-Math-- SparkleRL-Stage 1, and SparkleRL-Stage 2. uncover the specific capabilities enhanced by RL by dissecting reasoning through SPARKLE.
5.3
How does RL Impact Plan Following?
100
Owcn 2.5 Math 7B
21.8 02.4
additional plan w/ additional plan
SparkieRL-Stage
80
SparkleRL Stage
20.21.2716
Avgela
7.5.0
40
10.3
6
20
AIME24
AMC23
MATH500
GSMBK
OlympiadBench
Figure 4: Performance comparison of Qwen-2.5-Math-778 SpankleRL-Stage 1, and SparkleRL-Stage 2 with and without additional planning information. RL-tuned models (Stage and Stage 2) maintain performance with planning guidance, while the base model shows performance degradation in four out of five benchmarks when provided with plans. AIME24 exhibits the most pronounced effect where even RL-tuned models perform better without externally imposed plans.
RL-funed models demonstrate improved flexibility in plan following and execution. Planning is integral for problem-soll soliinsoovnn.... Surprisingly, we find that a valid plan, derived from human solutions, is necessarily good plan for models execute. Without externally imposed constraints, models can generate more reliable planning structures, such as functional Python code. Using predefined planning templates, while seemingly advantageous, paradoxically increases the likelihood of overlooking
'It is crucial to provide complete reasoning chunks rather than only the initial steps. as partial augmentations may limit the benefit of guided reasoning.
7

---

corner cases, resulting in incorrect final answers (Table Figure 4 quantifies these effects across benchmarks. The base model's performance drops every task except GSM8K. For elementary tasks like GSM8K (difficulty 1/10), the base model already possesses basic planning ability and benefits from explicit step-by-step instructions. For example, when the base model fails by attempting simultaneous calculations, it succeeds once guided to decompose the steps.
RL-tuned models, however, display stronger plan-following flexibility. Their performance remains stable or improves slightly with additional plans-except on the most difficult task, AIME24, both Stage I and Stage 2 see performance drop (e.g., Stage 2 decreases from 50.4% 47.9% Importantly, RL-tuned models consistently perform best when allowed to develop their own planning strategies rather than following numan-derived ones. This suggests that RL fosters internal strategies aligned with the model's 's reasoning dynamics, while externally imposed plans may conflict with the heuristics learned during training.
Additional SFT experiments (Appendix G confirm that RI. enhances plan-following flexibility beyond instruction following and Appendix reports additional results for the 32B model.
5.4
Does RL Improve Knowledge Utilization?
100
96.2 88.0
Qwen-2.5 M.een2.5-Math--0 Math-78 78
91.8K 92.44
knowledge knowledge
SparkleRL-SSagg
85.0
80
SparkleeL-Sttgee
67.5
Awg88
60
50.450.4
46,
44.0
44.3
39.1
28.5
20
15.2
11.2
AIME24
AMC23
MATH500
GSMBK
OlympiadBench
Figure 5: Comparison of Qwen-2.5-Math-72. SparkleRL-Stage 1, and SparkleRL-Stage 2 with and without knowledge information. The base model shows consistent performance degradation (avg. 5.4% decrease) when provided with external knowledge, RL-tuned models show significant performance improvements (4.3% and 4.2% avg. gains for Stage and Stage 2 models, respectively).
RL-tuned models exhibit enhanced knowledge integration capabilities despite inherent knowledge limitations. gggurercompparss the performance of Qwen-2.5-Matth-77, SparkleRL-Stage 1. SparkleRL-Stage 2-pss with and without access supplementary knowledge. The base model's performance consistently declines when given external knowledge (average drop of 5.4% across five tasks), indicating fundamental limitations in its ability incorporate external information into its reasoning process efficiently. In contrast, both RL-tuned variants show substantial gains when provided with the same
wewlllddeeeggaaaageeeee improvements of 4.3% (Stage 1) and 4.2% (Stage 2). This is a critical distinction between base and RL-tuned models and suggests that while these RL-tuned models still exhibit knowledge limitations, they have developed robust mechanisms for integrating new information during inference. A practical takeaway: rather than relying solely on continued RL fine-tuning which may risk catastrophic forgetting pees--dioviiggg targeted external knowledge is a simple and effective way to enhance performance on knowledge-intensive tasks.
Appendix F further shows that for the 32B model, knowledge augmentation provides larger gains than planning for RL-tuned variants.
5.5 Can RL Solve Decomposed Hard Problems?
RL-tuned models still struggle with detailed subproblem resolution. To test whether RL improves systematic problem decomposition, compare performance full problems versus their constituent subproblems. Figure 66 shows results for 2.ten-255-Mth--777 SparkleRL-Stage and SparkleRL- Stage 2-pss original problems their ability to solve all subproblems of those same problems.
Let P be a problem which can be decomposed into subproblems (2,.,8,..... 8K)- The subproblem success rate SSR(P) is defined as: 1 if the model correctly solves all subproblems
8

---

100
Owen 2.5 Math 70 SparkleRL-Stage
91.4
original problem subproblems
06.2
Subproblem success rete
80
SperkleRL-Stage
80.0
B1.0
67.5
60
50.4
40
42.5
37.5
40.2
32.5
29
15.7
15.7
26.7
2.3
AIME24
AMC23
MATH500
GSMBK
OlympiadBench
Figure 6: Comparison of Dwen-2.5-Math-777 SparkleRL-Stage 1, and SparkleRL-Stage 2 with original problems and average subproblem success rate. Results show consistent performance gap between solving complete problems and successfully addressing all constituent subproblems.
and 0 otherwise. For set of N problems {Pi, P2...., Pv }. the average subproblem success rate SSR = N Σn SSR(P.). Across tasks, both base and RL-ttuned models show large gap between original accuracy and SSR. On AIME24 the hardest benchmark-QQenn-222 Owen-2.5-Matt-77 reaches 16.7% accuracy on full problems but only 3.3% on subproblems; SparkleRL-Stage 2-pss achieves 50.4% on full problems but just 17.5% on subproblems. Viewed alongside the planning results in Figure 4 this suggests that RL-tuned models benefit from high-level planning guidance but remain weak at detailed decomposition and resolution. Even when subproblems appear simpler in isolation, the difficulty lies in solving every component consistently. Overall, current RL methods favor autonomous high-level strategies that align with a model's internal dynamics, while effective decomposed problem solving would likely require methods tailored to ensure consistency across subproblems.
5.6
A Closer Look at Knowledge and Planning by Difficulty Level
100
Owen-2 Owen-2.5-Mann 2-5tMtth--T Vanila
SparkieRL Stage 7B plan
100
-10
Spark elti Stage 2 Vanilla
Owen-3 S-Mtth It innowredy
Stage add, plan Goiin from add pan Loss from add, plan
Spark eRL Stage 2 knowledge
pAN0 (93
Gain from acd. knowledee
Lese from add. knowledge
20-
10
10
difcuRty level
diauuit loval
(a) With and without additional planning information. -t-tee models maintain stable across difficulties; base model degrades difficulty increases.
(b) With and without knowledge information. Knowledge augmentation benefits RI. models more, especially on harder problems.
Figure 7: Base model vs. RL-tuned model pass@) @1 1 by difficulty level.
Knowledge integration becomes increasingly valuable as problem difficulty rises, while planning benefits remain relatively constant. Figure 7a| and 7b show how planning and knowledge augmentation varies with problem difficulty. Across levels 1-8 (where sample sizes are reliable), supplementary knowledge consistently outperforms planning. and this advantage grows tasks become harder. For example, at level 7 (41 problems), knowledge augmentation improves RL- tuned performance by +4.9% while planning reduces accuracy by -4.9%; at level 8 (40 problems), knowledge yields a dramatic +42.5% gain compared to +15% from planning. Weighted across levels, knowledge provides average improvement of +4.53% compared +2.50% from planning.
For RL-tuned models, these results highlight that external knowledge is key driver of performance on complex problems, while planning contributes smaller and less consistent gains. In contrast, base models show the opposite pattern: planning often harms performance as difficulty increases, and knowledge yields only modest improvements. This asymmetry indicates that base models

---

are less capable of leveraging auxiliary information effectively, whereas RL-tuned models develop mechanisms integrate knowledge ways that meaningfully enhance performance.
Together, these findings reinforce our broader thesis: Rl. fundamenetlllly reshapes how models process and integrate auxiliary information. As difficulty increases, this ability becomes a key differentiator between base and RL-tuned models, particularly in knowledge-intenet reasoning tasks.
Related Work
Understanding Reinforcement Learning for LLM Reasoning. Gradient-barl based policy optimization
algorithms [5
38
39 with verifiable objectives have shown remarkable performance on reasoning-
intensive tasks
2 37
Curriculum-bais
methods B 24
41
57
such as difficulty-awai
Despite these advances, the mechanism by which RL shapes reasoning remains an open question. Previous studies have explored the interplay between SFT and RL in text-based [52] and visual environments [4), but they only involve a single-stage RL and do not dissect RL's effects beyond overall accuracy. Yue et al. [54] argued that RL-tuned models mainly reweight reasoning paths rather than creating new capabilities. Several recent surveys provide broader overviews of this evolving area [2 [3 [50 [56].
sampling, have been used to improve SFT and RL training for LLMs 19
26
48 53
Our work moves beyond accuracy metrics for LLM reasoning under multi-stage RL.
Diverse Aspects of LLM Reasoning. Reasoning LLMs has attracted significant attention in recent years [2] At its core, reasoning is a cognitive process that integrates evidence, arguments, and
logic to reach conclusions or judgments. Research in cognitive science [7 30 35 44) highlights
the interplay of knowledge, planning, and problem decomposition fundamental components of human problem solving. In the context of LLMs, knowledge retrieval and utilization [4 25 [
subproblem decomposition 8 23 46 58). and planning [45] have also been explored individually.
However, remains largely underexplored how RL shapes these crucial dimensions- a critical gap
address in this work.
Mathematical Reasoning Benchmarks. Mathematical problem-solving has become central testbed for evaluating LLM reasoning capabilities [1] 27
29 42 While earlier benchmarks such as GSM8K (5] and MATH [18] target grade- school and competition-leve mathematics, newer models perform strongly on these tasks, necessitating harder benchmarks such as AMC12 12 and AIME (32. Recent models show impressive results: OpenAl-o3 model scored 91.6% AIME2024, and DeepSeek-RI (28) reached 97.3% on MATH500. Nevertheless, most of these benchmarks despite their piility-provide only caase-grainee signals of reasoning ability with little insight into internal processes, motivating our more fine-gra one-grained diagnostic framework.
7 Discussion and Conclusion
We investigated if and how reinforcement learning shapes the reasoning capabilities of LLMs. To this end, proposed SPARKLE, fine-grained analytic framework that decomposes reasoning into plan following, knowledge integration, and subproblem solving. By augmenting existing mathematical reasoning benchmarks with human verification, we built the SPARKLE benchmark for detailed analysis. Our findings show that RL improves flexibility in plan following and knowledge utilization, yet compositional subproblem solving remains fragile. Interestingly, human-crafted plans can hinder RL-tuned models, which prefer autonomous, high-level strategies aligned with their internal reasoning. In contrast, lightweight external knowledge injection proves more beneficial, particularly for difficult tasks. We further introduce SparkleRL-PSS, multi-stage RL pipeline that reuses hard problems with partial step acaffolding-avoiie
new data generation while effectively guiding exploration.
Looking forward, we highlight two promising directions: (1) Data perspective: developing methods make diverse and imperfect data to provide effective guiding signals for RL, while aligning models' intrinsic reasoning strategies; and (2) Training perspective: systematically incorporating auxiliary hints for difficult problems, such as high-level plans, modular knowledge, or subproblem cues, into RL training to induce richer and more adaptive reasoning behavior. Overall, our framework, method, and findings shed light on if and how RL contributes to reasoning, offering practical insights for developing more intelligent, efficieni,,,,,, and interpretable RL pipelines for LLMs.
10

---

Acknowledgement
The authors would like to thank NeurIPS anonymous reviewers for their insightful feedback and helpful discussions. We thank Aditya Goyal for data verification. We are grateful for the support of the NSF under #2106707 and #2402833 and the Defense Advanced Research Projects Agency (DARPA Young Faculty Award).
References
[1] Aops wiki: Competition ratings. http::///rreeeoooooeeeelllrrrrer php/ AoPS_Wiki Competi ratinng Accessed: 2025-09-03.
[2] Anthropic. Claude 3.7 sonnet, February 2025.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41 41-48. ACM, 2009.
[4] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint urXiv:2501.17166 2025.
[5] Karl Cobbe, Vineet Kosaraju, Mohit Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv: 10.111100111 2021.
[6] Antonia Creswell, Murray Shanahan, and Irina Higgins. innection-ifferenee:: Exploiting large language models for interpretable logical reasoning. The Eleventh International Conference on Learning Representations, 2023.
[7] Jifí Dostál. Theory of problem solving. Procedia Social and Behavioral Sciences, 2015.
[8] Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1251-1265. Association for Computational Linguistics, December 2022.
[9] Tom Everitt, Mareus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198(Suppl 27):6435-6467, 2021.
[10] Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel.
[11] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07983 2024.
[12] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages
PMLR, 2023.
[13] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[14] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval augmented generation for large language models: 2024.
survey, arXiv preprint

---

[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint 2.0i:::500112944.5 2025.
[16] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0387) 2021.
[18] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Proc. NeurIPS, 2021.
[19] Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4552-4572. Association for Computational Linguistics, November 2024.
[20] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049-1065, Association for Computational Linguistics, Jul 2023,
[21] Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, al. A survey of frontiers in Ilm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint MxXiv:2504.00037 2025.
[22] Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li. Chengwei Qin, PeiFeng Wang, silvio savarese, Caiming Xiong, and Shafiq Joty. A survey of frontiers in LLM reasoning: Inference scaling, learning to reason, and agentic systems. Transactions Machine Learning Research, 2025.
[23] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023.
[24] Pascal Klink, Haoyi Yang, Carlo D' D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforcement learning via constrained optimal transport. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 11341-11358. PMLR, 17-23 Jul 2022.
[25] Miyoung Ko, Sue Hyun Park, Joonsuk Park, and Minjoon Seo. Hierarchical deconstruction of LLM reasoning: A graph-based framework for analyzing knowledge utilization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4995- 5027, November 2024.
[26] Hanyu Lai, Xiao Liu, Iat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: A large language model-based web navigating agent. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5295 5500.53366500000 2024.
[27] Jia LI, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: http:://hgggnnaaccco CCIAAAAAAAAAMiiiiaaaaaaroooaaaaa
com/ project-numingg aimo http://wwww...g.appaaaaaaaaaaaaa dataset. pdf) 2024.
12

---

[28] Wenfeng Liang, DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, and Junxiao et al. Song. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[29] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with 1.5b model by scaling rl. https::/ pttpss//pretty-rrrii-ooorfr radio- b75 notion site/
- SeepScaleR-Surppssiigg Surpassing 01 D1-Preview with a 1 5B Model o0t.:oaoooooooooooooooooo01.11o1 2025. Notion Blog.
RL 11011100001110000beaaaaa
[30] K. I. Manktelow. Thinking and Reasoning: An Introduction to the Psychology of Reason, Judgment and Decision Making. Psychology Press, 2012.
[31] Mathematical Association of America. American mathematics competitions, 2025.
[32] Mathematical Association of America. Maa invitational competitions, 2025. Information about the MAA invitational competitions including AIME, USAMO, and USAJMO that lead the selection process for the USA International Mathematical Olympiad team.
[33] Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models a survey. In Firsl Conference on Language Modeling, 2024.
[34] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50 2020.
[35] Allen Newell and Herbert A. Simon. Human Problem Solving. Prentice-Hall, Englewood Cliffs,
NJ, 1972.
[36] OpenAl. Introducing GPT-4.1 in the API, 4 2025.
[37] OpenAl. Introducing O3 and O4-mini, 4 2025.
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (tt.::/777.334// 777.06647, 2017.
[39] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
[40] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[41] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018
[42] Still Team. Slow thinking with 1lms: A series of technical report on slow thinking with 1lm. https://gi http///gtgtuuut.g thub. www/ Swa//RUCABBoo/SIU www.oowwwwionnn.iiiii.ee... witpg__wttt__________ 2025. RUCAIBox/Slow.- T'hinkingwwin
[43] Jianing Wang, Qiushi Sun, Xiang Li, and Ming Gao. Boosting language models reasoning with Chain-of-knowledde prompting. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 4958-4981, August 2024.
[44] Peter Cathcart Wason and Philip N. Johnson-Laird, Psychology of Reasoning: Structure and Content. Harvard University Press, Cambridge, MA, 1972.
[45] Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. PlanGenLLMs: modern survey of LLM planning capabilities. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 19497-19521. Association for Computational Linguistics, July 2025.
13

---

[46] Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8: 183-198, 2020.
[47] xAI. Grok 3 beta - the age of reasoning agents, February 2025.
[48] Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint ar 2rx.v:2252..47775 2025.
[49] Wei Xiong, Jiarui Yao. Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling reinforce. arXiv preprint (xXiv:2504.11443 2025.
[50] Fengli Xu, Qianyue Hao, Zefang Zong. Jingwei Wang, Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng. et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501. 109666, 2025.
[51] Evelyn Yee, Alice Li, Chenyu Tang, Yeon Ho Jung, Ramamohan Paturi, and Leon Bergen. Faithful and unfaithful error recovery in chain of thought. In First Conference on Language Modeling, 2024.
[52] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03377. 2025.
[53] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[54] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint trXhv:2504.1388 2025.
[55] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models the wild, 2025.
[56] Zhiyuan Zeng, Qinyuan Cheng. Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce ol from reinforcement learning perspective. arXiv preprint 2xXi::4422.1433: 2024.
[57] Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In Advances in Neural Information Processing Systems, volume 33, pages 13022-13033, 2020.
[58] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Lc, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023.
14

---

Appendix
Limitations and Societal Impacts
16
B SPARKLE Dataset Details
16
B.1 full example of the SPARKLE dataset
16
Experimental Details
22
Detailed Examples on Plan Following
24
Ablation: Impact of Model Size
29
Ablation: SFT vs. RL for Stage 2 training
29
Analyzing RL Gains: Instruction Following vs. Plan Following
30
Statistical Significance of Performance Gains
15

---

A Limitations and Societal Impacts
While our analysis offers detailed empirical view of how multi-stage RL shapes reasoning across plan following and execution, knowledge use, and problem decomposition, several limitations remain. We typically focus structured reasoning problems (e.g., math), and it may require adaptation for domains with less structured and explicit decomposition. The dataset construction process, though expert-validated, depends on human annotation and may face scalability challenges. Lastly, our findings are empirical; developing theoretical tools to characterize internal reasoning strategies remains an important direction for future work.
Limitations
Societal Impacts Our work contributes tools and insights for building more transparent and interpretable reasoning models. By identifying how RL enhances specific reasoning behaviors, framework can guide more targeted and efficient model development, especially in high-stake domains such education science. While datasets applying this methodology to broader domains will require careful attention to fairness and alignment. We hope this framework encourages more robust and trustworthy training practices for reasoning- capable LLMs.
math-focused and not privacy-sensitie
B SPARKLE Dataset Details
Following Section 2.2 we present more details for the constructed SparkleRL datasets.
SPARKLE is created based diverse mathematical problem benchmarks including AIME24 32 AMC23 (3I,, MATH500 (17), GSM8K (5). and OlympiadBench [16) (test splits). Each problem augmented with planning information derived from the groundtruth reasoning traces, relevant knowledge components, and sequence of subproblems curated via the pipeline introduced in Section 2.2 We also annotate the difficulty level (template shown in Figure () and mathematical domain following [II] (e.g.. linear algebra, geometry, number theory). The resulting SPARKLE benchmark contains 2,564 open-ended questions spanning 10 difficulty levels and domains.
B.1 A full example of the SPARKLE dataset
present a complete example of the SPARKLE benchmark in Figure 9 For the problem "One of Euler's conjectures.." we include its answer "144" and step-by-step solution "Taking the given equation modulo 2,3, and 5...". We also add difficulty level of "4" (averaged from three GPT-4.1 ratings), the domain "Number Theory + Congruences". and high-level solution plan "Step 1: Analyze the modular propertics... " related knowledge "Fact: Euler's sum of powers conjecture posited..." ", and smaller subproblems "Q1: What remainder patterns emerge..." This enable finer- grained evaluation of how reasoning models work and where they fail across different difficulty levels and domains.
Template for Labeling Difficulty Level
<system_role>>
You expert grader for mathematics problems. Given Problem and Solution, estimate the problem's difficulty on 1-10 scale according to AoPS standard.
Below is the AoPS standard for difficulty estimation: <requirements>
All levels are estimated and refer to averages. The following is rough standard based on the USA tier system AMC AMC AMC AIME USAMO/USAIMO- IMO, representing Middle School Junior High High School Challenging High School Olympiad levels. Other contests can be interpolated against this.
Notes: Multiple-choice tests like the AMC rated as though they are free-response. Test-takers can the answer choices as hints and therefore correctly answer more AMC questions than Mathcounts AIME problems similar difficulty. Some Olympiads are taken two sessions, with two similarly difficult sets questions numbered as one set. For these, the first half of the test (questions 1 1-3) is similar difficulty to the second half (questions 4-6).
Scale
1: Problems strictly for beginners, the casiest elementary middle-school levels (MOEMS, MATHCOUNTS Chapter, AMC 1-20, AMC 10 1-10, AMC 1-5, others involve
16

---

standard techniques introduced up to the middle-school level); most traditional middle/high--shhool word problems.
2: For motivated beginners; harder questions from the previous categories (AMC 8 21-25, harder
MATHCOUNTS States questions, AMC 10 11 11-20, AMC 12 5-15, AIME 1-3); traditional middle/high- school word problems with more complex problem solving.
3: Advanced-beginner problems that require more creative thinking (harder MATHCOUNTS Nationals questions, AMC 10 21-25, AMC 12 15-20, AIME 4-6).
4: Intermediate-level problems (AMC 12 21-25, AIME 7-9).
More difficult AIME problems (10-12); simple proof-hased Olympiad-style problems (early JBMO questions, easiest USAJMO 1/4).
6: High-level AIME-style questions (13-15); introductory Olympiad -eeeel questions (harder USAJMO
1/4 and easier USAJMO 2/5; easier USAMO and IMO 1/4).
7: Tougher Olympiad-level questions; may require more technical knowledge (harder USAIMO 2/5 and most USAJMO 3/6; extremely hard USAMO and IMO 1/4; casy-medium USAMO and IMO 2/5).
8: High-level Olympiad problems (medium-hard USAMO and IMO 2/5; casiest USAMO and IMO 3/6).
9: Expert Olympiad problems (average USAMO and IMO 3/6).
10: Historically hard problems, generally unsuitable for even very hard competitions (such as the IMO) due to being exceedingly tedious, long, and difficult (c.g., very few students worldwide are capable of solving them).
Examples
For reference, here are problems from each of the difficulty levels 1-10:
1: Jamie counted the number edges cube, Jimmy counted the number of corners, and Judy counted the number of faces. They then added the three numbers. What was the resulting sum?
I: Let trapezoid ABCD be such that AB || CD. Additionally, AC = AD = 5. CD 6. and AB = 3. Find BC.
How many integer values of satisfy |a| < 3n?
1: The whole number N divisible by 7. N leaves a remainder of when divided by 2, 2,3, 4, or 5.
What is the smallest possible value of N?
1: The value of two-digit number is 10 times the sum of its digits. The units digit is 1 more than twice the tens digit. Find the number.
1: The coordinates of AABC are A 7). B(1,, 7), and C(3. y) with y > 7. The area AABC
12. What is y?
1: How many different three-digit whole numbers can be formed using the digits 4, 7, and 9, assuming that no digit is repeated?
1.5: number is called slippy if its digits alternate between two distinct digits. For example, 2020 and 37373 are flippy, but 3883 and 1 1231 23 are not. How many five-digit flippy numbers are divisible by 15?
1.5: A rectangular box has integer side lengths in the ratio 1 3 : 4. Which of the following could he the volume of the box?
1.5: Two lines with slopes 1/2 and 5 intersect at (I, 1). What is the area of the triangle formed by these 4
two lines and the vertical line x = 5?
2: A fair six-sided die is repeatedly rolled until an odd number appears. What is the probability that every even number appears at least onec before the first occurrence of an odd number?
small airplane has rows of seats with 3 seats in each row. Eight passengers have boarded the plane and are distributed randomly among the seats. A married couple is next to board. What is the probability there will be two adjacent seats in the same row for the couple?
G. where b, and n are positive integers with f in lowest terms. What is the sum of the digits of the smallest positive integer for which is multiple of 1001?
2: Suppose that 2009 + 2012
2014
2.5: A, B, C' are three piles of rocks. The mean weight the rocks pounds, in the combined piles A and B is 43 pounds, and in the combined piles A and C' is 44 pounds. What is the greatest possible integer value for the mean, in pounds. of the rocks in the combined piles B and C??
40 pounds, B is 50
2.5: For some positive integer kt the repeating base-k representation of the (base-ten) fraction 1 is 0.23, = 0..232333.. .k. What is h??
3: Triangle ABC with AB 50 and AC 10 has area 120. Let 1 be the midpoint of AB, and let E he the midpoint of AC. The angle bisector BAC intersects DE and BC and G, respectively. What is the area of quadrilateral FDBG?
3: Wayne has green buckets, 3 red buckets. 3 blue buckets. and 3 yellow buckets. He randomly distributes 4 hockey pucks among the green buckets, with each puck equally likely to be put in each bucket. Similarly, distributes 3 pucks among the red buckets. 2 among the blue buckets, and among the yellow buckets. Once he is finished, what is the probability that green bucket contains more pucks than each of the other 11 buckets?
17

---

3: An object in the plane moves from one lattice point to another. At each step. the object may move one unit to the right. left, up, or down. If the object starts at the origin and takes ten-step path, how many different points could be the final point? Consider the integer
N = 9 99 999 9999
.9...99 99. digits
Find the sum of the digits of N.
3: Let ALMN have side lengths LM 15, MN 14, and NL 13. Let the angle biscctor of
Z. LN meet the circumcirele of ALMN a point T * L. Determine the area LLMT.
3.5: Find all thrce-digit numbers abc (with a * 0) such that a² + b² + c² a divisor of 26.
3.5: Consider polynomials P(æ) of degree at most 3, each of whose coefficients is in
{0. 2, 1, 5,6,77 7, 8,9). How many such polynomials satisfy P(-1) 1] -9?
3.5: Find the number of integer values of k in the closed interval 500, 500] for which the equation log(kz) log(2= + 2) has exactly one real solution.
3.5: In drawer there are at most 2009 balls, some white and the rest blue, which are randomly distributed. If two balls are taken at the same time, the probability that the balls are both blue or both white is 5. Determine the maximum possible number of white halls in the drawer such that the probability statement true.
3.5: Find three isosceles triangles, no two of which are congruent, with integer sides, such that cach triangle's ' area is equal to six times its perimeter. 4: Define sequence recursively by = 5 and
up 5x. + 4
xn-1
In + 6
for all nonnegative integers n. Let m the least positive integer such that
S 4 + 220
In which of the following intervals does lie?
4: An makes sequence of moves cube where move consists of walking from vertex an adjacent vertex along an edge. Initially the ant a vertex the bottom face and chooses one the three adjacent vertices as its first move. For all moves after the first, the ant does not return to its previous vertex but chooses to move to one of the other two adjacent vertices. All choices equally likely. The probability that after exactly 8 moves the ant is at vertex of the top face is m, where and n are relatively prime positive integers. Find m + n.
4: Find all real numbers a, b, c, such that
+ bbbb+cte + 20, ab + ad be + bd cd 150.
(C) [81, 242] 4: The vertices an equilateral triangle lie
(A) [9, 26]
(B) [27, 80]
(D) [243, 728
(E) 729,, 20)
hyperbola zy and a vertex this hyperbola the centroid of the triangle. What is the square of the area of the triangle?
4: Isosceles trapezoid ABCD has paraliel sides AD and BC, with BC < AD and AB = CD. There
is point P in the plane such that PA 1, PB = 2, PC = 3, and PD 4. What is BC/AD?
4.5: Find, with proof, all positive integers n for which 2n 12" + 2011" is perfect square.
4.5: Find the minimum value of
f(x) = x 2005
200T + 3x 2006
4x 2005
2006x + 2007 2008 + 2009
over all real numbers x.
4.5: Show that the cquation abb2 + 6² 2 + 302 - d² = 2005 has no integer solutions.
5: Triangle ABC has side lengths AB = 7, BC 8, and CA = 9. Circle passes through B and is tangent line AC at A. Circle we passes through C and is tangent to line AB A. Let K be the intersection of circles wi and wa not equal A. Then AK positive integers. Find m + n.
where mn n are relatively prime
5: A pair of integers (m, n) is called good if
m (n² + n) and (m + m).
Given relatively prime integers a, L, prove that there exists good pair (m, n) with a m and b n, but a / n and b m. 5: Let ABCD be a convex quadrilateral with ZDAC LBDC = 36". LCBD = 18 and LBAC 72". The diagonals intersect at point P. Determine the measure of ZAPD.
18

---

Call a positive real number groovy it be written in the form v vn + T for some positive integer n. Show that if x is groovy, then for any positive integer r, the number x" also gronvy. 5: Find all prime numbers p. q, such that 9 - r+1
4 1.
There are a + bowls arranged in row, numbered through a 6, where a and are given positive integers. Initially, each of the first a bowls contains an apple, and each of the last bowls contains a pear. A legal move consists of moving an apple from howl to bowl i + 1 and a pear from bowl j to bowl, j - 1, provided that the difference i is even. Multiple fruits may occupy the same howl. The goal is end with the first ò howls cach containing a pear and the last howls cach containing an apple. Show that this possible and only the product
even.
5: Solve the equation 3 5d = 2² in positive integers.
5: Find all triples (a, b, c) of real numbers such that
b b
a c'
b²
1 a²
62 62
1
5.5: Semicircle I has diameter AB length 14. Circle i lies tangent to AB at a point and intersects I at points Q and R. QR = 3/3 and <QPR 60%, then the area of 2 PQQ equals ovG where a and are relatively prime positive integers, and is positive integer not divisible by the square of any prime. Find + b + c. 5.5: Triangle ABC has LBAC = LCBA < 90", BC = 1, and AC 2 AB. Let H. I, and O be the orthocenter, incenter, and circumcenter of AABC, respectively. Assume that the area pentagon BCOTH maximized. What LCBA?
6: Given an acute triangle ABC. The incircle of AABC touches BC, AB at D, F, respectively. The angle bisector of /A meets DE and DF at K and I,, respectively. Suppose AA1 is altitude of AABC, and let. M be the midpoint of BC. (a) Prove that BK and CT. are perpendicular to the angle bisector of LBAC. (b) Show that A K. M L cyclic.
Let ABCD a convex quadrilateral. Let AC BD, and let E. H, F. and on AB, BC. CD, and DA, respectively, such that EF r GH = 1. If M = EG n AC and N = HF 0 AC, show that
AM
IN
1A
TM
CN
TC'
6: A 4 x 1 table is divided into 16 white unit square cells. Two cells are neighbors if they share side. A move consists of choosing cell and toggling the colors of its neighbors. After exactly moves all 16 cells are black. Find possible values of n.
6: A magic 3 x 5 board can toggle its cells between black and white. Define pattern to be an assignment of black or white to each of the 15 cells (so there are 2 patterns total). Every day after Day 1, at the beginning of the day, the board creates new pattern. However, the board always wants be unique and will die if any two of its patterns are fewer than cells diffcrent from each other. Furthermore, the board dies if it becomes all white. the board begins with all cells black on Day I, compute the maximum number of days it can stay alive.
Let a, b, e be positive real numbers such that + + = 4 Vabb.. Prove that
2(ab + be + ca) 44 min (²) 2 2 +2
6: Let MN be line parallel to side BC of triangle ABC, with M on AB and N on AC. The lines BN and C.M meet P. The circumeireles of triangles BM P and CN meet again at Q & P. Prove that ZBAQ = LCAP.
6: Let P be convex n-gon with n 2 3. Any set of n 3 diagonals of P that do not intersect in the interior of the polygon determines a triangulation of P into 2 triangles. If is regular and there is triangulation consisting only of isosceles triangles, find all possible values of 6: Let be the circumeircle of acute triangle ABC. Points D and E are on segments AB and AC, respectively, such that AD = -AE. The perpendicular bisectors of BD and C'E' intersect the minor ares AB and AC' of I at points J and G. respectively. Prove that lines DE and FG are either parallel or coincide.
6: Let AABC be an acute triangle with circumcirele w, and let H be the intersection of the altitudes of AABC. Suppose the tangent to the circumeirele of H BC at H intersects w at points X and Y. with HA 3, H = 2, and HY = 6. The area of ABC can be written in the form my/n, /n, where m and n are positive integers, and n is squarefree. Find m + n.
6.5: Let
28
P(x) = 24x22 +
> (24 22 + x 2.41)).
Let 21, R2. ....... 2. be the distinct zeros of P(z), and let 2े + but for ts = 1,2,..,,
.,, where
i = /-1 1 and de. by E R. Let
bul - m+ nyP.
19

---

where m,, rt, are integers and p squarefree. Find m + nt 6.5: Rectangles BCC, CAAICs, and ABB,A2 are erected outside acute triangle ABC. Suppose that
2BC1C ZCA B 180".
Prove that lines B2C,. A2, and B2 are concurrent.
7: We say that a finite set S in the plane is balanced if, for any two distinct points A, B E S, there exists a point C' E S such that AC = BC. We say that S is centre-free if for any three points A, B, C' S, there is no point P S such that PA PB PC. Show that for all integers 2 3, there exists balanced consisting of n points. Determine all integers centre-free set consisting of n points.
for which there exists balanced
7: Two rational numbers and positive integers. At any point, Evan may pick two of the numbers and y written on the board and write either their arithmetic
mn are written on blackboard,, where m and n are relatively prime
*+y their harmonic mean z+y on the board as well. Find all pairs
(m, n. such that Evan write the board in finitely many steps.
7: A 9 12 rectangle is partitioned into unit squares. The centers of all the unit squares, except for the four corner squares and the eight squares sharing a side with one them, are colored red. Is possible to label these red centers Ci, C2...., CaG in such a way that the following two conditions are both fulfilled: (i) the distances Ci Ca...., CosCos, CsaC1 are all equal / 13; (ii) the closed broken line C,C2 ... CoaCi has a center of symmetry?
7: Three nonnegative real numbers rı: r2; r3 are written on a blackboard. These numbers have the property that there exist integers a1, d3, not all zero, satisfying 41 + d2t2 dara = O. We may perform the following operation: find numbers 2 S y the blackboard, erase y. and write its place. Prove that after finitely many such operations. we can obtain least one 0.
7: Find the least possible area of a concave set in the 7-D plane that intersects both branches of the hyperparahola zyz = 1 and both branches of the hyperhola awy = 11 (A set $ in the plane is called convex if for any two points in S the line segment connecting them is contained in S.)
7: Find all integers 2 3 such that the following property holds: if we list the divisors of n! in increasing order as
<
de vol, then
d2 dj < dg d2
de dx-1-
7: Let P(z)) polynomial of degree >> with integer coefficients, and be a positive integer. Consider the polynomial (xx) P(P(... .. P(x)... ...)) Prove that there are most n integers / such
&
that (f) = t.
7.5: Let 2 be the set of integers. Find all functions f Z - E such that
f((2(()) -) v22x f(y)) f(x)² + f(y((N))
for all 2, Z with # O. 8: For each positive integer n, the Bank of Cape Town issues coins of denomination 1. Given finite collection of such coins (not necessarily of different denominations) with total value at most 99 + 2. prove that it is possible to split this collection into 100 or fewer groups such that each group has total value most Denote by S the set all positive integers. Find all functions S such that
f(f((m) (n)) m² + 2n² for all m, n S.
Prove that any monic polynomial of degree with real coefficients is the average of two monic polynomials of degree n with real roots. 8: Let H be an n x n matrix all of whose entries are +1 and whose rows are mutually orthogonal. Suppose H has an a x submatrix whose entries are all 1. Show that ab <
Let m positive integer. triangulation a polygon m-balanced triangles can be colored with m colors so that sum of the areas of triangles of same color the same for each color. Find all positive integers n for which there exists an on-balanced triangulation of a regular n-gon. (A triangulation of convex polygon P with 2 3 sides is any partition of P into 2 triangles by n diagonals that do not intersect in the polygon's interior.) 8: Given integer m. prove that there exist odd integers a, b and a positive integer such that
2m 19 + 99 + l 2000
8: Let S2...,, S100 be finite sets of integers whose intersection is nonempty. For each nonempty T C [S1,..., S100), ). the size of the intersection of the sets in T is multiple of (T). What is the least possihle number of elements that lie in at least 50 sets? 8.5: Let 7 be the incenter of acute triangle ABC with AB * AC. The incircle of ABC tangent sides BC, CAA. and AB E. and F. respectively. The line through D perpendicular EF meets
20

---

w at R. Line AR meets w again at P. The circumcircles of triangles PCE and PBF meet again at Q. Prove that lines DI and PQ meet on the line through A perpendicular to AI.
9: Let & be positive integer and let be finite of odd primes. Prove that there is most way (up rotation and reflection) to place the elements of S around a circle such that the product
any two neighbors is of the form x² + x + k for some positive integer æ. 9: For any a > 0, define the set S(a) = an] n = 1, 2, 2, ..... ..... Show that there are no three positive reals b, such that S(a) n ((b) = (b) n ((c) positive integer and real numbers 01 < 02 <
S(c) n (a)) = 1 and S(a) US(() U (.)(c) 21,2,3, ...... .J. 9: Given a
On such that Σ ns 1, prove that for any
x > 0,
Σ +
(a1 +
9: Point D is selected inside acute triangle ABC that LDAC LACB and LBDC 90" + ZBAC. Point E is chosen on ray BD so that AE EC. Let M line AB is tangent to the circumeircle of triangle BEM. A collection A1,...,AA Ax of (not necessarily distinct) subsets of ... n is called é-large if Aa 2 & for all sis k. Find, in terms of n and f, the largest real number such that
midpoint BC. Show that
9: Let n > 2 be an integer and let E
A n A A. |A,1
2
Σ 34
holds for all positive integers k. all nonnegative real numbers 21, 2i...., Lke and all é-large collections A1...., A. of subsets of (1. (1.2....,Te) (For a finite set S. denotes its cardinality.) 9: Let ABC be triangle with incenter / and excenters Ia. Io. Ie opposite A, B, and C, respectively, Given an arbitrary point on the cireumeirele of AABC that does not lic any the lines II. Isle. or BC, suppose the circumeircles of AII and A ADIIe intersect at two distinct points D and F. If E is the intersection of lines DF and BC, prove that LBAD = LEAC.
9.5: An anti-Pascal triangle is an equilateral triangular array of numbers such that, except for the numbers in the bottom row, each number is the absolute value of the difference of the numbers immediately below For example, the following is an anti anti-Pascal triangle with four rows which contains every integer from to 10.
6
8
10
9
Does there exist an anti- Pascal triangle with 2018 rows that contains every integer from 1 to 1 + ... + 2018? 9.5: Let ABC be an equilateral triangle. Let A1, B1, C1 be interior points of ABC such that BA1 A1C, CB, A,A, AC1 C₁ B, and
LBA,C + ACC.A+LCAA
480".
Let BC₁, and CB;, meet at Az, let CA₁ and AC; meet at B2. and let AB and BA, meet at Cz. Prove that if triangle A₁ B2Cr is scalene, then the three circumeircles of triangles AA, A2, BB, B2, and common points.
CC1Cs all pass through
10: Prove that there exists a positive constant such that the following statement truc: Consider an integer n > and a set of n points in the plane such that the distance between any two distinct points in S is at least 1. Then there is line i separating S such that the distance from any point of S to É is at least en 1/3 (A line separates set of points 10: Turbo the snail plays a game on a board with 2024 rows and 2023 columns. There are hidden monsters in 2022 of the cells. Initially, Turbo does not know where any of the monsters are, but he monster in each row except the first and last rows, and that each column
some segment joining two points in S crosses f.)
knows that there is exactly contains at most one monster. Turbo makes series of attempts to go from the first row to the last row. On each attempt, he chooses any cell in the first row, then repeatedly moves to an adjacent cell sharing a side (he may return to previously visited cell). If he reaches a cell with monster, his attempt start
ends and he is transported back to the first Turbo remembers whether each visited cell contains a monster. If he reaches any cell in last row, his attempt ends and game is over. Determine the minimum value for which Turbo has a strategy that guarantees reaching the last row on the n. attempt or carlier, regardless esss of the locations of the monsters.
attempt. The monsters do not move, and
10: Let Q be the of rational numbers. A function property holds: every æ, y E Q
- called aquaesulian if the following
f(x f(v)) = f(x) +y or flf(*) f(u)-
21

---

Show that there exists an integer e such that for any aquaesulian function f there are at most different rational numbers of the form / (r) + f((-) for some rational r. and find the smallest possible value of
10: Let v. be a positive integer. Nordic square is an n x board containing n², cach used exactly once. Two different cells are adjacent if they share an edge. Every cell that adjacent only to cells containing larger numbers is called a valley. An uphill path is sequence of one more cells such that. (i) the first cell in the sequence
integers from
valley; (ii) each subsequent cell is adjacent the previous cell; and (iii) the numbers written in the cells in the sequence
in increasing order. Find, as a function of n, the smallest possible total number uphill paths a Nordic square. 10: Let ABC be an equilateral triangle. Let A1, BI, C1 he interior points of A such that BA1 A,C, CB, = B,A.
C. and
LBAIC +CBIA+ = 480".
Let BC₁ and CB₁ meet at Ag, let CA₁ and AC1 meet B2, and let AB, and BA meet C2. Prove that triangle A B1Cr is scalene, then the three circumcircles triangles A.A1 A2, BB, B2, and CC all pass through two common points.
10: Let n be positive integer. A Japanese triangle consists of 1 2 ... + circles arranged in equilateral triangular shape such that, for each = 1, ...., nI, the 1t row contains exactly i circles, exactly one of which colored red. A ninja path in Japanese triangle cireles obtained by starting in the top row, then repeatedly going from a circle to one of the two circles immediatoly below it and finishing in the bottom row.
sequence of
10: Let > be integer and let d1...., On be nonnegative real numbers. Define S₁- Et (1) for k = 0, ,,..,.,. TR. Prove that
2h
Σ
Sk
(S.. So)
n
<'requirements>
The user will provide a problem and a solution. Your task is to estimate the problem's difficulty according to the AoPS scale described above. Output only single ISON object with the fields below. Do not include any extra text.
{
"difficulty"; <an integer from to 10, inclusive>, "reasoning": "<a concise explanation of the steps and logic used to assign the difficulty>"
}
If the difficulty seems borderline, choose the nearest integer (break exact ties upward). </system_roke role>>
puser_prompt>
MATH PROBLEM: {(math, prctchh__pbbmm
SOLUTION: {(solution)}
<'user prompt>
Figure 8: Instruction for grading problem difficulty level. {(math_probll problem)) and ((solution)) will be replaced with the specific question and corresponding solution during evaluation.
C Experimental Details
This section provides additional details on our training and evaluation following Section
Training and evaluation setup. We study two-stage RI. In Stage 1, we establish baseline model performance using a learning rate of lc-6 and KL loss coefficient of 0.001. For Stage 2, we investigate whether challenging problems remain valuable for a model already well-trained with large-scale RL from Stage To avoid substantial deviation from this well-trained model, we maintain the same configurations from Stage 1, except for increasing the KL loss coefficient to 0.01 to apply a stronger penalty on divergence. Throughout Stage 2, we use a sampling temperature of 0.6 and generate 32 samples per problem, prompt template as shown in Figure I0 During evaluation, we use a sampling temperature of 0.6 and a maximum context length of 16k tokens.
Training algorithm. To enhance the reasoning capabilities of our models, we employ Group Relative Policy Optimization (GRPO) (39], a variant of Proximal Policy Optimization (PPO) [8 that climinates the need for separate value model by estimating advantages through group-based
22

---

Problem:
One Buler's conjectures disproved in the 1960s by three American mathematiclans when they showed there positive Integer such that
183" + 110" + 84 +227
Find the valtre v..
Reasoning:
Taking the given equation modulo 3. and s. respectively, have
n' = (mod 2),"² (mud 3), (mod 5).
By either Fermat's Little Theorem (FLT) or inspection, we get (mod 2), 0 (mod 3),0 = (mod 6).
either the Chinese Remainder Theorem (CRT) or inspection, we get 24 (mod 30).
It is clear that 139, so the possible values for are 144, 174, 204, Note that
n² 135 110" + 84" + 27 199" 110" + (84 27) 15s' + 110" 111² x 3-155,,
from which
133
If 2 174, then
1.3
1.3
1..3 1.3 1.6- 1.6- 2.56 2.5
133
which arrives contradiction. Therefore, conclude that
144
Answer: 144
Difficulty Level:4
Domain: Number Theory
Congruences
Planning:
Knowledge:
Subproblems:
Step Analyze modular properties of both sides to determine posstble congruences for n modulo small primes.
Fact: Fuler's sum powers conjecture posited that least nth powers needed to sum to another nth power. Deftnitions:: 1) Modular arithmetic involves integers wrapping around upon reaching certain value, the modulus; The Chinese Remainder Theorem provides a way to solve systems simultancous congruences with pairwise coprime moduli.
Q:: What remainder patterns emerge when x² is divided by the primes and 5?
Qu: Based on the result from Qt, what is n modulo 30?
Step Apply Fermat's Little Theorem to deduce the congruences of modulo 2. and
Qs: What are values for greater
139 satisfy ne mod 30? an upper bound n by comparing the sum 185 110" 84"4 27 to n²?
Step Use the Chinese Remainder Theorem to combine these congruences and find specifte congruence modulo 30.
Q4:
Theorems: 1) Fermat's Little Theorem: If p is a prime number and an integer not divisible by p. then
Qs: What is the value of that satisfles all previous conditions and the original equation?
Step 4: Generate candidate values for that satisfy this congruence.
Step Estimate upper bounds for by comparing the sum on the hant-aan side to potential values of v², using inequalities.
(mod p); Chinese Remainder Theorem: If one knows the remainders of the division of am integer by several pairwise coprime integers, then one determine uniquely the remainder of the division of n by the product of these integers.
Step Test specific values within the narrowed range to identify the value of n that satisfies the original equation.
Figure Full example of benchmark instantiated from the SPARKLE framework. Given each problem and reasoning process (lop), we construct a high-level planning skeleton (bottom left) capturing the overall solution strategy, relevant knowledge (bottom middle) required for reasoning, and sequence of interconnected subproblems (bottom right) that decompose the solution process. We also annotated difficulty level (middle middle) and domain (middle right). This enables fine- grained understanding of reasoning capabilities and failure modes in reasoning models.
reward normalization. This design improves efficiency and stability in RL fine-tuning for LLMs. More specifically, for each question q. GRPO samples group of outputs {01, 02...... ...... oG) ) from the old policy Th and optimizes the current policy Ty by maximizing the following objective:
JGRPO (0) -E ((Q) (o. [G)
where (oR is given by:
G lod
no To q, Oi,tt) As, Te (or, q, 01,1)
min
0
To(oi. y, 0i,st) Tean (Oi, q, oi,et)
e, + +c) Ase BDD
clip
|| (ाe(]
Here, G is the number of sampled outputs per prompt, is the i-th sampled output sequence, and O is the token at step in that sequence. A is the advantage at token t, computed based on relative
23

---

Problem
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
Planning
First, determine the total number of eggs Janet has each day. Next, subtract the number of eggs Janet uses for breakfast and baking. The result is the numher of eggs available for sale. Then, multiply the number of eggs sold by the price per egg to determine the daily revenue from egg sales.
Model answer w/o provided planning
Model answer w provided plamning
First, let's determine how many eggs Janet sells at the farmers' market cach day:
First, we determine the total number of eggs Janet has cach day. Since Janet's ducks lay 16 eggs per day, the total number of eggs per day is 16.
Since Janet's ducks lay 16 eggs and she cats 3 eggs and bakes 4 eggs make muffins, she has 16 3 4 12 eggs to sell. [INCORRECT: Calculation failed,
Next, we subtract the number of eggs Janet uses for breakfast and baking. Janet eats eggs for breakfast every day and bakes muffins with 4 eggs every day, so she uses 3 + 4= 7 eggs for breakfast and baking. The number of eggs available for sale is the total number of eggs minus the number of eggs used breakfast and baking. So, the number of eggs available for sale is 16 7 9.
3 - 4 should be 9]
Perez's profit from selling these eggs: Since Janet sells each egg for $2, her daily profit is 12
x $2 $24.
Thus, the answer is $24
Then, we multiply the number of eggs sold by the price per egg determine the daily revenue from egg sales. Since Janet sells the remaining eggs at $2 per fresh duck egg, the daily revenue from egg sales is 9 x $2 $18.
The final result is [CORRECT: Answer is
18
Table 2: Comparison of solutions with and without planning information for the Qwen-2.5 2.5en2.5-Math--B model. The solution without planning information makes calculation error and arrives at an incorrect answer ($24), while the solution with planning framework correctly follows the structured approach and reaches the correct answer ($18).
rewards of the outputs inside each group (39.. Teet is the reference policy used for KL regularization, e is the PPO clipping threshold, and is the KL divergence penalty.
Software and hardware. We build upon the verl framework 40 for RL training. Our evaluation is based on modified version of im-evaluation-harnes. [3] using 8 NVIDIA H200, 15 NVIDIA A100-PCIE-40GB and NVIDIA A100-SXM4-40GH GPUs, with Python 3.10, PyTorch 2.4.0, and Transformers 1.47.1.
We conduct training and evaluation
Prompts and instructions. Following Section 5 we present detailed examples that support our findings. Figure [T shows instructions for providing additional planning information, Figure 12
presents instructions for providing additional knowledge information, and Figures [3 and 14 demonstrate instructions for asking subproblems when handling the first subproblem and sequencing subproblems, respectively.
D Detailed Examples on Plan Following
We present detailed example illustrating how RL-tuned models respond to planning information in Table 3 For challenging tasks like AIME problems, providing detailed planning information can actually impair performance. When presented with the problem alone, the RL-tuned model generates an appropriate solution strategy and arrives at the correct answer (809). However, when supplied with human-derived planning guidance, the model faithfully follows the general framework but fails to identify critical details specifically, it overlooks that n 0 (mod 5) are also losing ciattion-ultimatehhe yielding an incorrect result (405).
For simpler problems, both base and RL-tuned models benefit from additional planning when the guidance aligns with their internal reasoning processes. As demonstrated in Table 2 the base model leverages detailed planning information by decomposing the calculation 16 4, which it incorrectly computes as 12 without guidance. With detailed planning information, the model
24

---

Template for Training SparkleRL Models
A conversation between User and Assistant. The user asks math question, and the Assistant solves step by step. The Assistant first thinks about the complete reasoning process in the mind enclosed within <think> </think> tags. Then the Assistant provides a clear, concise answer to the user within <answer> </answer> tags, with the final result enclosed in \boxed notation.
For example:
<think>
reasoning process here
</think>
<answer>
The answer is \boxed(..... </answer>
User: {(question)) Assistant:
Instruction for training SparkleRL models. {{question)} will be replaced with the specific question during training.
Figure 10:
Template for Providing Additional Planning Information
conversation between User and Assistant. The user asks a math question, and the Assistant solves step by step. The Assistant first thinks about the complete reasoning process in the mind enclosed within <think> </think> tags. Then the Assistant provides a clear, concise answer to the user within <answer> </answer> tags, with the final result enclosed in \boxed{) notation.
For example:
<think>
reasoning process here
</think>
<answer>
The answer is \boxed(...}.
//answe>>>
User: {(question}}
Consider the following planning skeleton to guide your reasoning. You may adapt or extend this outline as needed based on your analysis of the problem: {(planning}}
Assistant:
Figure 11: Instruction for providing additional planning information. {[question]} will be replaced
with the specific question during evaluation, while { {planning)} replaced with a high-level solution plan for the given problem.
successfully breaks this into two steps: 3 + 4 = 7, followed by 16 - 7 = 9, yielding the correct result.
25

---

Template for Providing Additional Knowledge Information
A conversation between User and Assistant. The user asks math question, and the Assistant solves it step by step. The Assistant first thinks about the complete reasoning process in the mind enclosed within <think> </think> tags. Then the Assistant provides a clear, concise answer to the user within <answer> </answer> tags, with the final result enclosed [boxed notation.
For example:
<think>
reasoning process here
</think>
<answer>
The answer is \boxed(....:
//answer>
User: ((question)]
The following is reference knowledge base for concepts you might not be familiar with. If you already understand these concepts, you don't 't need to rely on this information. Otherwise, you refer this as needed: {(knoxledge))
Assistant:
Instruction for providing additional knowledge information. {{question}} will be replaced with the specific question during evaluation, while {(knowledge)} is replaced with relevant knowledge base content related to the problem.
Figure 12:
Template for Asking the First Subproblem
A conversation between User and Assistant. The user asks a math question, and the Assistant solves step by step. The Assistant first thinks about the complete reasoning process in the mind enclosed within <think> </think> tags. Then the Assistant provides a clear, concise answer the user within <answer> </answer> tags, with the final result enclosed in \boxed<() notation.
For example:
<think>
reasoning process here
</think>
<answer>
The answer is \boxed(..... </an </answer>
User:
Main Problem:
(main--poblest
We'll solve this incrementally. Your task is to solve only the current subproblem.
Current Subproblem:
ceurrent-subproble
Provide detailed solution showing your reasoning process. Express your final answer within \boxed(). Assistant:
Figure 13: Instruction template for handling the first subproblem. ((main-por (oaain-problem]] and question during evaluation, respectively.
are replaced with the specific main problem and current subproblem
26

---

Problem
Alice and Bob play the following game. A stack of tokens lies before them. The players take turns with Alice going first. On each trrn the player removes either token or 1 tokens from the stack. Whoever removes the last token wins. Find the number of positive integers less than equal to 2024 for which there exists strategy for Bob that guarantees that Bob will win the game regardless of Alice's play.
Planning
First, determine which positions are winning and which are losing for Alice by simulating the game's moves and marking outcomes recursively for small values of n. Identify the pattern that arises for losing positions. Generalize this pattern algebraically (modulo arithmetic).. Count the number of losing positions (where Bob can guarantee a win) for all n up to 2024 by applying the discovered pattern and summing such cases.
Model answer w/o provided planning
Model answer w provided planning
To solve this problem, need to analyze the game and determine the winning and losing positions. A position a losing position the player whose turn it is to move will lose if both players play optimally. [..]
To solve this problem, we need to determine the number of positive integers n I 2024 for which Bob has a guaranteed winning strategy. We can analyze the game by considering the possible moves and the resulting positions.
We can use dynamic programming determine the
1. Define Winning and Losing Positions:
winning and losing positions. Specifically, we can create an array dp where dp [i] is True if the position
A position n is a winning position if there is at least one move that leads to a losing position for the opponent.
i is a winning position and False if it is a losing position.
A position is losing position if every possible move Icads to winning position for the opponent.
The base case is dp [o] False, because if there are no tokens left, the player whose turn is to move
Base Cases:
loses.
If n. = 0, the player who is about to move loses because there are tokens left.
For each position i, we can check if there is a move
that leads to losing position:
If n 1, the player can remove token and win.
If 2 1 and dp [i-1] False, then dp[i]
n 4. the player can remove tokens and win. If n = 5, the player can remove I token, leaving 4 tokens, which is a winning position for the opponent.
True.
If i 2 1 and dp [i-4] = False, then dp[i] =
True. Otherwise, [i] False. We will then count the number of losing positions for
So, 5 is a losing position.
3. Recursive Cases:
For any n1, if
1 or n is losing position, then n is a winning position.
Bob, ie., the number of positions n I 2024 for which
dp [n] = False.
Otherwise, is losing position.
4. Pattern Recognition:
# Initialize the dp array
We can observe that positions that are congruent
dp = [False] * 2025
to 2 mod 5 are losing positions. This is because from any position n = 5k + 2, the only possible moves lead to positions 5k + 1, 5k 2, or 5 3,
Base case: dp[o] False dp [0] False
all of which winning positions for the opponent. [Failed to identify that n = 0 (mod 5) are also losing positions]
# Fill the dp array for i in range(1, 2025):
5. Counting Losing Positions:
if >= and not dp[i-1]:
We need to count the number of integers S 2024 that are congruent to 2 mod 5. The sequence af such numbers is
dp[i] True
elif i >= 4 and not dp
dp[i] True
2.7, 17,1,,..,,022.
else:
This is an arithmetic sequence with the first term
dp]i] False
a 2. common difference d = 5, and last term 1 2022.
# Count losing positions for Bob wiaiinngpssitios__ee count sum(
The number of terms in the sequence is given by
2022-2 +
2020 - + 1 = 404 + 1 = 405.
not dp[n] for n in range 2025))
INNORREET: Should count positions
mod 5 as well, yielding & of all positions]
print (wosing_ positionss_ccunt)
So, the number of positive integers n S 2024 for
# Output: 809
which Bob has guaranteed winning strategy is 405
The number of positive integers less than equal 2024 for which there exists strategy for Bob that guarantees that Bob will win the game regardless of
Alice's play is 809 [CORRECT: the answer is 809.]
Table 3: Comparison of solutions with and without planning information for problem in AIME24 for the SparkleRL-Stage 2-aug model. The solution without planning information arrives at the correct answer (809), while the solution attempting to follow the planning framework fails to identify all losing positions and reaches an incorrect answer (405).
27

---

Template for Asking subsequent Subproblems
conversation between User and Assistant. The user asks a math question, and the Assistant solves it step by step. The Assistant first thinks about the complete reasoning process in the mind enclosed within <think> </think> tags. Then the Assistant provides a clear, concise answer to the user within <answer> </answer> tags, with the final result enclosed {boxed{) notation.
For example:
<think>
reasoming process here
</think>
<answer>
The answer is \boxed(...].
</an /answer>
User:
Main Problem:
mmaai--probeen
We'll solve this incrementally. Your task solve only the current subproblem.
Below reference information about subproblems that we have solved and you can refer
needed:
Subproblem: (previiuu--eubrroblen))
Subproblem Answer:
Current Subproblem:
([current- sutrrenttsuuprobeemeee
Provide detailed solution showing your reasoning process. Express your final answer within
\boxed Assistant:
Figure 14: Instruction template for handling subsequent subproblems. The placeholder momain-problenile is replaced with the specific main problem, while (previous-Bne weteviiussuuuproo.e.o subproblent and wwwriiuussuuuprobo cwwroooblemaanee (emobleeemanswerr are replaced with all previously solved subproblems and their corresponding answers. For & previous subproblems, these placeholders are repeated times to provide complete reference information. The placeholder (ccurrent-subprobllee is replaced with the specific current subproblem question.
28

---

100
Dwen2.5-32
56.7
wio additional plan
Owen2.5-32B 2anm225-22--SmmgeRRl-oa
w/ edditional plan
70.9
Avg88
60
40
26.7
25.4 26.2
17.6
20.9
6.7
AIME24
AMC22
MATH500
GSMBK
OlympiadBench
Figure 15: Results with and without planning information for Qwen2.5-32B and Qwen2.5-32B- SimpleRI-ZZoo.
100
Qwen2.5-328
97.
w/o knowledge
Owen225-32B-Simglelle.
w/ knowledge
82.3
$3.3
80
66.7
Aug@a
60
47.0
41.3
25.4
24.3
20
10.6
AIME24
AMC23
MATH500
GSMBK
OlympiadBench
Figure 16: Results with and without knowledge information for Qwen2.5-32B and Qwen2.5-32B- SimpleRL-Zoo.
E
Ablation: Impact of Model Size
To study the impact of model size RL, conduct experiments with Qwen2.5-32B and Qwen2.5- 2DD-SimpleRL-Zoo [55]. Unlike our multi-stage RL approach with larger datasets, the SimpleRL-ZA model is trained on 8K dataset comprising GSM8K and MATH problems. The results are presented in Figures [53 In Consistent with our findings in Sections [5.3 [54] we observe that enowledge augmentation provides greater benefits than planning for RL-tuned models. Specifically, the vanilla performance of Qwen2.5-32BB 3-----2imppee--222-22
averages 63. 33.0% across the five tasks. When provided with additional planning information, the average performance increases 64.2% 1.2% gain), while additional knowledge information yields 99.8% performance (a 6.8% improvement over vanilla performance).
For the larger base model, we observe that both planning and knowledge augmentation provide benefits. This may be due increased model capacity that enables better utilization of auxiliary information. However, base models exhibit significant sensitivity to prompt variations due to limited instruction-follovinn capabilities, resulting high performance variance across different prompting strategies. In contrast, RL-tuned models demonstrate much more stable performance across varying instructions. For fair comparison, we report performance results in Figures 15 and 16
using the same
instructions in Figures II and 12
F Ablation: SFT vs. RL for Stage 2 training
Following Section 3 our best Stage 2 training (Stage 2-pss) focuses on the most difficult questions, where we augment the input with partial solutions. This choice is motivated by prior findings that RL methods with outcome- based rewards such as GRPO struggle to benefit from the hardest problems due to the scarcity of positive reward signals. A natural question then arises: if full solutions are available, why not directly apply supervised fine-tuning (SFT) on them?
While appealing principle, SFT is ineffective our setting for two main reasons:
Trace quality. The Stage training set contains over 40K math problems, with solution traces of highly variable quality-ranging ranging from incomplete or noisy
outputs

---

Model
AIME24
AMC23
MATH500
GSMSK
OlympiadBench
SparkleRI-SStage
46.67%
67.50%
80.00%
91.77%
39.11%
SparkleRL-Staee 2-pss SFT on hard problems with solutions
50.42%
71.25%
81.00%
92 08.38%
40.11%
15.00%
53.44%
70.03%
88.30%
30.70%
Table 4: Performance comparison of SparkleRI. models and SFT-tuned models. We bold the best results in each column.
clean human- answers. Generating new traces by distilling from stronger LLMs is possible, but it would incur substantial compute cost and require extensive human validation ensure correctness.
Empirical evidence. We conducted an additional experiment applying SFT directly on these noisy traces. As shown in Table SFT leads to notable performance degradation compared to SparkleRL-Stage 1, confirming that noisy full-solution supervision is detrimental.
We observe that SFT hard problems from the SparkleRL-Stage 1 model leads significant performance degradation across benchmarks. In contrast, our Stage 2-pss method yields consistent improvement.
In contrast, our Stage 2-pss method consistently improves performance across benchmarks. The key difference is that provide partial solutions for the hardest unsolved problems as input augmentations, rather than fine-tuning on full solutions. This design keeps the model in an on- policy setting, forcing it to continue reasoning rather than memorizing, which in turn strengthens the RL signal and yields superior reasoning performance.
G Analyzing RL Gains: Instruction Following vs. Plan Following
To better understand the source of RL's gains, we conducted experiments comparing supervised fine-tuning (SFT) and RL in scenarios where models are provided with explicit planning information. Specifically, we fine-tuned the base Qwen-2.5-Math-7B model on chain-of-thoupg (CoT) traces for hard questions, followed by optional RL training. We also tested each model variant with and without access to an externally provided plan. Results are shown in Table
Model
ATME24
AMC23
MATH500
GSMBK
OlympiadBench
Avg.
Owen-2.5-Math-
16.67%
42.50%
44.03%
44.30%
28.65%
35.23%
0wen-2.5-Math-77 (add.plan)
9.58%
30.94%
41.45%
46.92%
18.85%
29.55%
Qwen-2.5-Math--7B
15.42%
50.63%
68.55%
85.03%
31.44%
50.21%
Qwen-2.5 2..55MMth-7BB-SEE (add. plan)
14.17%
50.31%
69.08%
88.84%
31.78%
50.83%
2.een-25-Mathh-B-.
33.33%
64.06%
77.33%
91.58%
36.74%
60.61%
Qwen-2.5-Math- -ah--Math--BB-TT-RRL (add. plan)
36.25%
67. 67.19%
79.45%
94.47%
39.80%
63.43%
Table 5: Comparison of base, SFT, and RI. models with and without additional planning information. We bold the best results in each column.
We observe that:
Overall, as expected, SFT improves over the base model in absolute accuracy, but remains consistently below RL across all benchmarks.
In terms of poll-wog flexibility, SFT is not necessarily able use provided plan just like the base model, while RL-tuned models consistently leverage such plans to improve performance.
This finding suggests that RL improvements (at least in terms of plan use flexibility) cannot be solely attributed to improved instruction following.
Together, these results confirm that the gains from RL go beyond what SFT provides, aligning with our central claim that RI enhances flexibility plan following through mechanisms distinct from instruction following alone.
30

---

H Statistical Significance of Performance Gains
To verify that the observed performance improvements of SparklcRL-PSS are statistically significant for results in Table we conducted series of Welch's 's two-sample t-tests comparing Stage 2-pss model against Stage model across all benchmarks.
Table summarizes the comparative statistics.
Task
p-value
Cohen's
Sig.
AIME24
3.81
0.0030
1.91
Yes
AMC23
4.58
0.0006
2.29
Yes
MATH500
77.88
< 0.0001
3.99
Yes
GSM8K
6.96
< 0.0001
3.48
Yes
OlympiadBench
6.15
< 0.0001
3.07
Yes
Table 6: Welch's t-test results comparing Stage vs. Stage 2-pss on all benchmarks. All differences
are statistically significant (p < 0.01).
These findings confirm that the performance improvements from Stage to Stage 2-pss are statistically significant and robust across all evaluated tasks.
In addition, to support the specific claim that "RL-tuned models consistently perform best when allowed to develop their own planning strategies rather than following human-derived ones", conducted a Welch's 's t-test comparing SparkleRI. Stage 2-pss with vs. without planning on AIME24. The results indicate that providing human-derived plans leads statistically significant performance drop = 3.38, p = 0.0052 < 0.05, d - -1.69, large effect size), reinforcing that autonomous planning emerges more effective and generalizable strategy under RL fine-tuning.
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA). The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] " providddd a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to claborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found. IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are well supporue by the comprehensive experiments and analysis in Section 5 and further analysis results in the Appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We provide limitations in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
32

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex: system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide experimental details needed in Apendix C to support the reprodueibility of our experiments. Datasets and codebase with detailed instructions will also be released.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
33

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Both datasets and our code with detailed instructions will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
34

---

Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All experimental details are included in Appendix Cto facilitate understanding the results this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Ys]]
Justification:
We provide full results in Appendix ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide hardware and software details in Appendix
C
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
35

---

The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors have read the NeurIPS Code of Ethics and made sure the paper follows the NeurlPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide detailed societal impacts in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
36

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly cites the original paper or sources whenever an asset is used.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpsswiteeetteeeeeeeeeeeep thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code, dataset and models will be released with well-documented instructions. Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
37

---

Justification: We have included the complete instructions given to our expert human annotators in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: We obtained proper ethical approvals before hiring expert human annotators validate our generated results, ensuring all participants were informed of potential risks and provided consent according institutional guidelines.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We mainly use L.LM to summarize, judge and verify the generated data and have provided detailed discussion in Appendix B
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
38


Reviews:
[{"review_id": "ZAcsclj1Oo", "reviewer": "Reviewer_SZLe", "summary": "This paper proposes a number of methods for analyzing strengths and weaknesses of language model reasoning, and uses these methods to explore the effect of RL training (through GRPO) on reasoning in math domains. They also propose and evaluate a method for training through RL on difficult problems based on providing partial solution scaffolds.\n- The first angle they explore deals with planning versus executing – they test models with and without plans being provided, and surprisingly find that providing plans can often actually hurt the performance of models, especially base (non-RL) models, presumably because models are better at following the plans that they come up with instead of provided plans. \n- The second angle they study is on the ability to recall relevant background knowledge – they test models with and without relevant theorems/definitions/etc being provided in the prompt, and find RL based methods are much more strongly able to make use of provided knowledge (in fact it often hurts the base models!).\n- The third angle looks into the problems with providing plans (the first angle) in more detail by exploring how well the model can solve a subproblem given the ground truth solution prefix up to that subproblem. They find that models are far better at solving overall problems than solving the subproblem sequence.", "questions": "- Relating to both 6.3 and 6.5, I'm curious if you have any qualitative observations from looking at what sorts of human-provided plans (6.3) or subproblems (6.5) are difficult for the model to work with. Do you have any observations on how these plans and subproblems differ from those that the model comes up with and prefers to work with?\n- On my minor weakness about the Hard-Augmented results, it'd be helpful to hear a bit more of what kind of settings motivate this, or where you see this result being useful?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall I'm very positive on this paper, I think that it's an excellent fit for NeurIPS. It is beautifully written and presents a series of interesting results pertaining to a very hot area in the field right now (effects of RL on language models). The paper reads like an excellent example of good science – clear questions, well-designed experiments, beautifully presented. I'm not an expert in RL for LLMs and may not know all the related work, but given my understanding this is a solid, original contribution to the field.\n\nStrengths\n- I find the overall framework compelling. It's true that we need to move beyond accuracy based metrics, and I think that this is a nice breakdown into different aspects of the problem (planning, background knowledge, ability to execute plans).\n- The dataset construction pipeline including a final check by graduate students with a math background is an excellent step of verification and one that many papers don't do. And likewise during the hard-solution curation there's a graduate student verification step to find flawed solutions.\n- Section 6 is a nice presentation of a number of interesting results. The ordering and flow of the section made it quite easy to read.\n    - The results in 6.1 on training on scaffolded problems shows that the proposed method provides a reasonable boost. However I feel the main contribution of the paper is primarily in the other results.\n    - The result in 6.3 that externally provided plans decrease performance for many base/RL models over many domains (but that RL models are more robust to this) is particularly interesting – the idea that actually models are particularly good at following their own plans, and being forced into an externally provided plan can decrease performance. I appreciated the granular explanation of why that trend doesn't hold for GSM8K.\n    - The section 6.4 result that external knowledge harms the base model but helps the RL models was also really interesting, as was the 6.5 results on models having more trouble solving all subproblems than overall tasks.\n    - 6.6 was an important analysis to include for shedding a bit more light on the 6.3/6.4 result trends, showing how they vary with problem difficulty.\n- The writing of Section 3.1 is excellent, each of the axes is presented quite clearly. The style works well – leading with a sentence about the more general reasoning phenomenon the axis is trying to capture then getting very concrete and specific. And the high quality writing carries through the rest of the paper. The figures are also beautiful, the paper overall has a very high level of polish.\n\nWeaknesses\n- I don't have any strong weaknesses. \n- The Hard-Augmented model results (6.1) are interesting but don't feel too significant. In part because the approach requires having solutions (or partial solutions) to these problems, at which point one could just train on those solutions in the first place.\n\nMinor edits and suggestions\n- line 208: difficulty -> difficult\n- Figure 6 isn't labelled as \"Figure 6\" anywhere\n- Figure 6a caption: \"stable\" -> \"stability\"\n- I wonder if 6.6 could be moved up to be right after 6.4, since it's about how 6.3/6.4 vary with difficulty? Possibly 6.5 could fit well right after 6.3, since it's a bit of a look in detail into how things break down when solving human-provided problems. Just a thought / suggestion though, may or may not be better.", "was_revised": true, "final_justification": "Maintaining my Accept rating – my minor concerns were resolved in the rebuttal.", "rebuttal": "Thank you for your valuable feedback, thoughtful comments, and your support of our work! We address each question in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.\n\n> Q1: Any qualitative observations from looking at what sorts of human-provided plans (6.3) or subproblems (6.5) are difficult for the model to work with. Do you have any observations on how these plans and subproblems differ from those that the model comes up with and prefers to work with?\n\nYes, we obtained several qualitative observations while comparing model-generated reasoning with human-provided plans and decompositions. \n\n\n**Planning:** \n\nA common pattern is that **high-level human plans often lack sufficient granularity, particularly for more difficult problems**. This can lead models to overlook important edge cases, especially in mathematically nuanced settings.\n\nFor example, in Table 3 of the supplementary materials (AIME24), the provided planning instruction states: \"*Identify the pattern that arises for losing positions.*\" Following this plan, the model correctly identifies only one of the two correct losing scenarios (n ≡ 2 (mod 5)), but misses the other (n ≡ 0 (mod 5)), leading to an incorrect final answer.\n\nIn contrast, when no plan is provided, the same model generates Python code to perform dynamic programming and simulates the game for all n ≤ 2024. This strategy leads to the correct result (809), successfully capturing both losing scenarios. \n\nThis example illustrates a broader trend: RL-tuned models often perform better when allowed to explore the problem space using their internal reasoning mechanisms (e.g., generating code or applying strategies they are familiar with), rather than following externally imposed plans. While human-written plans can be helpful for simpler tasks (e.g., GSM8K), they may restrict model flexibility or bias the model toward **unfamiliar** or **suboptimal** strategies on more challenging tasks.\n\n**Subproblem decomposition:**\n\nWe observe a similar pattern when studying subproblem decomposition (Sec. 6.5). While human-provided subproblems are often semantically meaningful, breaking a problem into isolated steps can disrupt the model’s ability to reason holistically. In the same AIME24 example, a corresponding subproblem is phrased as:\n\nQ: What is the pattern of losing positions for n tokens?\n\nDespite being logically sound, the model still fails to identify one of the two key scenarios. This suggests that isolated subproblems can inherit the same brittleness as high-level plans if they are not grounded in the model’s natural reasoning strategies.\n\n\n> Q2: What motivates the Hard-Augmented setting and the possiblity of training on these solutions.\n\nThank you for the thoughtful question! We are motivated by prior work that suggests RL methods with outcome rewards such as GRPO *cannot benefit from the hardest problems due to the lack of positive reward signals* (i.e., all trajectories in a group yield wrong answers). We explore whether these hard problems could still be leveraged *in a way that improves performance*—without compromising the capabilities learned by a well-tuned RL model from Stage 1.\n\nWe agree that a natural approach would be to train on available solution traces with supervised fine-tuning (SFT). However, SFT does not yield promising results in our setting for the following reasons:\n\n- The training set used in Stage 1 contains over 40K math problems, where many solution traces are noisy (e.g., incomplete solutions). These traces vary in quality---from simplistic chain-of-thought outputs to human-written final answers. While obtaining new traces by distilling from competitive LLMs is possible, it would require substantial compute and additional human validation to ensure correctness and clarity. \n- Empirically, we conducted an additional experiment by applying SFT on these noisy traces. The results are shown below:\n\n| Model                     | AIME24     | AMC23      | MATH500    | GSM8K      | OlympiadBench |\n| ------------------------- | ---------- | ---------- | ---------- | ---------- | ------------- |\n|**SparkleRL-Stage 1**| 46.67%| 67.50%| 80.00%     |91.77%|39.11%|\n|**SparkleRL-Stage 2-aug** | **50.42%**| **71.25%**| **81.00%**|**92.38%**| **40.11%**|\n|**SFT on hard problems with solutions**|15.00%|53.44%|70.03%|88.30%|30.70%|\n\nWe observe that SFT on hard problems from the SparkleRL-Stage 1 model leads to **significant performance degradation** across benchmarks. In contrast, our **Stage 2-aug** method yields consistent improvement.\n\nThe key distinction in Stage 2-aug (see L202–216) is that we augment the hardest unsolved examples from Stage 1 with partial solutions provided in the input prompt, rather than full solutions used for fine-tuning. This design preserves on-policy learning, which is important for effective RL training, thereby further boosting the reasoning performance.\n\n> Typos\n\nThank you! We will carefully revise and correct all typos in the final revision."}, {"review_id": "sQVOEm5LhV", "reviewer": "Reviewer_wRv5", "summary": "The paper presents a framework for analysing the reasoning behaviours of LLMs along three dimensions: planning and execution, knowledge utilisation, and chain of subproblems. The authors also explore the use of curricula and solution guidance to improve the learning potential of hard problems when conducting RL for reasoning. Results demonstrate that these approaches improve the utility of hard problems during training and reveal that RL training reliably enhances two out of the three reasoning behaviours considered.", "questions": "Why does performance by difficulty level have a U-shape with knowledge augmentation?", "limitations": "Limitations are discussed in an Appendix, but should feature in the main text.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nBy not focusing on benchmark accuracy alone, the paper reveals some qualitative behaviours, which can be quantitatively measured, that are enhanced by RL finetuning. These insights provide intuitive insights into the general capabilities that translate to improved performance on reasoning tasks. The results also demonstrate that the proposed problem difficulty curriculum and solution augmentation do indeed improve learning from hard problems.\n\nWeaknesses:\n\nThe dual focus of analysing reasoning behaviours and improving learning from hard problems is disjointed. The proposed framework is also not comprehensive, as it fails to account for aspects such as the faithfulness of reasoning. While the paper reveals that problem decomposition is not useful, this unintuitive result warrants further investigation, or could even motivate an approach to making better use of problem decomposition. The impact of additional training is not accounted for; comparisons to SFT would and other offline methods such as DPO would be necessary to robustly show that it is RL training specifically that reinforces these behaviours. Though not necessarily a weakness, the findings of the paper are not particularly surprising and so their impact may be minimal.", "was_revised": true, "final_justification": "The paper makes progress towards quantitatively evaluating aspects of reasoning models besides their final performance, but the categorisation of three particular types of behaviours seems limited and is not likely to be of significant impact in its current state.", "rebuttal": "We appreciate your comments and suggestions and we address each question and provide further clarifications in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.\n\n> W1: The dual focus of analyzing reasoning behaviors and improving learning from hard problems is disjointed. \n\nThe dual aspect here **is intentional; in fact, we believe it is crucial for our overall findings**. Our work is centered on analyzing mathematical reasoning under reinforcement learning (RL). When applying RL to a set of problems, the key knobs are (i) the components necessary to solve a problem (planning, knowledge, etc.,) and (ii) the raw difficulty of the problems. A full characterization of RL for reasoning is not possible without addressing both of these elements. \n\nOur Hard-Augmented Stage 2 setting is not proposed as a general solution to the reasoning challenges observed, but rather as a complementary contribution: it demonstrates a practical method for leveraging hard problems in RL training—an area prior work has found difficult due to absent reward signals. Together, our dual contributions—(1) analysis framework and benchmark of reasoning behaviors across planning and execution, knowledge utilization, and chain of subproblems; and (2) a method to incorporate hard examples—provide stronger empirical evidence for understanding the effect of RL. The availability of two distinct RL-tuned models (SparkleRL-Stage 1 and SparkleRL-Stage 2-aug) allows us to conduct more fine-grained analysis than is typically possible in existing work.\n\n> W2: The proposed framework is also not comprehensive, as it fails to account for aspects such as the faithfulness of reasoning.\n\nWe appreciate this observation. Our work focuses on three core aspects of mathematical reasoning captured by the SPARKLE framework: plan-following, knowledge utilization, and subproblem decomposition. While faithfulness is indeed an important dimension, it is orthogonal to the goals of our current study. We believe that a fine-grained and dedicated analysis of these three components is equally valuable, as they represent foundational building blocks of mathematical reasoning. \n\nOur goal is not to exhaustively cover all aspects of reasoning—necessary trade-offs must be made to support a *focused investigation* with a clear central theme. We hope our work can serve as a cornerstone for future research to extend the framework, including incorporating dimensions such as faithfulness.\n\n\n> W3: While the paper reveals that problem decomposition is not useful, this unintuitive result warrants further investigation, or could even motivate an approach to making better use of problem decomposition.\n\nWe would like to clarify that our paper **does not claim that problem decomposition is not useful**. Instead, our findings reveal a **more nuanced perspective** on how decomposition interacts with RL-tuned models.\n\nIn our analysis of plan-following (Sec. 6.3), we observe that **RL-tuned models exhibit greater flexibility in generating and executing their own plans**. Interestingly, even valid human-written plans do not always yield improved performance—particularly on harder benchmarks like AIME24—because high-level plans may overlook critical edge cases. However, for simpler datasets such as GSM8K (difficulty level 1 out of 10), explicit plans consistently improve performance for both base and RL-tuned models. In these cases, planning helps reduce cognitive load by structuring the reasoning into manageable steps.\n\nIn our subproblem decomposition analysis (Sec. 6.5), we consider a problem unsolved if any subproblem is answered incorrectly. Under this strict criterion, RL-tuned models often struggle, especially when the decomposition is overly granular. Combined with our plan-following results, this suggests that **fine-grained decompositions may disrupt the model’s internal reasoning flow**, rather than supporting it.\n\nWe view this as an valuable direction for future research: rather than relying on human-authored decompositions, it could more effective to design model-aligned decomposition strategies that are better matched to the model’s inductive biases. However, proposing a new algorithm that leverages this phenomenon would require dedicated study and is beyond the current scope of our work. \n\n> W4: The impact of additional training is not accounted for; comparisons to SFT and other offline methods such as DPO would be necessary to robustly show that it is RL training specifically that reinforces these behaviors. \n\n\nThanks for bringing this up! We would like to clarify that our paper does not evaluate or include SFT or DPO methods **by design**. To avoid introducing confounding effects, we deliberately focus on comparing base pretrained models and RL-tuned models, isolating the impact of RL. This design choice complements prior studies that specifically analyze the interplay between SFT and RL [1,2] (see l.194–195).\n\nTherefore, it is **not** our objective to claim that RL, rather than SFT or DPO, **uniquely** reinforces these behaviors. Instead, our goal is to provide a focused, fine-grained analysis of reasoning behaviors that emerge specifically from RL training.\n\n\nTo ensure a clean comparison, we initialize both the base and RL models from the same pretrained checkpoint (Qwen-2.5-Math-7B) and apply RL training only, without any intermediate supervised fine-tuning (SFT) or preference optimization (e.g., DPO). This setup allows us to **attribute the behavioral differences directly to RL**, without contamination from other training paradigms.\n\nWhile comparisons to SFT or DPO are indeed important in the broader landscape, they are **outside the scope of this work**. Our goal is not to establish whether RL is superior to SFT or DPO, but rather to provide a **fine-grained analysis of how RL shapes reasoning behaviors**—an area that remains underexplored despite RL's widespread use in modern LLM pipelines.\n\nWe hope this clarifies our motivation and experimental design. We believe that narrowing the focus to RL is not a limitation but a necessary step toward understanding its unique contributions.\n\n\n[1] Chu et al., Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025\n[2] Yeo et al., Demystifying long chain-of-thought reasoning in llms, 2025.\n\n\n> Q: Why does performance by difficulty level have a U-shape with knowledge augmentation?\n\nThank you for this thoughtful question. The observed U-shaped trend in Fig. 11 primarily reflects the **natural imbalance in difficulty levels within existing math benchmarks**. Our framework is designed to **augment** existing problems—*not to construct new ones or rebalance the dataset*—so it inherits a skewed distribution, with only 4 total problems at levels 9–10. This sparsity at the top makes accuracy volatile and can visually exaggerate trends like the U-shape. We will clarify this in the revised manuscript.\n\nHowever, **this pattern does not contradict our central finding**: *knowledge augmentation is more beneficial than planning, especially for harder problems*. For example:\n- On level 8 (40 problems), knowledge augmentation improves performance by **+42.5%**, compared to a **+15%** gain from planning.\n- On level 7 (41 problems), knowledge gives a **+4.9%** gain, while planning causes a **−4.9%** drop.\n- When weighted averaged across all levels, knowledge yields a **+4.53%** gain, compared to **+2.50%** from planning.\nThese trends remain robust outside the sparsely populated level 9–10 range.\n\nThese trends are *robust across sufficiently sampled levels (1–8)* and support our conclusion that *external knowledge access is a key driver of performance on complex tasks*(l.334).\n\nWe also emphasize that difficulty levels are assigned based on AoPS Competition Ratings [[AoPS](https://artofproblemsolving.com/wiki/index.php/AoPS_Wiki:Competition_ratings?srsltid=AfmBOopoJxcbOKLhC-4La5S32ov_LvXuDDy5BLb-45hGUO9Q0pxDTh2Z)], which map to the structure of major U.S. math competitions: Levels 1–2 correspond to AMC 8, early AMC 10 and AMC 12 problems (e.g., standard middle school math), Levels 4–6 span through later AMC 12, AIME and intro level Olympiad-styple problems, and Levels 7–10 include USAMO, IMO, and other hard Olympiad-level problems.\n\nWe *deliberately avoid* using metrics like pass@k of model performance to define difficulty levels, as they may introduce model-induced inductive bias. Instead, we rely on competition-based standards to provide an objective and interpretable standard. While pass@k filtering would allow easier balancing, it introduces model-induced inductive bias, which we explicitly avoid."}, {"review_id": "dnj5nZGYsg", "reviewer": "Reviewer_1rRb", "summary": "This paper analyzes various interventions in RLVF posttraining on top of Qwen-2.5-Math-7B in order to better understand what limits reasoning performance: planning, knowledge, problem decomposition, or problem difficulty. The paper proposes a new dataset of 2,564 questions called SPARKLE which augments popular math benchmarks (such as GSM8K, MATH, AIME, etc.) with (1) a suggested plan for solving the problem; (2) relevant knowledge/concepts needed to solve the problem;  and (3) a set of subproblems and corresponding answers that need to be solved in the course of solving the whole problem. Ablations that either include or exclude the plans, knowledge, and subproblems demonstrate that: plans harm the base model’s performance while leaving the RL-trained model performance unchanged; knowledge harms base model performance but slightly improves RL-trained model performance; and all models seem unable to reliably solve subproblems. Additionally, the paper demonstrates that continued RL training on the most difficult problems augmented with hints can lead to the best performance.", "questions": "### Unsupported or overly broad claims\n\n* I think the drops in performance seen by the base model might just be due to poor instruction following. I therefore think that (1) the paper should state this is the reason for the drop in base model performance, and (2) Qwen-2.5-Math-7B-Instruct should really be included as a baseline in order to disentangle whether it is the lack of instruction-tuning that leads to these drops or the lack of reasoning. My prediction is actually that including things like the plan will help Instruct quite a lot, but maybe not as much as doing RLVF (i.e. base \\< instruct \\< instruct+plan \\< rl $\\\\approx$ rl+plan). **I would need to see Instruct added as a baseline to consider increasing my score**.  \n* Some of the claims need to be backed up by statistical tests—e.g. In the discussion in paragraph starting line 285, it’s not clear to me that the increases/decreases in performance are statistically significant. If you would like to make claims like “RL-tuned models achieve superior performance when permitted to develop autonomous planning strategies” then it is important to establish whether that superior performance is statistically significant. (Also, in that particular case it would also be helpful to clarify that you are specifically talking about AIME24). **I would need to see statistical tests included throughout the paper in order to consider increasing my score**.   \n* “Improved sampling efficiency alone cannot fully explain the observed gains” (line 267\\) → Why not? I didn’t understand this argument, or how the subsequent experiments disprove it.  \n* “RL-tuned models show greater flexibility in following different plans compared to the base model” (line 285\\) → how do you know they are actually following the plan rather than just ignoring it and doing their own thing?  \n* “Knowledge integration is a key driver of performance on harder tasks” (line 334\\) → I’m not sure this conclusion is correct. It is hard to know for sure without seeing the distribution of difficulty, but based on the pass@1 rates it looks like there might be very few problems at higher difficulty levels (e.g. maybe just one at difficulty 10, and three at difficulty 9?). If so, the fact that the solve rate *looks* high might just be the result of high variance. It would be much better to (1) report the avg@8 score (as in the other plots), (2) to include error bars, and (3) to report how many questions there are at each difficulty level, in order to make sure the interpretations are correct.  \n* Some of the language used in the paper is overly broad given that only a single model (Qwen-2.5-Math-7B) was used. For example, the abstract says “RL-tuned models exhibit greater robustness… than their base counterparts” (line 11), but there is only one base model examined\\! The language in the paper needs to be softened to clarify that Qwen 7B is the only model studied: I think this is ok because the benchmark is still interesting, but the claims in the paper need to be made more precise because without testing other models it is not possible to generalize the results in the paper to them.\n\n### Missing details\n\n* Significantly more detail is required to explain how SPARKLE is constructed. For example, which prompts are used to produce the plans/knowledge/decomposition? What prompts are used for the verification agent? How do you ensure that the subproblems all have correct answers and are solvable? How do you determine difficulty level? This doesn’t need to be in the main text, but at least should be in the appendix. **I would need to understand these details before considering raising my score.**  \n* I don’t understand how the “partial solution guidance” works. Could you please provide an example of what the prompt looks like? How do you choose the boundaries for the “chunks”? How is it different from methods for reverse curriculum training, e.g. [https://arxiv.org/pdf/2402.05808](https://arxiv.org/pdf/2402.05808) ? (Some of the details are scattered throughout the paper, e.g. where the reasoning traces come from—it would be really helpful to include a dedicated section in the appendix restating some of those things and explaining the whole procedure in more detail). **I would need to understand this better to consider raising my score.**  \n* What is the distribution of difficulty level (i.e. how many questions at each level)? It would be helpful to include a histogram in the appendix.  \n* Are the results reported in Figure 6 over the entire SPARKLE benchmark? It would be helpful to be more clear about this in the captions and text.\n\n### Other questions/comments\n\n* Please include error bars in Table 1 and all of the figures.  \n* In Figure 3, 4, & 5 (and in the appendix too) it would be helpful to include an “all” group which shows the average across all tasks.  \n* For the subproblem decomposition experiments, what is the average solve rate for the subproblems individually (not altogether)? And how many subproblems are there per problem (maybe you could include a histogram in the appendix)? It might be interesting to discuss your results in relation to [https://arxiv.org/pdf/2410.01748v1](https://arxiv.org/pdf/2410.01748v1) which I think has some similar findings.  \n* “RL tuning methodologies should prioritize high level reasoning strategies rather than detailed subproblem processing” (line 325\\) → couldn’t you draw the opposite conclusion, i.e. that we should include more compositional training data to improve the models’ performance on solving compositional tasks like these?", "limitations": "The paper should acknowledge that it only studies a single model (Qwen) and therefore it is unclear how well the results generalize.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* Clarity: I think the paper is structured well and it very clearly lays out the dimensions that are being studied.  \n* Significance: understanding RL reasoning performance at a more fine-grained level—beyond just accuracy on various benchmarks—is an important and burgeoning area of research.  \n* Originality: The various findings with Qwen 7B are very interesting and I believe will be of interest to the RLVF community: (1) RL performance can be boosted on hard problems if partial solutions are available; (2) performance of RL-trained models seems to be limited more by knowledge retrieval ability rather than planning ability; and (3) all models struggle with solving decomposed problems.\n\nWeaknesses:\n\n* Quality: there are a number of unsupported or overly broad claims. I detail these in the “Questions” section below, but will call out the two most important ones:  \n  * The paper is missing an important baseline, which is to compare to Qwen-2.5-7B-Math-Instruct. Some of the claims made in the paper about the base model may just be due to poor instruction-following capability, and it’s not clear whether the RL-trained models do better because they are better at instruction-following or because they’ve been trained with RLVF. Including an instruction-tuned baseline would help to clarify this.  \n  * There are no error bars or statistical tests reported in the paper, and some of the claims are made based on very small differences that could be due just to statistical error.  \n* Clarity: there are a number of missing details that make it difficult to fully evaluate the paper. I detail these in the “Questions” section below, but most importantly, there needs to be more detail about (1) how the SPARKLE dataset was constructed, and (2) how the “partial solution guidance” works.  \n* Quality: only a single model (Qwen-2.5-7B-Math) is studied in the paper, which limits the generality of the claims.", "was_revised": true, "final_justification": "My original concerns were: (1) lack of comparison to an SFT baseline, (2) no statistical tests, (3) missing details about how SPARKLE was created, and (4) missing details about partial solution guidance. In the rebuttal, the authors reported an analysis addressing (1), have promised to add (2), and have clarified (3) and (4). Thus all my primary concerns have been addressed and I recommend acceptance.", "rebuttal": "We sincerely appreciate your thoughtful comments, detailed feedback, and support of our work! Due to word limit, we provide concise responses below. We are more than happy to elaborate further in the discussion phase.\n\n> Q1: Could the drop in base model performance stem from poor instruction-following? Please include Qwen-2.5-Math-7B-Instruct to isolate instruction tuning from reasoning.\n\nWe did not include Qwen-2.5-Math-7B-Instruct in this work because, despite its name, it is **not a pure instruction-tuned model**. As noted in [1], it was trained by SFT and GRPO, incorporating additional CoT and tool-use supervision. This makes it difficult to disentangle instruction tuning from more general improvements in reasoning capability.\n\nOur focus is on comparing **base pretrained** vs. **RL-tuned** models to isolate the effect of RL on reasoning behavior. Notably, even the **base model shows substantial instruction-following ability**. In Fig. 2, scaling up the number of samples allows the base model to match RL-tuned model performance. On GSM8K, performance improves when planning information is provided—demonstrating plan-following capability even without SFT or RL (l.279–284, Fig.3).\n\nWe will clarify this distinction in the final version and are happy to include results for Qwen-2.5-Math-7B-Instruct if the reviewer would find it helpful.\n\n> Q2&O1: Claims like “RL-tuned models achieve superior performance...” need statistical backing. Include error bars in Table 1 and figures.\n\nThank you for this important suggestion. We have conducted detailed statistical analyses and updated the paper accordingly.\n\nTo support the specific claim in l.289—that “RL-tuned models achieve superior performance when permitted to develop autonomous planning strategies”—we conducted a Welch’s t-test comparing SparkleRL Stage 2-aug with vs. without planning on AIME24. Results show: t = 3.38, p = 0.0052 (< 0.05), Cohen’s d = −1.69 (large effect size). This confirms the performance difference is statistically significant, supporting our conclusion.\n\nIn addition, we now include mean ± standard deviation in Table 1 and have added error bars throughout the revised figures:\n|Model|AIME24|AMC23|MATH500|GSM8K|OlympiadBench|\n|-|-|-|-|-|-|\n|Qwen-2.5-Math-7B-SparkleRL-Stage 1|46.67% ± 2.11%|67.50% ± 1.58%|80.00% ± 0.13%|91.77% ± 0.12%|39.11% ± 0.21%|\n|Qwen-2.5-Math-7B-SparkleRL-Stage 2-aug|50.42% ± 0.99%| 71.25% ± 1.12%|81.00% ± 0.27%|92.38% ± 0.17%|40.11% ± 0.32%|\n\nWe also ran Welch’s t-tests comparing Stage 1 vs. Stage 2-aug across all datasets:\n|Task|t|p-value|Cohen’s d|Sig.|\n|-|-|-|-|-|\n|AIME24|-3.81|0.0030|1.91|Yes|\n|AMC23|-4.58| 0.0006|2.29|Yes|\n|MATH500|-7.98|<0.0001|3.99|Yes|\n|GSM8K|-6.96|<0.0001| 3.48|Yes|\n|OlympiadBench|-6.15|<0.0001|3.07|Yes|\n\nThese results confirm **performance improvements from Stage 1 to Stage 2-aug are statistically significant and robust across tasks**. We will incorporate these findings and the requested error bars in the final version.\n\n> Q3: Why not attribute the gains to improved sampling efficiency alone?\n\nWe agree sampling efficiency can contribute to performance gains (as seen in Fig. 2). However, our goal is to explore *what additional capabilities RL may confer*. Key evidence:\n- Plan handling: RL-tuned models exhibit greater robustness and lower variance across planning variants (Sec.6.3, Fig.3).\n- Knowledge use: RL-tuned models can leverage external knowledge more effectively than base models (Sec.6.4).\n\nThese differences suggest that RL affects **model behavior**, not just sampling variance. This motivates our SPARKLE analysis, which decomposes reasoning across plan following, knowledge use, and subproblem solving.\n\n> Q4: How do you know the model is following the plan rather than ignoring it?\n\nWe define *plan-following flexibility* as the model’s ability to **interpret, adapt, or selectively use planning information**. All models receive identical prompts (see Fig.9), including: “Consider the following planning skeleton... you may adapt or extend this outline.”\n\nUnder this setup, RL-tuned models are more robust and consistent (Fig. 3), implying they can integrate or disregard plans as appropriate. Base models show higher variance, often underperforming when given plans—suggesting lower flexibility or misalignment.\n\n> Q5: How are difficulty levels assigned? Are there enough high-difficulty samples to draw conclusions?\n\nWe assign difficulty levels using AoPS Competition Ratings [2], which aligns with U.S. math competitions (e.g., AMC, AIME, USAMO, IMO). To avoid inductive bias, we *deliberately do not use pass@k to assign difficulty*--ensuring *objective and interpretable levels*.\n\nSince SPARKLE enriches existing benchmarks without creating any new problems, our dataset inherits their natural skew--only 4 problems are level 9–10. We avoid drawing conclusions from the tail and focus on levels 1–8, where sample sizes are more reliable (e.g., 40–41 examples at levels 7–8).\n\nAt these levels: \n- *Knowledge augmentation* improves RL-tuned performance by *+4.9% (level 7)* and *+42.5% (level 8)*. \n- *planning* causes a *−4.9% drop (level 7)* and a *+15% gain (level 8)*.\n- Weighted across levels, knowledge yields an average gain of *+4.53%* vs. *+2.50%* from planning.\n\nThese trends support our claim in l.334 that *knowledge integration is key to solving harder problems*. We will include difficulty histograms and clarify this in the revision.\n\n> Q6: Some claims are too broad given only one model was tested.\n\nWe will revise the language to clarify this. We also include results for Qwen-2.5-32B in App. D.2, which exhibit consistent trends.\nWe experimented with other open models (Mistral-7B, LLaMA-3.1-8B), but they fail to achieve non-trivial performance even after RL tuning (e.g., avg@32 0.2% on AIME24), making meaningful reasoning analysis infeasible. This further motivates our focus on models with strong mathematical priors.\n\n> Q7: What prompts are used for plan/knowledge/decomposition? How do you ensure correctness?\n\nSPARKLE is constructed using a pipeline (Sec. 3.2) where GPT-4.1 (with Internet access) is given: the original question, ground-truth answer, and expert step-by-step solution.\n\nWe prompt it to generate:\n- Planning skeleton: 2–3 sentence outline of the reasoning path\n- Subproblem decomposition: step-by-step breakdown into well-posed, answerable subquestions\n- Knowledge extraction: key definitions, facts, or theorems (retrieved or inferred)\n\nEach annotation is then checked by a second GPT-4.1 verifier agent (prompted to assess correctness, completeness, coherence, pedagogical value). Human validators (math grad students) perform final verification.\n\nWe will incorporate more of these implementation details in the revised appendix.\n\n> Q8: Please clarify how partial solution guidance works. How's it different from [3]?\n\nOur partial solution guidance augments selected training problems with intermediate solution chunks.\n\nGiven a solution (excluding the final answer), we heuristically split it into 4 segments (e.g., by sentence) and create variants:\n- V1: question only\n- V2: question + chunk 1\n- V3: question + chunks 1–2\n- V4: question + chunks 1–3\n- V5: question + chunks 1-4\n\nThis differs from [3] in two key ways:\n- We do **not enforce an easy-to-hard progression** or assume the model can solve subgoals in isolation.\n- We *train in a single stage (Stage 2)* with mixed supervision levels, rather than gradually increasing difficulty.\n\nOur method provides a clear and practical way for incorporating hard problems during RL. Even when the traces are human-written and misaligned with the model’s internal reasoning, RL-tuned models still learn useful strategies and show improved performance.\n\n> Q9&Q10: Is Fig 6 averaged over all tasks? Can you add an “All” group to Fig 3–5?\n\nYes, Figure 6 reports results over the entire SPARKLE benchmark—we’ll clarify this in the caption and text. We will also add an “All Tasks” average to Figures 3–5 and include summary plots in the appendix.\n\n> Q11: What’s the subproblem-level solve rate? How does it relate to [4]?\n\nThank you for this suggestion and we present the subproblem solve rate with average difficulty level for each task below:\n|Task|AIME24|AMC23|MATH500|GSM8K|OlympiadBench|\n|-|-|-|-|-|-|\n|Avg. Difficulty Level|3.6|2.1|1.5|1.0|3.1|\n|Subproblem Solve Rate|53.94%|72.96%|84.92%|95.60%|66.96%|\n\nWe find that **subproblem solve rate drops as difficulty increases**, showing that decomposition alone is not sufficient to solve complex tasks—models still struggle with intermediate steps, especially under higher cognitive load. Most problems decompose into 4–6 subproblems; we’ll include histograms in the appendix.\n\nCompared to [4], our decompositions:\n- Are semantically aligned, derived from expert step-by-step solutions.\n- Span harder benchmarks beyond GSM8K (e.g., AIME, Olympiads).\n- Are studied under RL, which show that solving full problems does not imply solving subproblems (a gap unseen in prior work). This shows RL-tuned models may solve holistically, but are brittle under explicit decomposition.\n\n> Q13: Couldn’t your results suggest we need more compositional training data?\n\nGreat question—we believe both directions are valid. However, our evidence suggests that *not all decompositions are helpful*. Rigid, human-written breakdowns can **hurt** model performance (e.g., −4.9% on Level 7), likely due to mismatch with internal solution paths.\n\nThus, we argue for RL methods that emphasize **strategic flexibility**—the ability to adapt to guidance, retrieve knowledge, and reason through difficult tasks. Compositional training remains promising, but is suggested to align with model reasoning rather than impose rigid scaffolds. We’ll clarify this in the revision.\n\n[1] https://arxiv.org/abs/2409.12122\n\n[2] https://artofproblemsolving.com/wiki/index.php/AoPS_Wiki:Competition_ratings\n\n[3] https://arxiv.org/pdf/2402.05808\n\n[4] https://arxiv.org/pdf/2410.01748v1"}, {"review_id": "kA5g0pHjoc", "reviewer": "Reviewer_qeem", "summary": "This paper employs LLMs to synthesize three complementary components for mathematical questions, namely, the planning skeleton, relevant knowledge, and sub - problems. By leveraging these complementary elements, along with the original questions, as input information, the paper attempts to analysis diverse aspects of the performance of LLMs in mathematical reasoning, extending beyond mere accuracy. \nThis paper adopts a multi-stage training approach to explore the influence of question difficulty on the efficacy of training. Through the augmentation of partial-solution guidance, challenging problems can be made relevant for reinforcement learning (RL) training.", "questions": "1. How do the multi-staged training experiments integrate within your Sparkle framework? They appear to involve distinct objectives being presented within a single research paper. It seems as if two experiments are combined to amass the contributions of this paper.\n2. Given that, when complementary information is used as input, a decline in performance is observed in SFT models. Does this not suggest that the instructions are of poor quality? The complementary information is generated by LLMs, which may contain toxic and distracting elements. Does the absence of human verification not undermine the effectiveness of your framework?\n3. The analysis framework is contingent upon the quality of the data synthesized by this LLM. This data serves as the gold standard for each aspect of reasoning, which can be brittle and biased. How do you ensure fairness and prevent bias?\n4. How do you justify the conduction of providing partial solutions to the RL process? Partial solutions may train models to rely on step-by-step hints, reducing autonomy—contradicting RL’s goal of fostering self-generated strategies. Why not SFT instead if you have the \"gold standard\" solution? Could SFT models perform better with human-authored (non-LLM) plans? The paper assumes LLM-generated plans are \"valid,\" but human planners might structure steps differently.", "limitations": "data dependency assumptions (LLM-generated gold standards) and under-explored tradeoffs in training design (e.g., partial solutions)", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\nThe paper is well written.\nAnalysis of mathmatical reasoning beyond accuracy is a welcoming topic\n\nWeaknesses:\nOverreliance on LLM-generated annotations for metrics analysis, relying on a single LLM (GPT-4.1) for annotation may embed biases and artifacts.\nThe claim that \"RL improves plan flexibility\" is conflated with \"SFT struggles with LLM-generated plans,\" leaving open whether the deficit is model- or input-specific.\nNo direct comparison between LLM-generated vs. human-generated plans leaves ambiguity on input quality’s role.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and suggestions and we address each question and provide further clarifications in detail below. Please let us know if you have any additional questions or comments and we are more than happy to discuss further.\n\n> Q1: How do the multi-staged training experiments integrate within Sparkle framework? They appear to involve distinct objectives being presented within a single research paper.\n\nThank you for the thoughtful question. The dual focus is intentional and essential to our study. Our work aims to understand mathematical reasoning under RL. When applying RL to a set of problems, the key knobs are (i) the components necessary to solve a problem (planning, knowledge, etc.)—captured through the SPARKLE analysis framework—and (ii) the raw difficulty of the problems, which motivates our use of multi-stage RL training. A full characterization of RL for reasoning is not possible without addressing both of these elements.\n\nTo unify these aspects, we adopt a curriculum-style two-stage RL training setup, inspired by prior works (Sec. 2), and analyze its effects of RL using the SPARKLE framework.  Specifically, Stage 1 establishes a strong RL-tuned model using diverse math problems, while Stage 2 explores three strategies for fine-tuning on a subset of the most challenging problems. We explore three data strategies (L184-188) for the second-stage training, from naive RL training with no augmentation, to including partial-solution augmentation. Our results show that without augmentation, outcome-based RL struggles to learn from difficult problems due to sparse or absent reward signals (Table 1). In contrast, partial-solution augmentation emerges as a simple yet effective method for enabling RL to extract training signal from these challenging cases.\n\nThe SPARKLE framework is applied consistently across models trained at each stage, allowing us to systematically analyze how different training strategies influence specific aspects of reasoning. Rather than representing distinct objectives, the RL experiments and SPARKLE framework are tightly coupled to provide a unified and comprehensive analysis of RL’s impact on LLM reasoning.\n\n> Q2.1: Given that, when complementary information is used as input, a decline in performance is observed in SFT models. Does this not suggest that the instructions are of poor quality? \n\nWe would like to clarify that **our paper does not evaluate or include SFT models**. To avoid confounding effects from supervised fine-tuning, we focus exclusively on *comparing base models with RL-tuned models*. This design complements prior studies that analyze the interplay between SFT and RL [1,2] (l.194–195).\n\nAll results involving planning and knowledge interventions (e.g., Sec. 6.3 and 6.4) are based on this Base vs. RL-tuned comparison. (summarized in l.220–237).\n\nIn Sec. 6.3, we find that providing external plans can degrade performance—particularly on the most challenging benchmark (AIME24). However, this is **not** due to poor instruction quality, but rather reflects a **misalignment** between externally provided plans and the model's preferred internal plans.\n\nMoreover, the base model itself exhibits some instruction-following ability: for example, its performance improves with planning on GSM8K (Fig. 3, l.279–284), and with sufficient sampling (Fig. 2), it can match RL-tuned model performance.\n\n\n> Q2.2&Q3: The complementary information is generated by LLMs, which may contain toxic and distracting elements...Does the absence of human verification not undermine the effectiveness of your framework?\n\n**Human verification** is an integral part in the benchmark construction process and has been included in this work.\n\nWe would like to clarify that we utilize a web agent to \"extract a planning skeleton and summarizing key reasoning steps\" (l.145-146) from **existing step-by-step solutions in the test sets**, instead of prompting LLM to generate a plan *without reference*. Moreover, we address this risk through a rigorous multi-stage annotation and verification pipeline, as described in Sec. 3.2 (l.149–153). Specifically:\n- **A second verification agent** checks each generated annotation for correctness, coherence, completeness, and pedagogical soundness. If any aspect fails, the annotation is regenerated.\n\n- **Final validation is conducted by graduate students** with advanced mathematics backgrounds to ensure that each annotation faithfully captures the required reasoning and knowledge.\n\n\nIn addition, to curate the training set for the second-stage RL tuning, we also involved \"human verification to further filter out items with flawed solutions\" (l.207-208). We believe this step is important for mitigating the potential brittleness of the GPT-4.1-based web agent.\n\n\nWe hope this clarifies that human verification is not only present but plays a critical role in ensuring the reliability and effectiveness of our framework. We would appreciate if you could re-evaluate our paper in light of the human verification measures we have in place.\n\n> Q4: How do you justify the inclusion of partial solutions in RL? Partial solutions may train models to rely on step-by-step hints, reducing autonomy—contradicting RL’s goal of fostering self-generated strategies. \n\nThank you for raising this point. Our motivation is grounded in prior work showing that curriculum-based approaches can significantly enhance the effectiveness of RL training for LLMs (see Sec. 2). However, outcome-based RL methods such as GRPO often struggle to learn from the most challenging problems due to the sparsity of positive reward signals (i.e., most sampled trajectories in a group result in incorrect answers).\n\nThis raises the question: can we still leverage these hard problems in a way that improves performance *without compromising the capabilities already learned by a well-tuned RL model*?\n\n\nWe agree that overusing partial solutions may reduce autonomy if applied indiscriminately. However, our approach is **targeted and carefully scoped**. Partial-solution augmentation is **not** used as a general strategy for RL training (i.e., Stage 1), which could encourage dependency on hints. Instead, we introduce partial-solution scaffolding **only in Stage 2**, and **only on the hardest problems** that the Stage 1 RL-tuned model consistently fails to solve.\n\nTherefore, We would like to clarify that partial solution augmentation is not used as a general approach for RL training on all problems (Stage 1), which \"may train models to rely on step-by-step hints\". We only perform partial solution scaffolding in the second stage, **on the hardest problems the first-stage RL-tuned model cannot solve**, to further boost the reasoning performance. \n\nSpecifically, given an RL-trained model, we have explored three data strategies for the second-stage RL training (l.184-188):\n \n- (1) Hard-Only: Training on difficult problems without any augmentation.\n - (2) Mixed Difficulty: Combining easy and hard problems for exposure diversity.\n - (3) Hard-Augmented: Introducing partial solutions to help the model navigate complex reasoning paths.\n\nOur results (Table 1) show that the Hard-Augmented strategy significantly outperforms the other two, demonstrating that partial-solution scaffolding can effectively improve reasoning when the model is **otherwise unable to learn from sparse feedback**. In this context, partial augmentation serves as a practical curriculum tool to extend the learning frontier—rather than undermining reasoning autonomy.\n\n\n\n> Q5: Why not SFT instead if you have the \"gold standard\" solution?\n\nWe agree that a natural baseline would be to apply SFT using available solution traces. However, there are both **practical limitations** and **empirical drawbacks** that make SFT sub-optimal in our setting.\n\n- The training set used in Stage 1 contains over 40K math problems, for which \"gold standard\" solutions are largely unreliable: many solution traces are noisy or incomplete. These traces vary in quality---from simplistic chain-of-thought outputs to human-written final answers. While obtaining new traces by distilling from competitive LLMs is possible, it would require substantial compute and additional human validation to ensure correctness and clarity.\n- Empirically, we observe that applying SFT on noisy traces does not lead to improved performance. To evaluate this, we conducted a controlled comparison between three approaches:\n\n\n| Model                     | AIME24     | AMC23      | MATH500    | GSM8K      | OlympiadBench |\n| ------------------------- | ---------- | ---------- | ---------- | ---------- | ------------- |\n|**SparkleRL-Stage 1**| 46.67%| 67.50%| 80.00%     |91.77%|39.11%|\n|**SparkleRL-Stage 2-aug** | **50.42%**| **71.25%**| **81.00%**|**92.38%**| **40.11%**|\n|**SFT on hard problems with solutions**|15.00%|53.44%|70.03%|88.30%|30.70%|\n\n\n\nAs shown above, SFT on hard problems from the SparkleRL-Stage 1 model leads to significant **performance degradation** across all benchmarks. In contrast, our **Stage 2-aug** method yields consistent improvement.\n\nThe key distinction in Stage 2-aug (details in l.202–216) is that instead of fine-tuning on full traces that are noisy, we augment the hardest unsolved examples from Stage 1 with partial solutions in the input prompt. These partial solutions does not disrupt agent's on-policy learning, which is important for effective RL training. This preserves the benefits of RL, which encourages models to develop robust, self-directed reasoning strategies—given hints (partial solutions) from the challenging problems.\n\nIn contrast, SFT fine-tuning on noisy or overly prescriptive solutions can misalign with the model's learned heuristics, disrupting the internal reasoning behaviors acquired during RL training.\n\n\n\n[1] Chu et al., Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025\n\n[2] Yeo et al., Demystifying long chain-of-thought reasoning in llms, 2025."}]

Author rebuttal:

Meta-review: This paper investigates the benefits of RL training. In particular, the paper explores these benefits in the context of plan-following and execution, problem decomposition, and (3) improved reasoning and knowledge utilization. Experimental evidence is provided for each of these aspects. Interestingly, the paper shows that plans derived from human solutions are not necessarily good for the model to execute, which is not conceptually surprising. The paper also considers the effect of training exclusively on hard problems and using hard problems augmented with partial solutions and provides interesting insights. The reviews for the paper are mixed. While the problem setting is interesting, the reviewers were concerned about insufficient ablations, novelty of the findings and overall presentation of the paper. 

After going through the paper, it is somewhat surprising that the main body almost omits the key details about the experiments (e.g. details about the plans and how they are used). There are many design choices involved in the experiments and careful review of them is required to ensure robustness of these findings across settings (especially with respect to different design choices). I'm happy that this discussion came during the rebuttal process. It is important to address these concerns before publication of the final version because in the current form the paper seems somewhat misleading. It is important to clearly highlight the limitations of the work raised during discussion with Reviewer wRv5 and Reviewer 1rRb.
Decision: Accept (poster)