Title: Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Mathematical Reasoning

Reinforcement Learning

Evaluation
Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RL enhances performance is still lacking. To bridge this gap, we introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models to formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RL enhances models' ability to integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems---those yielding no RL signals and mixed-quality reasoning traces---can still be effectively used for training. We introduce SparkleRL-PSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide a principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: https://sparkle-reasoning.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper employs LLMs to synthesize three complementary components for mathematical questions, namely, the planning skeleton, relevant knowledge, and sub - problems. By leveraging these complementary elements, along with the original questions, as input information, the paper attempts to analysis diverse aspects of the performance of LLMs in mathematical reasoning, extending beyond mere accuracy. \nThis paper adopts a multi-stage training approach to explore the influence of question difficulty on the efficacy of training. Through the augmentation of partial-solution guidance, challenging problems can be made relevant for reinforcement learning (RL) training.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\nThe paper is well written.\nAnalysis of mathmatical reasoning beyond accuracy is a welcoming topic\n\nWeaknesses:\nOverreliance on LLM-generated annotations for metrics analysis, relying on a single LLM (GPT-4.1) for annotation may embed biases and artifacts.\nThe claim that \"RL improves plan flexibility\" is conflated with \"SFT struggles with LLM-generated plans,\" leaving open whether the deficit is model- or input-specific.\nNo direct comparison between LLM-generated vs. human-generated plans leaves ambiguity on input quality’s role.\n\nQUESTIONS:\n1. How do the multi-staged training experiments integrate within your Sparkle framework? They appear to involve distinct objectives being presented within a single research paper. It seems as if two experiments are combined to amass the contributions of this paper.\n2. Given that, when complementary information is used as input, a decline in performance is observed in SFT models. Does this not suggest that the instructions are of poor quality? The complementary information is generated by LLMs, which may contain toxic and distracting elements. Does the absence of human verification not undermine the effectiveness of your framework?\n3. The analysis framework is contingent upon the quality of the data synthesized by this LLM. This data serves as the gold standard for each aspect of reasoning, which can be brittle and biased. How do you ensure fairness and prevent bias?\n4. How do you justify the conduction of providing partial solutions to the RL process? Partial solutions may train models to rely on step-by-step hints, reducing autonomy—contradicting RL’s goal of fostering self-generated strategies. Why not SFT instead if you have the \"gold standard\" solution? Could SFT models perform better with human-authored (non-LLM) plans? The paper assumes LLM-generated plans are \"valid,\" but human planners might structure steps differently.\n\nLIMITATIONS:\ndata dependency assumptions (LLM-generated gold standards) and under-explored tradeoffs in training design (e.g., partial solutions)", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes various interventions in RLVF posttraining on top of Qwen-2.5-Math-7B in order to better understand what limits reasoning performance: planning, knowledge, problem decomposition, or problem difficulty. The paper proposes a new dataset of 2,564 questions called SPARKLE which augments popular math benchmarks (such as GSM8K, MATH, AIME, etc.) with (1) a suggested plan for solving the problem; (2) relevant knowledge/concepts needed to solve the problem;  and (3) a set of subproblems and corresponding answers that need to be solved in the course of solving the whole problem. Ablations that either include or exclude the plans, knowledge, and subproblems demonstrate that: plans harm the base model’s performance while leaving the RL-trained model performance unchanged; knowledge harms base model performance but slightly improves RL-trained model performance; and all models seem unable to reliably solve subproblems. Additionally, the paper demonstrates that continued RL training on the most difficult problems augmented with hints can lead to the best performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Clarity: I think the paper is structured well and it very clearly lays out the dimensions that are being studied.  \n* Significance: understanding RL reasoning performance at a more fine-grained level—beyond just accuracy on various benchmarks—is an important and burgeoning area of research.  \n* Originality: The various findings with Qwen 7B are very interesting and I believe will be of interest to the RLVF community: (1) RL performance can be boosted on hard problems if partial solutions are available; (2) performance of RL-trained models seems to be limited more by knowledge retrieval ability rather than planning ability; and (3) all models struggle with solving decomposed problems.\n\nWeaknesses:\n\n* Quality: there are a number of unsupported or overly broad claims. I detail these in the “Questions” section below, but will call out the two most important ones:  \n  * The paper is missing an important baseline, which is to compare to Qwen-2.5-7B-Math-Instruct. Some of the claims made in the paper about the base model may just be due to poor instruction-following capability, and it’s not clear whether the RL-trained models do better because they are better at instruction-following or because they’ve been trained with RLVF. Including an instruction-tuned baseline would help to clarify this.  \n  * There are no error bars or statistical tests reported in the paper, and some of the claims are made based on very small differences that could be due just to statistical error.  \n* Clarity: there are a number of missing details that make it difficult to fully evaluate the paper. I detail these in the “Questions” section below, but most importantly, there needs to be more detail about (1) how the SPARKLE dataset was constructed, and (2) how the “partial solution guidance” works.  \n* Quality: only a single model (Qwen-2.5-7B-Math) is studied in the paper, which limits the generality of the claims.\n\nQUESTIONS:\n### Unsupported or overly broad claims\n\n* I think the drops in performance seen by the base model might just be due to poor instruction following. I therefore think that (1) the paper should state this is the reason for the drop in base model performance, and (2) Qwen-2.5-Math-7B-Instruct should really be included as a baseline in order to disentangle whether it is the lack of instruction-tuning that leads to these drops or the lack of reasoning. My prediction is actually that including things like the plan will help Instruct quite a lot, but maybe not as much as doing RLVF (i.e. base \\< instruct \\< instruct+plan \\< rl $\\\\approx$ rl+plan). **I would need to see Instruct added as a baseline to consider increasing my score**.  \n* Some of the claims need to be backed up by statistical tests—e.g. In the discussion in paragraph starting line 285, it’s not clear to me that the increases/decreases in performance are statistically significant. If you would like to make claims like “RL-tuned models achieve superior performance when permitted to develop autonomous planning strategies” then it is important to establish whether that superior performance is statistically significant. (Also, in that particular case it would also be helpful to clarify that you are specifically talking about AIME24). **I would need to see statistical tests included throughout the paper in order to consider increasing my score**.   \n* “Improved sampling efficiency alone cannot fully explain the observed gains” (line 267\\) → Why not? I didn’t understand this argument, or how the subsequent experiments disprove it.  \n* “RL-tuned models show greater flexibility in following different plans compared to the base model” (line 285\\) → how do you know they are actually following the plan rather than just ignoring it and doing their own thing?  \n* “Knowledge integration is a key driver of performance on harder tasks” (line 334\\) → I’m not sure this conclusion is correct. It is hard to know for sure without seeing the distribution of difficulty, but based on the pass@1 rates it looks like there might be very few problems at higher difficulty levels (e.g. maybe just one at difficulty 10, and three at difficulty 9?). If so, the fact that the solve rate *looks* high might just be the result of high variance. It would be much better to (1) report the avg@8 score (as in the other plots), (2) to include error bars, and (3) to report how many questions there are at each difficulty level, in order to make sure the interpretations are correct.  \n* Some of the language used in the paper is overly broad given that only a single model (Qwen-2.5-Math-7B) was used. For example, the abstract says “RL-tuned models exhibit greater robustness… than their base counterparts” (line 11), but there is only one base model examined\\! The language in the paper needs to be softened to clarify that Qwen 7B is the only model studied: I think this is ok because the benchmark is still interesting, but the claims in the paper need to be made more precise because without testing other models it is not possible to generalize the results in the paper to them.\n\n### Missing details\n\n* Significantly more detail is required to explain how SPARKLE is constructed. For example, which prompts are used to produce the plans/knowledge/decomposition? What prompts are used for the verification agent? How do you ensure that the subproblems all have correct answers and are solvable? How do you determine difficulty level? This doesn’t need to be in the main text, but at least should be in the appendix. **I would need to understand these details before considering raising my score.**  \n* I don’t understand how the “partial solution guidance” works. Could you please provide an example of what the prompt looks like? How do you choose the boundaries for the “chunks”? How is it different from methods for reverse curriculum training, e.g. [https://arxiv.org/pdf/2402.05808](https://arxiv.org/pdf/2402.05808) ? (Some of the details are scattered throughout the paper, e.g. where the reasoning traces come from—it would be really helpful to include a dedicated section in the appendix restating some of those things and explaining the whole procedure in more detail). **I would need to understand this better to consider raising my score.**  \n* What is the distribution of difficulty level (i.e. how many questions at each level)? It would be helpful to include a histogram in the appendix.  \n* Are the results reported in Figure 6 over the entire SPARKLE benchmark? It would be helpful to be more clear about this in the captions and text.\n\n### Other questions/comments\n\n* Please include error bars in Table 1 and all of the figures.  \n* In Figure 3, 4, & 5 (and in the appendix too) it would be helpful to include an “all” group which shows the average across all tasks.  \n* For the subproblem decomposition experiments, what is the average solve rate for the subproblems individually (not altogether)? And how many subproblems are there per problem (maybe you could include a histogram in the appendix)? It might be interesting to discuss your results in relation to [https://arxiv.org/pdf/2410.01748v1](https://arxiv.org/pdf/2410.01748v1) which I think has some similar findings.  \n* “RL tuning methodologies should prioritize high level reasoning strategies rather than detailed subproblem processing” (line 325\\) → couldn’t you draw the opposite conclusion, i.e. that we should include more compositional training data to improve the models’ performance on solving compositional tasks like these?\n\nLIMITATIONS:\nThe paper should acknowledge that it only studies a single model (Qwen) and therefore it is unclear how well the results generalize.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for analysing the reasoning behaviours of LLMs along three dimensions: planning and execution, knowledge utilisation, and chain of subproblems. The authors also explore the use of curricula and solution guidance to improve the learning potential of hard problems when conducting RL for reasoning. Results demonstrate that these approaches improve the utility of hard problems during training and reveal that RL training reliably enhances two out of the three reasoning behaviours considered.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nBy not focusing on benchmark accuracy alone, the paper reveals some qualitative behaviours, which can be quantitatively measured, that are enhanced by RL finetuning. These insights provide intuitive insights into the general capabilities that translate to improved performance on reasoning tasks. The results also demonstrate that the proposed problem difficulty curriculum and solution augmentation do indeed improve learning from hard problems.\n\nWeaknesses:\n\nThe dual focus of analysing reasoning behaviours and improving learning from hard problems is disjointed. The proposed framework is also not comprehensive, as it fails to account for aspects such as the faithfulness of reasoning. While the paper reveals that problem decomposition is not useful, this unintuitive result warrants further investigation, or could even motivate an approach to making better use of problem decomposition. The impact of additional training is not accounted for; comparisons to SFT would and other offline methods such as DPO would be necessary to robustly show that it is RL training specifically that reinforces these behaviours. Though not necessarily a weakness, the findings of the paper are not particularly surprising and so their impact may be minimal.\n\nQUESTIONS:\nWhy does performance by difficulty level have a U-shape with knowledge augmentation?\n\nLIMITATIONS:\nLimitations are discussed in an Appendix, but should feature in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a number of methods for analyzing strengths and weaknesses of language model reasoning, and uses these methods to explore the effect of RL training (through GRPO) on reasoning in math domains. They also propose and evaluate a method for training through RL on difficult problems based on providing partial solution scaffolds.\n- The first angle they explore deals with planning versus executing – they test models with and without plans being provided, and surprisingly find that providing plans can often actually hurt the performance of models, especially base (non-RL) models, presumably because models are better at following the plans that they come up with instead of provided plans. \n- The second angle they study is on the ability to recall relevant background knowledge – they test models with and without relevant theorems/definitions/etc being provided in the prompt, and find RL based methods are much more strongly able to make use of provided knowledge (in fact it often hurts the base models!).\n- The third angle looks into the problems with providing plans (the first angle) in more detail by exploring how well the model can solve a subproblem given the ground truth solution prefix up to that subproblem. They find that models are far better at solving overall problems than solving the subproblem sequence.\n\nSTRENGTHS AND WEAKNESSES:\nOverall I'm very positive on this paper, I think that it's an excellent fit for NeurIPS. It is beautifully written and presents a series of interesting results pertaining to a very hot area in the field right now (effects of RL on language models). The paper reads like an excellent example of good science – clear questions, well-designed experiments, beautifully presented. I'm not an expert in RL for LLMs and may not know all the related work, but given my understanding this is a solid, original contribution to the field.\n\nStrengths\n- I find the overall framework compelling. It's true that we need to move beyond accuracy based metrics, and I think that this is a nice breakdown into different aspects of the problem (planning, background knowledge, ability to execute plans).\n- The dataset construction pipeline including a final check by graduate students with a math background is an excellent step of verification and one that many papers don't do. And likewise during the hard-solution curation there's a graduate student verification step to find flawed solutions.\n- Section 6 is a nice presentation of a number of interesting results. The ordering and flow of the section made it quite easy to read.\n    - The results in 6.1 on training on scaffolded problems shows that the proposed method provides a reasonable boost. However I feel the main contribution of the paper is primarily in the other results.\n    - The result in 6.3 that externally provided plans decrease performance for many base/RL models over many domains (but that RL models are more robust to this) is particularly interesting – the idea that actually models are particularly good at following their own plans, and being forced into an externally provided plan can decrease performance. I appreciated the granular explanation of why that trend doesn't hold for GSM8K.\n    - The section 6.4 result that external knowledge harms the base model but helps the RL models was also really interesting, as was the 6.5 results on models having more trouble solving all subproblems than overall tasks.\n    - 6.6 was an important analysis to include for shedding a bit more light on the 6.3/6.4 result trends, showing how they vary with problem difficulty.\n- The writing of Section 3.1 is excellent, each of the axes is presented quite clearly. The style works well – leading with a sentence about the more general reasoning phenomenon the axis is trying to capture then getting very concrete and specific. And the high quality writing carries through the rest of the paper. The figures are also beautiful, the paper overall has a very high level of polish.\n\nWeaknesses\n- I don't have any strong weaknesses. \n- The Hard-Augmented model results (6.1) are interesting but don't feel too significant. In part because the approach requires having solutions (or partial solutions) to these problems, at which point one could just train on those solutions in the first place.\n\nMinor edits and suggestions\n- line 208: difficulty -> difficult\n- Figure 6 isn't labelled as \"Figure 6\" anywhere\n- Figure 6a caption: \"stable\" -> \"stability\"\n- I wonder if 6.6 could be moved up to be right after 6.4, since it's about how 6.3/6.4 vary with difficulty? Possibly 6.5 could fit well right after 6.3, since it's a bit of a look in detail into how things break down when solving human-provided problems. Just a thought / suggestion though, may or may not be better.\n\nQUESTIONS:\n- Relating to both 6.3 and 6.5, I'm curious if you have any qualitative observations from looking at what sorts of human-provided plans (6.3) or subproblems (6.5) are difficult for the model to work with. Do you have any observations on how these plans and subproblems differ from those that the model comes up with and prefers to work with?\n- On my minor weakness about the Hard-Augmented results, it'd be helpful to hear a bit more of what kind of settings motivate this, or where you see this result being useful?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}