Title: GPO: Learning from Critical Steps to Improve LLM Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language model

explainable reinforcement learning

fine-tuning
Abstract: Large language models (LLMs) are increasingly used in various domains, showing impressive potential on various tasks. 
    Recently, reasoning LLMs have been proposed to improve the \textit{reasoning} or \textit{thinking} capabilities of LLMs to solve complex problems. 
    Despite the promising results of reasoning LLMs, enhancing the multi-step reasoning capabilities of LLMs still remains a significant challenge. 
    While existing optimization methods have advanced the LLM reasoning capabilities, they often treat reasoning trajectories as a whole, without considering the underlying critical steps within the trajectory. In this paper, we introduce \textbf{G}uided \textbf{P}ivotal \textbf{O}ptimization (GPO), a novel fine-tuning strategy that dives into the reasoning process to enable more effective improvements. 
    GPO first identifies the `critical step' within a reasoning trajectory - a point that the model must carefully proceed so as to succeed at the problem. We locate the critical step by estimating the advantage function.
    GPO then resets the policy to the critical step and samples the new rollout and prioritizes learning process on those rollouts. 
    This focus allows the model to learn more effectively from pivotal moments within the reasoning process to improve the reasoning performance.
    We demonstrate that GPO is not a standalone method, but rather a general strategy that can be integrated with various optimization methods to improve reasoning performance. 
    Besides theoretical analysis, our experiments across challenging reasoning benchmarks show that GPO can consistently and significantly enhances the performance of existing optimization methods, showcasing its effectiveness and generalizability in improving LLM reasoning by concentrating on pivotal moments within the generation process.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Guided Pivotal Optimization (GPO). GPO stands as a general strategy integrable to existing optimization methods by identifying critical steps and prioritizing the learning process on the rollouts of these critical steps, which are identified by estimating the advantage functions per step. This work also provides a theoretical analysis that proves the online GPO as a form of advantage-weighted RL in the offline preference learning setting. The evaluation is extensive, where GPO demonstrates consistent improvements over optimization baselines. Lastly, it includes a user study with 50 participants analyzing the alignment between the GPO critical step identification and human judgment.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. Automatically identifying the critical step via advantage in an online learning setup is integrable with many existing optimization methods.\n\n2. The theoretical analysis is useful in bridging the gap between the proposed online learning and the offline preference learning.\n\n3. The evaluation dataset and baselines provide a good coverage of tasks and existing learning approaches to reflect the generalization of the proposed method.\n\nWeakness:\n\n1. As acknowledged by the author, integrating GPO inevitably introduces roughly twice the computation cost. \n\n2. The critical steps are classified by splitting the new line in Algorithm 1. I find this definition may be adequate for mathematical reasoning tasks. However, I doubt the generalization of this definition to a wider array of tasks, such as the coding task or factual QA tasks.\n\n3. The evaluation is only conducted on a single base model DeepSeek-R1-Distill-Qwen-7B model. It is still necessary to conduct the experiments on other families of models to provide sufficient evidence of the generalization of GPO.\n\nQUESTIONS:\n1. Could you also include the standard deviation in Table 1?\n\n2. How would you justify splitting critical steps by newlines?\n\n3. I see you set a maximum number of steps around 10 to 15. However, for logical reasoning methods, can you justify whether your method scales to tasks with a larger number of steps?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel fine-tuning strategy to improve LLMs’ multi-step reasoning abilities, GPO. While existing methods such as PPO or DPO optimize reward or preference over entire reasoning trajectories, GPO aims to identify and prioritize critical steps—pivotal moments in the reasoning chain that heavily influence the final result. GPO uses an advantage estimation (from RL) to find these steps, then resets generation at these points to sample and train on new continuations. Extensive experiments covering seven diverse reasoning datasets indicate that GPO consistently and significantly improves test accuracy over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe idea of focusing learning on “critical steps” of reasoning (rather than on the entire trajectory) is original and reasonable. \n2.\tGPO is a play-and-plug module compatible with both online RL (PPO) and various offline preference learning algorithms (DPO, SimPO, ORPO, KTO), which broadens its applicability.\n3.\tThe paper not only provide empirical results, but also offers regret bounds for the proposed method and links per-step preference learning to advantage-weighted RL objectives.\n4.\tExperiments are thorough, covering 5 different optimization algorithms and 7 datasets. Strong and consistent improvements are shown.\n\nWeaknesses:\n1.\tThe key step of GPO - Monte Carlo estimation of future returns at each inference step - adds a non-negligible computational cost (~1.8-1.9x). While the authors believe this is manageable, this may be limiting for very long trajectories or large-scale settings. The tradeoff between benefits and computational needs discussion.\n2.\tSome failures might be dispersive or cumulative (multiple small errors). Could GPO be further improved by handling multiple critical steps, or scoring all steps proportionally?\n\nQUESTIONS:\n1. Why only one critical step is selected in each reasoning path? Would it be better to use the top 3 or threshold method? For example, when there are multiple critical steps in the reasoning path.\n\nLIMITATIONS:\nYes, the authors fully address the limitations of their work and the potential negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce guided pivotal optimization (GPO), a fine-tuning strategy for reasoning pathways which can be operationalized with policy-based and preference-based optimization algorithms. The method first identifies crucials point within CoTs, and then generates training data as continuations starting from this step. Then, this data is used to tune the network using PPO or DPO. The authors evaluate the utility of their method on DeepSeek-R1-Distill-Qwen-7B across 7 datasets and consider 5 different tuning algorithms to operationalize their strategy. The authors show that their method consistently improves upon tuning with the base algorithm and conduct ablation studies comparing how their algorithm behaves with respect to number of MC samples and model size.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The idea of identifying crucial points within the reasoning process and resampling is interesting\n- The proposed algorithm outperforms on the base tuning methods\n- The authors conduct experiments across a large number of datasets and add supporting ablation studies\n\nWeaknessess\n- The writing, especially in the part outlining the algorithm, could use improvement. Parsing the paragraph which describes identifying the critical step via advantage is difficult. Entire Section 4 would benefit from a rewrite, as it is very dense.\n- The experiments are conducted on a single model family, and while on some datasets the performance gain is significant, on others it is pretty negligible. I would also suggest reporting average gain in a separate column to emphasize the performance improvement.\n- The idea of identifying critical steps in CoTs has been explored before, limiting the novelty of the proposed approach [1]\n\n[1] Wang, T., Chen, J., Han, X., & Bai, J. (2024). CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks\n\nQUESTIONS:\n- How are the positive and negative trajectories obtained? Or, how are the samples ensured to be positive, and negative trajectories? Or, more plainly, how do we ensure that the new trajectory will improve on the previous one?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}