GPO: Learning from Critical Steps to Improve LLM Reasoning
Authors: Jiahao Yu
Zelei Cheng
Xian Wu
Xinyu Xing
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
explainable reinforcement learning
fine-tuning
Primary area: deep_learning
TL;DR: We improve the fine-tuning performance of reasoning LLMs by identifying the critical steps

Abstract: Large language models (LLMs) are increasingly used in various domains, showing impressive potential on various tasks. 
    Recently, reasoning LLMs have been proposed to improve the \textit{reasoning} or \textit{thinking} capabilities of LLMs to solve complex problems. 
    Despite the promising results of reasoning LLMs, enhancing the multi-step reasoning capabilities of LLMs still remains a significant challenge. 
    While existing optimization methods have advanced the LLM reasoning capabilities, they often treat reasoning trajectories as a whole, without considering the underlying critical steps within the trajectory. In this paper, we introduce \textbf{G}uided \textbf{P}ivotal \textbf{O}ptimization (GPO), a novel fine-tuning strategy that dives into the reasoning process to enable more effective improvements. 
    GPO first identifies the `critical step' within a reasoning trajectory - a point that the model must carefully proceed so as to succeed at the problem. We locate the critical step by estimating the advantage function.
    GPO then resets the policy to the critical step and samples the new rollout and prioritizes learning process on those rollouts. 
    This focus allows the model to learn more effectively from pivotal moments within the reasoning process to improve the reasoning performance.
    We demonstrate that GPO is not a standalone method, but rather a general strategy that can be integrated with various optimization methods to improve reasoning performance. 
    Besides theoretical analysis, our experiments across challenging reasoning benchmarks show that GPO can consistently and significantly enhances the performance of existing optimization methods, showcasing its effectiveness and generalizability in improving LLM reasoning by concentrating on pivotal moments within the generation process.

GPO: Learning from Critical Steps to Improve LLM Reasoning
Jiahao Yu Department of Computer Science Northwestern University j jiahao. wwwnortweestern.ncommmm edu
Zelei Cheng" AI Foundations Capital One
Department of Computer Science
Northwestern University
zelei. cheng@capital com
Xian Wu Meta AI
Xinyu Xingi
Department of Computer Science
xiansul23@metta com
Northwestern University xinyu. wwwgiiiiinnnnnitteeeeeeee edu
Abstract
Large language models (LLMs) are increasingly used in various domains, showing impressive potential on different tasks. Recently, reasoning LLMs have been pro- posed to improve the reasoning or thinking capabilities of LLMs to solve complex problems. Despite the promising results of reasoning LLMs, enhancing the multi- step reasoning capabilities of LLMs still remains a significant challenge. While existing optimization methods have advanced the LLM reasoning capabilities, they often treat reasoning trajectories critical steps within the trajectory. In this paper, we introduce Guided Pivotal Optimization (GPO), novel fine-tuning strategy that dives into the reasoning process to enable more effective improvements. GPO first identifies the 'critical step' within reasoning trajectory point that the model must carefully proceed to succeed at the problem. We locate the critical step by estimating the advantage function. GPO then resets the policy to the critical step, samples the new rollout, and prioritizes the learning process on those rollouts. This focus allows the model learn more effectively from pivotal moments within the reasoning process improve the reasoning performance. We demonstrate that GPO is general strategy that can be integrated with various optimization methods to improve reasoning performance. Besides thcoretical analysis, our experiments across challenging reasoning benchmarks show that GPO can consistently and significantly enhance the performance of existing optimization methods, showcasing its effectiveness and generalizability in improving LLM reasoning by concentrating on pivotal moments within the generation process.
whole, without considering the underlying
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, such code generation [1], navigating web pages [2], and question answering [3]. However, achieving reliable multi-step reasoning remains a significant frontier in the LLM research [4, 5].
Work done while Northwestem University.
Cooreeponndecce to: Xian Wu <xianwul23 1anmwll33@ wtt.:////mmmm..mmm..mmm. meta.com>, Xinyu Xing exinyu.ing@noortweestto
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Complex problem-solvine tasks, such as mathematical proofs and editing large codebases, often require generating coherent and logically sound sequences of intermediate steps, forming a reasoning trajectory. While L. can produce fluent text, ensuring the correctness of these multi-step reasoning trajectories challenging, as subtle errors introduced at intermediate steps can lead to the failure of the entire reasoning process [6].
Current state-of-the-art methods for aligning LLMs with desired behaviors, including complex reasoning, often rely on reward modeling fine-tuning techniques [7] like Proximal Policy Opúmization (PPO) [8]. or preference-base based methods like Direct Preference Optimization (DPO) [9]. While effective, these methods typically optimize the model based on preferences or rewards over entire generated trajectories. However, LLMs are prone to making mistakes intermediate steps, which can lead to the failure of the final answer. These fine-tuning methods are not able to effectively pinpoint and focus on these steps to learn how to handle these points.
In this paper, we propose GPO: Guided Pivotal Optimization, a novel fine-tuning strategy designed to improve LLM multi-step reasoning capabilities by Focusing on Pivotal Moments. Instead of treating reasoning trajectories as a whole, GPO breaks down the process to focus on key moments that are crucial for problem-solving. It first identifies the 'critical step' from the reasoning trajectory generated by the LLM. These critical steps are pivotal moments where the model must proceed with precision so as to succeed the problem, and thus, the model should give special emphasis those steps. We identify the critical step by estimating the advantage function of each step.
Second, GPO resets the trajectory at the critical step and generates new trajectory by continuing from the critical step. The intuition is that by focusing the learning process on trajectories after these crucial moments, we can more effectively teach the model to navigate challenging reasoning pathways and improve performance. Note that GPO is general framework that can be integrated into existing fine-tuning methods.
While GPO is as simple to implement for most existing fine-tuning optimization algorithms, we provide a theoretical analysis of the GPO for both online learning and offline preference learning settings. Empirically, we rum GPO on diverse reasoning tasks and existing fine-tuning algorithms to show the effectiveness and generalizability of GPO.
Specifically, we make the following key contributions in this paper:
Proposal of GPO: We introduce GPO, novel fine-tuning strategy that improves LLM reasoning by identifying critical steps in trajectories and prioritizing learning from these pivotal moments to improve the reasoning performance.
Theoretical Analysis: Under natural assumptions, we provide a theoretical analysis of the GPO for the regret bound in the online learning setting, and prove that GPO can be interpreted as a form of advintage-weighted RL in the offline preference learning setting.
We conduct extensive experiments on 7 diverse datasets, including general reasoning, mathematical problem solving, and STEM tasks, across 5 different fine-tuning algorithms to validate the effectiveness of GPO.
Empirical Validation:
We observe that by strategically focusing on learning through critical points, GPO offers a more targeted and effective learning strategy towards enhancing the complex reasoning capabilities of LLMs across diverse optimization frameworks.
Observation:
To improve transparency and inspire future research, we also release the code and data² to facilitate reproducibility and further research.
2 Related Work
Since our work aims to improve the reasoning capabilities by identifying critical steps, our work is related to research in reasoning with LLMs, post-training techniques to enhance LLM reasoning, and methods for identifying critical steps in RI..
LLM Reasoning. The foundation of LLM reasoning is Chain-of-Thouse (CoT) [10), where models are prompted to generate intermediate step-by-step reasoning before the final answer, which can

---

boost the performance on complex reasoning tasks. It aligns with how humans reason, where we break down the problem into smaller steps and reason about each step before arriving at the final answer. Subsequent works follow this direction to work on prompting strategies to enhance reasoning [11, 12, 13]. Beyond prompting, there is growing trend towards developing and fine- tuning LLMss specifically optimized for complex tasks with multi-step reasoning processes. Models like OpenAI 01 [14] or DeepSeek RI [15] are examples of such models fine-tuned with high-quality reasoning trajectories to achieve state-of-the-at performance. Even without explicit CoT prompting, these models are able to generate step-by-step reasoning trajectories. Our work falls into this category by providing fine-tuning strategy to improve the multi-step reliability of reasoning trajectories.
Post-training LLMs for Reasoning. To supervised fine-tune LLMs for reasoning tasks, it typically requires high-quality annotated datasets [16, 17, 18]. However, the annotation costs of this approach can be significant. To reduce the cost, one method to synthesize high-quality data from L.Mss. One approach uses stronger "teacher" L.MSs (e.g.. GPT-44o, Gemini) to generate reasoning demonstra- tions| [19, 15, 20, 21]. However, the cost of these strong L.LMs especially for those commercial L.L.s is still high, and recent work also reveals that the fine-tuning performance may be suboptimal due to the large capacity gap between the teacher and the student LLMs [22. 23, 24, 25, 26]. Thus, recent work starts to explore the LLM self-improvement, where models learn from their own generated data. These works often include methods that filter or refine self- eeff-genertted samples based on feedback or heuristics [27, 28, 29, 30, 31, 6] or employ advanced prompting techniques during data genera- tion [32, 33, 32]. Models will be trained on the refined samples to improve themselves. Another line of approach, different from data synthesis, is online RL learning [15, 34, 35, 36, 37], where the model interacts with the environment to learn the optimal policy guided by the reward function.
Among the above works, one closely related work to ours for self-im saff-improvement is Satori [6], which employs a strategy of randomly resetting the reasoning process at various points and then exploring alternative paths from those reset points to improve the quality of the self-generated data. While similar in use of reset strategy, GPO differs significantly by identifying the critical steps. Besides, as we will demonstrate in $6.2, this random reset strategy is not optimal compared with GPO. Furthermore, Satori only focuses on the offline method without accompanying theoretical analysis, while our method is suitable for both online and offline RL methods, and both provide theoretical analysis.
Critical Step Identification. The concept of identifying critical steps within a sequence is not new in Explainable Reinforcement Learning(RRL), where understanding agent behavior often involves pinpointing critical states or decisions in a trajectory. Various methods have been developed and can be categorized into two types: model-based and model-free methods. For model-based models, they often train a local model to predict the important steps within trajectory [38, 39, 40, 41]. For model-free methods, they often rely on value functions to identify the critical steps [42, 43, 44]. Our method aligns with model-free techniques. However, directly applying traditional XRL methods to LLM reasoning is challenging because treating generating a single token as an action lacks the semantic meaning in reasoning process. In $4, we will detail how GPO adapts the core ideas of critical step identification from XRL to the context of multi-step reasoning in LLMS.
3 Preliminaries
Markov decision process and problem setting. In this work, we consider a finite-horizon episodic Markov Decision Process (MDP) defined as M = (S. A, (P),, (r),, H, do), where S is the state space, A the action space, H is the episode length, (P), A denotes the transition dynamics, fr), is the reward function, and do denotes the distribution of initial state. Given a policy æ, the agent seeks to maximize the expected cumulative reward, expressed as Eando (s)). where the value function
defined as V25 (s) E Σn rh (s2,,a)) so 8, an ~ Tx(: sn) and (5h, ax) denotes the reward at step / under mtte-action pair (s), an).
In our setting, du denotes the distribution of prompt su = x. Given a reasoning problem x do. the goal is to improve base policy aee into refined policy π that maximize the expected reward over generated responses y ~ ( xx), where y = (yo, ,I,.... yH-1) E )' represents a sequence of reasoning steps (up to H), typically separated by newlines. Importantly, rather than treating each generated token as a step, we define each reasoning segment as step. Since generation is auto-
regressive, each step can be interpreted as an action taken by the agent in an MDP with deterministic
3

---

Policy
Q:Ii was Sept. 1st, 2021 week ngo. What the date 24 hours later?
hours exactly day
Sept. 9th.
Lets's is think step by step
Wait, think is asking the date after today
Correct Answer!
S
So today is Sept. 8th.
Rollout
S2
However, question asking
24 hours after Sept.
Update
State
33
Yes, regardless of today' 's date. I just need to find the date Sept.
State transit original trajectory
S
24 hours later the next day
State transit
56
exploration
So it is Sept 2nd.
Dutuse//Online butfer
trajectory
Sg
Answer!
Wrong
Overview of our method. Given an initial trajectory generated by the policy π for a reasoning task, GPO segments the trajectory into steps. It then identifies the most critical step via the MC simulation and resets the policy to the critical step to generate a new trajectory. The new trajectory is then added the dataset online buffer.
Figure 1:
transitions. Specifically, we treat the prefix (Ir yo. ................................ Uh- as the current state 8h. and the next reasoning step yh | 8%)) as the action taken at sh, resulting in the next state 8/+1-
Online policy learning and preference optimization. For standard MDPs with known reward functions, variety of online policy gradient algorithms- - including Proximal Policy Optimization (PPO) [8] and its recent variant Group Relative Policy Optimization (GRPO) [34]-have been proposed iteratively improve policy through direct interactions with environments. These methods have also been extensively applied in LLM training to solve complex mathematical or coding tasks [34, 15, 45], where clear binary reward function can be easily defined by comparing the LLM- --M--eeertted output with gold standard solutions. Formally, PPO optimizes the following objective function:
1i-11
molyil, Miss) Tew (Jk (a, Mes) e, +e) 11) -1
(0) Egmao. youn (1e)
Σ min
Tea (M.la, A,, clip
(I)
where and Teara are the current and old policy models, and are questions and outputs sampled from the question dataset and the old policy Th... respectively. 8 is a delipping-relateee hyper-parameter introduced in PPO for stabilizing training. A; is the advantage function.
The MDP formulation of preference learning was recently explored [9, 46, 47]. In this setting, the true reward function is typically unobservable; instead, we are given an offline dataset of trajectory pairs {xx y* y } labeled with human preferences. Prior approaches reinforcement learning from human feedback (RIHF) [48, 7] typically follow two-stage pipeline: (1) learning a reward function from preference data via the Bradley-Terry(B1) model [49], and (2) training a policy via the PPO algorithm to maximize the learned reward. In contrast, [9] establishes direct connection between the optimal policy x² and its associated reward function, and proposes surrogate objective, referred to Direct Preference Optimization (DPO) to directly learn the optimal policy from the offline preference pairs:
min (TPO(TT= E. (a (oaatt-)aa )>D )>
( ( (aeryt+ x))
o log
x.)
Blog T
x.) xz)
Teer (y
where (-) denotes the sigmoid function, / is temperature parameter, and Tee is fixed reference policy. There are variants DPO that use different loss functions like SimPO [47] and ORPO [46], or eliminate the need for paired samples, like KTO [46]. We introduce them in detail in Appendix D.
4 Method
At high level, given a reasoning trajectory y generated by current policy æ (e.g., π mer)) we identify the most critical reasoning step yn and refine æ from this point using either on-policy

---

Algorithm 1: GPO Optimization Framework
1: Procedure-I: Online Policy Training (PPO-based)
2: Input: Initial L.M policy x² = Tiff reasoning dataset Dr
3: for iteration 1. ..... T do
4: D --9 0
5: n= ... do
6: Random sampling a question æ from the reasoning dataset Dr
7:
Run x' to generate a K -tep reasoning trajectory y/ = (yo..... ... 1/ -1) split by newlines Identify the critical step ym with maximal advantage A (z, V0: ye)
8:
9: Reset x' ym and roll-out x' to generate trajectory y' (ym, --mm----KKKK) VK-
10: Add trajectory y/ and the final reward to D
11: end for
12: Optimize n² with respect the policy gradient loss (e.g., PPO loss) in Eqn. 1 on D
13: end for
14: Procedure-II: Preference Data Generation and Optimization (DPO-based)
15: Input: Supervised-finetumee base policy Tref, reasoning dataset D, D 9
17: for iteration = 1, ...... T do
18: Repeat the sampling, trajectory generation using wet and critical step identification as in Procedure-I to extract the important step ym from trajectory
Generate two continuations starting from yn to obtain positive trajectory
yt (yo...., Bm........ .... VK--) and negative trajectory y = (yo..... Vm..... F
20:
Add the preference pair (z, y² y to
21: end for
22: Optimize with respect to the preference loss (e.g,, DPO loss) in Eqn. 2 on D
algorithm such as PPO or an offline preference method such as DPO. Intuitively, revisiting these pivotal steps enables exploration of potential alternative reasoning paths, overcoming the training bottlenecks of current policy. To illustrate the core mechanism of GPO, consider the example shown in Figure 1. The task It was Sept. 1st, 2021 a week ago. What is the date 24 hours later? An initial trajectory sampled from the policy æ generates a multi-step reasoning process towards the question; however, it misinterprets the question and gives the wrong answer. GPO first segments this trajectory into multiple reasoning steps (e.g., Si.... Sa). Here, the state is the sequence of all previous reasoning steps, and the action is the next reasoning step to be taken. Then, it identifies the most critical reasoning step via Monte Carlo (MC) estimation of the advantage function from RL. S₂ identified as the most critical, as the alternative continuation after S2 could yield a correct final answer, while the continuation after other steps cannot during the MC simulation. Next, it resets the trajectory at S2 and generates a new trajectory y by continuing from S2 with the current policy æ, then adds the trajectory y' to the dataset online buffer. By focusing on trajectories associated with the critical step, GPO directs the learning process towards the specific reasoning step where the policy should focus, thereby enhancing the reasoning performance of the policy.
Identifying the Critical Step via Advantage. We measure the importance of each reasoning step using advantage functions in RL. For any candidate step yi within a predicted reasoning trajectory y. its advantage quantifies the incremental value of taking that step, defined as the relative
change in Q-value when adding yi to the current partial sequence yo:i-1, 1, i.e., A (I., 3;
Q" (x, 10--1: 94) - Q" (z, 30:1-2: 14 expected future reward of taking action ys after observing prefix y0:i-1 under auxiliary policy Formally, given a problem x with the gold answer Ugold, and a predicted trajectory y sampled from new
Here, the Q-function Q"(z, 90:- y.) estimates the
the policy म, we define: Q" (ax, 10:4-1i Mi) = Egrr, ( [r (Ivos, Vi+1.H -1 3gott)] The reward function r(.) compares the completed trajectory (including sampled future steps) with the ground-trutt solution Ygold- The policy л governs how future steps are sampled; It can be unbiasedly
'In RL, the advantage function is detined as A(s:, at) = Q(s, at) - Vs.).. In our setting, with deterministic transitions and zero intermediate rewards, this expression simplifies to the difference between consecutive Q- values: Q(st, ax) Q(s--1, a.)
5

---

estimated via MC simulations [50] by sampling multiple continuations from the current step under policy и. In $6.3, we will evaluate how the number of MC simulations affects the performance of GPO.
Fine-Tuning with PPO DPO. Once the most critical reasoning step in trajectory is identi- fied-formally, the step with the highest advantage-we refine the policy by exploring alternative continuations from this step. Instead of treating the entire trajectory uniformly, we reset the policy to a critical step and sample new rollouts conditioned on it. As we will demonstrate in $5, this -syyle sampling strategy reduces the regret of the final converged policy and enables more efficient online policy improvement. The resulting high-quality trajectories are then in- corporated into the training set guide policy updates. Our framework supports two complementary optimization methods: (i) online policy-gradient optimization like PPO, which updates the policy based on reward feedback (Procedure-I in Algorithm 1) and (ii) offline preference optimization like DPO, which leverages pairwise preferences (Procedure-II in Algorithm 1).
5
Theoretical Analysis
In this section, we present theoretical results and insights related to Algorithm 1.
5.1 Online policy gradient algorithm
Before proceeding, we generalize Procedure-I in Algorithm by sampling the critical step with probability proportional to 274" where is temperature value. The original algorithm can be viewed as the limiting behavior for sufficiently large y. Since we operate in finite-horizon episodic MDP, we evaluate the performance of the online policy gradient algorithm via its regret:
Regret- Ei= (Vo" (8o) Vo (so)). We begin by stating the following assumption regarding the Q-function.
Assumption 5.1 (Bounded Q-value). Suppose we have function class F and Q F holds for the Q function of policy x', Vt 1,2, .,, T. We assume that S Q% (sh, on) S T'ax for all
Q (sh, an) E F, sh E S. an E A
Assumption 5.1 reasonable because reasoning tasks typically involve a bounded final reward. We further present Theorem 5.2 to bound the regret for the online policy gradient algorithm.
Theorem 5.2. Under Assumption 5.1, with probability 1 - 8, we have the following regret bound:
Tlog A
TH
0 (so) vor (5o)) S m' Hy
CT max log(
/Wman (?) (3)
2
where C' constant and (maz(1) (7) represents the step-wise concentrability [51] between the optimal policy n² and our policy. We also show that an increasing 7 will tighten the regret bound in $C.1, which validates the importance of our advantage reweighting technique.
5.2 Preference optimization
For the DPO-based optimization in Procedure-Il of Algorithm 1, we consider a conceptual variant inspired by per-step DPO [52, 21]. which introduces preference comparisons at each individual reasoning step, in contrast to the standard DPO that operates over full trajectories. This step-wise modeling leads to the following result:
Theorem 5.3. Let D consist of tuples (v,, v,1,, [yo:--1, 171, where Yo:i-1 Tef and both continuations yi are drawn from Tec x. yos--1), with preferences determined by the advantage Afred (I,, :.) Then, the optimal policy from minimizing Eqn. coincides with the solution to the following udvantage-weighie RL objective:
xr, 110 exp ( Tree (II 20 100---10 15 3i)
max Egwdo, yo
Σ
log n (yi
3

---

Table I: Comparison of GPO-enhanced methods against baselines. The better results between GPO-enhanced and baseline methods are highlighted in bold. Each training result is the average of runs with different random seeds.
Test Accuracy (%)
Algorithms
BBH
MATH
GSM8K
MMLU
MMLUPro
AIME-2024
AIME-2025
Base Model
59.97
71.60
86.50
54.09
38.80
13.33
16.67
PPO 61.82
79.60
86.96
56.66
47.47
26.67
23.33
GPO-PPO
63.48
87.80
87.44
59.39
51.05
30.00
26.67
DPO
63.20
82.40
86.05
57.08
48.28
20.00
20.00
GPO-DPO
64.25
86.80
88.48
58.93
51.93
26.67
26.67
КTO
62.86
77.20
89.31
59.42
49.02
20.00
20.00
GPO-KTO
64.31
79.60
90.25
61.35
50.52
23.33
26.67
SimPO
61.97
72.20
86.58
56.93
45.70
20.00
23.33
GPO-SimPO
62.58
74.00
88.35
57.44
47.74
23.33
26.67
ORPO 61.75
75.20
87.26
57.72
46,66
20.00
20.00
GPO-ORPO
62.28
78.20
88.17
58.72
48.65
23.33
23.33
In summary, Theorem 5.3 shows that per-step DPO, with preference determined by the advantage function, corresponds to advantage-weighter RL, where the advantage-based preferences implicitly reweight the ligeeiill at each step. Compared to standard offline supervised fine-tuning (SFT), this reweighting allows the model to focus more on critical decision points, leading to more targeted updates and enhanced overall performance [53, 54]. We provide the proof details in Appendix C.2.
6 Experiments
Implementation Details. We primarily employ the PrepSeek-R--Distill- Dristill-Qwen-71 model as the base model for our experiments, selected for its cnstructionfollowwiin and reasoning capabilities, as well as training efficiency.
further enhance data quality and reduce training costs, we follow prior works [19, 55, 6] to filter the data based on question difficulty. For advantage function estimation, we use MC samples for each step. Additional implementation details are available in $E.2.
Baseline Methods. To evaluate the enhancement provided by GPO, we compare the performance of several established fine-tuning algorithms against their GPO-enhanced counterparts. The baseline methods include online RL method PPO, and preference-based algorithms DPO, KTO, SimPO, and ORPO. Here, we only consider one online RL method because PPO is one the most popular online RL methods widely used in the community, and the computation resources needed for online RL are much more significant than those for offline RL methods. Consistent with prior work on reasoning meibitasks [15, 55, 19], we utilize rule-based reward function for the PPO implementation. For fair comparison, identical hyperparameters are used for each baseline method and its corresponding GPO-enhande version. We use the LoRA method [56] for fine-tuning. Detailed hyperparameter configurations are provided in $E.3.
Dataset and Evaluation Metrics. We evaluate the effectiveness of our method on 7 diverse datasets covering range of reasoning tasks. For mathematical problem solving, we use GSM8K [57], MATH-500 [58], AIME-2024 [59], and AIME-2025 [60]. For general reasoning, we utilize BIG- Bench Hard (BBH) [61]. For STEM problem solving. we employ MMLU [62] and MMLUPro [63]. Standard train/test splits are used for GSM8K, MATH, MMLU, and MMLUPro. Following prior work [19, 64, 65], the AIME training set consists of problems from 1983-2023. For BBH, we randomly split the dataset into training set and test set by sub-task. Further dataset statistics can be found in SE. Accuracy evaluated using zero-shot pass@1 accuracy via greedy decoding. We use different random seeds for training and report the average performance over 3 runs.
7

---

87.9
67.5
Satori
87.5
Satori
Accuracy (36)
65-
64.2
61.8
64.3
62.3
53.1
60.3
66.2
78.2
57.5
17.5
52
70
MATH
Figure Ablation study results on BBH and MATH. We compare the performance of the standard GPO method and Satori's strategy that randomly identifies the critical step in the trajectory. Each bar represents the average performance of 3 runs, with error bars indicating the standard deviation.
6.1 Main Results
The results are presented in Table The table clearly demonstrates the effectiveness and gen- eralizability of GPO. Across all datasets and all 5 optimization algorithms, integrating GPO consistently leads to improved test accuracy compared to the respective baseline method. This con- sistent improvement underscores the robustness of leveraging critical step identification enhance fine-tuning.
Notably, the performance gains achieved by GPO are often substantial. For instance, GPO-PPO and GPO-DPO show significant accuracy increases on the MATH dataset compared to standard PPO and DPO. Similar positive trends are observed across other datasets like MMLUPro and the AIME benchmarks. While the magnitude of improvement varies depending on the specific dataset and baseline algorithm, the consistent improvement validates our core hypothesis: focusing learning on critical reasoning steps provides a more effective training signal, leading to enhanced reasoning capabilities in the fine-tuned models.
6.2 Ablation Study
To better understand how learning from the critical steps contributes performance improvement, we conduct an ablation study on the BBH and MATH datasets to analyze the impact of critical step identification. Satori [6] has shown that randomly locating step in the trajectory, then resetting and exploring the trajectory from that step, could help augment the training data and improve the performance of RLHF. Inspired by this, randomly locate the critical step in the trajectory and compare the performance of the standard GPO method. The results are presented in Figure 2.
The GPO method consistently outperforms Satori's random selection baseline across both datasets. The difference is particularly obvious on MATH, where PPO with GPO achieves 87.9% accuracy, significantly higher than the 79.9% achieved when using Satori's 's strategy. These findings suggest that the performance gains of GPO are not merely due to the random resetting mechanism itself. Instead, the identification and learning from critical steps, leading more effective training signal for crucial points, is a key factor driving the improvement observed during training.
6.3 Monte Carlo Simulation and Model Size Scaling
To understand the scalability of GPO, we investigate its performance to two key factors: the number of MC simulations used for critical step identification and the size of the base model. First, vary the number MC simulations from 2 to 16. Second, we apply GPO to models of varying scales, specifically D DeepSeek-BB ----iitiil-LLaate 70B model. These experiments are conducted on the MATH dataset, using both DPO and KTO as the underlying optimization algorithms. The results presented in Figure 3.
DeepSeek- -I-isstil---ti-eeereeeer
series (1.5B, 7B. 14B, 32B parameters) and the
The illustration reveals that increasing the number of MC simulations generally improves the per- formance benefit GPO. This suggests that a more accurate estimation of the advantage function, derived from more simulation samples, can lead to better training performance. However, the perfor-

---

Cotimization Alportthm
Dplimization Algorihm
6PO-0R3
* GPO-70
Simulation Sample Scaling
Model Size Scaling
Accuracy (%)
85.0
50
80.0
65-
775
3.50
140
120
702
Simulation Samples
Model Size
Figure 3: Scaling behavior of GPO. Performance impact of varying number of MC samples (left)
and applying GPO across different model sizes (right) on MATH using DPO/KTO.
mance gains appear to saturate beyond 12 simulation samples, potentially because the estimation of the advantage function converges. suggests that the overhead of GPO can be reduced by achieving a balance between the number of simulation samples and the performance gain.
Furthermore, the results also demonstrate that GPO consistently outperforms the corresponding baseline optimization algorithms (DPO/KTO without GPO) across all tested model sizes, from 1.5B 70B parameters. This consistent improvement highlights the robustness of the GPO and its applicability to larger, more capable language models.
User Study
We explore whether GPO's identification of 'critical steps' aligns with humans. We conduct user study to evaluate the correlation between steps identified by GPO and those by human evaluators.
Study Setup. We designed study involving 50 participants, recruited from college students. Participants are presented with five reasoning problems, each accompanied by a corresponding reasoning trajectory generated by the base LLM. For each trajectory, participants are asked to identify the single step they believed was the most critical point during the reasoning process for the final answer. They presented with four optional steps for each problem: one step identified by GPO, while the other three options are randomly selected. Further details regarding the specific questions and trajectories are provided in Appendix F.
Results and Discussion. The results indicate strong alignment between the critical steps identified by GPO and human judgment. Across the five questions evaluated, the percentage of participants who selected the GPO-identified step as the most critical was 44%, 68%, 88%, 76%, and 56%, respectively. This high degree of agreement suggests that the steps pinpointed by process are also often recognized by humans as the crucial points. These findings provide qualitative validation for the core mechanism of GPO, supporting the hypothesis that its empirical improvements come from.
8 Discussion and Limitations
While GPO shows promise in enhancing LLM reasoning, we acknowledge several limitations and areas for future work.
key limitation is the computational overhead from the Monte Carlo estimation of future returns at each step. This adds a neg-nggligible cost, increasing PPO training time by approximately 1.9x and offline data preparation by 1.8x, common challenge for related methods [21, 66, 32]. However, this trade-off is manageable. First, as shown in $6.3, performance gains saturate after certain number of simulations, allowing for a practical balance between accuracy and cost. Second, to adapt GPO for very long trajectories, we employ simple heuristic of grouping multiple generation steps into single logical step. We validated this approach on a challenging long-context reasoning subset of BigBBenchExtremeHHard (BBEH) [67]. where this simple grouping strategy enabled GPO to achieve 36.0% accuracy- substantial +7% improvement over the DPO baseline. This result confirms that GPO can be effectively scaled to complex tasks, and we provide the experiment details in $E.5.

---

Furthermore, there are several promising avenues for future research. To further enhance efficiency in the online setting, the current Monte Carlo estimation could be replaced with a more sample- efficient alternative like Generalized Advantage Estimation (GAE)[8), leveraging the value network trained by the PPO algorithm. Beyond efficiency, open questions remain regarding the identification of critical steps. For instance, could model- -ased explainability techniques[38, 39] offer higher fidelity compared to our model-free approach? Investigating hybrid methods, such as using powerful commercial LLMs like GPT-4o to assist in identifying critical steps, could also be fruitful direction.
Finally, evaluating the quality of identified critical steps currently relies on downstream task per- formance and expensive, hard-to-scale human judgment. Benchmarks like ProcessBench [68] only focus on identifying first incorrect step failed trajectory. which of critical step identification. Developing automated metrics or benchmarks to reliably assess critical step quality and relevance would significantly benefit future research and accelerate iteration on methods like GPO. We hope our work encourages further exploration of these important questions.
fully aligned with the scope
Conclusion
In this work, we introduce GPO, a strategy to enhance LLM reasoning by identifying critical steps within generation trajectories. Supported by theoretical guarantees, GPO demonstrably boosted per- formance across seven diverse reasoning datasets when integrated with five optimization algorithms, highlighting its generalizabilit.. Furthermore, the study confirms that the critical steps identified by GPO align well with human judgments of pivotal moments in reasoning failures. We believe GPO represents valuable step towards more robust and reliable reasoning in LLMs, and we hope inspires further research into targeted trajectory optimization and analysis.
Acknowledgement
This work was supported in part by NSF Grants 2225234 and 2225225, This research was also supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility Northwestern University which jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. We thank the anonymous reviewers for their constructive feedback and valuable suggestions that helped improve this work.
10

---

References
[1] Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[2] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk. Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
[3] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
[4] Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for Ilms with deliberative planning. arXiv preprint 2tt.:22006.14283. 2024.
[5] Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx. Summer Yue, and Hugh Zhang. Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409. 200.0240,,333,, 2024.
[6] Maohao Shen, Guangtao Zeng. Zhenting Qi, Zhang- Wei Hong. Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain of aattinooo----t--------e------oe- saction-thought enhances Ilm reasoning via autoregressive search. arXiv preprint arXiv:2502.022 02500, 2025.
[7] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 2022.
[8] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[9] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2023.
[10] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of of-houht prompting elicits reasoning in large language models. Advances neural information processing systems, 2022.
[11] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 2023.
[12] Macicj Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[13] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Lc, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. / 2022.
[14) Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Hl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint 2rXiv:2412.16720) 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.

---

[16] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05563. 2023.
[17] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint 1..iv:2339.122888 2023.
[18] Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. Unleashing reasoning capability of Ilms via scalable question synthesis from scratch. arXiv preprint MwXiv:2410.18691 2024.
[19] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025.
[20] Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492, 2025.
[21] Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold. Advances in Neural Information Processing Systems, 2024.
[22] Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03880 03898, 2024.
[23] Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. Advances in Neural Information Processing Systems, 2023.
[24] Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Wei Wu, Benyou Wang, and Dawei Song. Lifting the curse of capacity gap in distilling large language models.
[25] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.0885. 2023.
[26] Alyona Vert. Everything you need to know about knowledge distillation, 2025. Accessed: 2025-05-04.
[27] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. arXiv preprint arXiv:2401.0133. 2024.
[28] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401.00 08967, 3, 2024.
[29] Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Oleksandr Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. Learning math reasoning from self-sampled correct and partially-correet solutions. arXiv preprint arXivv2205./4333 2022.
[30] Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXhv:2312.06585, 2023.
[31] Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving chain-of-thougle reasoning in llms. Advances in Neural Information Processing Systems, 2024.
[32] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 2022.
[33] Fangkai Jiao, Zhiyang Teng, Bosheng Ding. Zhengyuan Liu, Nancy F Chen, and Shafiq Joty. Exploring seprrsuper logic-enhanced training for large language models, arXiv preprint 17711 2023.
12

---

[34] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000, 2024.
[35] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679 2024.
[36] Jonathan D Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D Lee, and Wen Sun. Dataset reset policy optimization for rlhf. arXiv preprint arXiv:2404.08495, 2024.
[37] Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy- explore. arXiv preprint urXiv:2507.00017 2025.
[38] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. Edge: Explaining deep reinforcement learning policies. Advances in Neural Information Processing Systems, 2021.
[39] Jiahao Yu, Wenbo Guo, Qi Qin, Gang Wang, Ting Wang, and Xinyu Xing. {AIRS): Expla- nation for deep reinforcement learning based security applications. In 32nd USENIX Security Symposium (USENIX Security 23), pages 7375-7392, 2023.
[40] Zelei Cheng, Xian Wu, Jiahao Yu, Sabrina Yang, Gang Wang, and Xinyu Xing. Rice: Breaking through the training bottlenecks of reinforcement learning with explanation. arXiv preprint arXiv:2405.03064 2024.
[41] Zelei Cheng, Xian Wu, Jiahao Yu, Wenhai Sun, Wenbo Guo, and Xinyu Xing. Statemask: Explaining deep reinforcement learning through state mask. Advances in Neural Information Processing Systems, 2023.
[42] Alexis Jacq, Johan Ferret, Olivier Pietquin, and Matthieu Geist. Lazy-mdps: Towards inter- pretable reinforcement learning by learning when to act. arXiv preprint arXiv:2203.08542 2022.
[43] Sandy H Huang. Kush Bhatia, Pieter Abbeel, and Anca D Dragan. Establishing appropriate trust via critical states. 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE.
[44] Dan Amir and Ofra Amir. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th international conference on autonomous agents and multiagent systems, pages 1168 1176, 2018.
[45] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 2rXiv:25033/97788 2025.
[46] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXlv:2402.01306, 2024.
[47] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 2024.
[48] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017.
[49] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
[50] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
[51] Hyungkyu Kang and Min-hwan Oh. Adversarial policy optimization for offline preference-based reinforcement learning. In Proc. of ICLR, 2025.
13

---

[52] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step- dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406. 18629, 2024.
[53] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator- weighted offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning. PMLR, 2022.
[54] Zhang- Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harness- ing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306. / 13085, 2023.
[55] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach scaling up reinforcement learning on the base model. arXiv preprini arXiv:2503.24299, 2025.
[56] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
[57] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[58] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
[59] AIME. Aime 2024 problem set. http:://nnngiigggeeecc////tttttt RAME_2024, 2024.
[60] AIME. Aime 2025 problem set. http:://wagggggggeeec.o/aatttttp AIME2025, 2025.
[61] Mirac Suzgun, Nathan Scales, Nathanael Schärli. Sebastian Gehrmann. Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quae V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
[62] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[63] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[64] Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 7b tir, 2024.
[65] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 2024.
[66] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship learning mathematical reasoning with large language models. arXiv preprint 23Xi:2330.008820 2023.
[67] Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Peter Chen, al. Big-bench extra hard. arXiv preprint 2901 2025.
14

---

[68] Chujie Zheng, Zhenru Zhang, Beichen Zhang. Runji Lin, Keming Lu, Bowen Yu. Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint urXiv:2412.06559, 2024.
[69] Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. Proc. of ICLR, 2023.
[70] Xuefeng Liu, Hung TC Le, Siyu Chen, Rick Stevens, Zhuoran Yang, Matthew R Walter, and Yuxin Chen. Active eowaantage-alignnn online reinforcement learning with offline data. arXiv preprint arXiv:2502.07937, 2025.
[71] Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv.2403.07691, 2024.
[72] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Illmmfactorry: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly state the scope of the paper and the main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In $8, we discuss the limitations of the work performed by the authors, including the overhead of the proposed method as well as other potential limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: In $5, we provide the full set of assumptions and a complete (and correct) proof for the main theoretical results. In $C.1, provide complete proof for the main theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In $6, we provide the full set of experimental results for the proposed method. In Appendix E, we provide the implementation details for the experiments. We also release the code for our method in the supplemental material. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the code as well as the data the supplemental material, with instructions for how to run the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /ttpps://niie cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. cctpublcc/ggeeeeccccccccccccccce mublic/guides / iettmmsssinPaa. asionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Appendix E, we provide the experimental setting/details for the proposed method, including train/test splits, hyperparameters, and optimizer settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In our experiments, we report the mean results across runs and show the error bars in the figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix E, we provide the compute resources used for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https htt///neuripppp cc heabblii/tthishhiieeee.
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and we believe that the research conducted in the paper conforms the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix A, we discuss the potential positive societal impacts and negative societal impacts of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The data we used in our experiments are commonly used benchmarks in the community and do not pose any safety risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly credit the creators of the assets and mention the license and terms
of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcom thcode. com/coo/////ooooo com/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have a detailed README file for the assets we released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Appendix F, we provide the full text of instructions given to participants as well details about compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Appendix B, we discuss that the user study we conducted exempted from IRB review and poses no harm to the participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only use LLM for formatting purposes during paper writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: hhppss///nuripss cwwpurips.cc/oofffeee for what should or should be described.
A Broader Impact
Beyond enhancing the reasoning capabilities of L.MMs, GPO holds significant potential for advancing the trustworthiness and transparency of these models. By employing our method, we can identify and highlight the critical steps within the reasoning processes of LLMs. This capability not only aids in demystifying the pocision-making pathways of these models but also empowers users to gain a deeper understanding of how conclusions are reached. Consequently, this increased clarity can foster greater trust in the outputs generated by LLMs. Furthermore, by making the reasoning process more transparent, stakeholders can more easily verify and validate the model's 's decisions, thereby enhancing the overall reliability and acceptance of LLMs in various applications. There could be potential negative societal impacts of our work. For example, if the critical steps are not properly highlighted, it could lead to misinformation and harm the trust in LLMss However, believe that the potential benefits of our work outweigh the potential risks, as we open-source our method and inspire research the critical steps identification in LLMs.
B
Ethics Considerations
Our work involves the use of user study to evaluate whether the identified critical steps align with human preferences. We have detailed the instructions we gave to participants in Appendix F to make transparent and reproducible. The questions we designed only evaluate the critical steps; thus, they pose no harm to the participants. We consult the IRB office at our institution and receive an exemption for this study. Moreover, we do not collect any sensitive information from the users.
C Theory
C.1
Online policy gradient algorithm
First, we introduce the performance difference lemma from [36].
Lemma (performance difference lemma [36]). For any policy nt' and reward function r, we have
1-1 E, - KKQ ( T/. (sw)))
(So)
Vo
(5)
h=0
22

---

Based on the performance difference lemma, we can rewrite the regret as
regret= (Vo (sn) Vo¹ (so)) =
en " [10 (8%), (T)))))))))))))))))))))))))))( (s.)))
(6)
h=0
where 27 (.) is the estimated Q function for the policy x' and di denotes the state-action visitation measure given the optimal policy x². So we decompose the above equation be
regret-
1(27% (s (sh)))
(term( ())
T
H-1 E
+
Σ
an 11Q2 (sh) Q (sn), T77 - (s,))))
(2))
We present the following theorem to bound the term (1). Theorem C.I. Suppose Assumption 5.1 holds, have
T
H-1 E sn [ (8n), (sh) T (Sh))) S Tmaz H
T log [A]
Proof of Theorem C.I. First, we decompose term (1) into two parts.
(QT (sn), (sa) & (sn))
=QQ (sn), (sm) - // (sn)) + (Q% (sn), +++ (sn) - (sn)) S(QT (sn), (s)) (sh)) 11 (sn) || (sn) कं (sn)lla
By Assumption 5.1, we have 11 (sx) S Tman, which further implies
(Q (sn), ()()) (sw))
SI¹T (sn), म (sx)
T/
++ (sx)) + Tmax||TT (+++++11 (sx)) क (sx)lla
Note that the policy update formula for the online policy gradient algorithm has closed-form expression [51]
Th (sn)
(i (sh)) ( (ow)
Z (s,) Ti
(8)
where Z) (sh) a normalization factor. Take the logarithm over both sides, and we have
'T (sn) log (sa) + log Th (sn) log (sh)
Thus,
(Q (sn), (8n) - - 1+1 (sa.)
(log Zj (sh) log m/ (sn) log / (sn))s=7(ss)) m7(()) =
Note that log (x(s))
Eal |T' (alsk) (a((s))) O. We can reorganize the above equation (Ô (sn). (5a) T (sh))
- U
(sx) -KL
77
23

---

Note that by Pinsker's inequality, we have
|| ++1 (ss) (oW))l S K ( (sn)ll
(10)
2
We further obtain the following bound for (º% (sn): (sh) - t (sn)))
(2% (sn), / (sh))
S (8h)))
" ( (sh)) + (T((m(())))))(()))))
(s)) - (x)l
21
Therefore, we have the following upper bound for (Qh (sn), (86)) =b()))
((())) = -i (sn), Th (sn) tht (sn)) + (Q) (st), (sn) - x/((5))
(QR (sn))
1 [K LKK (sn (1 (bn)) KL(T (sn)]) + imax|1=Hh 1 (sp) - (/))l)
1 || (sh) TT (sn)
2m
1 KoANE (sn)] + + 5" max
Summarizing over all horizons and over all iterations, have
T #-1
E d
(QR (sk), (sn) - Tf.(
L -KT)) (T( (st))
I Enn d
-nmr max
#-1
[KL(M - (=))) (+=m)+
T maxx"
0=
Note that initially xt r'' (an|sn) is uniform distribution over the action space A. We further have
E
(QT (sx), Th m (sn))
A=0
log |Al
E sn
+ 2 mrmuss?
17
0=0
2 log AAl
Let 17
V
and we get
E ay m (º (sn), (sh)) ((()))
H-1
log [A|
+ 2 r 2ax T)
Bun
17
h=0
T Tlog [A
-T
2

---

Algorithm 2: Q function estimation
Input: Num of rollout K, Current policy x², Reward r. Hyperparameter y
2: Initialize: D. = 0
3: for & === .......... K do
4: Collect one trajectory {( ag st.
s #- a - })}.
5: Compute the advantage A" (s, a) = Q" (s, a) V* (s) for each (s, a).
6: Sample (s, a) with probability proportional to exp(> a)). Denote the sampled state-action pair as (sm ).
7: Reset r' to * and follow n' to generate trajectory {(s a am....y s a)))
H-1 8: Compute 4t = "j=mm ry and add (s am, y ( into Dr.
9:
end for
10: Compute Q argminkE ([f(s, -
We present Algorithm 2 to obtain the estimate Q function.
Note that under Algorithm 2, we actually reweight the state-action occupancy, i.e., reweighted state-action occupancy
de,(s,a) d% (s, (s.a) (s)
where (s) is normalization factor
Zn(s)= Σ
dh (s,a)e7dtiaal aEA
Then, we have the following lemma: Lemma ([69]), With probability 1, 6, Algorithm guarantees that, for every t 1, 2, ..... , and
h 1..,.....
E Lon Ny (U (sn. on) Q (sn. an) s C'r2 max log(T F1/88
K
where C² is an absolute constant.
Now, we bound term (2). Note that
dgy [(QT' (sn) Q (sn). (T - (sx))}
E sn 10% (sk) Q (sn), T (sn)) + sn 1Q (sn) - Q4 (sn). (sw))
-IB =I 1Q an) Q (sn, an}il
E an on an 1QTh (sh, an) - Q (sh, anlll
Bn [ (sh, an) Q (Sh, an)) )
Bxgnag" (amlo) (9T (sn, un) Q (sn, un))
2
+
a(s,a,h) B Bisnoandagg [ (sn, (0 - (sh. an) )
27
<2
max NE[H]
sup (s.a)ESxA
2 max log( (F1())
max KE(M]
sup (s.a)ExxA
w(s,a,h,7)
K
where w1s a, h,,))
d3 (s.a)
is the density ratio between di (x, a) and (8, a) and C' is constant.
2 (x.a)
25

---

Therefore, with probability 1 - 8, we have
T
H
E
wuy 110% (sx) Q% (sn), (1hh) - (sh)))
(11)
A=0
Cr
log (1/
STH
max
max NE(M) (a.a)ESXA
sup a(s,a,h,,)
(12)
K
(13)
Now, we would like to show that the density ratio will decrease with an increase in y.
Based on Theorem in [70], we can rewrite d' (s, a) as (8, a) dz (s, (ala)) The density ratio w(s is
di (s, a) EA dh (s, a))
0% (s, a a (a|s)>
Suppose di (s, a) & exp(3 (s, a)) and - + (a|s) x exp(32A" (s, a)) for some parameter 31 < B2. This reasonable since the updated policy π ++ better than the current policy x'. Take the logarithm and we have
log w(s, a, h 7) log di (s, a) + log Σ (0 cl0 (ama) - (B1 + 327) (s,a) (14)
Partial derivative of log w(s a, h,,,,, 7) with respect to is:
ΣA A" (5,4))((++ (e.a)
By llg w(s, 7)) = 32
AT
(15)
Σn e (s,a)
Note that the largest density ratio happens for a" argmar, (s, a) Due to the softmax function in the gradient, see that for a", the derivative negative, meaning that by increasing 7, the regret bound will decrease.
C.2 Preference optimization
Proof. We follow the proof strategy outlined in Theorem 6.1 of [21]. To derive the desired result, we begin with the key observation that DPO [9] is equivalent to optimizing a KL-regularized expected reward objective, where the reward function is used to define preferences via the Bradley-Terr -erry model. Specifically, the optimal policy n" (-)-) that maximizes the following regularized objective:
max Exop, (-on(- [r v)] 3DDL (#(- (r) m w:))
given in closed form
(r(x,)) v) B
(y x)) merr x)) exp
(16)
This optimal policy can be recovered by applying DPO to preference- pairs (z, Vı, y2). where preferences are sampled from the Bradley-Terry model [49] defined by the reward function
exp(r(x,
plun ž u2 xx) (xp(r((,)))) exp(r(xx y2))
(17)
Given this background, we consider preference pairs of the form (z, [Jo:i-1, 1, vith, [Vo 1, I),, where both continuations are sampled from the Tref: y,* Teee(- xz, (o:.---)) 1). y4 Teer(- xr, No.s i). and the preference is determined based on the advantage estimates Ar (x,, vo: Combined with Eqn. 16, this yields the following equation:
AArr (x,, yo.--1:9s)
ग (s 90:4-1) x Tet(y; vo--1) exp
(18)
B
Moreover, since the optimal serantage-weightee policy that maximizes Eqn. 18 coincides with the solution in Eqn. 4, the proof is complete.
26

---

D Additional Preference Optimization Objectives
Beyond the DPO objective described in the main text, several other preference optimization methods have been developed. These methods also typically operate on an offline dataset of preference pairs D = (xr y* y ) where y² is the preferred response and y is the dispreferred response given an input z. The policy being optimized is denoted by To, with 0 being its parameters, and Teat is fixed reference policy.
ORPO [71] introduces an objective that penalizes the model for assigning low likelihood to preferred responses, while simultaneously ensuring that preferred responses have higher odds than dispreferred ones. For the single preference pair (x, y y the ORPO loss is:
Po (x)) pa (yt
pe (xT) po (
CORPO(TO; ug," , ") Dogpo (1y*t111)) (x)
logg
log
log
where po(y)æ) exp log (v(( is length- normalized likelihood for sequence y, log To Te (y/z)
is the sum of propabbbabilities of tokens in y, (-)) is the sigmoid function, and À is weighting coefficient. This formulation directly encourages the policy to generate y" and ensures its odds are favorable compared to y
SimPO [47] offers modification of the DPO-style loss by incorporating sequence length normaliza- tion directly into the log- log-probability difference and adding a margin term 7. The SimPO loss for a preference pair is:
Lsipo (T; v. y² ,y" 1 log (3 log No (y+1z)
log Toe (y (x) -7)-
ly+
where 3 is constant that controls the scaling of the reward difference. This objective aims to maximize the margin between the morgh-nnomaalied log-likelihood the preferred response and that of the dispreferred response.
KTO [46] introduces an alignment approach rooted in prospect theory. Instead of optimizing preference likelihoods, KTO focuses on directly maximizing the utility of each individual generation y given an input x. distinctive characteristic is its reliance on a binary signal for every input-output pair (I, v) E 'D classifying
either desirable or undesirable for Consequently, KTO does not inherently require paired preference data (i.e., yt vs y The loss for single such sample (I, y) formulated to be minimized and is given by:
(KTO(TO, Teet v) = (A (1 - (ro(z,) v) zx))) if y is desirable for æ Au (1-o (3 (zo - (p(x,,))) if is undesirable for x
In this formulation, ro y) = log xe(glw)
)a(()=))) represents the log-probability ratio of the current policy Ty against a fixed reference policy Tea. The term 2n serves as a reference point related to the KL estimate. 3 is hyperparameter modulating risk aversion, and Àp, Au positive hyperparameters that weight the contributions from desirable and undesirable outputs, respectively. This structure allows KTO process feedback individual generations, and if paired data is available, each part of the pair (y+, ,y-) would contribute its own loss term based on its desirable/undesirabh status.
E Additional Experiment Details
In this section, we provide additional details on the experiments.
E.1
Dataset Details
As mentioned in $6, we utilize established train/test splits for several benchmarks. For GSM8K. MATH, MMLU, and MMLUPro, we adopt their standard train/test distributions. Specifically for BBH, the dataset is randomly partitioned into training and test sets at the sub-task level. For AIME dataset, problems from the years 1983-2023 constitute the training set, while problems from 2024-2025 form the test set. Detailed statistics for each dataset, including the number of samples in the training and test sets, and the source of the dataset, presented in Table 2.
27

---

Table 2: Dataset Statistics: The table presents the number of training and test samples for each dataset, along with the source of the dataset.
Dataset
Train Samples
Test Samples
Source
GSM8K
7470
1320
https:/
httgs////gggigaac/eeee wtdddaaatttpppeaaia/aaeeeee gamBk
MATH
12000
hatps://githun cwwp//ona///ceeeeeeeeeeeeaaaaaat pprm00kk
MMLU
99842
14042
httpe: attpe://hugsaa /hwgggiggaaceeeccccc//// co/ cetlassts/cais/nnen
MMLUPro
9625
2407
https://huue httg////gggggggg///eeeee
BBH
3261
3250
Hard
AIME-2024
903
bttps:
attpe://huggingfase ace. co/
AIME-2025
903
30
https://huggingfee wtt.acc.e...eeaeeeeeeeeeeeeeeeee www.a/aaeettt///onnngnn//aaaaaaa
E.2
Implementation Details
Here we provide further details on the data processing and algorithm implementation for the experi- ments.
Question Filtering To construct a training set of appropriate difficulty, we apply a filtering process to each dataset. For every question in the initial training pool, generate 8 responses using the base model with a sampling temperature of 0.7. Questions that are solved correctly across all eight attempts, conversely, incorrectly across all eight attempts, subsequently excluded from the training set. This procedure aims to retain questions that are neither trivially easy nor prohibitively difficult for the base model, thereby focusing the fine-tuning process on more informative problem distribution. This filtering protocol is applied uniformly to create the training data for both the baseline methods and their GPO-enhanced counterparts.
Step Grouping for Trajectory Segmentation Reasoning trajectories generated by the models are segmented into multi-steps using the following procedure. First, each trajectory split into steps based on newlines. Multiple consecutive newline characters are collapsed into a single newline. To avoid overly short steps, any step consisting of fewer than 30 words is merged with the previous step. Furthermore, to maintain computational feasibility, we impose maximum number of steps: 15 for offline preference optimization methods and 10 for online PPO. If a trajectory exceeds this maximum step count, all subsequent steps beyond the limit are concatenated to form the last step.
Preference Data Preparation for Offline Methods For the offline preference optimization al- gorithms DPO, SimPO, and ORPO, positive and negative preference data are sourced during the question filtering process described above. Since the filtering retains questions for which the base model produces mix of correct and incorrect responses across the generated samples, these naturally provide pairs of successful (positive) and unsuccessful (negative) trajectories for the same input question.
For KTO, which requires demonstration data rather than explicit preference pairs, we adapt the approach from the original KTO paper [46]. For the baseline KTO, each preference pair (y y from the DPO dataset is decomposed into two separate demonstrations: y* with the positive tag and y with the negative tag. For the GPO-enhanced KTO, first apply the GPO strategy to form paired preference data. These n preference pairs are then similarly decomposed into 2 n individual positive and negative demonstrations to train the GPO-enhanced KTO model.
E.3 Hyper-parameters for Experiments
We list the main hyper- paper-parameters for the experiments in Table 3. The GPO-enhanced methods are trained with the same hyper-parametesn as the baseline methods. For those unmentioned hyper- parameters, we use the default values provided by the LLaMA -aaat-F framework [72].
E.4 Running Environment
Our training process primarily utilizes the LLaMA-Factory framework [72]. The experiments are conducted on a server equipped with four AMD EPYC 7702 64-Core CPU Processors and eight NVIDIA H100 80GB GPUs. The total computational resources consumed include approximately 1800 GPU hours and 2 TB of storage for model checkpoints.
28

---

Table 3: Hyper-parameters for Different Datasets and Methods.
Method
Dataset
Hyper-parameter
GSMSK
MATH
MMLU
MMLUPro
BBH
AIME-2024
AIME-2025
LoRA Alpha (o)
2
2
2
2
LoRA Rank
8
8
LoRA Target
all
Common
Optimizer
Adam
Adam 4096
Adam 2048
Adam
Adam
Adum
Adam
Sequence Length
4096
2048
2048
4096
4096
Clip Value
0.2
0.2
0.2
0.2
0.2
KL Divergence Coeff.
0.05
0.05
0.05
0.05
0.05
0.05
0.05
PPO
Leurning Rute
le-5
le-5
2e-5
2e-5
le-5
le-5
le-5
Batch Size
32
64
64
64
128
Epochs
5
Beta (3)
0.1
0.1
0.1
0.1
DPO
Leurning Rute
le-5
le-5
le-5
le-5
le-5
le-5
le-5
Batch Size
16
16
Epochs
10
10
10
Desirable Reward Scular Undesirable Reward Scalar
L0
1.0
1.0
KTO
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Leurning Rute Batch Epochs
le-5
le-5
le-5
le-5
le-5
le-5
le-5
10
10
2
10
10
10
10
SimPO
Reward Margin
0.5
0.5
0.5
0.5
0,5
0.5
0,5
Leurning Rute Batch Epochs
le-5
Se-6
Se-6
le-5
le-5
le-5
le-5
16
10
10
5
10
10
10
10
ORPO
Leurning Rate
le-5
le-5
le-5
lc-5
le-5
le-5
le-5
Batch Size
2
16
16
16
Epochs
10
10
Table 4: Accuracy on a long-context renngoonntxxreasonnnnn subset of BBEH.
Method
Accuracy (%)
Base Model (DeepSeek- -R
24.5
DPO Baseline
29.0
GPO-DPO (with step grouping)
36.0
E.5 Experimental Details for Tong-Trajectory Reasoning
We provide supplementary details for the experiment discussed in Section 8, which was designed to validate the scalability GPO tasks involving very long trajectories.
Experimental Setup. To create a challenging long-contex: reasoning benchmark, we selected sub-dataset from BIG-Bench Extra Hard (BBEH). This dataset chosen for its particularly long problem descriptions (averaging approximately 1,700 tokens) and lengthy, complex reasoning chains (averaging approximately 190 lines). The experiment conducted using training set of 1,000 questions and a test set 200 questions. The base model used for all experiments was DeepSeck-RR -inadk-RR--istil---ee----
Step Grouping Heuristic. To adapt GPO for these long trajectories without incurring prohibitive computational costs, we implemented a simple step grouping heuristic. For the GPO-DPO method, we grouped every 15 lines the generated reasoning chain into a single logical "step". This approach maintains the core principle of identifying pivotal moments for targeted feedback while significantly reducing the number of points at which future returns need to be estimated.
The results, summarized in Table 4, demonstrate that even with this straightforward heuristic, GPO delivers substantial performance gains over both the base model and the DPO baseline. This confirms that GPO can be effecttivly adapted for long-trajectee reasoning tasks.
Results.

---

User Study
To evaluate if the human annotators agree with the critical steps identified by method. conducted a user study. This section details the setup of the study and the format used for presenting questions the participants.
F1 User Study Setup
Before participating in the study, participants presented with an informed consent form, outlining the purpose of the research, the nature of their participation, and confirming that no private information would be collected. Participation voluntary.
The study consisted of five questions and designed to take approximately five minutes to complete. For each question, participants were presented with the following:
Task description: A brief overview of the problem type.
Question: The specific question posed to the LLM.
Ground truth answer: The correct answer the question.
LLM's reasoning trajectory: The step-by-step reasoning process generated by an LLM,, with each step clearly numbered. This reasoning leads to incorrect answer.
Selection task: Participants are then shown four potential critical steps. Their task is to select the single step they believe is the most critical LLM's s reasoning towards the result. Within the four steps, there is one step that is identified by our method, and the other three steps are randomly selected from the of the steps in the trajectory.
F2 User Study Questions and Results Format
the questions and the results format below. We highlight the critical step identified by our method in red for the paper presentation, and note that the participants will not see the red-highlighted text.
Question 1: Infer the date from context
Task description: Infer the date from context.
Q: It was Sept. 1st, 2021 week ago. What is the date 24 hours later in MM/DD/YYYY?
Options:
(A) 03/09/2022
(B) 09/10/2021
(C) 08/31/2021
(D) 09/09/2021 (Ground Truth)
(E) 09/02/2021
(F) 09/17/2021
The Large language model's reasoning trajectory with labeled step number is: 1. Okay, so need to figure out the date 24 hours after September 1st, 2021. Let me break this down step by step.
First, the question says it was September 1st, 2021, a week ago. Wait, does that mean today is September 8th, 2021? Because week is seven days, adding seven days to September would land on September 8th. But the question asking about 24 hours after September 1st, not today's date. Hmm, maybe I'm overcomplicating it.
3. Wait, no, the question is straightforward. It says, 'It was Sept. 1st, 2021 week ago. What is the date 24 hours later in MM/DD/YYYY?" So, regardless of today's date, just need find the date that is exactly 24 hours after September sst, 2021.
So, 24 hours later would the next day, September 1st, plus one day September 2nd,
2021. Let me check the options provided.
5. (A) 03/09/2022- That's March 9th, 2022. That's way too far in the future.
30

---

6. (B) 09/10/2021 10th, 2021. That's nine days later, not just
(C) 08/31/2021-August 31st, 2021. That's the day before September 1st, that's day earlier, not later.
8. (D) (9/09/22011Seeiemmerr 9th, 2021. Again, that's eight days later.
9. (E) 09/02/2021-Septenee
2nd, 2021. That's exactly day after September 1st.
10. (F) 09/17/2021-Septenn
17th, 2021. That's 16 days later.
11. So, the correct answer should be option (E), which 09/02/2021. I think got it right.
Just adding one day to September 1st gives September 2nd.
12. The date 24 hours after September 1st, 2021 is September 2nd, 2021, which corresponds to option (E).
Participant task: There are several steps listed; please select which step you prefer that is more critical to the LLM's failure.
Step 2
Step 3
Step
Step 11
Results for Question 1:
Step 2(Our method): 44%
Step 3: 32%
Step 5: 6%
Step 11: 18%
Question 2: Answer questions about causal attribution
Task description: Answer questions about causal attribution.
Q: How would typical person answer each of the following questions about causation? Drew, Kylie, Oliver, and Jen are regular customers small, local coffee shop. Given the selling price of the coffee and the cost of daily operation, the coffee shop will turn profit anyone orders coffee on given day. Only one person ordering coffee needed for the coffee shop to turn a profit that day. Drew, Kylie, Oliver, and Jen all usually order coffee on Tuesdays. This Tuesday, as usual, Drew ordered coffee. The same day, Kylie ordered coffee, Oliver ordered coffee, and Jen also ordered coffee. Since at least one person ordered coffee on Tuesday, the coffee shop made profit that day. Did Drew ordering coffee on Tuesday cause the coffee shop to make profit that day?
Options:
Yes (Ground Truth)
. No
The Large language model's reasoning trajectory with labeled step number is:
1. Okay, so I'm trying to figure out whether Drew ordering coffee on Tuesday caused the coffee shop to make a profit that day. Let me break this down step by step.
First, the coffee shop makes a profit
least one person orders coffee. So, the key here is that only one order is needed for profit. Now, this particular Tuesday, Drew, Kylie, Oliver, and Jen all ordered coffee. That means there were four orders that day.
3. The question is specifically about Drew's order. Did Drew's 's action cause the profit? Well, since the shop only needs one order to make a profit. Drew's order alone would have been sufficient. But in this case, there were three other people who also ordered coffee. So, even if Drew hadn't 't ordered, the shop would still have made profit because of the others.
This makes me think about the concept of causation. For Drew's order to be a cause, should be necessary for the outcome. But since the outcome (profit) would have happened regardless of Drew's order, his action isn't the necessary cause. It's 'ss more like contributing factor, but not the sole necessary one.
31

---

5. In this case, Drew's order is sufficient its own, but it's not necessary because others also ordered. So, while his order contributed, it wasn't t the cause in the sense that was required for the profit.
6. Another angle is think about what would happen if Drew didn't order. If the shop still made a profit because of the others, then Drew's s order wasn't 't the cause. It's like saying if you have multiple people contributing to event, each one isn't an't the sole cause unless their absence would change the outcome.
7. So, putting it all together, Drew's order wasn't 't the cause because the profit would have occurred without him. Therefore, the answer should be no.
8. The coffee shop made a profit because at least one person ordered coffee, but since others also ordered, Drew's order wasn't necessary for the profit. Therefore, Drew's action wasn't t the cause.
Participant task: There several steps listed; please select which step you prefer that is more critical to the LLM's s failure.
Step
Step
Step
Step
Results for Question 2:
Step 1: 2%
Step 4(Our method): 68%
Step 5: 22%
Step 6: 8%
Question 3: Questions that involve enumerating objects and asking the model to count them
Task description: Questions that involve enumerating objects and asking the model to count them.
Q: have a microwave, a lamp, four cars, a stove, a toaster, and a bed. How many objects do I have?
Ground Truth Answer:
The Large language model's reasoning trajectory with labeled step number is:
1. Okay, so need figure out how many objects the person has. Let me list them out one by one to make sure don't 't miss anything.
2. First, they mention microwave. That's 's one object. Then there's lamp, that's two.
3. Next, they have four cars. Wait, four would count four separate objects, right? So that adds four to the total.
4. After that, there's 's a stove, which is another object, making it five so far. Then toaster, that's six. Finally, a bed, which brings the total to seven.
5. Let me double-check to make sure didn't skip any. Microwave (1), lamp (2), four cars (4), stove (5), toaster (6), bed (7). Yep, that adds up seven objects in total.
6. The person has a microwave, lamp, four cars, stove, toaster, and bed. Counting each item individually, the total number of objects is 7.
Participant task: There several steps listed; please select which step you prefer that is more critical to the L.LM's failure.
Step
Step
Step
Step
Results for Question 3:
32

---

Step 2: 4%
Step 4(Our method): 88%
Step 5: 2%
Step 6: 6%
Question 4: Multi-step arithmetic word problem
Task description: Multi-step arithmetic word problem.
Q: Jeremy is driving 600 miles to visit his parents. His average speed for this trip is 50 miles per hour. Every two hours of driving, he makes a rest stop for 15 minutes. His car gets 18 miles per gallon of gas, and his gas indicator comes when he's used 15 gallons, which point he stops to refill, which takes 10 minutes. How long, in minutes, does he take to get to his parents' house if he starts out with full tank of gas?
Ground Truth Answer: 815
The Large language model's reasoning trajectory with labeled step number is:
1. First, need determine how long will take Jeremy to drive 600 miles an average speed of 50 miles per hour. Next, I'll calculate the total driving time by dividing the total distance by his average speed.
Then, I'll figure out how many rest stops he" Il need. Since he takes a 15-minute break every hours of driving, I'll divide the total driving time by 2 find the number of rest stops. After that, I'll calculate the total rest stop time by multiplying the number rest stops by 15 minutes.
3. Next, I'll determine how much gas he during the trip dividing the total distance by his car's fuel efficiency. Then, I'll find out how many times he needs to refill his gas tank by dividing the total gallons used by the capacity of his gas tank.
Each refill takes 10 minutes, I'll multiply the number of refills by 10 get the total gas stop time. Finally. I'll add up the driving time, rest stop time, and gas stop time to get the total time the trip minutes.
Let's break down Jeremy's s trip step by step to determine the total time he takes to reach his parents' house. ###1. Calculate the Total Driving Time Distance to Travel: 600 miles Average Speed: 50 miles per hour Driving Time Distance/Speed = 600 miles/50 mph
12 hours
6. ###2. Determine the Number of Rest Stops Jeremy makes
stop every 2 hours of driving. Number of Rest Stops 12 hours/2 hours per stop 6 stops. Each rest stop lasts
15 minutes. Total Rest Stop Time 6 stops 15 minutes 90 minutes 1.5 hours
###3. Calculate the Gas Consumption and Stops Car Fuel Efficiency: miles per gallon
Total Gas Used: Total Gas = 600 miles/18 mpg = 33.33 gallons
8. Number of Gas Stops: Since the gas indicator comes after using 15 gallons, we divide the total gas by the gas used per stop. Number of Gas Stops 33.33 gallons/15 gallons
2.22 stops
9. However, since he starts with full tank, he will need refill before the gas out. Therefore, he makes 2 gas stops. Each gas stop takes 10 minutes. Total Gas Stop Time stops * 10 minutes 20 minutes 1/3 hour
10. ###4. Calculate the Total Time Total Time Driving Time Rest Stop Time + Gas Stop Time Total Time = 12 hours 1.5 hours + 1/3 hours = 13.833 hours Convert the total time to minutes: 13.833 hours 60 minutes/hour 830 minutes
Participant task: There are several steps listed; please select which step you prefer that is more critical to the LLM's failure.
Step 3
Step 4
Step
Step
33

---

Results for Question 4:
Step 3: 14%
Step 4: 10%
Step 6(0ur method): 76%
Step 9: 0%
Question 5: Financial calculation word problem
Task description: Financial calculation word problem.
Q: Tom's 's cat needs an expensive surgery. He has had pet insurance for 24 months that costs $20 per month. The procedure cost $5000, but the insurance covers all but 20% of this. How much money did he save by having insurance?
Answer: 3520 (Ground Truth)
The Large language model's reasoning trajectory with labeled step number is: 1. First, need to calculate the total cost of Tom's pet insurance over 24 months. The insur- ance costs $20 per month, so multiplying that by 24 gives the total insurance expenditure. Next, I'll determine the cost of the surgery. The surgery itself costs $5000, but Tom's 's pet insurance covers all but 20% of this amount. To find the amount covered by insurance, I'll calculate 20% of $5000 and subtract that from the total surgery cost to get the out-of-pocket expense.
3. After that, I'll compare the total insurance with the out-of-pocket surgery cost find the savings. By subtracting the out-of-pocket expense from the insurance cost, can determine how much money Tom saved by having insurance. Finally. I'll present the calculated savings as the answer.
4. Let's break down the problem step by step to determine how much money Tom saved by having pet insurance. ###Step I: Calculate the Total Pet Insurance Cost Tom has pet insurance for 24 months at cost of $20 per month.
5. Total Insurance Cost 24 months * $20/month $480 ###Step 2: Determine the Surgery Cost Covered by Insurance The surgery costs $5000, and the insurance covers all but 20% of this amount. Insurance Coverage 100% 20% 80% Amount Covered by Insurance 80% * $5000
0.8 $5000 = $4000
7. ###Step 3: Calculate the Out-of- Pocket Expense Tom will pay the remaining 20% of the surgery cost out of pocket. Out-of-Pocte Expense = 20% $5000 0.2 $5000 = S1000
8. ###Step 4: Determine the Savings To find out how much Tom saved by having insurance, subtract the out-of-pocket expense from the total insurance cost. Savings Total Insurance Cost Out-offPocckt Expense = $480 $1000 = -$520
9. Wait, this result indicates that Tom actually spent $520 more with insurance compared to not having insurance. However, this doesn't make sense because insurance should reduce his expenses. Let's re-examine the calculations. Revised Approach The correct way to calculate savings is:
10. The correct way to calculate savings is: Savings = Surgery Cost - (Surgery Cost Insurance Coverage) Total Insurance Cost Savings $5000 (S5000 * 0.8) $480 $5000 $4000
$480 = $520
Participant task: There several steps listed; please select which step you prefer that is more critical to the L.LM's failure.
Step 4
Step
Step
Step 10
Results for Question 5:
34

---

Step 4: 6%
Step 6: 8%
Step 8(Our method): 56%
Step 10: 30%
F.3
Overall Findings
Overall, the results indicate a strong alignment between the critical steps identified by GPO and human judgment. Across the five questions evaluated, the percentage of participants who selected the GPO-identified step as the most critical was 44%, 68%, 88%, 76%, and 56%, respectively. The strong alignment between the steps identified by our method and those recognized by humans as critical points indicates that our process effectively highlights key reasoning steps. This alignment serves qualitative validation for the core mechanism of GPO, reinforcing the hypothesis that its empirical improvements are well-founded.
35


Reviews:
[{"review_id": "mn2mQiPRPy", "reviewer": "Reviewer_FbHp", "summary": "The authors introduce guided pivotal optimization (GPO), a fine-tuning strategy for reasoning pathways which can be operationalized with policy-based and preference-based optimization algorithms. The method first identifies crucials point within CoTs, and then generates training data as continuations starting from this step. Then, this data is used to tune the network using PPO or DPO. The authors evaluate the utility of their method on DeepSeek-R1-Distill-Qwen-7B across 7 datasets and consider 5 different tuning algorithms to operationalize their strategy. The authors show that their method consistently improves upon tuning with the base algorithm and conduct ablation studies comparing how their algorithm behaves with respect to number of MC samples and model size.", "questions": "- How are the positive and negative trajectories obtained? Or, how are the samples ensured to be positive, and negative trajectories? Or, more plainly, how do we ensure that the new trajectory will improve on the previous one?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The idea of identifying crucial points within the reasoning process and resampling is interesting\n- The proposed algorithm outperforms on the base tuning methods\n- The authors conduct experiments across a large number of datasets and add supporting ablation studies\n\nWeaknessess\n- The writing, especially in the part outlining the algorithm, could use improvement. Parsing the paragraph which describes identifying the critical step via advantage is difficult. Entire Section 4 would benefit from a rewrite, as it is very dense.\n- The experiments are conducted on a single model family, and while on some datasets the performance gain is significant, on others it is pretty negligible. I would also suggest reporting average gain in a separate column to emphasize the performance improvement.\n- The idea of identifying critical steps in CoTs has been explored before, limiting the novelty of the proposed approach [1]\n\n[1] Wang, T., Chen, J., Han, X., & Bai, J. (2024). CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks", "was_revised": true, "final_justification": "The authors have added experimental results on another model family and committed to clarify the description of their method, as well as add a discussion regarding a closely related work. While this response has mostly addressed my concerns, I would like to see the revisions in writing, as I believe exposition would benefit from improvement. However, my score remains a 4, while I am inclined to lean towards acceptance (5).", "rebuttal": "We thank Reviewer FbHp for the constructive and insightful comments. Please see our response to each of your questions below.\n\n> The experiments are conducted on a single model family \n\n**Response:** We thank the reviewer for this valid point. To demonstrate that GPO’s effectiveness is not limited to a single model family, we have conducted new experiments on **Llama-3.1-8B** and **InternLM-3-8B** over two datasets: MATH and GSM8K. We select three baselines for experiments. The average results **over three runs** are shown below:\n\n | Llama-3.1-8B  | GSM8K (%) | MATH (%) | \n |--------|-----------|----------| \n | Base | 69.60 | 62.80 | \n | PPO | 72.36 | 67.20 | \n | GPO-PPO | **74.17** | **69.80** | \n | DPO | 76.57 | 68.20 | \n | GPO-DPO | **77.30** | **72.40** | \n | KTO | 75.39 | 68.60 | \n | GPO-KTO | **76.57** | **74.20** | \n \n\n| InternLM-3-8B  | GSM8K (%) | MATH (%) | \n|--------|-----------|----------| \n| Base | 82.64 | 79.20 | \n| PPO | 83.53 | 82.60 | \n| GPO-PPO | **86.20** | **84.20** | \n| DPO | 87.15 | 86.80 | \n| GPO-DPO | **91.54** | **89.80** | \n| KTO | 89.32 | 86.40 | \n| GPO-KTO | **91.52** | **89.20** | \n\n\nFrom the results, we can observe that GPO consistently outperforms all baseline methods across both new model families, confirming its general applicability. We will integrate these results into the final version of our paper.\n\n> How are the positive and negative trajectories obtained? …… how do we ensure that the new trajectory will improve on the previous one? \n\n**Response:** We thank the reviewer for this insightful question. We would like to clarify that positive and negative trajectories are only required in the offline setting.\n\nIn the online setting, we do not constrain that re-exploration from the identified critical steps should improve the original trajectory’s reward. Instead, our goal is to reduce the overall policy regret as shown in Theorem 5.2. As we prove in Appendix C.1, resetting and exploring from advantage-based critical steps improves the initial state distribution for exploration, thereby leading to a better-learned policy.\n\nIn the offline setting, positive trajectories are those that succeed, while negative trajectories are those that fail. For a given question $x$, if continuing from step $y$ does not lead to a successful outcome, we use advantage values as a proxy to select trajectories with higher advantage values as positive trajectories. Theorem 5.3 provides the theoretical justification for using advantage as this selection criterion.\n\nWe would add this clarification in the next version.\n\n> The idea of identifying critical steps in CoTs has been explored before, limiting the novelty of the proposed approach [1] \n\n**Response:** We sincerely thank the reviewer for bringing CPL into our attention. While both our work (GPO) and CPL share a high-level goal of improving the reasoning process, they are built on fundamentally different principles and offer distinct contributions. In fact, CPL implicitly leverages the difference of advantage (i.e., $adv(s_t, a_t) - adv(s_t, a'_t))$ to update the policy through its per-step DPO function, and it is limited to offline RL setting and lack of theoretical guarantee. In contrast, our method is driven by a different motivation and design: we explicitly identify critical steps based on RL advantage, and then enhance the policy by further exploration from these key steps. Moreover, our algorithm is applicable to both online and offline settings, offering broader applicability and stronger theoretical guarantees. In addition, we present a case study to validate the correctness of our identified critical steps. We will incorporate the discussion into our next version.\n\n> The writing, especially in the part outlining the algorithm, could use improvement. \n\n**Response:** We thank the reviewer for their valuable feedback on the clarity of Section 4. To clarify, the critical step is identified via the advantage function, $Adv(s_t, a_t) = Q(s_t, a_t) - V(s_t)$. In our setting with deterministic transitions and zero intermediate rewards, this simplifies to the difference between consecutive Q-values: $Q(s_t, a_t) - Q(s_{t-1}, a_{t-1})$. Thus, we only need to estimate the Q-function, which can be done unbiasedly via Monte Carlo. The overview of this method is detailed in Algorithm 1. We will revise Section 4 to state this derivation clearly and improve its overall readability.\n\n> I would suggest reporting average gain in a separate column to emphasize the performance improvement\n\n**Response:** Thanks for this suggestion, below is the performance gain compared with the original optimization algorithms and we would add the table in our revised version.\n\n| Algorithm | BBH | MATH | GSM8K | MMLU | MMLUPro | AIME-2024 | AIME-2025 |\n|-----------|-----|------|-------|------|---------|-----------|-----------|\n| GPO-PPO | +1.66 | +8.20 | +0.48 | +2.73 | +3.58 | +3.33 | +3.34 |\n| GPO-DPO | +1.05 | +4.40 | +2.43 | +1.85 | +3.65 | +6.67 | +6.67 | \n| GPO-KTO | +1.45 | +2.40 | +0.94 | +1.93 | +1.50 | +3.33 | +6.67 | \n| GPO-SimPO | +0.61 | +1.80 | +1.77 | +0.51 | +2.04 | +3.33 | +3.34 |\n| GPO-ORPO | +0.53 | +3.00 | +0.91 | +1.00 | +1.99 | +3.33 | +3.33 |"}, {"review_id": "T8LWrV4wRN", "reviewer": "Reviewer_6HQB", "summary": "This paper introduces a novel fine-tuning strategy to improve LLMs’ multi-step reasoning abilities, GPO. While existing methods such as PPO or DPO optimize reward or preference over entire reasoning trajectories, GPO aims to identify and prioritize critical steps—pivotal moments in the reasoning chain that heavily influence the final result. GPO uses an advantage estimation (from RL) to find these steps, then resets generation at these points to sample and train on new continuations. Extensive experiments covering seven diverse reasoning datasets indicate that GPO consistently and significantly improves test accuracy over baselines.", "questions": "1. Why only one critical step is selected in each reasoning path? Would it be better to use the top 3 or threshold method? For example, when there are multiple critical steps in the reasoning path.", "limitations": "Yes, the authors fully address the limitations of their work and the potential negative social impacts.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tThe idea of focusing learning on “critical steps” of reasoning (rather than on the entire trajectory) is original and reasonable. \n2.\tGPO is a play-and-plug module compatible with both online RL (PPO) and various offline preference learning algorithms (DPO, SimPO, ORPO, KTO), which broadens its applicability.\n3.\tThe paper not only provide empirical results, but also offers regret bounds for the proposed method and links per-step preference learning to advantage-weighted RL objectives.\n4.\tExperiments are thorough, covering 5 different optimization algorithms and 7 datasets. Strong and consistent improvements are shown.\n\nWeaknesses:\n1.\tThe key step of GPO - Monte Carlo estimation of future returns at each inference step - adds a non-negligible computational cost (~1.8-1.9x). While the authors believe this is manageable, this may be limiting for very long trajectories or large-scale settings. The tradeoff between benefits and computational needs discussion.\n2.\tSome failures might be dispersive or cumulative (multiple small errors). Could GPO be further improved by handling multiple critical steps, or scoring all steps proportionally?", "was_revised": true, "final_justification": "I will maintain my recommended score.", "rebuttal": "We thank Reviewer 6HQB for the constructive and insightful comments. Please see our response to each of your questions below.\n\n> The key step of GPO - Monte Carlo estimation of future returns at each inference step - adds a non-negligible computational cost (~1.8-1.9x). While the authors believe this is manageable, this may be limiting for very long trajectories or large-scale settings. The tradeoff between benefits and computational needs discussion. \n\n\n**Response:** We thank the reviewer for this insightful comment. We agree that the trade-off between performance gains and computational cost is a critical aspect, and we appreciate the opportunity to elaborate on how GPO can address this, particularly for long trajectories.\n\n1.  **Managing Overhead via Simulation Samples:** As we discussed in Section 6.3 and demonstrated in Figure 3 (left), the computational overhead is manageable. Our experiments show that performance gains begin to saturate after a certain number of MC simulations. This allows for a practical balance between accuracy and computational cost, as one does not need an excessive number of rollouts to achieve most of the benefit.\n    \n2.  **Adapting to Long Trajectories with Step Grouping:** To address the challenge of very long trajectories, we employ an initial and simple heuristic: grouping multiple generation lines into a single step. This maintains the core idea of identifying pivotal moments without incurring significant computational cost for complex trajectories.\n    \n    To validate this, we conducted a new experiment on a challenging reasoning task. We used a sub-dataset 'boardgame' from BIG-Bench Extra Hard (BBEH) as it has the longest problem descriptions within BBEH (avg. ~1700 tokens) and lengthy reasoning chains to solve the problem (avg. ~190 lines). We randomly selected another 5 subtasks as the training set. The test set contains 200 questions, while the train set contains 1000 questions. We applied a simple heuristic of grouping every 15 lines into a single step for GPO. The results are as follows:\n    \n    -   **Base Model (DeepSeek-R1-Distill-Qwen-7B):** 28.5% accuracy\n        \n    -   **DPO Baseline:** 32.0% accuracy\n        \n    -   **GPO-DPO (with step grouping):**  **36.0% accuracy**\n        \n    This experiment confirms that GPO can be adapted for very long trajectories, yielding a substantial **+4% improvement** over DPO baseline. As grouping is a very initial mechanism, we would like to add the discussion and leave more complex strategies as future work.\n    \n3.  **Future Avenues for Efficiency:** Furthermore, there are promising avenues for future optimization. For the online setting (PPO-based), the current Monte Carlo estimation of the advantage function could be replaced with a more sample-efficient alternative, i.e., Generalized Advantage Estimation (GAE) [1], which may further reduce computational overhead. We can utilize the value network trained by the PPO algorithm to estimate the advantage more efficiently.\n    \nWe will incorporate the related discussion into the next version of our paper. We thank the reviewer again for their constructive feedback, which has helped us demonstrate the practical scalability of GPO.\n\n> Some failures might be dispersive or cumulate. Could GPO be further improved by handling multiple critical steps, or scoring all steps proportionally?\n\n**Response:** We thank the reviewer for this insightful question. The idea of handling multiple critical steps is an important but also challenging topic in explainable RL, as prior work [2,3] in XRL also chose to focus on one critical step.\n\nThe choice to focus on a single critical step is based on practical ground, which we are happy to clarify. LLM reasoning can be modeled as a sequential decision, where each step depends on the entire prefix. The \"criticality\" (i.e., advantage) of any given step is conditioned on the specific path taken to reach it.\n\nWhen we identify and reset from the first most critical step in a trajectory, the subsequent reasoning path is entirely new. Any other \"critical points\" identified in the original, flawed trajectory beyond this reset point may no longer be relevant, or even unreachable, in the new paths. Therefore, resetting at the point of highest potential improvement is the most logical and efficient way to explore the most impactful alternative reasoning branch. Tackling multiple pre-identified critical steps simultaneously is conceptually challenging due to this path-dependent nature of trajectory generation.\n\nAs to scoring all steps proportionally, our framework is flexible enough to accommodate it. As described in Section 5.1, our algorithm can be generalized to sample the reset point with a probability proportional to its advantage, controlled by a temperature parameter $\\gamma$. This provides a \"softer\" focus. As shown in Theorem 5.2, a larger $\\gamma$ implies a tighter regret bound which motivates us to select the step with maximal advantage in one trajectory in our main experiments. Empirical results show that selecting the step with maximal advantage provides the strongest learning signal by enabling the model to learn the most significant point.\n\nWe will add this clarification to the paper to make our design rationale more explicit. We thank you for prompting this important discussion.\n\nReferences:\n\n[1] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017).\n\n[2] Cheng, Zelei, et al. \"Statemask: Explaining deep reinforcement learning through state mask.\" Advances in Neural Information Processing Systems 36 (2023): 62457-62487.\n\n[3] Cheng, Zelei, et al. \"RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation.\" International Conference on Machine Learning. PMLR, 2024."}, {"review_id": "LcdhLigdxU", "reviewer": "Reviewer_r5W8", "summary": "This paper introduces Guided Pivotal Optimization (GPO). GPO stands as a general strategy integrable to existing optimization methods by identifying critical steps and prioritizing the learning process on the rollouts of these critical steps, which are identified by estimating the advantage functions per step. This work also provides a theoretical analysis that proves the online GPO as a form of advantage-weighted RL in the offline preference learning setting. The evaluation is extensive, where GPO demonstrates consistent improvements over optimization baselines. Lastly, it includes a user study with 50 participants analyzing the alignment between the GPO critical step identification and human judgment.", "questions": "1. Could you also include the standard deviation in Table 1?\n\n2. How would you justify splitting critical steps by newlines?\n\n3. I see you set a maximum number of steps around 10 to 15. However, for logical reasoning methods, can you justify whether your method scales to tasks with a larger number of steps?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: \n\n1. Automatically identifying the critical step via advantage in an online learning setup is integrable with many existing optimization methods.\n\n2. The theoretical analysis is useful in bridging the gap between the proposed online learning and the offline preference learning.\n\n3. The evaluation dataset and baselines provide a good coverage of tasks and existing learning approaches to reflect the generalization of the proposed method.\n\nWeakness:\n\n1. As acknowledged by the author, integrating GPO inevitably introduces roughly twice the computation cost. \n\n2. The critical steps are classified by splitting the new line in Algorithm 1. I find this definition may be adequate for mathematical reasoning tasks. However, I doubt the generalization of this definition to a wider array of tasks, such as the coding task or factual QA tasks.\n\n3. The evaluation is only conducted on a single base model DeepSeek-R1-Distill-Qwen-7B model. It is still necessary to conduct the experiments on other families of models to provide sufficient evidence of the generalization of GPO.", "was_revised": true, "final_justification": "The rebuttal has addressed my questions for this work. Given the limited backbone models studied in this work and the heuristics of the newline separation for steps, I am keeping my rating at 4 with an improved confidence.", "rebuttal": "We thank Reviewer r5W8 for the constructive and insightful comments. Please see our response to each of your questions below.\n\n> I doubt the generalization of this definition to a wider array of tasks, such as the coding task or factual QA tasks. How would you justify splitting critical steps by newlines.\n\n**Response:** This is a good point, and we thank the reviewer for raising it. Our choice to split steps by newlines is a practical heuristic for reasoning tasks where LLMs naturally structure their thoughts into distinct lines. In LLM generation, there is no inherent \"step\" as in traditional reinforcement learning. While generation is token-by-token, treating each token as a step is computationally expensive and semantically weak, as individual tokens rarely carry complete meaning. Therefore, we use newlines to segment the output into semantical steps of reasoning, and this strategy can also be found at prior works[1,2]\n\nWe agree that this specific heuristic may not be optimal for all tasks. However, the core idea of GPO is to identify the critical step and explore, where the definition of a ‘step’ can be flexible and application-dependent. For example, for the coding task, a step can be the generation of a complete function, a class or a code block. For long-form QA, a step could be a full paragraph.\n\nThe step-grouping heuristic we discuss below is a prime example of this adaptability, demonstrating that the step can be adjusted to suit the task’s complexity and structure. We will make this point about the flexibility of step definition explicit in the revised version.\n\n[1] Wang, Huaijie, et al. \"Offline reinforcement learning for llm multi-step reasoning.\" Association for Computational Linguistics: ACL 2025\n\n[2] Setlur, Amrith, et al. \"Rewarding progress: Scaling automated process verifiers for llm reasoning.\" ICLR 2025\n\n> Can you justify whether your method scales to tasks with a larger number of steps?\n\n**Response:** To address the challenge of very long trajectories, we employ an initial and simple heuristic: grouping multiple generation lines into a single step. This maintains the core idea of identifying pivotal moments without incurring significant computational cost for complex trajectories.\n    \nTo validate this, we conducted a new experiment on a challenging reasoning task. We used a sub-dataset 'boardgame' from BIG-Bench Extra Hard (BBEH) as it has the longest problem descriptions within BBEH (avg. ~1700 tokens) and lengthy reasoning chains to solve the problem (avg. ~190 lines). We randomly selected another 5 subtasks as the training set. The test set contains 200 questions while the train set contains 1000 questions. We applied a simple heuristic of grouping every 15 lines into a single step for GPO. The results are as follows:\n    \n-   **Base Model (DeepSeek-R1-Distill-Qwen-7B):** 28.5% accuracy\n        \n-   **DPO Baseline:** 32.0% accuracy\n        \n-   **GPO-DPO (with step grouping):**  **36.0% accuracy**\n        \nThis experiment confirms that GPO can be adapted for very long trajectories, yielding a substantial **+4% improvement** over DPO baseline. As grouping is a very initial mechanism, we would like to add the discussion and leave more complex strategies as future work.\n\n> Integrating GPO inevitably introduces roughly twice the computation cost.\n\n**Response:** Thanks for this comment. We think this can be alleviated by:\n\n1.  **Managing Overhead via Simulation Samples:** \n\nAs we discussed in Section 6.3 and demonstrated in Figure 3 (left), the computational overhead is manageable. Our experiments show that performance gains begin to saturate after a certain number of MC simulations. This allows for a practical balance between accuracy and computational cost, as one does not need an excessive number of rollouts to achieve most of the benefit.\n    \n2.  **Adapting to Long Trajectories with Step Grouping:** as we have shown.\n    \n3.  **Future Avenues for Efficiency:** Furthermore, there are promising avenues for future optimization. For the online setting (PPO-based), the current Monte Carlo estimation of the advantage function could be replaced with a more sample-efficient alternative, i.e., Generalized Advantage Estimation (GAE) [3], which may further reduce computational overhead. We can utilize the value network trained by the PPO algorithm to estimate the advantage more efficiently.\n    \nWe will incorporate the related discussion into the next version of our paper. We thank the reviewer again for their constructive feedback, which has helped us demonstrate the practical scalability of GPO.\n\n[3] Schulman, John, et al. \"High-dimensional continuous control using generalized advantage estimation.\" arXiv preprint arXiv:1506.02438 (2015).\n\n> The evaluation is only conducted on a single base model.\n\n**Response:** We thank the reviewer for this valid point. To demonstrate that GPO’s effectiveness is not limited to a single model family, we have conducted new experiments on **Llama-3.1-8B** and **InternLM-3-8B** over two datasets: MATH and GSM8K. We selected three baselines for experiments. The results are shown below:\n\n | Llama-3.1-8B  | GSM8K (%) | MATH (%) | \n |--------|-----------|----------| \n | Base | 69.60 | 62.80 | \n | PPO | 72.36 | 67.20 | \n | GPO-PPO | **74.17** | **69.80** | \n | DPO | 76.57 | 68.20 | \n | GPO-DPO | **77.30** | **72.40** | \n | KTO | 75.39 | 68.60 | \n | GPO-KTO | **76.57** | **74.20** | \n\n\n| InternLM-3-8B  | GSM8K (%) | MATH (%) | \n|--------|-----------|----------| \n| Base | 82.64 | 79.20 | \n| PPO | 83.53 | 82.60 | \n| GPO-PPO | **86.20** | **84.20** | \n| DPO | 87.15 | 86.80 | \n| GPO-DPO | **91.54** | **89.80** | \n| KTO | 89.32 | 86.40 | \n| GPO-KTO | **91.52** | **89.20** | \n\n\nFrom the results, we can observe that GPO consistently outperforms all baseline methods across both new model families, confirming its general applicability. We will integrate these results into the final version of our paper.\n\n> Could you also include the standard deviation in Table 1?\n\n**Response:** We appreciate this suggestion, and the standard deviation table below will be included in the next version.\n\n|  **Algorithms**  |  **BBH**  |  **MATH**  |  **GSM8K**  |  **MMLU**  |  **MMLUPro**  |  **AIME-2024**  |  **AIME-2025**  |\n|---|---|---|---|---|---|---|---|\n| PPO | 0.59 | 0.81 | 1.38 | 0.45 | 1.41 | 0.00 | 3.33 |\n| GPO-PPO | 0.45 | 1.63 | 1.00 | 0.82 | 0.55 | 0.00 | 3.33 |\n| DPO | 1.53 | 1.45 | 2.61 | 0.47 | 0.47 | 0.00 | 1.92 |\n| GPO-DPO | 0.75 | 1.79 | 1.03 | 1.05 | 0.52 | 3.33 | 3.33 |\n| KTO | 0.35 | 1.59 | 1.10 | 1.82 | 1.78 | 3.85 | 3.85 |\n| GPO-KTO | 0.94 | 2.57 | 0.47 | 1.56 | 0.49 | 1.92 | 1.92 |\n| SimPO | 0.74 | 2.69 | 1.42 | 0.83 | 0.35 | 3.85 | 1.92 |\n| GPO-SimPO | 1.02 | 2.57 | 1.23 | 0.30 | 1.40 | 0.00 | 0.00 |\n| ORPO | 0.85 | 3.81 | 2.99 | 1.16 | 0.58 | 1.92 | 1.92 |\n| GPO-ORPO | 0.98 | 2.05 | 2.71 | 1.51 | 0.91 | 1.92 | 3.85 |\n\nTo further and more rigorously quantify the significance of the improvements brought by GPO, we have also conducted a **paired t-test** for each GPO-enhanced method against its corresponding baseline.  Across the 35 pairs of comparisons (5 baseline algorithms on 7 datasets), our analysis reveals that all 35 comparisons have the p-value < 0.05 (a lower p-value indicates a better statistical significance).\n    \nThis analysis provides strong statistical evidence that the performance gains from integrating GPO are not due to random chance but represent a consistent and meaningful improvement over the baseline methods. We will add the discussion of the statistical significance in the next version. Thanks again for the valuable feedback."}]

Author rebuttal:

Meta-review: This paper proposes Guided Pivotal Optimization (GPO), a fine-tuning strategy that improves LLM reasoning by identifying and resampling at critical steps in chains of thought, using advantage estimation. GPO can be combined with both online RL (e.g., PPO) and offline preference-based methods (e.g., DPO, SimPO, ORPO, KTO). It is novel to focus optimization on critical reasoning steps rather than entire trajectories. Meanwhile, the proposed GPO is general and compatible with multiple optimization methods. Thorough experiments across diverse datasets and optimization algorithms, showing consistent gains. Supporting ablations further strengthen the evidence. Overall, all reviewers agree the proposed method is interesting, well-validated, and of broad potential impact. I recommend acceptance.
Decision: Accept (poster)