SSVPO: Effective Step-Level Credit Assignment for RL Training of Language Models
Authors: Yugu Li
Zehong Cao
Jianglin Qiao
Siyi Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Credit Assignment
Reinforcement Learning
Step-Level Reward
Large Language Model
Primary area: reinforcement learning
TL;DR: Step-level reward credit assignment for efficient reinforcement learning training of large language models

Abstract: Language models have shown strong performance on mathematical reasoning tasks. Post-training with outcome-based reinforcement learning (RL) can further enhance reasoning but is inefficient because it relies solely on final rewards. Recent credit assignment–based RL methods provide intermediate feedback, yet they often struggle to fairly evaluate each step’s importance, especially in partially correct reasoning chains. We propose Sequential Shapley Value Policy Optimization (SSVPO), a step-level credit assignment framework inspired by multi-agent RL. SSVPO introduces an insertion MDP and Sequential Shapley Values (SSV), which measure each step’s marginal contribution by reordering reasoning steps into alternative chains, ensuring fair credit assignment to all possible steps. By identifying steps with zero credit, SSVPO can shorten reasoning chains to improve training efficiency. We further provide a theoretical proof that SSV fairness to allocate credits and demonstrate that SSV as the new advantage baseline is consistent with Proximal Policy Optimization (PPO). Across 7 benchmarks, SSVPO outperforms state-of-the-art RL methods, both outcome-based (RLOO, GRPO, DAPO) and credit assignment–based (VinePPO, SPO), achieving up to an 11.6\% gain in accuracy, an 18.1\% reduction in token usage, and a 1.6× improvement in reasoning efficiency over vanilla methods. Our findings highlight that SSVPO provides effective step-level credit assignment, advancing post-training LLM reasoning performance while reducing token budgets.

Published as a conference paper at ICLR 2026
SSVPO: EFFECTIVE STEP-LEVEL CREDIT ASSIGN- MENT FOR RL TRAINING OF LANGUAGE MODELS
Yugu Li & Zehong Cao
School of CSIT, Adelaide University Adelaide, SA 5000, Australia {yugu. 1i, jimmy. cao (aaaoeaaeelllii adu.ualdeeeeuuuuu edu.au
Jianglin Qiao
Siyi Hu
ACFR, The University of Sydney Camperdown, NSW 2050, Australia School CSTT, Adelaide University Adelaide, SA 5000, Australia jianglin
School EECMS, Curtin University Bentley, WA 6102, Australia
School CSTT, Adelaide University Adelaide, SA 5000, Australia
edd..uu
siyi. siyi.huBcurtinn edu. au
ABSTRACT
Language models have shown strong performance mathematical reasoning tasks. Post-training with outcome-based reinforcement learning (RI.) can fur- ther enhance reasoning but is inefficient because it relies solely on final rewards. Recent credit assignment-based RL methods provide intermediate feedback, yet they often struggle to fairly evaluate each step's ' importance, especially in partially correct reasoning chains. We propose Sequential Shapley Value Policy Optimiza- tion (SSVPO), step-level credit assignment framework inspired by multi-agent RI. SSVPO introduces an insertion MDP and Sequential Shapley Values (SSV), which measure each step's marginal contribution by reordering reasoning steps into alternative chains, ensuring fair credit assignment to all possible steps. By identifying steps with zero credit, SSVPO can shorten reasoning chains to im- prove training efficiency. We further provide a theoretical proof that SSV faimness allocate credits and demonstrate that SSV the new advantage baseline con- sistent with Proximal Policy Optimization (PPO). Across 7 benchmarks, SSVPO outperforms state-of-the-art RL methods, both outcome-based (RLOO, GRPO, DAPO) and credit assignment-based (VinePPO, SPO), achieving up an 11.6% gain in accuracy, an 18.1% reduction in token usage, and a 1.6x improvement in reasoning efficiency over vanilla methods. Our findings highlight that SSVPO provides effective step-level credit assignment, advancing post-training LLM rea- soning performance while reducing token budgets.
INTRODUCTION
Lightweight large language models (LLMs), such as Qwen3 (Yang et al. 2025), DeepSeek-RI (Guo al. 2025a), and Gemini-2.5 (Comanici al., (025) with fewer than 7B parameters, have demon- strated comparable reasoning performance to much larger models. These compact models can be widely applied to many reasoning tasks, with mathematical reasoning serving critical bench- mark (Hendrycks et al. 2021 Cobbe et al.) 2021 Mathematical Association of America 2023] 2024 2025). Mathematics requires multi-step logical reasoning chains, handling symbolic manip- ulation, and solving structured problems, making it rigorous testbed for evaluating a language
model's reasoning capability (Wang & Lu 2023 Forootani 2025
Patil & Jadon 2025
For post-training LLMs (c.g., fine-tuning) approaches, particularly reinforcement learning (RL) with outcome-based methods (c.g., reinforcement fine-tuning), have shown promise in enhancing mathe- matical reasoning performance from 2024 (Gao et al. (2024; Lyu et al. 2025 Han & Zhang 2025)
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
such Reward Leave- Oneeee---n--ee Out (RLOO) (Ahmadian et al. 2024). Group Relative Policy Optimiza-
tion (GRPO) (Shao et al. 2.24). and Dynamic Sampling Policy Optimization (DAPO) Yuu et al.
2025). Unlike earhier work supervised fine-tuning (Zelikman et al. 2022. Yu et al.
2023 Yue
et al. 2023), outcome-based RL relies purely on a final correctness reward signal, avoiding biases
introduced by human preference feedback. This enables LLMS to tackle challenging math problems more effectively, such as algebraic manipulation, equation solving, and those tasks involving geome- try, number theory, and combinatorics (Yang et al. [2024] Zoo et al. 2024). However, outcome-based RL has significant limitations in training efficiency: rewards are only granted at the end of the rea- soning chain when the final answer correct (Lightman et al., 2023] Uesato al. 2022] Gao al.]
(024). There is no explicit signal provided to intermediate steps that help measure their contribution
to the full reasoning process.
(A) Fair Credit Assignment
(8) Efficient Reasoning Chain
Noth Prompt
True Soludones
Math Prompt To 12
Heward True Soation elsw
Reward 11f True Soluion
Reasoring Steps
Sakaton Correc Rensoning Correct Ansoee
Baps
Dasallns Soluaioon Reasoning Stape with Redandancy Correc Anewer
Greitt Angreen
Blart wet g Sx-+-8
iieerlly conticient
2 ten aurtanttiinn muipprt d6 add - they prn - and -3
2. Wr quadiati farrut eppiii Vit
Focter 33/2
Bet wanh lade aave x- 22-00 Oarz
Chid Astigrrant
V
(oomen
Simppyy
(Roduncaan Nitl that swiding by can withen awwaaroo
(T
Roseiino
SSVPO (our))
Sokaien Whongg Reasorin-- Whory Arener
Spli the into two casss T+l
2
I-1
7. (Podurcant Recion each fardonn is too stape Lond (Notucarinece sciudere mocrdin onder
2 hare -
a (Macnass Comora dicivaane
Heoutt a (b, 4).
a thare Tree via 7 sou
spuro Solation (ound) -4 Staps (Eficiunt Russcring Correct Amnovr
Resultx 13) twrong dus the riktak in
Start witn a 7a e 13
-P-2
hactee 186 4) - pen Jere
Rreaar Scude
(3, 1)
Figure I: Overview of the differences between existing RL methods and our proposed SSVPO in mathematical reasoning tasks. (A) Baseline methods evenly distribute credit across all steps in reasoning chain, equivalent to outcome-based final rewards. In contrast, SSVPO assigns credit ac- cording to each step's 's importance, ensuring fair evaluation and allowing correct early intermediate steps to receive positive credit even when the final answer is wrong. (B) Baseline methods explore all possible reasoning chains without emphasizing key steps, whereas SSVPO focuses training on important steps identified through fair credit assignment, reducing chain length (e.g., from 8 to 4 steps) while still producing the correct answer.
As result, post-training LLMs tend to treat all reasoning steps as equally important for obtaining the final answer, crediting the same reward to each step. This often leads them to elongate reasoning chains to maximize the reward accumulation, thereby generating many redundant steps. further consequence of these elongated chains is the risk that, as the number of reasoning steps grows, LLMs may fail to model coherent reasoning paths and instead directly memorize final answers (Wu) et al. 2025 Sui et al. 2025; Wang et al. 2025 Li et al. (025a). This becomes a significant problem when applying the model to unseen mathematical problems, where generalization ability a must.
To mitigate this issue and enhance L.M reasoning efficiency, very recent studies such as VinePPO (Kazemnejad et al.) 2025). Segment Policy Optimization (SPO) (Guo et al., 2025b), and Group Token Policy Optimization (GTPO) (Tan & Pan 2025), have introduced credit assignment for post- training. These credit assignment-be
RL methods aim to decompose the outcome reward and allocate intermediate rewards reasoning steps, using techniques such as Monte Carlo returns, value differences between adjacent steps, and dynamic entropy weighting. While they show empir- ical gains in reasoning performance, they lack fair estimation of step-level rewards derived from the final reward.
Key Challenge: Design credit assignment method with theoretical guarantees that faith- fully capture each reasoning step's marginal contribution, such that important steps can be leveraged to improve training efficiency in subsequent RL training of LLMs.
Inspired by prior work on fair credit assignment using Shapley values (Li et al. 2021 Wang et al. 2022a 2020 Han et al.] 2022) in multi-agent reinforcement learning (MARL) (Foerster al. 2018
2

---

Published as a conference paper at ICLR 2026
Sunehag et al. 2018 Rashid al. 2020: Wei al. 2022 Li al. 2025b), address this gap by modeling the reasoning process as sequence of agents, where each step in a Chain of Thought (CoT) Yao et al. 2023 Stern et al. 2019 Lightman et al. 2023) is treated as individual agent. To achieve this, we design a new form Shapley value that, unlike the classical formulation which as- sumes independent and exchangeable participants, extends from the spatial domain in MARL the temporal domain of reasoning, enabling exploration of all possible combinations of reasoning steps be new chains for computing their marginal contributions while explicitly capturing sequential dependencies and order sensitivity. This provides a theoretically fair credit allocation for each step when reordered into new chains. We hypothesize that such fair step-level credit assignment can im- prove reasoning efficiency over baseline methods and mitigate overthinking issues (e.g., redundant tokens) commonly observed in outcome-based RL. A detailed review of outcome- and credit assignment RI. training for LLMss, along with the adaptation of original credit assignment methods from MARL to LLMs, is provided in Appendix A
Therefore, in this paper, we develop Sequential Shapley value Policy Optimization (SSVPO) to achieve two goals: (1) Fair step-level credit assignment via the proposed Sequential Shapley value (SSV), which provides theoretical guarantees by allocating credit to each reasoning step based on its marginal contribution, and is applicable even to partially correct reasoning chains; and (2) Efficient reasoning through policy optimization, where SSV acts new value baseline for the advantage estimator. This ensures that steps with high marginal contributions are prioritized during training, leading to improved sample efficiency and shorter reasoning chains, as illustrated in Figure [
Our contributions are summarized as follows:
Fair credit assignment for each reasoning step. We offer an effective step-level credit assign- ment method (SSV) for outcome-based based RL. training of LLMs, which fairly evaluates each step's marginal contribution by constructing new chains from the insertion MDP.
Effective advantage estimator for RL training. We propose new advantage estimator (SSVPO) that leverages the SSV advantage function to guide policy optimization, providing an unbiased, minimal-varian variance value baseline for stable convergence and enabling shorter, more ef- fective reasoning chains.
Superior benchmark performance. On mathematical reasoning benchmarks, our proposed method achieves superior performance and efficiency compared existing outcome-based and credit assignment RL methods, which improves accuracy by 1 11.6%, reduces token usage by 18.1%. 1%, and achieves a 1.6-fold gain in reasoning efficiency over vanilla methods.
METHOD
Our proposed SSVPO method introduces a RL post-training paradigm that ensures the credit decomposed from the final reward is fairly allocated to each reasoning step under outcome-based RL settings. This faimess enables the formulation of new advantage function for fine-tuning (e.g.. policy optimization), thereby improving both reasoning performance and efficiency.
2.1 SSVPO WORKFLOW
Given a math prompt, we first use the vanilla model generate rollouts, then apply an extraction- and-aggregation procedure that compresses long reasoning chains (Appendix G.2) by identifying and merging intermediate reasoning steps. This reduces the number of steps involved in reordering, lowers computational overhead, and removes the final answer sentence from each reordered chain to prevent direct access to the ground-truth answer. This forms the SSVPO pre-stage workflow, followed by the three formal stages (Stages 1-3) of reasoning and training illustrated in Figure
In Stage 1, by using the extracted steps, we construct reordered chains based on the Insertion MDP (Sec. 2.2 In SSVPO, step reordering is not used to generate new trajectories. Instead, permutations are used solely to construct alternative prompts. This process does not rely on the language model's autoregressive next-token generation; rather, it enables modelling the marginal contribution of each step under different chain orders. In Stage 2, we feed these reordered chains to the LLM as prompts, require it to answer directly from the given chain, and evaluate correctness to obtain rewards. These rewards are then used to compute the Sequential Marginal Contribution (SMC) (Sec(.2) and sub- sequently the Sequential Shapley Value (SSV) (Sec. 2.3 for each reasoning step. In Stage 3, the
3

---

Published as a conference paper at ICLR 2026
SSVPO Workflow
Stage Sequeetiol Shapley Volue (SSV)
Model Rollout
Sampling Reasoning Steps
Start
WHH
3MC 0
Chock Answer:
- + 12 m0
Prompt
Solation
SMCo60u 90
aldnd m-44
SMC
wward
RM0l
SMC 30
the
BIN) bel of ooea tie neasoring stapa
Calcaale SEV el va sap at (22) porate ontengg)
f 14.5 a,p,y
(8 4.09
D.
Stage 1: Insertion MDP
Baccu Rest, x,ol)] [MM00(s..))
Previsation nasol ug chiil
Stage 3: Efficient Reasoning Choin with Higher S.V
Reward Scale
SBV hostiag all reasoringg steps
Reease chain beand 01 SMC sunciingg zape
SMRLAAI I RIV.- Rid
4 53002//044)0 (0t-aa00-00--00
Figure 2: Overview of the proposed SSVPO workflow. Given a math prompt, the LLM vanilla model first generates reasoning steps through rollout. The SSVPO workflow then proceeds in three stages: (1) represent the reasoning process as an insertion- based MDP, enabling the creation of new (re-ordered) chains by re-positioning the reasoning steps; (2) compute the Sequential Shapley value (SSV) for each step across multiple new chains, assigning fair credit by averaging its Sequen- tial Marginal Contributions (SMC); and (3) fine-tune LLMs vanilla model via policy optimization, where the SSV advantage serves as new value baseline to retain steps with higher contributions, ultimately yielding shorter and more effective reasoning chains that improve answer correctness.
SSV is used as step-level baseline, combined with its corresponding SMC to produce a fair advan- tage estimate (Sec.2.4) This advantage is then used in PPO-style policy updates, providing more granular step-level credit assignment within the standard Reinforcement Learning from Verifiable Rewards (RLVR) framework.
2.2 MODELING LLMS REASONING PROCESS AS AN INSERTION MDP
The reasoning process in language generation models is often formulated as a Markov Decision Process (MDP) (Kazemnejad et al. 2025: Guo et al. 2025b), where each token (or reasoning step) appended sequentially to the end of the current sequence (chain), and the state transition is defined by deterministic concatenation. However, under the standard next-token prediction perspective, such an MDP only cares about what token (or step) is generated at the end of the sequence, and the final reward is assigned only once after the entire reasoning chain is completed, making it difficult to distinguish the specific contribution of each intermediate step across different chains. To address this limitation, we propose
"Insertion MDP" that operates only the already generated candidate steps: in this new MDP, a state is a current reasoning chain, an action is to insert a candidate step at an arbitrary position in this chain, and by comparing the rewards obtained before and after inserting this step, we compute the marginal contribution of this step under the given context and insertion position. Note this new Insertion MDP itself does not involve the language model's generation process; instead, it is a credit-assigmment model that is specifically designed to finely model and estimate marginal contributions through reordering steps. The insertion process continues until all steps have been considered, thereby generating all possible re-ordered new chains. This formulation enables the estimation of each step's marginal contribution by evaluating its role across chains with different insertion orders. Importantly, these insertions are independent of semantics: we do not require the reordered chains to maintain a coherent or human- rradable reasoning structure. Instead, whenever the model is still able produce the correct answer from such reordered chain, interpret this as evidence that the permutation reveals an additional, model-specific reasoning path.
Specifically, we formalize the Insertion MDP tuple prompt, which serves as the input to the vanilla LLM. N' = S. R). Q denotes the math mwnn is a finite set of reasoning steps generated from the vanilla rollout, with |A| < co. S = {(Q, ni.:)) | E Zzo, ny E N, Vk e 1,....t} defines the state space, i.e., set of reasoning chains. Each state
(Q, n1:1) contains the prompt and the first t reasoning steps. A denotes the insertion action. For a given state s,
A(s) = ((,,.) E Nr E 1,..., + 11]. where each action (n, corresponds to
4

---

Published as a conference paper at ICLR 2026
inserting step n at position z in s, yielding new chain s' = Ins(s, (s, n, (Q, 111:2-1: 1, n, R2z:1) A + A(S) is the transition function of Insertion MDP, represent as: P(s' 8, (n,z)) 1fs' Ins(s; s, n, x)). R: S + {0, 1} is the reward function, where R(s) if the reasoning chain 8 produces a correct solution to 2, and R(s)) = O otherwise. To quantify the marginal contribution of a reasoning step under Insertion MDP, (R(s)) and after insertion (R(s))) This leads to the notion of the Sequential Marginal Contribution (SMC), defined below.
compare the reward of the chain before insertion
Definition (Sequential Marginal Contribution). Given an Insertion MDP (2, S,,,SS,A,T R), for
any current reasoning chain = (2, mi:e) E S and an insertion action a = (n,x) with æ E (I....,(++1. let s' Ins((s, 2) denote the new (re-ordered) reasoning chain after insertion. The Sequential Marginal Contribution (SMC) of inserting a reasoning step at position of the new chain s' defined as
SM R(s') R(s).
2.3 FAIR CREDIT ASSIGNMENT FOR EACH (INSERTED) REASONING STEP
In the Insertion MDP, reasoning step's marginal contribution must be evaluated fairly, as it depends on both the ordering of its predecessors and its specific insertion position. To ensure faimess, we introduce the Sequential Shapley value (SSV), which attributes credit each step within reasoning chain by accounting for order- and position-dependen effects. For each step, SSV is computed by averaging the SMC values across all possible re-ordered new chains under alternative insertion actions. The formal definition of SSV is provided below.
Definition 2 (Sequential Shapley Value). Consider an Insertion MDP S, R), an or- dering of N refers to permutation that arranges all reasoning steps into a chain, with each step included exactly once. Let G(A) be the set of all possible orderings. For n E N and a E G(N), let pred, (n) denote the predecessor of reasoning step n in ordering o, which indicates the ordered reasoning steps that appear before step n in a (c.g., if N = {n1, 72, n2, and (n2, R1, nz), then pred_(n,) (12) and pred, (na) = (n2, n1)). Set s = (Q, (ned,(n)). The Sequential Shapley value (SSV) of reasoning step n is then
(n = EEES(N) (xe((....ls.111)) (s.,,(s..)))
(2)
To guarantee that SSV achieves fair credit assignment in LLMs reasoning, impose the Shapley fairness properties (Wang et al. 2020) (see Appendix B.I on reasoning chains, with four criteria: (i) Sequential Efficiency: the final reward must be fully distributed across all reasoning steps, without under- or over-estimation; (ii) Sequential Symmetry: if two reasoning steps make equal contributions across all chains, then those two steps must receive equal credit. (iii) Sequential Additivity: when two reasoning chains are merged into a new chain, the credit of the merged chain must equal the sum of the credits of the originals; and (iv) Sequential Null Step: any step that does not affect the final reward must receive zero credit.
Theorem 1 (Fairness of SSV Credit Assignment). Given an Insertion MDP (Q,W S,A.P,R R),, let Øssv : > IR denote the credit assignment over all reasoning steps E N, where each éssv (n) is assigned credit to reasoning step n (computed according to Eq. 2 If Øssv satisfies the following four criteria, then Øssv constitutes a fair credit assignment:
1.1 Complete Credit Across Reasoning Chains (Sequential Efficiency). The final reward is fully allocated across reasoning steps:
EVEN (n)) Bo,x [R( - R R.
1.2 Equitable Credit for Equivalent Steps in All Chains (Sequential Symmetry). If two reasoning steps make equal contributions across all possible chains, they must be assigned same credit:
dssv (i) = dssv(t), Vi,j N.
1.3 Loss-Fred Credit Preservation in Chain Combination (Sequential Additivity). Credit is pre- served under the chain combination. For independent chains with disjoint step sets Nr and rewards R1, R2, N Ni U.2 and R = R1 + R2. Then
5

---

Published as a conference paper at ICLR 2026
EneN @ (n)v(n) =mEN, (ssv(n) EnENs (ssv
Rs
1.4 Zero Credit to Null or Redundant Steps (Sequential Null Step). Any step that never changes the final reward receives zero credit:
Øssv (n) O, if, =0 v(s,ar).
The complete proofs of Theorems 1.1-1.4 appear in Appendix D.I
2.4 FAIR ADVANTAGE ESTIMATION FOR EFFICIENT POLICY OPTIMIZATION
To leverage SSV fairesss as an Advantage for RL training of LLMs, SSV must serve as a value baseline that reduces variance without introducing bias, thereby facilitating convergence in policy optimization, such as Proximal policy optimization (PPO) (Schulman et al. 2017: Ouyang et al., 2022) (refer to preliminary Appendix B.2). Since SSV can be a step-level fair baseline, which invariant to the current reasoning chain and insertion position, we define the step-level advantage function as
A,SV = (se, - ussv(ne):
where SMC,, (st, xz1) is defined in Eq. [ and (iss((n)) is defined in Eq. 2 value baseline for advantage estimation, is independent of the insertion position and preserves the total credit across steps. This baseline guarantees that the direction of strategy optimization does not deviate, while reducing variance and guiding policy updates in accordance with the fairness principle (Theorem 1). Therefore, the RL training objective of SSV policy optimization (SSVPO) follows SSV advantage defined in Eq. 3 as the new value baseline while preserving PPO's clipping- based stability.
Note that SSV, as new
(O) min min(rr(0) (0) A 1lip(n(0,,-...))) A
A35V} DKL((TO( s1, Q) mer(: st, 2))
(n,, .4) is the insertion action, r0) = no(a; 8c: ②)
Toaa (a: st. Q) is importance ratio that reweights samples gathered under the behavior policy Taas to estimate the objective under the updated policy Te. Q denotes the prompt, Tref is a fixed reference policy, > is the clipping parameter, and B 2 controls the KL strength.
where 02
To justify SSV as new value baseline capable of inducing effective advantages, we provide two intuitive examples in Appendix(C] illustrating how SSVPO (i) handles useless steps and (ii) assigns credit to trajectories whose final answers are incorrect but contain partially correct reasoning steps. And we show SSVPO is both compatible with insertion-action optimization and variance- weriance-ffficien. First, the SSV baseline is independent of the sampled insertion action, replacing insertion- action advantage with A$SV leaves the expected policy gradient unchanged. Second, among all baselines that keep the policy gradient unbiased and that depend only on the reasoning step in a given math prompt Q and not on the insertion action, the SSV bascline uniquely minimizes the variance of the update. In practice, this reduces the frequency of PPO clipping and the resulting clipping bias, thereby improving policy optimization stability and sample efficiency. The following theorem formalizes that our advantage construction is unbiased and variance-minimal for policy optimization.
Theorem 2 (Unbiased and Variance-Minimal Advantage with SSV). Given an Insertion MDP (Q.N&S,A,7 R), let A$SV (Eq. 3) be an advantage function. Then, satisfied with:
2.1 Unbiasedness Dection-Independern the expected policy gradient unchanged
For any policy ma. replacing advantage by ASSV leaves
E V e log To (a, st) ASSV| E Va log na (a. SMC.).
2.2 Variance Minimality (Optimal Baseline). Among all value baselines b(n) that do not depend on the insertion position,
Øssv(n) arg min Vo log (o(a) (SMC, b(m:))]

---

Published as a conference paper at ICLR 2026
2.3 Convergence for Policy Optimization. Under standard policy- gradiett conditions (bounded wards (Sutton et al. 1999), pequurr--ntegrala scores (Borkar & Borkar 2008). Wobbins-Monn stepsizes (Robbins & Monro 1951)). using A,SV gives an unbiased, variance-minimal gradient
(Greensmith et al. 2004; homas & Brunskill 2017) estimator baselines, so stochastic gradient
ascent converges to stationary point of (ssvro(0) in Eq.
The complete proofs of Theorems 2.1-2.3 appear in Appendix D.2
EXPERIMENTAL SETUP
Base Models. We selected two widely adopted open-source pre-trained language models vanilla method, which are commonly fine-tuned for mathematical reasoning tasks: Qwen3-4B (Yang et al.) 2025) and DeepSeek-RI I-Distill-Qwee--..t (Guo et al., 2025a). Qwen3-4B (with thinking mode) is the latest generation of pretrained language models under 7B parameters for best reasoning perfor- mance according the technical report of Qwen3 Yang et al. 2025). while DeepSeck-RI-DE -I--ssiill--l Qwen-1.5B is even more lightweight parameters, yet still achieves competitive performance through knowledge distillation from much larger pretrained models (Guo et al. (025a))
Baselines. We compare our method against credit-assignment variants. Among the autcome-based RI approaches, we include RIOO (Ah- madian et al. 2024), GRPO (Shao et al. 2024), and DAPO (Yu et al., [025]_ For credit assign- ment-based RL approaches, we include VinePPO (Kazemnejad et al., (025) and SPO (Guo et al.) 2025b), both of which were released very recently. All baselines on-policy RL, and we further discuss how SSVPO can extend to off-policy settings in Appendix H..1
the-hhtattt art outcome- mused RL approaches and their
Datasets and Benchmarks. We train and evaluate SSVPO and all baselines on seven mathemat- ical reasoning datasets. For training, we use: (1) GSM8K (Cobbe et al. 202T), which contains 7,500 grade-school math problems reguiring multi-step CoT reasoning and widely used in RL for LLMs; (2) MATH (Hendrycks et al. (202T) contains 7,500 challenging problems spanning algebra, geometry, number theory, and other advanced topics, many of which require 10- 20 reasoning steps, which often evaluate on a representative subset, MATH-500 (Lightman et al. 2023 ; (3) MultiArith (Rae al. 2021), with 420 arithmetic word problems; and (4) SVAMP (Patel al. 2021), with 700 problems testing shallow reasoning. To assess generalization, we also evaluate on three competition- style benchmarks: AMC'23 (Mathematical Association of America, 2023). A1ME'24 (Mathemati cal Association of America 2024), and AIME 25 (Mathematical Association of America [2025), which feature more challenging integer-answer problems requiring long CoT and advanced reason- ing. Since no training data are available for these competitions, they are used only for testing in zero-shot setting. We also discuss extending SSVPO to zero-RL scenarios in Appendix H.2.
Training Details and Source Code. Across all experiments, set the training batch size match the generation batch. The learning rate is fixed at 5 x 10-7, and we use ADAM (Kingma 2014) as the optimizer. Full training configurations provided in Appendix G We also release open-source code with instructions and scripts in an Anonymous GitHub repository, detailed in Appendix
Evaluation Metrics. We evaluate SSVPO and all basclines along two dimensions: reasoning accu- racy (performance of the finetuned base model) and reasoning efficiency (tokens required to reach a correct answer). We further assess performance under constrained token budgets to analyze the curacy and efficiency trade-off. In addition, we evaluate the quality of fair credit assignment through case studies. We evaluate each experiment three times and report the mean performance. To ensure a fair and consistent comparison with baseline methods, our experiments fix the total computation budget for each reasoning benchmark by setting maximum number of samples for all algorithms throughout the entire training process, detailed in Appendix G.1
4 RESULTS
4.1 PERFORMANCE IN OUTCOME-BASED RL TRAINING
In this set of experiments, we evaluate SSVPO by comparing its reasoning performance with other mutcome-based RL approaches, including RLOO, GRPO, and DAPO. We focus on reasoning ac- curacy, which is defined generating the correct final answer to a given math prompt, while also

---

Published as a conference paper at ICLR 2026
considering the token usage required to produce the solution (answer). All methods are based on the same base model, Qwen3-4B, with the vanilla method also included as a base model. Results are summarized in [
Accuracy. We first examine the overall performance of SSVPO across seven mathematical reason- ing benchmarks. Although each benchmark differs in question difficulty, average reasoning length, and topic coverage, SSVPO achieves state-of-the-ar art performance on six of them, ranking second only RLOO on SVAMP. On challenging benchmarks, such as MATH-500, AIME 2024, and AIME 2025, SSVPO outperforms the previous SOTA algorithm by a large margin (3%-5% abso- lute accuracy). Averaged across all seven benchmarks, SSVPO clearly leads, improving over the vanilla method by 4.6% and over the previous SOTA RLOO by 1.8%. Further accuracy analysis of the case study results provided in Appendix L.1
Efficiency. We next evaluate reasoning efficiency, measured by the number of tokens required to generate a correct answer. As shown in Table each benchmark reports the average token usage per question. Among the seven benchmarks, SSVPO achieves the best efficiency on five, with only two (MultiArith and SVAMP) slightly behind RLOO. The most significant gain observed on GSM8K, where SSVPO reduces token usage to 1,320 per reasoning chain, compared to 2,215 for the base model and 1,665 for RLOO. On average, SSVPO delivers the best efficiency, reducing token usage by 18.1% relative to the baseline and outperforming all existing methods aimed at improving the reasoning potential of pretrained L.MMs. Further efficiency analysis of the case study results is provided in Appendix 1.2
GSMBK
MATH-500
AIME24
AIME25
0.6
0.6
84
0.4
0.9
0.3
50.P0
D2
0.1
0.1
6:0
8DC
1000
1K
2K
3K
68
5K
10K
Todour Liange
Figure 3: Reasoning accuracy comparison between SSVPO and Qwen3-4B (vanilla) under varying reasoning length constraints (measured by token usage) on four benchmarks: GSM8K, MATH- 500, AIME 2024, and AIME 2025. SSVPO consistently achieves higher accuracy across all length constraints. Higher curves indicate better performance a given token budget.
(e) GSMBK
(b) MATH500
(c) AIME24
(d) AIME2S
Ovenn Doen Pertmanee
6.2
2v0gk 36
Pargk
15
Persg:
36
tangk
26
Figure 4: Reasoning accuracy with K queries (success if at least answer is correct), reported as Pass@ K. Bars show Pass through Pass @ 16 under the same token cap. Higher bars indicate the performance gains achieved by SSVPO.
Reasoning with constrained token budget. To further examine reasoning performance under lim- ited token resources, we constrain the token budget: the fine-tuning methods must produce a correct
Table 1: Performance comparison between SVPPO and nutcome-based RL. approaches for mathe- matical reasoning across seven benchmarks. "Ace" denotes accuracy, and "Tokens" denotes token count (token budget fixed at 16144 tokens per sample). The best performances are shown in bold, while suboptimal results are marked with underlines. In the last column, red indicates the average improvement over the base model (vanilla) across all benchmarks, whereas blue indicates a decrease.
Method
MultiArith Tokess
SVAMP Tokens
GSMBK
MATH-500 Token:
AMC 2023
AIME 2024
AIME 2025
Overall
Aze
Tokess
Acc
Ape
Trkkna
1632
2215
91.2
5167
12205
82.1-9.0
-0%
100.0
984
94.8
1529
94.6
1665
92.2
4966
94.1
63.3
11573
12159
34.9
-2.
5650
10.225
GRPO
99.8
1012
94.2
1615
94.5
2117
92.4
5058
91.6
6634
61.1
12125
300
12668
85.4
5894
DAPO
99.5
1022
93.8
1642
94.8
2312
91.8
6742
91.5
94054
62.2
14445
47.3
12920
83.0
6828
SSVPO
100.0
096
94.6
1585
95.2
1320
95.0
4185
95.0
6436
66.5
10168
61.1
11559
N6.7
5178.
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison between SVPPO and other credit assignment algorithms on the DeepSeek backbone for mathematical reasoning.
Mothod
MulüArith Auc
Tokens
Adc
SVAMP
Tokens
Acc
GSMBK
Tokens
Acc
MATH--50
Tokens
Acc
AMC 2023
Tokens
Auc
AIME 2024
Tokens
Aec
AIME 2025
Tokems
Acc
Overall
Tokens
Vanilla
92.1
324
74.0
4399
76.1
1617
09.0
6018
52.2
8819
23.3
13702
13.3
14450
57.1
6481
VinePPO
95.8
307
14.5
429
85.4
1462
67.5
8413
23.3
12518
20.0
12477
68.7.60
5750
SPQ
94.3
311
75.0
7137
17679
20.0
11420
5402 -10050
SSVPO
95.5
298
76.6
8059
20.8
13377
13035
433.6
5838
55.9%
Reward
Mr-GSM8K: Question-ID-2e4
Ground Truth
SSVPO
0.5
GRPO
Step
We are tioning
we let :0.
We are given sivven...40000 $400,000
Substituting this vatue 240000
Step 1-2 are cornect
Step 3-4-5 are wrong
Step 6-7-8-9 are correct
Stop 10-11-12-19-14-15 are wrong
Figure 5: Credit assignment results reasoning sequence that fails solve the math problem. Among 15 reasoning steps, steps 1,2,6-9 are correct, while steps 3-5, 10-11 are wrong. Compared to GRPO, SSVPO provides estimates that are closer the ground truth on all reasoning steps.
answer using fewer tokens than a specified cap. The results are shown in Figure We observe that SSVPO achieves steady accuracy gains across all token budgets in all tested benchmarks in- cluding GSM8K, MATH-500, AIME24, and AIME25. Even under severe constraints (40%-60% of the tokens typically required), SSVPO still outperforms its base model (vanilla) by significant margin (10%-30%). This result is consistent with SSVPO's reasoning path, which emphasizes key reasoning steps reach the answer efficiently when tokens are limited.
Reasoning with multiple trials. Another statistical analysis we conduct to more comprehensively evaluate reasoning capability is Pass@1 K, which measures the probability that model generates at least one correct answer over multiple trials. We evaluate this metric on GSM8K, MATH-500, AIME 2024, and AIME 2025, with results shown histograms in Figure 4 As K increases from to 16, reasoning accuracy improves for both the base model and SSVPO. Importantly, SSVPO shows con- sistent gains over the base model (vanilla), with particularly strong improvements on GSM8K and MATH-500, where it achieves over 10% average accuracy increase compared to Qwen3-4B. This experiment demonstrates that while SSVPO emphasizes key reasoning steps for accuracy and effi- ciency, it also retains the capacity to organize more diverse reasoning chains, suggesting substantial room for further improvement when given additional time and resources.
4.2 EFFECTIVENESS IN CREDIT ASSIGNMENT
In the second part of the experiment, we replace the base model Qwen3-4B with DeepSeek-RI. Distill- Souill-Qwen-I.5B, a lighter-weight backbone specialized for reasoning tasks with a stronger fo- cus credit assignment, aligning with existing work (Dang & Ngo 2025 Chen et al. 2025 Liu et al. 2025b Ren et al., 2025). Based on the new backbone model, we aim to (1) discover the accuracy efficiency trade-off in the reasoning process by comparing SSVPO with other credit- assignment variants, including VinePPO and SPO, and (2) visualize how credit assignment operates the fairness during reasoning and how SSVPO discovers better strategies compared to the baseline.
Accuracy and efficiency trade-off. Results in Table that existing credit assignment meth- ods achieve lower token usage than SSVPO (e.g., SPO on AMC 2023 and AIME 2025, VinePPO on AIME 2024). However, these efficiency gains often come at the cost of reduced accuracy per- formance, as they skip some reasoning steps that are crucial for reaching the correct answer. For instance, on AMC 2023, SPO uses fewer tokens than SSVPO, but accuracy drops from 79.6 to 77.5. observed AIME 2025, where SPO achieves 20.0 accuracy with 12,420
A similar pattern tokens, while SSVPO achieves 24.4 with only slightly more tokens (13,035). Another observation is on AIME 2024, SPO reaches 19.3 with 12,679 tokens, whereas SSVPO achieves 28.8 with 13,377

---

Published as a conference paper at ICLR 2026
tokens, showing that small increase in tokens yields substantial performance gain. Overall, these results suggest that efficiency gains often come at the expense of accuracy, particularly on com- plex reasoning tasks. SSVPO, however, balances this trade-off more adaptively: easier tasks, achieves both higher accuracy and lower token usage, while on more challenging tasks, it main- tains accuracy with only minimal additional token usage. Further accuracy and efficiency trade-off analysis of the case study results is provided in Appendix 1.3
Fairness through step-wise alignment to ground-truth reward. SSVPO incentivizes fair credit assignment in long reasoning chain. Here, present fairness property, complementing our theoretical analysis (Sed2.3). As shown in Figure we sample question ID 248 from Mr-GSM8K (Zeng al. 2023 We let SSVPO and GRPO generate reasoning chains and estimate the contribution each reasoning step. Unluckily, both SSVPO and GRPÒ arrive at incorrect final answers. However, their treatment of this failure differs. GRPO treats all steps in the reasoning chain as equally useless, assigning each a reward of O. By contrast, SSVPO distinguishes among steps: even though they do not ultimately lead to the correct answer, some steps are credited for making meaningful progress in the reasoning chain. Such steps may not solve this particular problem, but could be crucial solving others.
study to illustrate the empirical
To validate this, we overlay human- labeled ground-truth reward curve (e.g.. process reward mod- els) on the reasoning chain produced by SSVPO. Unsurprisingly, SSVPO's credit assignment closely aligns with this ground truth: steps that are logically correct still receive positive credit, regardless of the final outcome, especially in cases where the final answer is wrong. similar effect occurs when the final answer is correct but some intermediate steps are flawed: SSVPO identifies these and assigns them lower credit. Thus, SSVPO enhances LLMs reasoning by ensuring that each step is rewarded fairly by assigning more credit to important steps and penalizing misleading ones. This strategy ultimately leads to higher accuracy and greater efficiency in mathematical reasoning and beyond.
4.3 ABLATION STUDY
To evaluate SSVPO's generalization across model scales and task difficulties, we also train Qwen3- 1.7B (Yang et al. 2025), with full results in Appendix E We further study the effect of the SSVPO- specific hyperparameter reorder_num using 2-, 3-, and 4-step reordering configurations, report- ing accuracy (Acc) and response length (Len) GSM8K. MATH-500, AMC23, AIME24, and ALME25. As shown in Table increasing the number of reordered steps consistently improves accu- harder benchmarks-Bae but also increases generation cost. Gains range from 1.3 points on GSM8K to 5.0 on AMC23 and 3.3 on AIME24/25 (7-8% relative). Thus, deeper reorder- ing yields modest benefits on everyday reasoning but substantial improvements competition-level
racy-paatee pyytpaaicullarly
tasks. Additional training dynamics appear in Table 5
Table 3: Performance comparison of different update steps on mathematical reasoning benchmarks.
GSM8K
MATH-500
AMC 2023
AIME
2024
AIME 2025
Method
Acc
Len
Acc
Len
Acc
Lens
Acc
Len
Acc
Len
Steps
90.5
1492
79.4
3909
77.5
7084
43.3
11088
40.0
12986
Steps
91.3
1617
80.0
3996
80.0
8109
43.3
13057
43.3
13424
4 Steps
91.8
1692
82.0
4166
82.5
8766
46.6
13603
43.3
13742
CONCLUDING REMARKS
This work introduced SSVPO, a sequential Shapley value-based credit assignment framework for enhancing post-training LLMss reasoning. We provided theoretical proof showing that SSVPO ensures fair step-level credit allocation within reasoning chains and establishes variance-efficient value baseline for policy optimization, enabling shorter and more effective reasoning processes. Em- pirical evaluations across diverse math reasoning benchmarks demonstrate that SSVPO consistently outperforms existing RL training methods in terms of accuracy, token efficiency, and reasoning qual- ity. Our findings highlight the importance fair credit assignment as a guiding principle for future RI. approaches in L.Mss reasoning.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken extensive measures to ensure reproducibility. The detailed related work on RL for LLMs post-training and MARL is summarized in Appendix A, and the necessary preliminaries for SSVPO (including Shapley value axioms and advantage functions) are introduced in Appendix B. Intuitive examples illustrating SSVPO credit assignment retical foundations, including formal claims and complete proofs of Theorems and 2, are presented in Appendix D. Additional experimental results on Qwen3-1.7B and training dynamics analysis reported in Appendix E. Implementation details such as the full algorithm, pseudocode, and anony- mous source code are provided in Appendix F, with the repository including all scripts for training and evaluation to be made publicly available upon publication. Appendix G reports full experimen- tal settings, including random seeds, hyperparameters, computational infrastructure (hardware, op- erating system, and software libraries), and the partition strategy for reasoning chains. Discussions on SSVPO applicability to zero-RL and off-policy settings are provided Appendix H. Finally, we empirically validate all theoretical claims by reporting reasoning accuracy and efficiency, with extended result analysis and additional case studies given in Appendix I.
provided in Appendix C, while theo-
ETHICS STATEMENT
We propose SSVPO, method that introduces fair credit assignment mechanism enhance the efficiency of LLMs reasoning. All experiments are conducted on open-source LLMs and publicly available mathematical reasoning datasets, which contain no personally identifiable or sensitive formation. No new data involving human subjects were collected, and thus, no privacy concerns arise. We therefore do not anticipate any negative ethical impacts from this work.
ACKNOWLEDGEMENT
This study is supported by Australian Research Council (ARC) Projects DE220100265, DP250103612, and LP240200636.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in Ilms. arXiv preprint S4V::2200.1174400 2024.
Vivek S Borkar and Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 100. Springer, 2008
Jacopo Castellini, Sam Devlin, Frans A Oliehoek, and Rahul Savani. Difference rewards policy gradients. Neural Computing and Applications, 37119)1 2025.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving rl ri-like reasoning models. arXiv preprint arXiv:2503.cm 045488 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprini urXiv:2507.06261. 2025.
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.07668 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ims: What works and what doesn't. tt. arXiv preprint arXiv:2503. /6219, 2025.

---

Published as a conference paper at ICLR 2026
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang. Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint wrXi:2557.198499 2025.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 10978, 2025.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Ali Forootani. survey on mathematical reasoning and optimization with large language models. arXiv preprini arXiv:2503. 17726, 2025.
Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115, 2024.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learning Research, 5:1471 2004.
Daya Guo, Yang Dejian, Zhang Haowei, Song Junxiao, Wang Peiyi, Zhu Qihao, et al. Deepseek- rl incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633-66 2025a.
Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Ef- fective segment-level credit assignment in for large language models. arXiv preprint arXiv:2505.23564, 2025b.
Songyang Han, He Wang, Sanbao Su, Yuanyuan Shi, and Fei Miao. Stable and efficient shapley value-based reward reallocation for multi-agent reinforcement learning of autonomous vehicles. In 2022 International Conference on Robotics and Automation, pp. 8765-8771. IEEE, 2022.
Yifu Han and Geo Zhang. Reinforcement learning fine-tuning of language model for instruction following and math reasoning. arXiv preprint arXiv:2506.21566, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In The Conference and Workshop on Neural Information Processing Systems, 2021.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rihf framework. arXiv preprini 1rXiv:2405.11143, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in rl training of 1lms. The International Conference on Machine Learning, 2025.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Huihan Li, You Chen, Siyuan Wang, Yixin He, Ninareh Mehrabi, Rahul Gupta, and Xiang Ren. Diagnosing memorization in chain-of-thouuhh reasoning, one token time. arXiv preprint arXiv:2508.00000 2025a.
Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 934-942, 2021.
Yugu Li, Zehong Cao, Jianglin Qiao, and Siyi Hu. Nucleolus credit assignment for effective coali- tions in multi-agent reinforcement learning. arXiv preprint arXiv:2503.00 arxiv:2503.00337, 2025b.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
12

---

Published as a conference paper at ICLR 2026
Licheng Liu, Zihan Wang. Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. A simple" try again" can elicit multi-turn Ilm reasoning. arXiv preprint rrXiv:2507.14295 2025a.
Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, and Lu Hou. Quantization hurts reasoning? empirical study on quantized reasoning models. arXiv preprint arXiv:2504. 0442233444, 2025b.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781 2025.
Mathematical Association of America. American mathematics competitions (amc), 2023.
Mathematical Association of America. American invitational mathematics examination (aime), 2024.
Mathematical Association of America. American invitational mathematics examination (aime), 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. 35:27730-27744, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2090-2094. Association for Computational Linguistics, 2021.
Avinash Patil and Aryan Jadon. Advancing reasoning in large language models: Promising methods and approaches. arXiv preprint arXiv:2502.03667, 2025.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint ar.iv:2 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
Qingyu Ren, Qianyu He, Bowei Zhang, Jie Zeng, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. Beyond the trade-off: Self-supervised reinforcement learning for reasoning models' instruction following. arXiv preprint arXiv:2508.021500 2025.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pp. 400-407, 1951.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint a:Xiv:2402.03300, 2024.
Lloyd S Shapley et al. A value for n-person games. Princeton University Press Princeton, 1953.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning. pp. 5976-5985. PMLR, 2019.
Yang Sui, Yu-Neng Chuang. Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
13

---

Published as a conference paper at ICLR 2026
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech M. Czamecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- decomposition networks for cooperative multi-agent learning, In The 17th International Confer- ence on Autonomous Agents and Multiagent Systems, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
Hongze Tan and Jianfei Pan. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508. 04349, 2025.
Philip S Thomas and Emma Brunskill. Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines. arXiv preprint arXiv: (rrXiv:7706.00 776.06643, 2017.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang. Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-baat based feedback. arXiv preprint arXiv:2211.14273 2022.
Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7285-7292, 2020.
Jianhong Wang, Yuan Zhang. Yunjie Gu, and Tae-Kyun Kim. Shaq: Incorporating shapley value theory into multi-agent q-learning. Advances in Neural Information Processing Systems, 35: 5941-5954, 2022a.
Tianduo Wang and Wei Lu.. Learning multi-step reasoning by solving arithmetic tasks. arXiv preprint urXiv:2306.01707 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, celf-connistency improves chain of thought reasoning in language models. arXiv preprint (tXXi::2233.1172 2022b.
Yibo Wang, Li Shen, Huanjin Yao, Tiansheng Huang, Rui Liu, Naiqiang Tan, Jiaxing Huang, Kai Zhang, and Dacheng Tao. RI-compress: Long chain-of-thought compression via chunk compres- sion and search. arXiv preprint arXiv:2505.1688: 2025.
Qinglai Wei, Yugu Li, Jie Zhang, and Fei- Yue Wang. Vgn: Value decomposition with graph attention networks for multiagent reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 35(1):182-199 2022.
Mingqi Wu. Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. Reasoning or memorization? unreliable results reinforce- ment learning due to data contamination. arXiv preprint arXiv:2507.10555 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. Formal mathematical reasoning: A new frontier in ai. arXiv preprint arXiv:2412.16075. 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36:1 1109-1 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
14

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14477 2025.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05663, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 35:15476-15488, 2022.
Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. Mr-gsmök: A meta- reasoning benchmark for large language model evaluation. CoRR, abs/2312.17080, 2023, doi: 10:4550/ARXX12331.1700..
Jia Zou, Xiaokai Zhang, Yiming He, Na Zhu, and Tuo Leng. Fgeo-drl: Deductive reasoning for geometric problems through deep reinforcement learning. Symmetry, 16(4):437, 2024.
15

---

Published as a conference paper at ICLR 2026
DECLARATION OF LLMS USAGE
The development of SSVPO did not rely on LLMs as essential, original, or non-standard components of the research. GPT-5 was used solely to assist with proofreading tasks, including polishing text, correcting grammar and spelling, and ensuring consistency in American English usage.
A RELATED WORKS
Butcome-Based RL Training of LLMs Post-training for LLMs is shifting from supervised fine- tuning toward RL with verifiable rewards. Among these approaches, outcome-basee RL has become central to enhancing LLMs reasoning capabilities. By rewarding only the final outcome (e.g., the correct answer to a math prompt), it encourages models to implicitly connect intermediate reasoning steps, thereby mimicking the human CoT process. Representative methods include Reward Leave- One-Out (RLOO) (Ahmadian et al.) 2024), which introduces leave-one-out baselines for improved trajectory-level variance reduction, and Group Relative Policy Optimization (GRPO) (Shao et al.) 2024). which reduces variance and instability by sampling multiple trajectories relative to token- level PPO baselines. More recently. Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) YYu et al. 2025) extends GRPO with overlong reward shaping. improving stability with- relying KI. regularization. While these outcome-based RL methods improve reasoning by leveraging final rewards, they still face optimization challenges. Most notably, the difficulty of de- termining each step's 'ss true contribution within reasoning chain. Accurate estimation of step-level contributions is crucial for maintaining efficiency, as it allows redundant or even detrimental steps to be pruned, while preserving those that meaningfully contribute to the final answer.
Credit Assignment in RL Training of LLMs To improve LLMs' reasoning efficiency, recent works have introduced credit assignment strategies into outcome- RL (Kazemnejad et al. Guo et al. 2025b; Tan & Pan 2025). These methods explicitly model the contribution of each reasoning step by decomposing the final reward signal of the entire CoT into step-level credits. Specifically, VinePPO (Kazemnejad al. 2025) defines intermediate steps and generates multiple
2025
continuations (vine rollouts), using Monte Carlo returns to estimate per-step advantages. Segment Policy Optimization (SPO) (Guo et al. 2025b) introduces probabilistic masking framework to derive advantage estimators based on value differences between adjacent reasoning steps. More recently, Group Token Policy Optimization (GTPO) (Tan Pan 2025) applies dynamic entropy weighting, reweighting returns within correct trajectories to reduce variance and stabilize training.
Although these credit assignment methods outperform purely outeome-based RL approaches, they still fall short of fairly and accurately estimating each step's true contribution. For example, VinePPO aggregates Monte Carlo returns, which tends to overweight early steps in a reasoning chain while under-crediting steps closer to the terminal state, thereby misrepresenting their marginal value. SPO provides more reliable credit when reasoning chains are fully correct, but fails in par- tially correct cases, when early step is logically valid but later steps lead answer, its contribution is ignored. GTPO, by contrast, uses entropy as heuristic, which can over- credit lexically diverse but unimportant steps while under-crediting crucial low-entropy ones. Such misalignments underscore the need for a step-level credit assignment method with theoretical guar- antees of fairness and provable stable variance in policy optimization.
incorrect final
We also note that recent studies, such as Serial-GRPO (Dai et al., (025) and Unary Feedback as Observation (UFO) (Liu al., 2025a), extend the single-turn paradigm to multi-turn setting in order to capture intermediate reasoning steps. However, they still rely on additional policies or feedback improve performance, indicating they cannot be regarded as pure RL methods with credit assignment driven solely by outcome-basse signals. More recently, LLM agent work has explored leeel fine-grained credit assignment, including Agentic Reinforced Policy Optimiza- tion (ARPO) (Dong et al., 2025) and Group-in- iaap-----rruppp Policy Optimization (GiGPO) (Feng et al.) 2025). These methods refine trajectory-level returns into step-level credit in agent-based environ- ments, thereby improving the scheduling of multi-turn LLM tool use. Specifically, they leverage explicit environment states and interaction signals (e.g., retrieval success, subgoal completion) and derive step-level credit through anchored states, within within-group comparison, or group-in- group aggre- gation. This enables higher success rates and better sample efficiency in retrieval and web-operation tasks. However, such approaches assume that each step has an observable external success criterion
16

---

Published as a conference paper at ICLR 2026
(e.g., whether a tool invocation succeeds), which fundamentally differs from mathematical reason- ing tasks. In reasoning, intermediate steps rarely provide verifiable immediate feedback, making it difficult deliver consistent and accurate supervision at the step level. In contrast, our focus is on a single-problem, single-turn setting, where we derive step-level credit assignment purely from outcome-based signals.
Original Credit Assignment from MARL to LLMs The concept of credit assignment originates from multi-agent reinforcement learning (MARL) (Foerster et al.) 2018: Rashid al.) 2020 Li et al. 2025b), where the goal is to evaluate each agent's ' actual contribution to the team's 's perfor- mance. For example, the difference reward method (Castellimi et al. 2025) defines an agent-specific utility function by contrasting the team return with and without that agent, directly measuring its incremental contribution. The counterfactual advantage method (Foerster et al., 2018) employs a centralized critic that holds other agents fixed and marginalizes over the focal agent's is action to con- struct counterfactual baseline, attributing only the advantage beyond this baseline that agent. More recently, the Shapley-Q approach (Wang et al., 2022a) distributes the team reward according to each agent's expected marginal contribution across coalitions and orderings, aligning well with fairness principles central our work.
However, directly adapting this notion from MARL to LLMs reasoning is non-trivial. Original Shapley values do not capture the inherently temporal nature of reasoning processes such as CoT, where steps unfold sequentially and exhibit strong positional dependencies. Inspired by insertion- based text generation [Stern al. (2019) and reasoning path search methods (Wang al., 2022b Yao et al. 2023). we assume each reasoning step can be an "agent" and insert those steps into the position to form a reasoning chain, which can be "action." This formulation allows the design of a new Shapley value tailored for LLMs training, while preserving the axiomatic fairness properties of the original Shapley value in MARL, when computing the expected marginal contribution of each reasoning step in CoTs.
B PRELIMINARY
B.1 SHAPLEY VALUE AND ITS AXIOMS
We recall the Shapley value from cooperative game theory (Shapley et al. 1953), widely used as fairness reference for credit assignment in MARL (Wang et al. (202a).. At a high level, the Shapley value measures how much extra value each participant contributes average across all cooperative contexts. Imagine forming a coalition by letting players join one at a time in a uniformly random order; whenever a player joins, we record the incremental payoff (marginal contribution) created by their participation. A player's Shapley value is the average of this marginal contribution over all possible arrival orders. Its appeal as a fairness rule stems from the fact that it uniquely satisfies four natural axioms: efficiency, symmetry. additivity, and the dummy (null player) property. Formally, let A be a finite player set and let 2 + be a (coalitional) set function that assigns value to every coalition S C. For i E N, the Shapley value is defined by the original subset-sum formula
|S| (IV) - [S] - 1)) (())
(.(v)
Σ
(5)
WW
SCNVO)
where the factorial weight depends only [S] and [W].
Shapley Value Axioms. value assignment +(v) = (v))ee) called Shapley it satisfies
the following axioms:
Axiom 1 (Efficiency). The split neither creates nor destroys value: the total credit equals the coali- tion's 's total value (relative to the empty set).
Σ TEN
(v(()) v(N) - v(0))
In particular, if v(2) = O, then EieN (x()) = v(N).
Axiom 2 (Symmetry). Players who are indistinguishable in their impact every coalition receive the same credit. If j.j e N satisfy v(S (1) = v(S (i)) for all C W (i.j) then (1(v) (r())
17

---

Published as a conference paper at ICLR 2026
Axiom 3 (Additivity). Credits add across games: for any v, 2N + R and every E .,, one has 61(v + w) = (x(v) + (i(w..
Axiom (Dummy (Null Player)). A player who never changes any coalition's 's value receives zero credit. If vS U {i)) v(S) for all S C N {i), then (: = 0.
Fairness of the Shapley value. By the classical characterization of Shapley et al. (1953), the Shapley value in Eq. Slis the unique credit assignment that satisfies the four axioms stated above: Efficiency (Axiom IT. Symmetry (Axiom 2 Additivity (AxiomB) and Dummy (Null Player) (Ax- iom D. Consequently, the Shapley value constitutes principled and fair rule for credit allocation, and adopt it fairness reference throughout.
B.2 ADVANTAGE FUNCTION IN RL TRAINING OF LLMS
We place the advantage function at the center of outcome- based RL fine-tuning for LLMs. Beyond serving as the signal that tilts probability mass toward better actions, a well-chosen baseline yields lower-variance, more stable updates. We first discover review policy gradients and the invariance to action-independent baselines, then introduce Monte Carlo (MC) and generalized advantage es- timation (GAE). We finally review PPO's clipped surrogate (with optional KL regularization) and assume SSV enters value baseline. Throughout, notation matches the Insertion MDP: denotes the before insertion chain and a, the insertion action (e.g., ag = (n4. )))).
Why advantage matters. Intuitively, the advantage function A, measures how much better (or worse) the sampled action is than the state's average. In long-horizon reasoning with final-reward LLMs settings, deriving the advantage value from a strong baseline can drastically reduce gradient variance, thereby stabilizing training and improving sample efficiency.
Policy gradient, advantage, and baseline invariance. For stochastic policy ne, the policy- gradient theorem gives
(() = E log (o(a, (8)) 47),
(6)
where the step-level advantage compares action- and state-values,
A7 Q" (st,at)
VT(ss)
Subtracting any baseline b(s;) that does not depend on the sampled action az preserves the expected policy gradient and typically lowers variance:
Vo log no (ax (t)) (47 b(ss)) = V log (a (st) A7].
(8)
Practically, b(-) is learned a value function (-) replaced by task-specific baseline (e.g.. SSV
in our method). The identity in Eq. follows from Ee-x((ls)] V log (a {)] =
Monte Carlo and GAE estimators. and define the (bootstrapped) return
Let R, be the scalar reward at time f, (0, 1] a discount,
T G; = Σ k Rs + -TT- Ve(ST) (V()) (if not terminal at T'). k=t
(9)
A basic estimator is A, G. -G-VVel (st)), which is low bias, high variance for The generalized advantage estimator (GAE) with X E [0, 1] trades bias and variance via temporal- difference (TD) residuals
reasoning.
d R + Vo Volss),
(10)
T-1
A GAE
- 6, + 7d AGAE
(11)
In the policy loss, A, (or AGAR is used with stop-gradient For numerical stability, it is standard to normalize advantages per batch to zero mean and unit variance.
18

---

Published as a conference paper at ICLR 2026
Practical birs-variance guidance. Monte Carlo (MC) is unbiased but may have very high vari- ance on long sequences; GAE with y = and [0.9. 0.97] is a robust default for LLMs training. Ensure reward and value scales match (value-loss weighting reward scaling help), and con- sider grouping normalization by prompt to avoid cross-task scale interference.
PPO clipped surrogate and clipping bias.
Let the importance ratio be
(0) = To (a(a 8t)
(12)
To (a: 3t)
and let be the clipping parameter. The PPO surrogate maximizes
(8) = E min (x(()) A., clip(r:(0), 1 1- c, c, 1+ A,
(13)
Clipping prevents excessively large updates and implicitly enforces a trust region, but introduces clipping bias: positive advantages are truncated more often than negative ones, nudging updates toward conservatism. Reducing the variance of Å, (e.g., through stronger baseline such as SSV) lowers clipping frequency and the induced bias, improving the sample efficiency.
KL-regularized variant and early stopping. A KL-penalized objective augments Eq. 13 with
3 DKI (. 8t) To (- |s))),
(14)
monitors DKL for early stopping when it exceeds threshold. An adaptive that targets desired per-update KL (e.g., per-token) is often effective, especially for text policies where distributional drift must be tightly controlled.
Plugging in SSV as value baseline. In Insertion MDP, a reasoning step's contribution depends on order and position. The SSV baseline (defined earlier) removes position-dependees noise while remaining action- ieiin-indeeeedent for position-only updates. Concretely, we substitute the baseline in the advantage with
ASSV (s.1,,)) Øssv (n+),
which preserves unbiased policy gradients and uniquely minimizes update variance among base- lines that depend only on (2,n), under standard conditions discussed earlier. In practice, AASVV reduces clipping frequency and clipping bias in Eq. [3 improving optimization stability and sample efficiency.
Implementation notes for Insertion MDP. If the policy factorizes as Te | s) (n
T 8) and only the position component is updated while T s) is held fixed, then (n)v(n) action-independent and safe use baseline. If both components are updated jointly, either (i) use separate baselines that are independent of each component's action, (ii) stage the updates that the iition-independdnne condition holds for the component being updated.
C INTUITIVE EXAMPLES OF SSVPO CREDIT ASSIGNMENT
C. EXAMPLE 1: DISTINGUISHINO USELESS STEPS
Consider four reasoning chains: A + correct, B + correct, A + B + correct, B + A + correct.
Let the reward be for correct answers. When optimising the trajectory A + B + correct, VinePPO obtains
(AM) 1,
(B) 1
The advantages for A and B are:
A MC (M, B) = A + 7 (B) MC (C < 0,
MC (B, terminal state) rinall + 7
state) PM = 0.
This means VinePPO cannot accurately detect useless steps: on the trajectory A - B - correct, both the essential step A (which already makes the answer correct) and the redundant step B (which
19

---

Published as a conference paper at ICLR 2026
does not change the outcome) receive the same zero advantage. The VinePPO estimator advantage does not distinguish between redundant steps and necessary steps, so useless step will have zero- advantage does not imply that VinePPO can reliably identify useless steps in practice.
In contrast, SSVPO yields:
SMCA(0,0) =
SMC O,
and
(M. (SV(A) = (S. SMC +
(B, = 3
SBV( = (SSMCB(0,0) SMC,( (4,,(4,0) =
The advantages become:
A SM,) - Øssv (4) =
A SMC Øssv (B) (B)= =0
Thus, SSVPO assigns negative advantage to the redundant step B in the trajectory that needs optimisation, while giving positive advantage to the useful step A. This illustrates why SSVPO provides more accurate step-level credit than VinePPO.
C.2 EXAMPLE POSITIVE ASSIGNMENT FOR USEFUL STEP IN AN INCORRECT TRAJECTORY
We give an intuitive example showing why SSVPO (which uses SMC) offers more accurate step- level credit assignment than VinePPO. Assume four possible reasoning paths for the same math problem: A - correct, + correct, A + B - correct, B + A - incorrect.
Let the reward be for correct answers. Consider optimizing the trajectory B A incorrect.
VinePPO value estimates:
MC 3
BC) =
VinePPO advantages:
(A) MC (B) 0, A MC -B
MMC (A, terminal state) = > V.MC (terminal state) PM (A) = -
Thus, VinePPO fails to assign positive credit to step B, even though contributes to correct rea- soning in other trajectories.
SSVPO (SMC-based) value estimates:
SMCA = 11
SMCn(200) = =1,
and the corresponding Sequential Shapley values are
SSV(A) = 3 (SM 0) + SMC,(B,1) + (B,) 3
SBV(B) = 3 (SMC&(0) + SMC3(A, 1) + SMC =
SSVPO (SMC-based) advantages become
ASS SMC 0) Øssv (B) 1 1=
SSV SMCA(B: 1) - Øssv (A) = - -1-
This shows that even if the final answer is wrong, SSVPO correctly identifies as useful step and assigns strong penalty to the emrnneos step A.
D THEORETICAL ANALYSIS
D. PROOF OF THEOREM 1
For an insertion MDP (Q,W,S,A,7 P. chain (Q, R1:)), an action (n, x)) with æ (1,..., t + 1}, and s' = Ins(s, n, x), the sequential marginal contribution (SMC) is
(M) R( nn)))) R(s).
20

---

Published as a conference paper at ICLR 2026
Let ((V) be all orderings of / and, for a E ((V), let pred, (n) be the ordered list of steps before n under d; set 8o := (Q. pred, The SSV step n is
Φssv(n) (n) = BeyG(N) (s1 (so.
(15)
Eq. 13 can be written as the following explicit finite sums.
|s/l+1
(1)v((1)
Σ
R (Inns(S,,,,,))) R((aa)
(16)
[G(W]
rEB(N)
Equivalently, let pos, (n) = |8~1 + be the position of in (one plus its predecessor length);
then
pas, (m)
(ssv(n)
Σ
R (TTn((8o, R(so))
(17)
(G)
REG(N) pos, (n)
(2) Sum grouped by predecessor length. For k E (O....,,WII 1}. let O&(N) {nt) be the set of all ordered k-tuples (orderings) drawn from N and for (o),..., ob) E Og write 8o (Q, 01, ....... ok). The probability that uniformly random permutation has as the exact ordered predecessor of (IVI
1//[W//
Hence, the full calculation of SSV can be rewritten as follows:
W (W
Øssv(n) (n)
k 1)!
Σ
R (I, R(so)) (18)
WI
oEOL (NV (M))))
D.1.1 PROOF OF THEOREM 1.1: COMPLETE CREDIT ACROSS REASONING CHAINS (SEQUENTIAL EFFICIENCY)
Lemma 1 (Complete Credit Across Reasoning Chains). Let W and II 6(N) be the set all orderings of N. Sample a = (oI...., .n) ~ UnI and, conditional on a, sample
X, Unif(1,..., independently for L ,,..., n. Define the insertion process
8 (),
of Ins Ins(61-11 oe. Xt) 1,=1,.....
Then
(oss(u)) uEN
Σ
Eor Xi.! ( -R(si)]
In particular, Rf()) = R(2, 0) then
Σ Øssv (u) F.,,, XI... [((*)]]
WEN
Proof. We proceed in four steps. Throughout, we assume R is integrable so that interchanging sums and expectations is justified.
Step Unfold the SSV definition. By Eq. 15 for each u E W and ordering a let T(u,a) {1. nZ be the unique index such that (r(u,)) u, and let
So (u) = (2, pred,(((()))
T((,,)-11
Then
Øssv (u) = E, Exe(I.... (w)))))) (s.
(19)
Step 2: One-step increment identity. Fix and t = T(u,o). Since Iso (u)l = t- and X, ~
[n..(1......
Ex (MC..(s Ex,
R[Ins(sp_1n; R(+211)
= Ex. R(s?) R(s
(20)
where the last equality uses the construction of = Ins( Xx) and the fact that dt-
21

---

Published as a conference paper at ICLR 2026
Step 3: Reindex by insertion time and telescope. Summing Eq. 19 over N and applying Eq. yields
Σ
ossv(u)
= Ess
Σ
E Ex((..)))
((()((m))) - R( ((0F((m))-))
WEN
uEN
E
Ex, (((s)- - -R(69-1)
E. Ex
Σ (R(s)) R(({-1))) t=1
Ea, Xim
R(0h) R(+))
where we used that (T(u, o): W/ = (I and the inner sum telescopes.
Step 4: Conclude the stated identities. This proves the main identity. If R(()) = 0, the special case follows immediately.
D.1.2 PROOF OF THEOREM 1.2: LOSs-FREE CREDIT PRESERVATION IN CHAIN COMBINATION (SEQUENTIAL ADDITIVITY)
Lemma (Loss-Free Credit Preservation in Chain Combination). Let R1, R2 : S + be reward functions and let a TR. Define the combined reward Re Rı RR2. Then for every N,,
@ssv (n) = a @ssv (n) + 3 Bssv (n). In particular, for a = 3 = we have R Ra ØssV
(n) = @ssv (n) + (n).
Proof. We establish that the SSV operator is linear in the reward.
Step I: Linearity of SMC in the reward. For any chain E S, admissible position æ E [l,...,|s|4 1}, and E N.
SMC n (s,,) = Re( Ins(s, n, x)) - Rg(s)
= a R1( Ins(s, n)) - R1(s) + 8 R2 Ins(s, n, - R2(s))
a SMC. (s, + B SMC (s,
(21)
This uses only the algebra of differences and does not depend on the dynamics.
Step Take expectations the SSV definition. Fix E N and write sa (2, (n)) By
Eq. 15
and Eq. 21
@ssv (n) = EOES(A)) EDEEL.... 1e SMOR (so, v)
E, Ez (a (so, + S (So, x)
=a E. E, SMCR (s), m)] + Bo E, SMC R2 (so: x²)
R2 = (n),
where linearity of expectation justifies moving outside the integrals.
Step 3: Conclude. The desired identities follow immediately, including the additivity case B 1.
D.1.3 PROOF OF THEOREM 1.3: EQUITABLE CREDIT ACROSS ALL POSSIBLE CHAINS (SEQUENTIAL SYMMETRY)
Lemma 3 (Equitable Credit Across All Possible Chains). Suppose i,j N are symmetric in the following sense: for every chain E S and every admissible position v, if swap is obtained from by swapping the labels <> j. then
R(s)
R(s and R( (ns((8, R( j,,)))
22

---

Published as a conference paper at ICLR 2026
Then @ssv (1) (ssv
Proof. Let i.j E N satisfy the symmetry condition of Lemma 3 We prove Φssv(i) = Øssv (i)
by constructing a weight-preserving bijection between i-contexts and j-contexts that preserves marginal contributions.
Step 1: Triple spaces and SSV regrouping. Define
T: = ()) S SWW (1), TE GES), 0 k S [$]]
T, {(T, TW (i), G(T), 0 [T]}
For (S, let (S, r) be the predecessor chain formed by ordering S according to 7 (suppressing Q in notation). For [i,j] define
SMC.(S, T, k) == R( Ins(s(S, u, k++)) R(s(S, r)),
where gaps are indexed by k E (O...., [S]] and Ins uses 1-based positions. Regrouping the SSV expectation over orderings and positions yields
w(S = (WW | 1)!
dssv (u)
Σ (S,T,A)ETE
w(S, k) (S,,(,
W
[S] +
so wSSkk) k) depends only |S| and the gap index k.
Step 2: Construct a bijection & T Tj. Given (S, k)) Tr define ((S,T, = (T, ar,am)) m)) as follows.
(a) Set mapping.
S T
j S, ((S) ())
j E S.
(b) Sequence and gap mapping. Let repl(r, j ++1 be the sequence obtained by replacing j with i while preserving the relative order of all other elements.
j $ S,
k
repl(T, j j-,,,, j E S,
Since swapping labels does not change the chain length nor the set of gaps, the gap index is pre- served.
Example. I (a, j, b, c), then (a, i, b, c) and the gap mapping is m = & for all k (O....,4). 4).
Step 3: Equal contributions under symmetry. By construction, s(T, a) is obtained from a(S, r) by swapping labels i ++ j while fixing all other labels and their order. By the lemma's symmetry condition on R (for the predecessor chain and for single-step insertions at the same gap),
R( Ins(s(S, i),,, ((s(S,()) r)) R(
R(s(T.o)).
with k. Hence (S,,(S, SMC,(T,
Step 4: Injectivity, surjectivity, and weight preservation. Define Φ-1 :T, - Te by symmetrically swapping ++ j: for (T, a, m). set
TT S=
T. TET,
T= [0, oo,
i T, repi(a, i+tj), i T,
k= m.
(T (1),
Then क 8- (S,,7,, and ( (T, m)) = (T, a, m), so is a bijection. Moreover, [T| [S] and = l imply w(T, m) k))
Step 5: Conclude by summation. Summing equal marginal contributions with preserved weights, Øssv (1)
Σ
w(S,k) SMC,(S,,,)
Σ
w(T,k) (MK))MC k) = Øssv (j)-
(S.T,K)ETT
(T,
23

---

Published as a conference paper at ICLR 2026
1.1.4 PROOF OF THEOREM 1.4: ZERO CREDIT TO NULL OR REDUNDANT STEPS (SEQUENTIAL DUMMY)
Lemma 4 (Zero Credit to Null or Redundant Steps), if E N satisfies (I,nsss, 1, x)) = R(s) for every s and every (I...../ss |s| (equivalently, (s, for all admissible (s, z)), then Øssv (i) 0.
Proof. Step 1: SSV as an expectation over i-contexts. For fixed i, define the context space
Ω ((,,x): G(V), x {1, (I,.... [s)( 1}},
where So (i) (2, pred (i)) is the predecessor chain of i under a. Let μt₃ be the product measure induced by Unif and, given a, x U (t) 1}. Define
[((, SMC, (s.((),
By the definition of SSV (Eq. we have
Φssv(i) a) x)].
Step Apply the null (redundant) condition. By hypothesis, for every admissible (s, x) one has SMC, = 0. In particular, for all (a, E Ω4
f( (s x) = 0.
Hence
(ssv(1) = Ex, = [p,[0] = 0,
which proves the claim.
D.2 PROOF OF THEOREM 2
Fix the insertion MDP N, 8, A, R). For n E N, chain = =Q, E S, and an insertion position (L...., 11.
SMC. R(Ins(s,,, - R(s).
Let ((N) be all orderings of and, foro E G(N), sct = (Q. pred, (n)). Define the Sequential Shapley Value (SSV)
(nsv(n)= == (y.-umi(((())) ExIp-Unii((,....//ll111: [SMC, x)],
8y = (2. pr)dyy()).
We also write Y, = SMC, (SE, X) and g = aQ n) (allowed information: prompt and step identity).
We prove the following three claims for the SSV advantage A,SV := SMCn, (st-1, 0z) Øssv (nt), under the assumptions that (i) insert transitions are deterministic, (ii) éssv (n,) depends only
(2, ne), is independent of It, and is used with stop-gradient, and (iii) its Monte Carlo estimate (when used) is computed from data independent of the rollout being updated.
D.2.1 PROOF OF THEOREM 2.1: SSV AS AN UNBIASED VALUE BASELINE )
Let the rollout be at state 8; = (Ω, n14-1) :) E S and sample at ~ To(- (t). For step E N,, let its realized marginal contribution this rollout be
SMC
SMC,(s roal ax real).
Let the baseline B, E {(sss(() (n), 2ssv. bosvvx(nn (n)) depend only on (Q, n), be independent of the current action, and be used with stop-gradient. Define A,, SMCreal B,, We aim to show
F Vo log Ta(at) | se) An] V log (a 8&)) SMCreal]
24

---

Published as a conference paper at ICLR 2026
Lemma 5 (Action iectioninddeeendence baseline = unbiased policy gradient). If baseline B is mea- surable w.r.t. G := o(2,)), n). independent of the current action, and used with stop-gradient, then E log no (az (8)) B] = 0; hence using A, = -MMC, B yields an unbiased policy-gradient estimator.
Proof. Step 1: Tower property (condition on (st, Ba)). With Z g = (st, Bn).
Vo Volog Ta(a | 8x) An and
Z = B 9|] E E Vo log (al 8t) An Se, B ]]
(22)
Step 2: Expand An and separate the baseline. Since A, = SMC B,,
E v log π An
St, Bn
I V log π SMC,
St, Bn|
16 V log Bn,
St. Bn
V log π SMC
8t, B.] - B, E V log ग
8r, B,)
(23)
where V log æ abbreviates V log no (at
82) and B,, is constant under the inner conditioning.
Step Use independence and the score-function identity. Independence broos-fitting/stop-eee implies
V log Se, Ba] = E V log ग 8t] = Σ (a 81) Vo log (a(a 8t)
(24)
aa(a
84) = Vol
(25)
Hence the baseline term Eq. 23 vanishes and
log क A, 8t, B,
E
V log π SMC,eal 8t, B,
(26)
Step 4: Remove conditioning via the tower rule. Substitute Eq. 26 into Eq. 22 to obtain
ViVo log Ta(ac 84) An]
E V log (a (x) SMC,
(77)
D.2.2 PROOF OF THEOREM 2.2: SSV A MARIANCE-MINIMAL VALUE BASELINE (OPTIMALITY)
Let G(N) be all orderings of M. For Σ ~ Uni)( set sy (Q, (n)d and draw X Σ Unif(1...... (ss] + 1}. Define the random variable
Yn SMC, X).
(28)
Let the allowed information be the alagebrra G == a Q n). Consider the class of legal baselines
B bb (Q. + R and is G-measurable }
(29)
We aim to minimize, over E B, the residual variance
v(b) Var(Y. n))
(30)
Lemma 6 (G-measurable baseline => variance is minimized if b dssv). If baseline b is G- measurable, then Var(Y, - b) E[[Var Yn G)] + Var(E[Y, G] - b). and thus the variance is minimized if and only if b(Q, n) EYY, G (n))
Proof. Step I: Law of total variance. Apply the law of total variance to Z = Y, b(2,m) with respect to G:
V(b) R[Var(2 z [ + Var E ())
Var Var (Yn b(Q,n) 9) + Var(E(X,b(Q,n)) -(2.n)
(31)
25

---

Published as a conference paper at ICLR 2026
Step 2: Use G-measurability of b. Since b( b(2,1) n) is G-measurable, it is constant under G-
conditioning, Hence
Var Yn b(2,)) ( = Var(Y, 19),
(32)
E Y,, b(Q, n) G] B[Y, | b( n).
(33)
Substitute Eq.
32 Eq. into Eq. 3T
V(b) = E Var(Y, 9]] Van g b(2, n))
(34)
independent
Step Pointwise minimization. For each realization of (2, n), the second term in Eq. 34 nonnegative variance which is minimized to O if and only if
b(2,n) Y |
[E,x(SMC,
2,n].
(35)
Therefore, the unique minimizer is
m), E|Y,, Q.n] (ss)
min V(b) = Var((,,, 19)].
(36)
BEB
This proves the variance-optimality of SSV within the allowed information class.
D.2.3 PROOF THEOREM 2.3: CONVERGENCE THE UNBIASED, MARIANCE-MINIMA SSV VALUE BASELINE (POLICY OPTIMIZATION)
Let (M,, Xx) be i..dd copies of (2, X) defined above and set
(ssv,
Σ SMC, (S5,, K k=1
8%,
(Q, predd,,
(37)
All draws are generated in an evaluation procedure independent of the rollout being updated (cross- fitting), and the estimate is used with stop-gradient.
Lemma 7 (Independent i.i.d. orderings = unbiased MC SSV with 0(K-1/2) rate). If (E,, K k=1 i..dd and independent of the rollout being updated, then Āssv. (ssv.x(n)) (n) K Ei= (SC, Xx)) satisfies E[ōssv
Q, = (ssv(n) and Var (arsssv (n)
2,n) = Var(Y, Q n)K (hence sd = 0(K-1/2)).
Proof. Step 1: Unbiasedness. Conditioning on (2, n) and using linearity of expectation,
Øssv K (n)
Q, n
E|SMC, Xx.) Q, n]
K
EE (SMC, (SE, X) Q, n| = (ssv(n).
(38)
Step 2: Variance and OKK11/2) rate. By independence of the K summands and identical condi- tional variance,
1
Var @ssvv (n) | 2,n)
Var
SMC, Xx) 2,n
K
Var(SMC, Xx) Q,))
K2
1
Var(Yn
Q.
(39)
K
Thus sd( @Sss (n)) = 0(3-)
Step 3: Consistency and CLT. By the strong law of large numbers (SLLN),
Øssv. Jov.KKle (n) 2.s. dssv(") (n)
K
(40)
26

---

Published as a conference paper at ICLR 2026
By the (Lindeberg Feller) central limit theorem,
VK K (Ōssv, (n) Φssv(n)) + N(O, Var(Yn Qn))
K - co.
(41)
Step 4: Impact on the advantage variance (with cross-fitting). Let the step-level advantage used for policy updates be
An == SMC, - xssv, SMO SMC, (s , xxronn).
(42)
If (ssv.x(m) (n) is computed independently of the rollout (cross-fitting), then SMC, 1 (issv,.(1)) and
Var(An) Var (SMC + Var Var(fssv.xi (x))
E Var(Yn 2.
E[Var(Yn 2n
(43)
K
information--heorrtii lower bound from Eq. 36
MC estimation noise
Hence, the advantage variance approaches the theoretical lower bound at the O(1/K) rate.
E ADDITIONAL EXPERIMENTAL RESULTS
E.1 EVALUATION ON QWEN3-1.7H
To examine the robustness of SSVPO across model scales and task difficulties, we conduct additional experiments using the Qwen3-1.7B backbone, one of the strongest publicly available small models, and extend the evaluation to OLYMPIADBENCH, a benchmark featuring olympiad-leve scientific problems. We compare four methods under unified training and evaluation setup: Vanilla, RLOO, VinePPO, and SSVPO. For each dataset, we report accuracy (Acc) and average response length (Len). The results are summarised in Table
Table 4: Performance of Vanilla, RLOO, VinePPO and SSVPO on Qwen3-1.78 across all bench- marks.
MultiArith
SVAMP
GSMBK
AIMF24
ATME25
OlympiadRench
Method
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Acc
Len
Vanilla
94.7
1034
94.0
1631
90.6
2437
77.4
5503
75.0
91.59
40.0
13697
23.3
13724
55.9
9635
RLOO
98.9
966
93.3
1537
90.1
2265
78.4
4925
77.5
8866
43.3
13599
33.3
13285
56.2
9408
VinePPO
99.4
93.3
1123
90.4
1692
79.6
4077
75.0
7085
43.3
12788
33.3
12130
55.0
7717
SSVPO
99.4
740
99.4
1258
91.8
1706
82.0
4218
80.0
8110
43.3
13027
40.0
12987
57.6
9257
As shown in Table SSVPO performs strongly across all benchmarks under the Qwen3-1.7B set- ting. On simple reasoning tasks, SSVPO achieves the best performance on both MultiArith and SVAMP, reaching 99.4% accuracy on each. Notably, these gains come with the shortest generation lengths among all methods (740 tokens on MultiArith and 1,258 tokens on SVAMP), indicating that SSVPO improves reasoning efficiency rather than relying on longer offiin----thought.
For meedimn-dffficule tasks, SSVPO consistently outperforms all baselines on GSMSK and MATH- 500. In particular, SSVPO attains 91.8% accuracy on GSMSK and 82.0% on MATH-500, surpassing both outcome-basse methods (Vanilla, RLOO) and the credit-assigmment baseline VinePPO, while keeping response lengths comparable or shorter than strong baselines.
On competition level tasks, SSVPO continues show clear advantages. It achieves the highest accu- racy on AMC23, AIME25,, and OlympiadBench, while matching the strongest baseline on AIME24. For example, SSVPO improves AMC23 accuracy to 80.0%, raises AIME25 accuracy to 40.0%, and reaches 57.6% accuracy on OlympiadBench. Across these harder benchmarks, SSVPO also main- tains competitive or shorter response lengths compared to RLOO and VinePPO, suggesting that its step-level credit assignment allows the model to solve more challenging problems without resorting to excessively long reasoning.
Overall, these results demonstrate that under unified Qwen3-1.7B configuration. SSVPO delivers top-1 accuracy on all evaluated benchmarks while preserving or improving token efficiency, sup- porting its robustness and scalability across wide range of reasoning difficulties.
27

---

Published as a conference paper at ICLR 2026
E.2 TRAINING DYNAMICS AND STEP EFFECTIVENESS
We further analyse the training dynamics of SSVPO on a new backbone (Qwen3-1.7B) and study how the number of reordered steps affects optimisation. We vary the step-merging rule using three settings: relaxed, standard, and strict, which correspond to 2-step, 3-step, and 4-step reordering regimes, respectively. These settings control how many merged steps are subject to reordering during training. We track the validation score (Acc), average response length (Len), and KI. divergence with respect to the reference policy (KL), and compare SSVPO against RLOO under the same configurations. The numerical results summarised in Table
Table 5: Training dynamics of SSVPO with different reordering regimes (2-step, 3-step, 4tsp)) Qwen3-1.7B, compared with RLOO.
Step
2-step
2-step
2-step
3-step
3-step
3-step
4-step Ace
4-siep
4-sep
72.0
74.6
4725
0.0031
72.4
484]
000037
75.2
73.8
+0.0288
74.3
-0,0225
73.0
4322
-00046
70,8
-0.033
73.5
4791
+0,0299
160
75.0
4141
0,9026
74.5
4106
200046
75.2
4293
00003
74,4
4807
+0,125
180
76.6
4299
-0.0009
76.0
4227
-000155
77.2
4103
-00020
75.2
4856
+00334
200
78.0
4012
+0.0001
77.8
4010
200012
75.2
4055
-10005
76.8
4980
+001367
220
78.6
3888
+0.001
79.0
3990
-0,0002
$1.0
4097
-00002
774
4812
+0,0233
240
79.4
3909
+0.0018
$0.0
3996
-0.0013
82.0
4166
+0.004
77.8
4796
+0.026
Accuracy vs. step length. As the number of reordered steps increases -2-> 3 4), accuracy improves monotonically, but the response length also grows. For example, at step 240, the ac- curacies the 2-step, 3-step, and 4-step regimes are 0.794, 0.800, and 0.820, while their average lengths are 3909, 3996, and 4166 tokens, respectively. This indicates that larger reordering regimes enable stronger reasoning ability but require more tokens. By contrast, while RLOO's accuracy also improves with training, its response length remains around 4.8k tokens and shows ability to shorten outputs.
Entropy divergence dynamics. SSVPO displays characteristic "positive negative posi- tive" KL trajectory, whereas RLOO remains strictly positive and monotonically increasing. This means that SSVPO initially suppresses high-probability regions of the reference policy, indicating wider exploration, and then gradually increases KI. once better solutions are found. RLOO, in contrast, performs mostly one-sided exploitation around the reference policy, which may lead to tropy collapse and reduced output diversity. The negative-then-posite KL pattern of SSVPO thus provides a clear exploration advantage during training.
F IMPLEMENTATION DETAILS
F.1 PSEUDO CODE
We provide pseudo-code of the proposed Sequential Shapley Value Policy Optimization (SSVPO) in Algorithm 1.
28

---

Published as a conference paper at ICLR 2026
Algorithm Sequential Shapley Value Policy Optimization (SSVPO)
Require: Query Q, policy Te. rollout count N
Ensure: Updated parameters
1: Sample & pool. Generate N CoT responses {or {r'} with on Q. Apply extra to obtain
normalized step sequences p² = (s))))...,;.(1)) (exclude the answer step). Let N the set
of distinct step ids aggregated from all {p²}.
2: Construct all chains (batched). Let ((V) be all orderings of N. For each n N and
a E ((N), set 80 (2, pred,(n)). Form the batched set
B (8o ((N)) U { (Ins(s,, n,
G(V), E,W, x (1,...., 1
3: Answer oracle (batched, same policy). Define (z):: append the fixed exit prompt (timeout; answer now; </think>) to chain æ, then decode with the same policy ny deterministically once to return R(x) E {0,1}. Forward all B in batched calls to obtain (R(x))uEB) cache R(.))
4: Compute SMC table. For every (a, (,,n,)) with E (1,...,188 [1]1+1], set (o,,
((Ins(so,n,2)) R(sa). 5: Compute SSV by Eq. For each n E N. ossv (n) = BOGG(A) (1....... (S,C,
Advantages on original samples Eq and PPO update Eq/4
7: for to N do
8: for
T.j do Th. +- -ii(s(()); 81-1 t- chain p with step / removed (original order preserved);
9:
choose 24 such that Ins((t (St-1, ne, 24) = =p"
10:
(s R4, 21) + SMCn, (s1-1, 22)) ASSV + A (8t-1, ne, xe) - ØssV (nz;) end for Map
11:
13:
{.4$VV} } to tokens with standard GAE (per $3.3), then update 0 using the PPO clipped
objective:
(SsVPo E [min [ (
ASSVV1 1s
end for anorymous www.m. Sccienncerr/SS-CC//
F.2 SOURCE CODE
We release the source code of SSVPO an Anonymous GitHub repository (https://
The repository includes complete installation instructions and versioned dependency requirements. Our implementation built upon OPENRLHF (Hu et al.) 2024) and extends it to support step-level credit assignment, whose core additions are: (1) a robust module for extracting mathematical reasoning steps, (2) operations for reordering the reasoning chain, and (3) computation of per-step advantage signals with propagation to the cor- responding steps for learning. We also provide out-of-the-box training and evaluation scripts to reproduce our main results GSM8K, MATH, AMC, and AIME.
www.scieeccco
G DETAILED EXPERIMENTAL SETTINGS
G.1 TRAINING CONFIGURATIONS
Hyperparameters. We provide the detailed hyperparameters for all experiments conducted in this study in T Unless otherwise noted, the global sampling budget and the maximum promp- Uresponse lengths kept the same across all experiments, while per-round sampling, number of rounds, training batch size, and the number Monte Carlo (MC) samples are specific to the outcome-based and credit-assignment RL methods.
29

---

Published as a conference paper at ICLR 2026
For QWEN3-4B, of 0.01, and an initial KL coefficient of 0.01. The log-probability clipping range is set to [-10, 10). For DEEPSEEK-RI - SEETIILL-WEN-133.
sampling temperature of 0.6, the ADAM optimizer with learning rate
we adopt the same hyperparameters as QWEN3-4B. For VINEPPO, we draw three rollouts from each intermediate state. For SPO, we set the number of segments sample three rollouts per segment, and apply probability-mask threshold of 0.9. All other sampling settings (c.g.. number of rollouts/rounds), epochs, and maximum generation lengths are exactly as reported in Table 6 Both VinePPO and SPO are trained with the random seed fixed to 42. For all methods, we use a discount factor of y 1.
Computing Details. For all datasets, experiments with QWEN3-4B and DEEPSEEK- R1- DISTILL-QWEN-I.JB are run on 4x H200 GPUs per run: 1 GPU hosts the reference model, 2 GPUs host the actor model, and 1 GPU runs the vLLM server. The experiments are conducted on UBUNTU 22.04 operating system. For more details on the dependency libraries and configurations, please refer the anonymous repository provided in Appendix F22
G.2 PARTITION STRATEGY FOR REASONING CHAINS
Step Segments from Initial Reasoning Chains We extract candidate reasoning steps solely based on mathematical segments to obtain a granularity that is robust to formatting and line breaks. Concretely, we first scan inline and display formulas (S.../, SS... well struc- tural constructs (e.g.. \frac, \sqrt, matrices/vectors, sets/intervals/uplee)) in reasoning chains. Segments containing (boxed(.... from the reasoning steps, preventing the "answer" from being treated as intermediate reason- ing steps. The extracted segments are then lightly normalized by removing TeX delimiters and redundant formatting commands (c.g., /Lght//right, \quad), unifying multiplication sym- bols and whitespace, and folding equivalent forms (e.g., (a + b)/c and (frac(atb) Hc], a/2 and (sgrt(a)). Each segment is assigned a type label grguuttioeepp///acctoonummbbebee ty/set/interval or hteec//aiiiieecttrrrraaia/ooonnn subsequent granularity reduction and training-signa backfilling. The extraction pipeline guarantees determinism and idempotence: the same input under the same configuration yields the same step sequence and type distribution; when no recognizable mathematical anchors are present, only the
used only obtain the final answer and excluded
etc.) to serve as the coordinate system for
Table 6: Hyperparameter settings for all experiments. Global sampling budgets and maximum sequence lengths are follows: MultiArith (prompt 2048, response 2048, budget 2000); SVAMP (prompt 3072, response 2048, budget 3000); GSM8K (prompt 5120, response 4096, budget 30000); MATH (prompt 10,000, response 8196, budget 30000). "Samples/Round" denotes the product of problems per round and rollouts per problem. For methods with internal structures, the type of rollout is indicated parentheses. Notex: "probs" problems per round; "rollouts" = responses problem; "inter." intermediate rollouts; "cutpoint" cutpoint rollouts; "reorder" = reordered chains.
Dataset
Method
Samples/Round
Rounds
Batch
RLOO GRPO/ DAPO
64
VinePPO
16 inter. (16)
7
256
225
MultiArith
SPO
16 x cutpoint (9)
15
128
$128
SSVPO
16 x reurder (8)
27
64
S64
RL00 GRPO DAPO
64
VinePPO
16 inter. (16)
256
$256
SVAMP
SPO
16 x cutpoint (9)
22
128
$128
SSVPO
16 x reorder (8)
44
64
$64
ELOO//RPOO DAPO
16 4
469
64
64
VinePPO
4 inter. (128)
512
$512
GSMBK
SPO
16 x culpoint (9)
234
128
112
SSVPO
reorder (32)
469
64
$128
RLOO/ GRPO/ DAAO
16
407
64
64
VinePPO
4 x inter. (512)
60
512
$112
MATH
SPO
16 x cutpoint (9)
205
128
1128
SSVPO
2 reorder (128)
205
128
<56
30

---

Published as a conference paper at ICLR 2026
candidate
Aggregating
We ane glvan polysemaat
Ecokin
We
The "tree of carer", the "predicst of sarcr" and "wan of cooiidiera" ur
Ecodon
"yonterapp" the grech monra
We to tin wale of
Coudon
Bu, 8
Ccolin
Pij 2a15
Loi Aurooota bu L- Than
EXPR
MAZ
"Bunral tr
"Prooded momen Sr. (1mo))177) -hoo00011 --8
Ecotin
_00000 wwwf-fnndeee200 -1asd0000
We alrondy product zeros
Coolon
The "saan coolkduurta be
88 bo b 10:89
Lef's theso equallonn
shwogg)))) Fighorm troepert Wiihhoomm
Etelion
b- 10 Hightrree - 1i palanoow
Fivdl
Figure 6: Partition strategy to obtain reasoning steps.
final answer (if any) is recorded. This extraction layer is decoupled from the SSV algorithm itself, facilitating replacement and reuse.
Aggregation into New Reasoning Chains. We apply conservative granularity reduction rule by aggregating (merging) adjacent steps of the same type into single "merged step," preserving the original order and the semantic continuity of equation/derivation chains. To achieve cross-sample alignment, we classify each merged step using the content of its last original step: we first check whether contains parsable numeric solution (c.g., x = 24, 6 -38, r ss 1.732, y E {2, 5)); if variable-value pair can be extracted, steps are grouped the same step by normalized key (vari- able name and canonicalized value, with a tolerance for approximate numbers). If no numeric solu- fall back comparing
tion is present, and e + d b + are considered the same step because the chosen side is identical), clustering by the algebraically normalized single-side expression. If neither criterion applies, the step is recorded as an isolated entry using its normalized text. This "merge + upper-level computation - backfill" structure anchors training objectives a stable granularity, reduces variance introduced by format- ting and filler text, and provides robust cross-sample anchors without altering the intra-sample order or the computation of learning signals on merged steps to new reasoning chains. We provided an example to illustrate the entire processing procedure in Figure
side of equation/inequaliy only (e.g., a atc c b+
H SSVPO IN ZERO-RL AND OFF-POLICY SETTINGS
H.1 OFFLPY RL SETTINGS
We now analyse the effect of off-policy sampling, which is closely related to offline RL scenarios where trajectories are collected under fixed behaviour policy.
31

---

Published as a conference paper at ICLR 2026
Effect unbiasedness. Under standard off-poliey correction with proper importance sampling, SSV as baseline does not alter the expected value of the policy gradient and remains an unbiased estimator. If asyne or partial rollouts are used as approximation and the importance weights are not exact, then the overall gradient becomes slightly biased; however, this bias arises from the off- policy approximation itself, not from SSVPO. Importantly, SSV does not introduce any additional bias beyond what off-policy PPO already incurs.
Effect on varianee. Theoretically, SSV is the conditional expectation of SMC under the sampling distribution and reward, making it the minimum-variance baseline among all baselines depending only on step identity. When the sampling distribution shifts due to off-policy or asyne rollout, the "strict minimum" becomes "approximately minimum", Nonetheless, SSV continues to provide greater variance reduction than conventional baselines (e.g., value-functing baselines or mean base- lines), which is particularly beneficial in offline or replay-based settings where variance can be high.
Effect on convergence. As in standard PPO, asyne or partial rollout introduces lag between the behaviour policy and the update policy. This increases the deviation between the empirical gradient and the theoretical gradient, and can cause more noise the lag grows too large. However, this is property of PPO-style asyne sampling itself, not specific to SSVPO. SSV does not worsen this behaviour; it simply inherits the same convergence characteristics as PPO under asyne or offline- style rollout.
Overall, off-policy or asyne sampling does not fundamentally conflict with SSVPO: the estimator remains unbiased (with proper importance sampling). retains its variance-reduction benefits, and behaves similarly to PPO in terms of convergence stability in offline and off- off-policy regimes.
H.2 ZERO-RL TRAINING SCENARIOS AND OTHER RL DOMAINS
Zero-RL training scenarios. The short answer that SSVPO is potentially applicable in zero- RL training scenarios, but not directly in its current form. At this stage, SSVPO is designed for mathematical reasoning tasks with well-structured formats, e.g., regulated answer extraction, clear step divisions, and consistent symbolic patterns. These assumptions do not fully hold in a zero-RL training scenario where the model lacks any prior long-form reasoning ability. However, SSVPO can be applied after appropriate cold-start SFT stage, where the model is first trained to produce mathematical reasoning steps in a format that can be reliably extracted and recognized. Once this minimal structure is established, SSVPO can be used for further post-training enhance reasoning quality, as seen in prior work.
Beyond LLM post-training (e.g., robotic control). This is very insightful question. In principle, SSVPO could be applied beyond LLM post-training, including to domains such as robotic control. Although our work focuses on LLM reasoning, the underlying idea of marginal-contribution credit assignment is inherited directly from cooperative MARL, where such credit mechanisms are well- established and not specific to language models.
In fact, any sequential decision-making problem could, in theory, be compatible with SSV-style credit assignment. The main practical challenge is not the credit formulation itself, but how con- struct permutations of trajectories under physical or dynamical constraints. Unlike LLM reasoning, where the set of reasoning steps discrete and easily reorderable, robotic trajectories exist in contin- uous state-action spaces and obey strict physical dynamics. Determining how to permute or reorder such trajectories meaningfully, and how to discretise them into a comparable "step set" would a non-trivial challenge similar to what current spatial-AI and muttinooss-cnntr methods face today.
Thus, while SSVPO is not limited to LLMs in theory, applying it to embodied or continuous RL settings requires additional research on trajectory representation and feasible permutation schemes.
32

---

Published as a conference paper at ICLR 2026
1 EXTENDED RESULT ANALYSIS
We conduct three series of comparative case studies, each comprising five cases, to evaluate reason- ing accuracy, efficiency, and the trade-off between them, using representative benchmark results.
I.1 REASONING ACCURACY COMPARISONS
We select five representative cases (Figures D to analyze typical failure outcomes from Vanilla and GRPO. These failures generally fall into two categories: (i) Underweighted early effective steps: Vanilla and GRPO often apply weak updates to seemingly minor algebraic rewrites equivalent transformations, which causes the derivation to drift and ultimately terminate with an incorrect an- swer. (ii) Redundancy and branching accumulation: extended reasoning chains frequently spawn ineffective branches and repeated computations, exhausting the token budget before convergence.
By contrast. SSVPO employs step-level credit assignment to consistently allocate effective return to transitions that meaningfully alter the chains. This suppresses ineffective expansions and enables steady progress in algebraic manipulation and geometric relation reconstruction. Consequently, SSVPO more reliably reaches the correct solution and achieves higher accuracy on complex prob- lems, demonstrating the effectiveness of its fair credit assignment.
AMC23-20 Q: Janet rolls standard 6-sided die 4 times and keeps running total of the numbers she rolls. What is the probability that at somee point, her running total will equal 3? The final answer can be wrilten in the form WMrra(mMK)) where m and are relatively prime positive integers.
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Okay, Janet is rolling standard 6- sided die times, and she keeps a running total of the numbers she rolls. The question is asking for the probability that some point during these rolls, her running total equals 3. Then, we need to express this probability as reduced fraction m/n and find n. Alright, let me to figure this out step by step. IninFirst. 0.393. These different. Wait, but in the inclusion- exclusion, the calculation of P(A2) 8/36 However, the total number of sequences where the running total is 3 at the second roll is 8, which iincludes sequences where was already 3 at the first roll. Therefore, the
Okay, so Janet is rolling standard 6- sided die four times, and she keeps running total The question is asking for the probability that some point during these four rolls, her running total will equal 3. Then we need to express this probability as a reduced fraction min and find m Alright, let me figure this out step by Ininiitt comilted>15 10
Okay, so Janet is rolling a standard sided die four times, and she keeps running total. We need to find the probability that at some point during these four rolls, her running total equals 3. The answer should be in the m
numbers she rolls.
form m/n and then find m n. Alright, let's start breaking this down.First First,
coprime,
<omitted>Since relatively prime, this is the reduced form Step Final AnswerinvnThe problem asks for the sum where the probability is written as Mffrc(m]()) lowest terms.m 49, lquuad 216, liuuad 216 Wboxed(265)
and
<omitled>its
92. Correct.
dp[4][]3] 107. Which dp[3][[2] +
dp[3][11] dp[3][10] dp[3][9] + dp[3] [8] + dp = 21 + 22 21
+ 10 107. Correct, dp[4][14] = 115.
Which is dp[3][13] dp[3[[12] + dp[3]
49
[11] + dp[3][10] dp[3][9] dp[3][8]
18 + 21 22 21 18 15 115.
number of sequences where is at the second roll but not at first 8
Correct. dp[4][15] 113, Which is dp[3]
[14] dp[3]13]
X
Token counts: 144
Token counts: 16, 16,144
Token counts: 14,053
Figure 7: Reasoning accuracy case (AMMC23-Qussiinn))
33

---

Published as a conference paper at ICLR 2026
AIME25-13 Let ABCD be tstrahedron that AB=CD= lisqrt(4 1), AC=BD= WBgrr((B0) BC=AD= Wsarr(89)) There exists a point I Inside the tetrahedron such that the distances from to each of the faces of the tetrahedron are all equal. This distance can written the form lifrac(m llagrt nýp), where m, n, and are positive integers, m and p are relatively prime, and n is not divisible by square any prime. Find m+n+p?
Vanilla
Vanilla + GRPO
Vanilla+SSVPO
Okay, need to find the distance
Okay, so need to find the distance from a point inside tetrahedron its faces, distance is equal for all faces. The tetrahedron has edges AB = CD = v41, AC = BD Y80, and BC
Okay, need to find distance
from the point inside the totrahedron
from the point inside tetrahedron ABCD to each of its faces, given the edge lengths. The tetrahedron
ABCD to each of
The
ABCD
ketraleddron has edges AB CD v41.
AC BD = v80, and BC AD = \89. The distance supposed to be equal to all faces, so think this point incenter of the tetrahedron. But wait, in tetrahedron, the incenter is the point equidistant from all four faces, right? need to find inradius this tetrahedron. cenirrronnnnniiiinnnn mninFirst-omittdd Then subtract 36,000: 240,800 - 36,000 204,800 IninSo det(minort)= 204, 800, IninNext, compute
CD v41, AC = BD = -80, and BC = AD v89. The distance is supposed to be the form mvnn with and p coprime, and n square-free. Then find m + n p.IninFirst. <omitted*### Step 5: Final Answerknin The inradius is (irao(20 so in the form Mtrac(m\Vsrr(/]HHo)
v89. The answer should be in the form mshn p. with m and p coprime, square- free, and then find <omitted>Scoond term: -9 (B0"(-89) (-48)r89] = -9 [-7120 4272] =
the
IninFirst.
(-2848) 25, Third term: [-3120
have:- $m 20in- = 21 p = 63 Check:- 21 square-free. m 20 and p 63 are coprime.IninSo.ng 63 lboxxe((14))
(-00( (-80) (-3310
7120] = 9 4000 = -36, 0000nn00,,,,, D2
" 215,168 25,632 36,000
20 21
dellminnor2)tnin4411 -41 -9/inin|80
(215, 168 25, 632) (222)336,)) 36,000 = 240,800
488 -48)lnin|B9 39 -89/ininLet me
36,000 204, 800t8000..inext,
compute this determinant. Again. expand allong the first row.lnin11
minor13:3ninfnn(44 -41, -11.))9 48,
In8, 39, -89]yajniinCoppuee
det(48, -48, 39, -89) - (-41) dat(80, -48
determinant D3. 1nn det[[4,, 48), [39, -89] (-41) del[[60
Token counts: 16, 14,144
Token counts: 16,144
Token counts: 9,899
Figure 8: Reasoning accuracy case 2 (AIME25- 100ME222-Qu
AIME25-19 Q: Suppose lttriagge ABC has angles liangle BAC 84%rcir <omitted> label(1"H!", H. (-2, 1))dabel(("J) (8))
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Okay, so need to solve this geometry problem. Let me first try to understand what's given and what's being asked.IntnWe have triangle ABC with angles A, B, and C being 84%, 60". and 36", respectively. Then D. E, midpoints of BC, AC, and AB, The circumcircle of triangle DEF intersects
Okay, need to find the value of arc DE 2*arc HJ 3"arc FG, where all ares are measured in degrees on the circumcircle triangle DEF. The triangle ABC has angles given: angle BAC is 84", angle ABC 60', and angle ACB is 36'. D, E, are midpoints of BC, AC, AB respectively. The circumcircle of DEF intersects BD, AE, AF G, H, respectively. Then, maybe the arcs are not thought. For example, are FG could be different. Wait, arc FG
Okay, so need to find the value the expression: arc DE + 2"arc HJ + 3'arc FG, where each arc is measured in degrees. The problem involves triangle ABC with given angles, midpoints D, of sides, and the circumcircle of triangle DEF intersecting BD, AE, AF at points G. H, respectively. iinFiirst, should probably sketch triangle visualize everything. But since can't draw here, I'll try to imagine Triangle ABC with angles A: 84", B: 60', C: 36'. Then D. E. F are midpoints. So DEF is the medial triangle
BD, AE, and AF al points G, H, respectively. Then, the points G, D, E. H. F divide the circumcircle of DEF into six minor arcs. The question asking for the sum of arc DE plus twice
p somited-nnAterrativn
arc-omitted>ange
is 60", angle at F is 36", established before. uninBut the arc DE in the circumcircle of DEF corresponds to twice the angle at which is 36", so arc DE would be 72", which matches my previous calculation. Similarly, arc would twice the angle D, which
between F and G. In our calculation, are 72", which
same as
The <omitted> Thus, the expression becomes:72 +2 Modot 24 3 Wodot 72 72 + 48 216 336 #ed Final Answer lbboxe(3336)
are DE. But according to the central angles, arc FD is 120', arc DG is 48", FD DG something? Wait, no
Token counts: 14,
Token counts: 16, 144
Token counts: 16,043
Figure 9: Reasoning accuracy case 3 S0AiE225-Quuttionn1 19)
34

---

Published as a conference paper at ICLR 2026
AIME25-26 Let 2lidots (_11) non-corvex 11 11-gon
The area 1A, [i+1] for We
We 10, Vicos(Vanglee A_ i_ 1A_ [++1)=iffac(12211 for each 2 We i We 10, The perimeter of A_1A A__ooos A A_(11) is 20. If
A_1A 2+ (11) can be expressed as (ifrac(mllqgn(nn)-p for positive integers m.n.p.q with squarefree and ligod(m.p.q)=1, find m*n+p*q.
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Okay, let's try to tackle this problem. It's about non-corivex 11-gon with some specific area and cosine conditions. The goal to find the
Okay, so need solve this geometry problem about an 11-gon. Let me try to parse the problem step by step. 'ninWe have non-convex 11-gon labeled A1Az..A1.. The key information given is:nim1. For each 2 s sis 10, area of triangle A AAAAA_A [I+1) 1n2 For each 2 si 10, the cosine of the angle ZA iA [i+1] 12/13.Ind 3.33.
Okay, so need to solve problem about non-convex 11-gon. Let me try to parse the problem step by
step IninFirstt, the problem says:
A_1A_2 A_ 1A_ 1A_(11) expressed in particular form and then compute m + n + p q IninFirstt let me parse the problem again to make sure undersland all conditions. WninWe have an 11-gon A_ 2ldots A (11) S, which is non-convex. For each 2
AiAz...Ai be non-convex 11-gon
such that the area of triangle
10. AAiA_ [[+1) for each
Also, the casine of the angle
ZA, (AIA_[I+1) is 12/13 for each 2 s 10<omittedd> From expression InS Wirac[9 1H44 We have: m In- n= In p= q 4 Check Nged(9, 1, Thuss, square-free. Thus, the final answer is:m
perimeter of the 11-gon is 2n. We need to find A1A2 + AiAn expressed the form (mvin p)qq, where m, n. P. q are positive integers, is square-
Weq Weq the
triangle
A A_iA_A,, (A_((+1) is 1. Also, the cosine of
and 5
the angle somitted>.n138t 9.360
free, and god(m. q) =)=1. Then compute
-13059.
1:3059.6nnn22044 6inin2704. Total: 9142.8577 8579
q.IninFirst. I, let try to visualize the problem. There's an t<omitted>quare rootinin9 "qrrtyy-
+ p q = 5 Nboxed(19)
11911664 64 13059.6 2704 =
29142.8579 1191.64) + (-13059.6
2704) + 4939 10355.6
20) = ytninSquare both sides:'nln811(* (a 20) (20
-21.102 1021 < 0.1ninSo, between 3.1 and
3.2, f(a) crosses from -21 to +469.
y>*'innExpannd both sides/nin81y" 1620 = 400 - 40y yAninBBring all
Let's by
terms to
Token counts 16,114
Token counts: 16,144
Token counts: 12,724
Figure 10: Reasoning accuracy case 4 (AIME25-
AIME25-20 Q: Circle llomeaa with radius 6 centered at point A is interally tangent at point <omitted> label(("Hi", H. (-8, -8);; label((1iomega_11 1V", (9, -5)); sabel(("lomega_2"" 21", (-1, -13.))
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Circle llomega_1 1 with radius 6 centered point A internally tangent at point B to circle Vlomega_ 2 with radius 15. Points and D Nomega_2 2 such that Vaverline(BC) diameter of Nlomeaa 2 and (Voverline(BC) liperp Noverline(AD)). The rectangle EFGH is inscribed in Nooegaa 1 such that Noverline(EF)) closer hoverline(GHi) than to (EFF),
Circle Voomeaa with radius $6$ centered at point A is internally tangent at point B to circle llomeaa 2 with
Circle liomega 1 with radius centered at point A is internalyy tangent at point B to circle llomeaa_ 2 with
radius Points and$D Nommga_2 2 such that loverline(BC) is a diameter of llomega 2$ and (Voverline(BCC lpperp Noverline(AD)). The rectangle EFGH inscribed in llomeg 1 such that Voverline(EFF lperp \overline(BC), C is closer to
radius 15. Points C and
Vomega 2 such that lioverlineBC) is diameter of Vlomeaa and (Voverline(BCC lperpp NoverllneeADD). The rectangle EFGH is inscribed in Nomega 11 such that liaverline(EF)
iperp Noverline(BC).
Uperp hooerllineBCCs. <omitted> So:tnillex((Area) (Mracc24)((sarr(()) Mcdot Wfrac(12)(ssqr((5)) Wfrac(288) (5) Thus, 288 n and
Moverline(GH) than to NloverlineEEF), and $D$ closer lloverlineFGG) <omilted>-fracc (1a)s///)))
and closer Noverline(FG).
<omitted=inin192*= 12P(1 1/22 3(1
1/2 = OlninExpand 121P(1 + 2t H)
ltext(Aree) = (frrac24)()ssrr((5)) Vicdot Mfrcc (2))ssrr((S)) = Vfrac((28)({5) Thus, the area (2 and since 288 and 5 are coprime, the final answer is:m 288 5
293 inin--ninttttt *FFinl Annswrr"
12f' + 24 + 1nnn" -3(1
=) - 30 overaltinin1922
+ 12224 + 24t + 12F 33 6t 3P =
OlnvnCombine terms:inin12MM 24t + (192P+ + 12F - 3P") (-61)
Nboxed(293)
Token counts 16,114
Token counts 14,036
Token counts: 7,835
Figure 11: Reasoning accuracy case 5 AAMMF25-Questione
35

---

Published as a conference paper at ICLR 2026
1.2 REASONING EFFICIENCY COMPARISONS
We select five representative cases (Figures 12 in which all methods obtain the correct answer but differ in efficiency. Vanilla and GRPO typically produce valid yet longer derivations, contain- ing more redundant equivalent steps. In contrast, SSVPO maintains accuracy while more quickly identifying effective steps, reducing unnecessary exploration of the solution space. As result, it generates shorter and more efficient reasoning chains. This effect is particularly evident in Fig- ure T6] although all three methods solve the same problems, SSVPO shows little to no redundant back backtracking repetition, indicating stronger ability to capture effective steps. By comparison, GRPO and Vanilla often reach the answer with additional equivalence expansions, repeated checks, or excessive exploration.
AMC23-0 Q: Cities A and B are 45 miles apart. Alicia lives in and Beth lives in B. Alicia bikes towards at 18 miles per hour. Leaving same time, Beth bikes toward A miles per hour. How many miles from City they be when they meet?
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Okay, Alicia and Beth live cities and B, which are 45 miles apart. Alicia starts biking from towards at 18 mph, and Beth starts biking from B towards A at 12 mph. The question is asking how many miles from city A each
Okay, Alicia and Beth live in cities, A and B, which are 45 miles apart Alicia starts biking from city A lowards city
Okay, so Alicia and Beth
cities
and B, which are 45 miles apart. Alicia is biking from A to B at 18 mph, and Beth biking from to
18 miles hour,
mph. They leave at the same time, and need to figure out how many miles each other.
and Beth starts biking from city B towards city
miles per hour. The question asking how many miles from city A they will be when they meet. Hmm, let me think about how approach this IninFirst.
they will be when they other. Hmm, let me think about how to approach this. Fiirs,t,, remember that when objects each other, their relative speed is the sum of their individual speeds. maybe
from City A they'll
<omitted>
we can calculate the distance Beth
towards
travels same
remember that when two objects move relative
( = 12 tttimes 1.5
towards each other, speed the sum of their individual speeds <omitted>lexti miles)Adding both distances 'n27 miles) This matches the original distance between the cities, confirming the calculation is correct Final Answer Wboxed(27)
18 Wiee miles]Adding these
use that concept here. <omitted-This This means in the time they meet, Alicia covers Wrrac(3y5) of total distance. (DisttDis from A) Wffra(3y5) Vitimes 45 27 litex(( miles))--Ininntte "Final
distances:277
Weex miles)
Thin matches the total distance belween the cilies, confirming our answer correct.inin- innnnttiinn "Final
Weex
Answer* Wboxed(27)
Answer"" \boxer")))))ed227)
Token counts 1,854
Token counts 1,684
Token counts: 1,189
Figure 12: Reasoning efficiency case HAMMC2--Questone
36

---

Published as a conference paper at ICLR 2026
AMC23-17 Q: There is unique sequence inlegers a licdols a (2023) such Ihatvnl//tan22233 Wrracfa Wan
* 3 W 5 litann5 x + Vioott (2023) Vttnn(2023) x)(1 a Vitan*2 x Vitann4 Wcdots a (2022) (2tan(2022) xj(]whenever Vian 2023x$ is defined. What is $a_ S2_(22
Vanilla
Vani
Vanilla+SSVPO
Okay, I need to find the coefficient
Okay, so need to find the value of
Okay, so need to find the coefficient a (202)))) in the unique sequence of integers a_1, that satisfies the equation/Htnn202kx Wfrac[a_ 1 Wan x a__ Waan'3 x + a
a (2023) in the unique sequence of
(2023) in the unique sequence integers a_ a Wdots, (2023)
integers a a_1, a_ Viloos, (2023)
Wioots, a (2023)
that satisfies the equation:\tan2223x Mracca 1 Mat x + a WMla"3 x a_5
that satisfies the equation/ltan2023>>
Mfraa(a Vitn x a Iitan33 x a_5
litan
(2023)
Vitan X loodots + a a_(2023)
Iaan'5 Niddos a (2023)
(itan")0223) xX{ a__ 12tan a_4
(2023 x){1 + a_2 W x a_4
Mtan"(023) x)(1 + Vitann2 a_4
MMtan4 + licoots + a_[222
M4ln'4 ficdots + a (2022)
Ittan Nodoos + a [2022) <omitted>
(2022) x)whenever Itan 2023x
(2022) xjyn$Sininwheneeer
W2innom/2023220223) Since
is IninFist, remember that
<omitted=tep 2: Identify the Coefficient
Wbiinom/2223/2023) = and and $1011S
there are formulas for multiple-angle tangent identities. For example,
Waan22033)+ We are interested in the coefficient of \ttnn(223)> in
is odd, we have:n(-1)M11011 -11
Therefore, the coefficient a (22023) a (2023) = 1 Final Answer
the numerator. This corresponds to the term where the exponent 2023 S. so:2m 1 2023 WRighttrrow
(binom(2023(2023) = 1, and since $1011S is odd, we havera (2023) (-1)(10111 Vicdot =1 ### Final Answer (boxee(-1))
Wboxed(-1)
m= 1011 Now compute the coefficient: (-1y(10111) Wbinom/2023/2022 (-1)(10111) liedot = -1 ### Final
Answer liboxed(-1)
Token counts :7,460
Token counts 5,293
Token counts: 4,820
Figure 13: Reasoning efficiency case 2 (AMC23-Question11)
graph of yelllog [2)x. midpoint Noovrrine/ABB (6, 2). What is the posilive difference between the x-coordinstes of A and B? The final answer can written in the form m llsqrt(n). where and are relatively prime positive integers. What is m+n?
AMC23-21 Q: Points
Vanilla
GRRO
Vanilla+SSVPO
Okay, need to find the positive
Okay, so need to solve this problem where points A and B are on the graph of y log base of x. The midpoint of segment AB (6, positive difference between the x- coordinates of and B, and express in the form miln, then find m + Alright, let me think step by step.Inin IninFirstt let
Okay. let's try to tackle this problem. So, have points A and the graph of y logax. The midpoint of segment AB is 2). need find the positive difference between the coordinates of A and B. and express it in the mnnn where and n are coprime positive integers, then find n. Alright, let's break this down. down.kninFiree minFiirtt, since both points A and B are the graph of logux. their coordinates must satisfy that equation. denote the coordinates A <omitted>nWe are told the answer should be in the form mitsqrt(n). are relatively prime positive integers. m = 4, n = 5, and since 4 and are coprime, we have:m 5= Wboxed(9) ### Final Answer Vboxed(9)
difference between the x-coordinates of points A and on graph of logix, given that their midpoint is (6, 2). The answer should be in the form mvn. where m and are coprime, and then find m Alright, let's start breaking this down. down.ininFirst ininFirtt, since points A and on graph of logex, their coordinates must satisfy this equation. Let me denote point A as (a, logea) and point as (b, logab).
have to find the
that the
midpoint of two points (xi. y1) and (xa. y2) is given by ((x) + x2) (y) + y2) Here, midpoint given as 2). So, If let point A be (a, logea) and point B be (b. logab). then according to the midpoint formula:/nnnt (a b)/2
midpoint
of AB is given (6, 2). The midpoint formula is ((x) + x2)/2, (y: + y=)22). comitted>= [4%lsqr((5] 41ssqrt(5) Thus. the positive difference belween the x -coordinates is 41laqrt(5) which is form mlssgr((n) with m 4 = 5 ### Final Slep: Compute m + n = (boxed(9)
(logua logab)/2 )= (6. 2)ininSo, that gives me two equations:nnin1. (a b)/2 12<omitted>
where
6 are relatively prime positive
Integers.Herd m = 4, n 5, sorm nn liboxed(g)
Token counts: 2,465
Token counts: 2,454
Token counts: 2,403
Figure 14: Reasoning efficiency case 3 (AMC23-Quession22)
37

---

Published as a conference paper at ICLR 2026
MATH500-1 Q: Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $tr,Wiheea), where $r > We Wheta Mpp..
Vanilla
VRni
Vanilla+SSVPO
Okay, I need to convert the reclangular coordinate (0, 3) to polar coordinates. Hmm, polar coordinates given as (r. 0), right? Where is the distance from the origin to the point, and
Okay, need to convert the rectangular coordinate (0, 3) to polar coordinates. Hmm, polar coordinates are given as (r. 8), right? Where is the distance from the origin to the point, and the angle made with positive x-axis. Let me recall the formulas for converting between rectangular (x, y) and polar (r. 0) <omitted-Uhe angle WhetaaS is:intnSSWiltheea VMfra((pp((2)Th is the standard angle measured counterclocwwiee from the positive x- axis to the point on the positive y- axis # Step 3: Check the Conditionsinin- r = ### Final Answer Viboxe{(33, (Mrrc(Mpi)22))
Okay, so need to convert the point (0. 3) from reclangular coordinates to polar coordinates. Hmm, polar coordinates... right, that's where you have a radius and an angle e, remember
angle made with positive x-axis, Let me recall the formulas for converting between rectangular (x, y) and polar (r. coordinates-omiin
formulas to convert from rectangular (x, y) to polar (r. 8) are equals the square root of x squared plus y squared,
Hin- In- Weq
arctangent of y over x <omitted> the correct range Weeq Wheta 20pi ininn-Ininntfit Final Answer [[oxed (Mfra((ppi(2))
Whhhe < 20pp VMrra((pi)22) is within required range + Final
Answer boxed((3. (Mfra((ppi(2))
Token counts: 1,719
Token counts: 1,879
Token counts 1,495
Figure 15: Reasoning efficiency case 4 IMATH500-Questoon
AIME25-15 Q: Six points A, B, C, D. E, and F lie in straight line in that order. Suppose that G a point not on the line and that AC=26, BD=22, CE=31, DF=33, AF=73, CG=40, and DG=30. Find the of litriangle BGE.
Vanilla
Vanilla+GRPO
Vanilla+SSVPO
Okay, so I need to find the area of triangle BGE. The points B, D, E. are in straight line that order.
Okay, let me try to figure out how to solve problem. So, we have six points straight line: B, C, D, E, in that order. There's a point not on line. We're given several distances: AC-26, BD-22, CE=31,
Okay, so need to find the area of triangle BGE. Let me start by trying visualize the problem. There points A, B. C, D. E, F on straight line that order. Then there's a point on the line. The given distances are AC=26, BD=22, CE=31, DF=33,
six
There's also a point G not on the line.
They've ve given me several distances: AC=26, BD=22, CE=31, DF=33, AF=73, CG=40, and DG=30, Hmm,
DF=33, AF=73, CG=40, and DG=30.
okay. try visualize First.
comitted-triangle Wrriingle BGE
AF=73, CG=40, and DG=30. need to find the of triangle BGE,Firrt <omitted>
maybe I should
using the coordinates:'n- BB (18. 0)
skeich-omitted> triangle with coordinates :Ntex((Area) (Mrrc[1)22
Sin- $ G (58, 24) 5in S E (57. 0) SintnWe use "base-height
S to the x-axis, which the y- coordinate of G. so Wetxt(Area) (Mrac(1)2) Wiims Vitex(bbas) Wiimes (teet(height)) Wirra(112) Witmess Viimes 24 Wffrac(336(22 468 ### Final Answer (boxed(466)
[Wleft] x_nl)xx____222 10__11yy21111 2_333)+ x__ 2(y__ 3- y___1) ))+
formula* The base $ BE = 57 18
x_3(y_1 1 2-_ \rrigtt) Plug in the
39 The height is the y-coordinate of
coordinates
\frrc(1)2)
G. which (since the distance from the x-axis) Wtext[Area]- Mrac((1)2) Wiimes Wtex((asee) lilimes (text(height) (ifrac(1)(2) Vitimes 39 Viimes 24 = Wfrac(936)(2)==
[Weft] 18(24- 0) 58(0 0) + 57 - 24)
\rrihh|| Wrrrc (1)) Weft] 18 Vidol 24-
57 licdot 24 Wright| Wfrra(112) Weft]
432 - 1368 \riiht)== = \ifrac(122) Vicdot
936 468 Final Answer (ooxed(468)
468'n Answer: "nswer: (lboxee(468))
Token counts 6, 6,002
Token counts 5,419
Token counts:5,28
Figure 16: Reasoning efficiency case 5 (A1ME25-Qucstion155
38

---

Published as a conference paper at ICLR 2026
1.3 REASONING ACCURACY AND EFFICIENCY TRADE- OFF COMPARISONS
We select five cases (Figures 17 to examine how SSVPO achieves the correct answer within reasonable token budget, compared existing credit-assignmen methods that consume fewer tokens but often return incorrect answers. Our observations reveal a consistent pattern across these and other cases: SPO typically generates shorter reasoning traces but, due early termination or over-aggressive simplification, skips critical intermediate decision points needed to reach the correct solution, leading to failure. By contrast, SSVPO produces traces that are slightly longer yet still compact. It consistently concentrates updates on transitions that genuinely alter the reasoning chain, preserves the necessary intermediate steps, and thereby converges to the correct solution. In practice, SPO often stops after few aggressive equivalence rewrites that commit to incorrect path, whereas SSVPO continues past the pivotal turn and successfully arrives at the correct path.
As summarized in Table 2 existing credit- assignment baselines (c.g.. SPO) appear token usage, but this "efficiency" often arises from pruning mid-chain segments that actually encode crucial decision points, ultimately causing failure. In contrast, SSVPO strikes more balanced trade-off between accuracy and efficiency: a modest increase in token budget leads to a substantial gain in accuracy.
efficient in
AMC23-49 Q: What is the area the region the coordinate plane defined by |||11111 1 + 1lyl- We 1?
Vanilla
Vanilla+ SPO
Vanilla+SSVPP
Okay, so I need to find the area of the region defined by inequality [x]
Okay, so need find the area region defined the inequality [x] 1 + lyl | 1. Hmm, that looks a bit complicated with values. Let me try to break it down step by step.IninFirst,, remember that often
area of the region defined by the inequality || | || [+I I |- We 1 Hmm, that looks complicated
Okay, so need to
[y| Is Hmm, that looks bit complicated with the absolute values. me try break it down stap by step. Inrss.. somitted>the region.IninThe point (1. 1): 11 1| |1
absolute
absolute values. Let me try to break it down step by step. Ininiirstt, remember that absolute
absolute value expressions be handled by <omitted>i, symmetric in all four quadrants, and we have already accounted for the entire region in the first quadrant, the *"tota area" region is:lall((Toaal Area) 2.nim Final Answer \boxed(2])
inequalities often define regions that are symmetric in some way. Since there are absolute values around both and y.. <omitted> This is the area in the "'fist
1|
inside. Ihen point
(2.1): [2- 1 11 = 0 1.155 So
it' on the boundary.vninThh point
(1.2):
wnenThe point
(0.1): 10 1|+ 11- 1|= 0 1 So
quadrant" μM# Step Total Area Since the region symmetric in four quadrants, the total area is:4 Nitimes Final Answer (bbxee
it's the boundary.ininThe point (0.5,
0.5): |0.5 1| |0.5- 1 0.5 0.5
1. So this point is on the booundary.ninnWaiit
the region inclluee points like (0.5, 0.5) which in the center? Wait, but according to equation in the uv-plane, the region is ju 1| [v- 1| So (0.5
X
Token counts: 144
Token counts 8,736
Token counts: 10,042
Figure 17: Reasoning accuracy and efficiency trade-off case 1 (AMC23- (ueeti--
39

---

Published as a conference paper at ICLR 2026
AIME24-86 Q: Alice chooses a set SA$ of posilive integers. Then Bob lists all finile nonemply sets SB$ of posilive inlegers with the property that the maximum element SB$ belongs SAS. Bob's list 2024 sets. Find the sum of the elements of
Vanilla
Vanilla+SPO
Vanilla+SSVPO
Okay, have this problem where w Alice chooses a set A of positive integers. Then Bob lists all finits nonempty sets B of positive integers with the property that the maximum element belangs Bob's list has 2024 sots. I need to find the sum of the elements A. <omitted> 28 64 32 8iJinThe corresponding exponents are 10, 9, 8,8, 7. 6, 5, 3. Thus, the set M. A W) WM (11, 10, 7, 40J9) The elements of W A W) is:3/4 6 + 7 + 8 10 11 55W] Therefore,
Okay, I've come across this problem where Alice chooses a set A of positive integers, and then Bob lists all finite nonempty sets B of positive integers such that the maximum element B A. It's given that Bob's list has 2024 comilted>>bt that doesn't directly correspond to a sum of distinct [a-1), because the subtraction isn't allowed in sums of positive integers, perhaps
Okay, I have this problem where Alice chooses a set A of positive integers. Then Bob lists finite nonempty sets B of positive integers such that the maximum element of B is Bob's list 2024 sets, I need to find the sum of the elements of A.nnHHmm, let's break this down. somitted>ithee
in It's given
tums out
A must be: W33, 5, 6, 7, 8, 9. 101
integers.Inin In'n Therefore, where each contributes a distinct power of 2, so sum just sum of dislinct 2*a- 1). So, (*aa-) must be subset of the powers of 2
Summing the elements of W. A M)M13
A
5 6 7 9 + 10 48%]
of the
Therefore, the sum of the elements of M 1)) [sbo[oxxxd(48))
sum of the elements of 10( A W) is (1boxed(55))).
Token counts : 16,144
Token counts: 2,701
Token counts: 3,084
Figure 18: Reasoning accuracy and efficiency trade-off case 2 SAIME24-Questione
AIME25-1 Q: Find the sum all integer bases b>9 for which 17_b divisor 97_b.
Vanilla
Vanilla+SPO
Vanilla+SSVPO
Okay, so I need to find the sum of all integer bases W. b. W) for which M0 17_ b W) is divisor W 97_ b Hmm, let me make sure understand the problem correctly. IninFirst, W 17_b 17_b W) and W. 97 b W) are numbers in base WK b W). need to treat these numbers as base
Okay, so have this problem here: need to find the sum of all integer bases W is divisor of VV 97 b ()). Hmm, okay. Let me think about how to approach this. innsFistt. know that when number written in base W. W). each digit represents a power of W. b N). So. for example. 17_ (1) would be equal to W Wimes b 7 W) in Docimal--miitodd-nnvaiii as W Wee 9 NyyThus, the valid bases are 98 and
AIright, so I have this problem here: need to find the sum of integer bases W. b 99 W) for which W. 17_b 1i) divisor of M 97 1i). Hmm, okay. Let me to parse this out step by step. AninFirstt remember that when we write number a certain base W W). each digit represents a power of MV b Wt). So, for example, V. 17 b_bb W)) would equal Vi Vitmes b*1
() which WV, 17 W)
values and see when MM 17 b W) divides W 97 b_ li). That means W 17 W) should be factor 10 b W)
Jecimall.nninAligg so let recall how to convert numbers from base W.
Viime bn0 (t), which simplifies to W b
N). Similarly, VM 97_b
W) t<omitted-Waii. let
19. The sum of these bases is:1l[ 98
<omitted>
IninWait, W d = 9 k W.. Since W. 1)) is positive integer (as it's a divisor in the equation V. 9b 7 = k(b
19 = 117 WiThe final answer is:1l
Thus, we have:- 1V b b 28 1t)
[Nboxed([11])]
implies W. b 21 1i-- 1/ b + 56 W
implies
49 WiThe possible bases
+ 7) W)). W/( k ligeq 1 W). Therefore, li( d
are 21 and 49. Summing these bases,
=9- k Weeq (ly Ininuu also, since MM b > WJ. from the expression 1y b
we get'l[ 21 49 70 WJThus, the
sum of all intager bases W M/b W) is W (\lboxed(70ji)
|frac[7(K- - 1))d) W). M dl) must be posilive. So, W is a positive
Token counts : 16,114
Token counts: 2, 2,130
Token counts 4,801
Figure 19: Reasoning accuracy and efficiency trade-off case 3 (AIMF:25-uussiin)
40

---

Published as a conference paper at ICLR 2026
AIME25-25 Q: Let SSS be the set of verlices of regular 24- 24-gon. Find the number of ways to draw 12 segments of equal
lengths so that each vertex S is an endpaint of exactly one the 12 segments
Vanilla
Vanilla+SPO
Vanilla+SSVPO
Okey, need to figure out how many
Okay, I need figure out how many ways there are to draw 12 segments of equal length in regular 24-gon, such that each vertex is the endpoint of exaclly one segment. Let me try to break down step by tinrint let me visualize a regular 24-gon <omiteed>Summing Up All Valld Cases text(Total)) 2 + 4 8 16 64 1 (9oxed(95) #šي Final Answer (boxed(35)
Okay, need to figure out how many ways there are to draw 12 segments of equal lengths a regular 24-gon, where each vertex is endpoint of exacily one segment, Hmm, let me start by understanding
ways there are to draw 12 segments of equal length a regular 24-gon, such that each vertex exacily one segment. Let me try to break this down step by step. ininFist,, let me. <omilted>Since there are ""two" such step sizes (2 and 10), the total number of matchings is:Inin4 #M# Final Answer \iboxed(8)
endpoint of
problem better better.iminFirrs. IninFirst,, a regular 24-gon has 24 vertice, <omitted>
Ntimes = 16 In- k 16 = 16 In- = 1 Wimmes 64 = 64 k
Vitimes
=12: Wtimes = Adding these:
+ 16 16 64 1 Wboxed(113)
ииa Final Answer (boxed (13)
Token counts 10,100
Token counts 13,666
Token counts 11,,433
Figure 20: Reasoning accuracy and efficiency trade-off case 4
AIME25-18 The product((Morod" (k=4) Vrac/Wloo_ k (5^(k*2- 1WHilog 1) (5*(k*2
4]) Mfrac(Vllo_44 4 (51(15))
(Vlog 5 (5*(12)) licdot |frac[Wlogg_ 5 (5124))(ilogg 6 (5)loo Ifrac(Woog 6 (5535))
7 (5*(32)))
Vicdots
irrac(ilog_ (63) (3966))) (ilog_[64) (5966))))) equal $iltfrac mn,S where m and are relatively prime positive
integers. Find
Vanilla
Vanilla+SPO
Vanilla+SSVPO
Alright, so I have this product to
Okay, so have this product to
Okay, need to compute this big
(aapaatee/ini/@nnpproe(33333 (k=4)
(ooputtes) (k=4)
product from k=4 to k-63. Each term in product fraction where numerator log base k of 5*(k" -1)
Wirra(Ullo__ k (5*(k*2- 1)))Wlog_(( [k+ (5*(k"2 - 43)yniijilnAAdd the problem says that this product is equal to V. ((tfrac(m)()))) where V(mml) and Winl) are coprime posilive inlegers, and need to find Wm nli). IninHmm. okay. So, first, need to undersland what this product is. ..omittee>>
Mfrac/illog (5*(k*2 1)JWWlog 1)
(5* 4))) \Mraa[WWoo 4 (51(15))
(Wlog (5)(1 Wedot Wrrac/Vooo_5
and the denominator is log base (k+1) of 5*(ke-4, -4). IninFirst
(5 (24)))oooo...o (51))) loodot
Mfrac(Voog_ 6 (3))3)) 7 (52)))))
the general term/ininlog, k(5*()² -1))
Modots Wfrac|VVoo (63) (5*(398)))
log_ (k+1)5 -4)) remember that b(a)= (In a) (In b)<omitted>
(Wolo (64) (54/3665))niijlnnnkk) problem comilled these results Vicdot Wiara(1)
the
Mracc - 1)kk + +1)][(k 2Xk + Thiis
but fraction form, m3nii. but hang on, 93/13, which is maii.i, 93 is 3'31, and 13 prime, so 93/13 is in lowest ferms. Thus, m=93, n=13, so n= 108. wait, I'm confused because 31 and 1/13 is equal to and 5/13, but is 93, so 93/13 is equal to 7 and 2/13 IninWait. but
producttelescopes to:4/Wracc(31) (13)y) Combining these results.
is 93, times 1/13 is
(13) Vidot 3 (rrrac(3 (273) Mrrac(1) (91))) Thus, the product simplifies to [Vfrac(1))91)) where 11/m 1'l) and (n = 911)). Therefore, W(m + n= 91 -S2N).\\boxxd((221)
we geti:([ffra((31)(( Vitmes 3 (frac(933(33M) Since 93 and are coprime, the fraction is in simplest form. Thus, 1i(m = 93%) and Win
13l).giving
106().1
[Nboxed[106]M
Token counts: 16, 144
Token counts: 2,816
Token counts 3,871
Figure 21: Reasoning accuracy and efficiency trade-off case 5 (AIME25- -uertttin
41


Reviews:
[{"review_id": "krY9s6B1mU", "reviewer": "Reviewer_BBy3", "summary": "This paper adapts the concept of Shapley value into LLM reasoning trajectory, using Qwen model (and its distilled version) on math reasoning benchmarks for training. It proposes something called the SSV value to perform multi-step credit assignment in the reasoning process, and uses it as the advantage estimator in RL. The paper provides some explanation claiming that this estimator is unbiased and has lower variance.", "questions": "Why would Shapley value work for multi-step reasoning credit assignment? \n\nWhy can token-by-token generation from the same policy be partitioned and treated as a multi-agent game?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "Applying Shapley value to LLM reasoning is indeed novel.", "weaknesses": "1. The modeling is unreasonable: Shapley assumes independent and exchangeable participants, but the LLM reasoning steps are strongly dependent and order-sensitive. The SSV estimation method enumerates all orderings, which is unnatural and violates the basic premise of sequential language modeling. Under such ordering, the MDP transitions are arbitrary concatenations, not reflecting the real generative distribution.\n\n2. shapley value is 'fair' only ensures that each step gets an equal share in total reward decomposition, but does not mean the step truly causes the result. In other words, a logically irrelevant or even wrong step can still get positive credit due to statistical averaging over permutations.\n\n3. computationally intractable: ssv requires averaging over all permutations—combinatorial explosion. Even with 10 reasoning steps, there are already millions of permutations.\n\n4. The effectiveness of this credit assignment: compared with REINFORCE-style RL algorithms that use (normalized) outcome rewards as advantage, this method adds the variance of SMC estimation, introducing more sampling noise. The paper provides no analysis or experiments for this.\n\n5. empirically: scenarios and models are very limited. Using Qwen3 for training on a few math datasets is questionable. two groups of experiments, each using only one different backbone.\n\n6. On results: many benchmark improvements are meaningless, the absolute difference is within 2%. Qwen models already saturate these benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWH2dIVNzI", "reviewer": "Reviewer_Xabj", "summary": "This paper introduces SSVPO, By using Shapley values, SSVPO fairly allocates credit to each reasoning step based on its marginal contribution, even in partially correct reasoning chains.\nThe key contributions include:\nCredit assignment through SSV to better estimate the utility of each reasoning step.\nValidation showing that optimizing the utility of each step through SSVPO leads to performance improvements in mathematical reasoning tasks.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces an innovative credit assignment method using Shapley values (SSV) to estimate the utility of each reasoning step. \n2. The paper provides empirical evidence demonstrating the effectiveness of the SSVPO method in improving reasoning accuracy and computational efficiency.", "weaknesses": "1. One important weakness is the lack of discussion on computational cost. The paper introduces SSV for step-level credit assignment, but methods like SMC require evaluating multiple reasoning sequences, which increases the computational load. If the goal is to improve training efficiency, the authors should compare the computational cost and ensure a fair comparison with baseline methods by keeping the training computational load consistent.\n\n2. I want to know the advantages of SMC over VinePPO, which uses search-based value estimation. A discussion on how SMC compares in terms of accuracy, efficiency, and computational cost would make its benefits clearer.\n\n3. The paper also lacks ablation studies to show how different hyperparameters affect SSVPO's performance. These would help assess the stability of the algorithm under different settings.\n\n4. Is SSVPO suitable for zero-RL training scenarios? All experiments in the paper are conducted on models that already have long-COT abilities. It would be useful to know how well the method performs on models without such prior capabilities", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4NXYtnnfoo", "reviewer": "Reviewer_K1FX", "summary": "This paper proposes a totally novel way of credit assignment when doing RLVR. The credit assignment is based on the marginal contribution of a specific step among many possible reorderings of all possible reasoning steps. This credit assignment is different from the traditional definition of the credit when doing RL which just indicates the change in value after an action. But, it captures how useful this action could have been in other sequences. Therefore, it detects more generally if an step is useful.", "questions": "1-Inserting some steps in an existing solution might make the solution completely wrong as the inserted step might not logically follow from the previous one. However, this is going to be missed with the current verifiable rewards for MATH. Is that fine under SSVPO? Am I understanding this correctly?\n2-Permutation of steps of reasoning makes the reasoning useless. However, the SSV operates based on this. Is my understanding correct? If yes, how is this fine?\n3-I am wondering if the SSVPO can be used in normal RL settings like training a robotic hand. Is there a reason/paper that this works well? It feels like changing the meaning of credit assignment might not play well in all deep RL settings. If that is true, why is it not the case for LLMs?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main strength of the method is that it is very novel and it also provides empirical evidence for its proposed methods. The baselines, VinePPO, SPO, GRPO, and RLOO are appropriate baselines and evaluations on AIME'24 and AIME'25 are strong evaluation benchmarks. The method is introducing a completely new way to see what credit assignment means. As an RL person, I am used to \"credit assignment\" meaning the delta in values of states before and after an specific action and this paper is challenging that assumption.", "weaknesses": "While the paper is novel and provides empirical evidence, the method is so radically new, that I think I need more theoretical and intuitions in accepting its soundness. I think the authors do not provide intuition or justification on the reason that their proposed method should work better than the baselines. VinePPO in particular, is able to detect useless steps: a useless step will have zero-advantage. Therefore, I cannot rely on just intuiting the strength of results of SSVPO based on assuming it detects useless steps more accurately.\n\nAdditionally, I need more explanation on how the credit is determined. The reward in verifiable reward tasks is based on the final answer in the last sentences. If I understand correctly, if we reorder sequences, it is only the last sentence that matters and dictates the reward. In other words, changing the order of all previous sentences should not change the reward that we get out of the current mathematical verifiable rewards. I think this has some problems? The response that is the result of such reordering has no meaning. Therefore, I don't understand how SSVPO can lean on permutations of steps as there is no mechanism that detects that permutation of steps is not a correct response. \n\nI think the paper fails to provide intuition on why it works beyond just providing the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtbKZaKh1e", "reviewer": "Reviewer_4L2k", "summary": "This paper presents SSVPO, a reinforcement learning algorithm aimed at improving reward allocation strategies. By modeling the generation process as an insertion MDP and using SSV as an advantage estimator, SSVPO computes more equitable token-level rewards by reordering the reasoning steps and calculation the relative promotion gained from the insertion of each step. The authors theoretically define the properties required for fair reward allocation and prove that SSV satisfies these properties. Comparisons with existing RL baselines show that SSVPO yields greater gains in sample efficiency and overall performance.", "questions": "4. Could you analyze the effect of off-policy sampling (especially async sampling, aka parital rollout) on SSVPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. By introducing game-theoretic principles, SSVPO provides a novel and intuitive framework for fairly distributing step-level rewards.\n\n2.  The experimental analysis is quite thorough. It includes results on multiple benchmarks; comparisons with both outcome-based RL methods and RL methods focused on credit assignment; and evaluations of sample efficiency and diversity (pass@n), which essentially cover the main interests of reasoning-oriented RL.\n\n3. The implementation details and visualization such as Figure 5 and Figure 6, help clarify the procedure of the algorithm a lot.", "weaknesses": "1. In the pass@K evaluation, there is no comparison against other RL baselines. In addition, pass@N is typically scaled exponentially (pass@{1,2,4,8,16}). Up to what extent can SSVPO maintain its exploration advantage?\n\n2. There is a lack of discussion of the training dynamics: how do key metrics such as val_score, response_length, and entropy evolve?\n\n3. The algorithm has certain limitations: the large number of sequences generated by reordering causes sample inflation, which in turn forces the number of reasoning steps to be capped at 3. To what extent does this affect training speed and performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)