Learning to Reason Efficiently with Discounted Reinforcement Learning
Authors: Alex Ayoub
Kavosh Asadi
Dale Schuurmans
Csaba Szepesvari
Karim Bouyarmane
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
reasoning
blackwell optimality
post training
Primary area: foundation or frontier models, including LLMs

Abstract: Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.

Published as a conference paper at ICLR 2026
LEARNING TO REASON EFFICIENTLY COUNTED REINFORCEMENT LEARNING
WITH DIS-
Alex Ayoub*
Kavosh Asadi Amazon
Dale Schuurmans University of Alberta
Amazon, University of Alberta
Csaba Szepesvári University of Alberta
Karim Bouyarmane Amazon
ABSTRACT
Large reasoning models (LRMs) often consume excessive tokens, inflating com- putational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses im- prove accuracy. By penalizing reasoning tokens using discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell opti- mality restricted policy classes, encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in stochastic shortest path problem. Experiments confirm theoretical results that this approach shortens chains of thought while preserving accuracy.
INTRODUCTION
Many sequential decision problems ask for policies that reach goal quickly while maintaining a high probability of success. Large reasoning models (LRMs) increasingly solve math and code problems by emitting intermediate reasoning tokens before final answer (Jaech et al.] (2024), where the goal correct terminal response. Reinforcement learning (RI. post training (Sutton Barto 2018) improves accuracy but can lengthen responses (Liuu et al. 2025a), raising inference cost and latency. Our objective is to train LRMs that reach correct answers effectively and efficiently: more concise reasoning with no loss in accuracy.
Longer chains of thought (Wei et al. (022) are not free: they inflate compute and memory (quadratic attention and growing key value (KV) cache), slow inference and reduce serving throughput. This makes the problem closely related to finding a stochastic shortest path (SSP): each additional reason- ing token an extra step, and we would like the shortest successful trajectory
correct terminal
state. Moreover, the role of length in accuracy is contested (Shao et al. 2024 Liu et al.) 2025h) Lu
al. 2025 Fatemi et al. 2025) with many claiming there an inherent tradcoff between length and accuracy. In this work we show that, up to regime determined by the model class and problem instance, there is no tradeoff between accuracy and path length. Namely, one can reduce response length up to certain instance dependent threshold without seeing a drop in accuracy. After the response length dips below this threshold, then accuracy begins to dip.
We model verifier based reasoning as a finite horizon Markov decision process (MDP) (Puterman 2014) with binary terminal reward, which can be viewed as a finite-horizon stochastie shortest path instance where the goal is terminal correctness and the path cost is reasoning length. We then train with a discount factor 7 < 1. This design is motivated by Blackwell optimality (Blackwell) 1962 Puterman 2014 Grand-Clément & Petrik 2023): near = 1, discounting should preserve accuracy (goal reachability) while preferring shorter successful trajectories. In practice, we only apply discounting to the environment (correctness) reward. The amount of discounting depends only on reasoning length, leaving intrinsic formating/shaping rewards undiscounted. Practically, we discount only reasoning tokens, regularize with KL penalty moving reference policy (Peters)
Work done during internship Amazon

---

Published as a conference paper at ICLR 2026
al. 2010) and ensure token budgets across methods are comparable for fair comparisons. Our can be summarized as follows:
Within any fixed (possibly restricted) policy class II, we show that Blackwell optimal poli- cies (optimal for all sufficiently close to 1) simultaneonsly maximize undiscounted suc- cess (probability of reaching the correct terminal goal) and, among accuracy maximizers, minimize expected trajectory length (a stochastic shortest path criterion). Thus, up to a regime determined by the class, there is no tradeoff between accuracy and path length. Our result calls into question the claim that there is a tradcoff' between accuracy and response length and establishes that one can shorten response length up to an instance dependent quantity as hypothesized by Lee et al. (2025)
For finite II, Blackwell factor "Tow < I exists such that y optimal policies are constant for all y E (Yow 1) and equal the Blackwell optimal set. We bound how close to 1 the discount must to maintain accuracy while shortening average response length. This clarifies how to choose 7 when the deployment class is restricted.
Using group relative policy optimization (GRPO) (Shao et al., 2024) with the discounted objective, substantially reduce mean response length GSM8K, MATH and addi- tional math benchmarks while matching the undiscounted pass @ baseline, in line with the shortest path prediction at fixed success probability.
Efficient goal-reaching (and, in particular, efficient reasoning) has been pursued via: (1) RL with length based penalties, which adds per token or per step penalties during policy optimization (Arora 2025 Su & Cardie 2025 Ling et al. 2025 Xiang et al.
& Zanette
2025); (ii) curated data ap-
proaches, which fine tune variable length compressed traces internalize concise reasoning
(Fatemi et al. 2025; Hammoud et al. 2025 ; Qiao et al. 2025 Lu et al. 2025 Zhao et al. 2025 Shri-
vastava et 2025 Dai et al.) 2025); ; and (iii) prompt control. which prompts the model to reason more conciscly Aggarwaal & Welleck (2025); Dumitru et al. (2025); Wu et al. (2025). We propose and analyze plain old discounting as principled, instance aware mechanism. In finite horizon MDPs with binary terminal reward, maximizing the discounted correctness reward and minimizing expected path length coincide as the discount factor approaches one. Moreover, a small per step negative reward in this setting is equivalent to discounting (Bertsekas) 2012). See Sui et al. (2025) for a broader overview of efficient reasoning methods.
2 SETTING AND NOTATION
We model reasoning as finite horizon discounted Markov decision process (MDP) (Puterman 2014] which is given by the tpll M action spaces, P S x A + A(S) is the transition kernel, r : A + R is a bounded reward (verifier), / is the horizon, 7 E [0, 1) the discount factor[] and E A(S) is the distribution over initial states (questions) where A(S) is the set of probability distributions over states.
H,,,H, Here and are finite state and
(possibly nonstationary) policy π = (m, (m)) consists maps TT (- 8) ((A) for each l. Fixing the start state, s, policy (or language model) induces distribution IP x, over trajectories
S1, A1, R1, ... Sn, An,
A ~ 7 ( R₃ r(S,, At), S41 P(St, Ar).
The (discounted) state value function of is
H
5 (s) = E
R
where Er,s is the expectation corresponding to PP - The weighted return is
(T) / (()s
We use 7 E [0, 1) for analysis; when defining accuracy also consider ? 1 finite horizon.
2

---

Published as a conference paper at ICLR 2026
2.1 LANGUAGE MODELING
In language modeling, actions are vocabulary tokens and states are token sequences. The next state the current sequence with the chosen token appended:
S = P(S., A.) S.A.
where we write zy for the concatenation of and y. The special action eos ends the episode and moves to an absorbing terminal state. After taking eos, the process remains in an absorbing state with zero reward for the remainder of the horizon. If eos not emitted by time H, we deterministically transition to terminal state that triggers the verifier.
In the so called RL with verifiable rewards (RLVR) (Lambert al. 2024), the verifier returns and only if the sequence at emission of eos contains correct final answer and 0 otherwise:
r St, eos) ((S,, contains a correct answer),
r(S,, a) = 0 for a of eos.
Under this reward, the undiscounted finite horizon return equals the success probability. We there- fore define the (Pass@ I) accuracy of as
Acc(T) = (T( = = P within H) p(ds),
i.c., the fraction of prompts (under μi) for which the first generated solution is verified correct.
BLACKWELL OPTIMALITY AND OUR MAIN THEORETICAL RESULTS
To formalize maximizing accuracy while minimizing mean response length, we use stronger notion of optimality than is standard in reinforcement learning: the notion introduced by Blackwell] (9622). henceforth Blackwell optimality (Puterman) 2014; Grand-Clément & Petrik 2023). A policy Blackwell optimal if it is optimal for all discount factors sufficiently close to one. This is relevant because the optimal policy in goal reaching MDPs (RLVR) at y maximizes success (accuracy), while- as we show below-the the optimal policy for < the one that reaches the goal via the shortest path. If policy is optimal both for < (near one) and for y 1, then it simultaneously maximizes accuracy and minimizes mean response length. The missing proofs of all our results can be found in our appendix.
Why Blackwell optimality? Discounting with y < 1 breaks ties between equally successful poli- cies by preferring earlier success. but if is not sufficiently close to it may instead prefer a shorter yet less successful policy. The following example, with a restricted stochastic three-policy class, illustrates both effects. We consider restricted stochastic policy classes this is simplified model of softmax policy classes which are standard when analyzing policy gradient methods (Sutton & Barto 2018 Mei al. 2020).
Proposition 3.1. Fix p E 10.1) and < 91 < 92 < 1, and consider the MDP Figure 7 with horizon H 2 4 and deterministic initial state 80. Let the restricted policy class be II { n2, T3). where at 80; for i E {1,2), n, selects ag with probability qi and 02 with probability
qG and ma selects 01 with probability p and as with probability 1 p. Let T(x) denote the time step which aend is taken under policy x. Then
J. (i Ji 1, E[r(m)) = + (i=14 1,2), Ji(TT) T(T3) = 2.
For all y E 1).
(T:) =(1 (1-9))²² 4 (i= 1,2), Ja(Ta) PY
Thus there exists a threshold 7' E (p. 1) such that for every y > 7 क. is both an optimal policy in II and shortest path policy. This example motivates Blackwell optimality: it selects the shortest policy among success maximizers (as f 1), while excluding policies that become optimal only by sacrificing success probability at smaller
3

---

Published as a conference paper at ICLR 2026
cont
Gend r
cont
0
dend;
0
0.4
Figure I: finit-hhorizon MDP illustrating the conffict between suceess probability and discounting. Green
(g) indicates the goal state (r = 1), while Red (f) indicates failure (r = 0).
Roadmap of the theory. Our analysis starts from a uniform Taylor expansion of (T( around Y = which makes explicit the stochastic shortest path (SSP) structure induced by discounting: maximize success probability and, among maximizers, minimize successful path length. We then relate this SSP objective to Blackwell optimality, and finally establish existence of Blackwell optimal policies for restricted finite policy classes.
We now introduce the formal definition Blackwell optimal policy. Recall that we assume finite horizon H < co, finite state and action sets, and bounded rewards.
Definition 3.2. Given y [0, 1), a policy T' E II. We call II, C the set of y discount optimal policies.
II is discount optimal if JT) 2 J. (TT' for all
Definition 3.3 (Blacwwell 1962)). A policy is Blackwell optimal if there exists 7 E [0, 1) such that æ E n for all y E 17, 1). We call IT. the set of Blackwell optimal policies.
Note that our definition of optimality is with respect to both an MDP instance M and a policy class II, whereas the usual notions of optimality (and the existence of optimal policy) (Puterman 2014 Bertsekas 2019 Szepesvári 2022) depend only on the MDP M.
3.1 MAIN THEORETICAL RESULTS
We now specialize to the setting of reaching goal (RLVR) where reward is deterministic binary terminal verifier.
Assumption 3.4. There exists a termination action Qterm E A (c.g., cos), an absorbing state sabs E S, and goal set G C S such that for all E S:
1. r(s, a) 0 for all + aterm:
2. taking Oterm transitions to the absorbing state, i.e. P(Sabs 8, (term) 1;
3. the terminal reward is deterministic and binary, r(s, Qterm) = I{s E G} E {0, 1}. More- over, the absorbing state yields no further reward and transitions itself: for all A,
Saba a) O, P(saba
Sabs: a) =
Let 7 < H be the (first) absorption time. Define the success probability and (conditional) successful path length
p(T)= = Pr, (success within H), L(r)= Er,a(T success).
with the convention that the product p(x) (L(xt) - 1) is interpreted as 0 when p(T) = 0. Call n a shortest path policy if it maximizes p(T)) and, among all maximizers p. minimizes L(T).
P+ == max p(T)) = 0, the shortest path condition reduces to the first criterion.
Lemma 3.5 (Uniform Taylor expansion). Let = 1- 7. Under Assumption 3.4 for every policy
JT(T)) ET [-- 11sccees)) = p(TT) (1 (I(x) - 11) Rr(E),
with remainder satisfying the uniform bound [R((e)] & CH e², where Cи (H 1) 2).
4

---

Published as a conference paper at ICLR 2026
Lemma 3.5 makes the SSP structure explicit: as 1, the leading term is p(T)) (accuracy), and discounting breaks ties among success maximizers using the next term, which depends on ((T) (successful path length). In other words, for sufficiently close to 1, maximizing J, implements the SSP objective "maximize success probability, then minimize (conditional) steps to success" (Bertsekas) 2012 Puterman 2014). We show that Blackwell optimality recovers the shortest- path SSP objective in goal reaching MDPs (RLVR).
Theorem 3.6. In finite-horizon MDPs with deterministic binary terminal verifier reward (Assump- tion 3.44 every Blackwell optimal policy is a shortest path policy:
hw S argmin L(T), n =Ellmaxy
where
Imaa p = argmax
sell
Theorem give our main theoretical guarantee. Namely that Blackwell optimal policies are accu- racy maximizing and have the shortest mean response length within the class of accuracy maximiz- ing (7 1) policies.
Proof intuition (sketch). Taking the Taylor expansion (Lemma 3.5 of J, - (z) gives
p(=*) p(T) (1 -1--10 (==()((()))()())))))
1) - ((=) - +
Since x² is Blackwell optimal it must be optimal for all y arbitrarily close to one. Thus if some π had p(*)) > p(x*), ) the leading term p(**)) - p(T) < 0 would make - < 0 for y close enough to 1, contradicting Blackwell optimality (Definition 3.3). Therefore π- arg max P(T). Moreover, among policies with p(T) = p(x=), the first term cancels and optimality for 7 + forces L(x*) < L(x).
Existence of Blackwell optimality for restricted classes. The SSP characterization above is use- ful only if Blackwell optimal policies exist in the restricted class II. We now adapt classical Black- well arguments (Blackwell] 1962, Zwick Paterson] 1996, Puterman 2014 Grand-Clément &
Petrik 2023) to the case where the admissible class restricted.
Assumption 3.7 (Finite policy class). The admissible class lI is finite: [II| < co.
Theorem 3.8. Given finite horizon MDP M, under Assumption 3.7 there exists 7 E [0, 1) and nonempty set 11 C such that for all ry E (7),
argmax JT( II
#Ell
Combining Theorem 3.8 with Theorem 3.6 yields that, in goal reaching MDPs (RLVR) with finite restricted policy thars,there exists a policy that is discounted optimal for all sufficiently close to 1, and this policy is shortest path policy in the sense above. We now introduce the Blackwell discount factor, first introduced by Grand-Clément & Petrik (2023).
Definition 3.9. The Blackwell discount factor is
7bw = inf { [0, II, = H1 V E7,1111))
where II; arg max (TT).
At a high level, the Blackwell discount factor Yow guarantees that any policy that is discount optimal for E [Tw, 1) is also Blackwell optimal. This reduces finding Blackwell optimal policy to solving for a discount optimal policy.
Instance hardness and uniform eritical discount. The quantity Thw is instance dependent since it depends on the MDP instance M and on the admissible class IT. This quantity captures how "hard" it is for discounting to reliably implement the SSP tie breaking without sacrificing success probability. Concretely, the closer 7ow is to 1, the smaller the margin separating the best SSP behavior from near-ties in 11, and the more carefully one must choose to avoid preferring shorter but less accurate behaviors.
Crucially, in setting each reasoning instance induces finite horizon MDP, so 7bw (M, II) for every instance. Moreover, for any finite collection of instances (M, NN one may define a single critical discount factor
Perit == max 7bw MM, II) < 1,
[
5

---

Published as a conference paper at ICLR 2026
and then any (Tarit: 1) is simultaneously "close enough to 1" for all those instances. In this sense, finiteness the problem family yields a single discount choice that preserves instance- wise SSP optimality across the entire finite set. We now state result that shows that for an arbitrary finite restricted policy class IT, the Blackwell discount factor exists.
Lemma 3.10. Civen finite horizon MDP M, under Assumption 3.7 the Blackwell factor "/ow exists and satisfies Yow < 1.
Proof. Theorem 3.8 ensures that 11;, is constant for all y sufficiently close 1, so the infimum in Definition 3.9 is well defined and strictly less than 1.
The next lemma establishes that for finite horizon problems, a Blackwell optimal policy must also be optimal for the undiscounted objective. Lemma 3.11. Blackwell optimal policy also optimal the undiscounted problem.
Proof. Suppose is Blackwell optimal: E [. Then for any policy n' we have (M)) (T)) 2 0 for all 7 E [Thw, 1). Therefore since (T) is well defined for finite horizon MDPs.
lim (T) (T')) 2 0.
We also know that J. - (=') is polynomial and therefore continuous. Thus, it must be that Jn (T)) J-=i (x') 2 0, i.e. π is also optimal in the undiscounted problem.
3.2 SOFTMAX TRAINING, GREEDY DEPLOYMENT
We now consider common setting in language model post training and deep reinforcement learn- ing where (Haarnoja et al. 2018). This setting is important as our experimental setup will train softmax poli- cies and evaluate their greedified variants. We fix deterministic tie breaking rule on A and define the greedification map on the states
softmax policies for training and then evaluate (or deploy) the greedified policy
Greed s) E arg max x(a (s) aEA
Vs S.
The deployment class is the image Σ == (Greed(T,-) stationary policies on the finite horizon MDP M. Each a E Σ corresponds one-to-one to a deter- ministic nonstationary policy. In our appendix, we also provide a bound on the Blackwell discount factor of the policy class Σ for completeness, see Theorem A.12 for more details.
E II,), subset of the deterministic
4 TRAINING METHODOLOGY
Guided by the theory in the previous section, we translate discounting into a practical training recipe for efficient reasoning with language models. Our design has four components:
1. Discount only the environment (correctness) reward. We apply a discount factor ? E (0, 1) to the environment reward but the learner's intrinsic formatiigg/saping reward. This preserves the incentive to produce well structured outputs while encouraging shorter, more efficient chains of reasoning.
2. KL regularization to changing reference policy. We KL regularization against reference model that is updated over training, following standard practice in policy gradient methods (Peters et al. 2010; Mei et al. 2020 Vieillard al.) 2020) Vaswani et al. 2022).
This viewpoint aligns with relative entropy policy search (Peters et al. 2010) and has also been adopted in recent language model alignment work (Gorbatovski et (025).
3. Discount only reasoning tokens. Discounting is applied exclusively to tokens used for reasoning; do not discount tokens required for prompt adherence, formatting, or final answer presentation.
4. Comparable token budgets across methods. To ensure fairness, we make token budgets across methods comparable: since discounting shortens reasoning traces, increase the number of rollouts for discounted methods so that the total tokens processed and hence training accuracy- comparable the undiscounted baseline.

---

Published as a conference paper at ICLR 2026
Objective. Because both the correctness and formatting signals computed only at the end of the trajectory, we use a sequence level return. Let my E (0, ] indicate whether token t is part of the reasoning span and define the number of reasoning tokens ((T) _ E. my, Let (T(r) be the unvironment/correeness reward and (T) the formatting/shaping reward, both evaluated at the end of the rollout 7. We discount only the environment reward function of reasoning length:
R(7)= K(r) (T) (T).
The learner then optimizes
Bs (St) [R(T)] B KL (TT x²),
(2)
where n' is reference policy that changes over training (defined below) and / > 0 sets the regu- larization strength. Equation ( applies discounting only through K(T), leaving formatting tokens undiscounted, in accordance with the Blackwell optimality perspective.
Implementation details. (i) Reasoning mask. The indicator my isolates tokens that perform latent computation (chain of thought tool use) from tokens required for formatting final answer emission. (ii) Reference updates. The reference n' = Ted is updated periodically (e.g.. at epoch or fixed step boundaries) to stabilize learning while allowing the target policy to improve. (iii) Comparable budgets. We report results under matched token budgets; if discounted training yields fewer reasoning tokens per generation, increase generations equalize total tokens before comparing accuracy. We now elaborate on each component.
4.1 EXTRINSIC VERSUS INTRINSIC REWARD
Extrinsic reward comes from the environment, whereas intrinsic reward is assigned by the learner to its own experience, usually to speed up learning exploration (Singh et al.) 2010: Barto 2012 Linke et al.] 2020). The goal maximizing correctness is extrinsic, since it comes from the en- vironment. By contrast, formatting rewards that encourage the learner to emit correctly structured reasoning and answer tags are intrinsic: they help the agent structure its reasoning and format the answer in way that satisfies the verifier. Only the correctness reward is necessary learn op- timal policy, but intrinsic rewards can guide the learner toward behaviors beneficial for learning. Since we care about learning Blackwell optimal policies, discount only the extrinsic correctness reward and leave intrinsic formatting rewards undiscounted. Popular frameworks that allow dis- counting, such as ByteDance's 's Volcano Engine Reinforcement Learning for LLMs library (Sheng et al. 2025), discount both extrinsic and intrinsic rewards.
4.2 KL REGULARIZATION
Discounting strongly nudges the model to shorten its answers. the policy moves too fast, it can collapse: it learns to stop early and forgets how to reason. We add KL penalty to a moving reference policy to keep updates small-like a trust region-so the objective changes gradually. The reference policy is not fixed: we periodically refresh it to the current policy the anchor follows progress without allowing single large drift. More specifically, every u training steps we perform
Tel stop_grad(T)).
the details of which can be found in Gorbatovski et (2025) in the TRL library (von Werra al. 2020
4.3
WHAT TO DISCOUNT
Discounting is applied only to reasoning (thinking) tokens:
[r]
K(T)
ma-
I(token t lies in the reasoning span).
1=1
In our experiments, we delineate the reasoning spans using explicit tags injected by prompting (c.g., creasoning: </ </reasoning>), Tokens required for prompt adherence, formatting and
7

---

Published as a conference paper at ICLR 2026
the final answer segment have m, 0 and thus are not discounted. Empirically, discounting the entire response slightly hurt accuracy (about a 0.5%-1.0% drop on GSM8K): the model would occasionally drop formatting tags required by the verifier or respond with an answer that was too short (e.g., dropping zeros from long integers).
4.4 COMPARABLE TOKENS
Discounted policies produce shorter traces, so for the same number of epochs (or passes over prompts) they experience fewer transitions/samples than undiscounted policies. This can make dis- counted methods look worse simply because they saw less data, not because the objective is inferior. To keep comparisons fair, whenever this discrepancy mattered during training we adjusted the num- ber of generations: either increasing generations for the discounted method or, when more sensible, decreasing generations for the undiscounted method so that the total samples/tokens observed were comparable.
In some settings, the discounted method still matched the undiscounted baseline despite seeing fewer samples-au informative robustness result. In others, we ensured sample counts were comparable to make a fair judgment.
Practical notes. (i) Choosing 7 In light the Blackwell analysis, we select 7 as far from as possible while preserving undiscounted training accuracy] This can be accomplished via sim- ple bisection search, adjusting until accuracy matches (or begins to dip below) the undiscounted training accuracy. (ii) Updating the reference policy. We choose the update frequency via abla- tions nanely, we find the best update frequency and | that maximize the undiscounted model's accuracy and apply the same values the discounted methods. (iii) No algorithmic change required. Any policy optimization algorithm-e.gg REINFORCE (Williams) 1992) and variants (such as RE- INFORCE Leave One Out (Ahmadian et al.,
2024 can be used with Equation [ our contribu- tion is training with the discounted return in Equation ( together with the masking and budgeting
rules above. In what follows, we employ GRPO as our policy optimization method.
5 NUMERICAL EXPERIMENTS
We empirically validate our theoretical prediction that discounting incentivizes efficient reasoning in large language models. Recall from Theorem3.6jthat in deterministic verifier MDPs, a Blackwell optimal policy prioritizes correctness and, among equally correct strategies, minimizes expected trajectory length. Our experiments test whether this pattern appears in practice when post training language models using GRPO.
Setup. We finetune and evaluate four instruction tuned models: Qwen2.5 7B- IB-Inttr and Qwen2.5
14B- Ins-tust (Yang et al., 2025). Llama 3 Instruttttt (Grattafior et al.) 2024) and Phi-4 (Abdin
et al. 2024), post trained via GRPO with and without discounting. The undiscounted case (7 1) optimizes correctness only, whereas ? < additionally rewards shorter successful trajectories. We evaluate grade school math (GSM8K) (Cobbe et al. (222I) and MATH (Hendrycks et al.) 2021 for Qwen2.5 7B and Llama 3 8B. We then train the larger Qwen2.5 14B and Phi-4 models on a subset of the DeepScaleR math dataset (Luo et al. 2025) and evaluate
AMC 2023, AIME 2025, MINERVA (Lewkowyez et al.] 2022) and OLYMPIAD (He et al. (202)) to test generality. We report Pass@ @1 I and mean response length. Pass @ 1 is the fraction of problems for which the first generated solution (one sample per prompt) is judged correct by the verifier. In our setting, the average Pass @ is the accuracy.
Implementation and benchmarking. We use Hugging Face TRL for GRPO fine tuning and vLLM (Kwon et al. 2023) for inference. inference, we use greedy decoding (temperature v = 0), consistent with Theorem A.12 We select Qwen2.5 7B-Instruct and Llama 3 8ns-rrtt estab- lished basclines for sanity checking our implementation and verify that our reimplementations meet or exceed published numbers on GSM8K and MATH. For Qwen2.5 7B-Instruct we compare against
PIn our empirical setup, we first tune the hyperparameters maximize the performance of the undiscounted method, then apply discounting with these hyperparameters.
8

---

Published as a conference paper at ICLR 2026
Dataset
Model
Undise. Pass@1
Undisc. Len
Disc. Pass@1
Disc. Len
GSM8K
Qwen2.5 7B-Instruct
91.06
217.60
91.07
170.08
Llama 8B-Instruct
80.87
125.43
81.07
108.67
MATH
Qwen2.5 7B-Instruct
64.80
491.32
64.55
384.96
Ilama 3 8B-Instruct
24.48
328.43
24.75
257.73
Table 1:
GSM8K and MATH: Pass@1 and mean response length (tokens) for discounted vs. undiscounted GRPO. Averaged over training seeds and 10 evaluation seeds per model; evaluation seeds are fixed across methods for paired comparisons.
VERL's official baselines; for Llama 3 SB-Instruct we follow Roux et al. (2025) Minor differences
may arise because we average multiple training and evaluation seeds, whereas prior ports use single seed estimates. For GSM8K we limit completion length to 786 tokens; for MATH to 2048 tokens; and for DeepScaleR to 4096 tokens.
Variance control and reporting. To obtain sta- ble estimates, we repeat each training run with random training seeds and, for each trained model, evaluate with 10 independent sampling seeds on GSM8K and MATH; report averages over 3 10 runs per condition and fix evaluation seeds across methods for paired comparisons. For AMC 2023, AIME 2025, MINERVA and OLYMPIAD, we aver- age over five evaluation seeds per model. This mat- ters because RL style post training and decoding in- troduce variance (Patterson al. 2024 He Lab) 2025) and single seed reporting can be misleading for both Pass and length statistics. When sweep- ing 7 we select and report a single discounted con- figuration per model/dataset using the following cri- terion: among all discounted settings whose train- ing Pass@ matches or exceeds that of the undis- counted run, choose the with the shortest mean response length. All tabled metrics are then computed the evaluation seeds for the selected configuration.
88
600
87
500
Accuracy (%)
85
400
84
300
200
82
100
1.0e-7 2.50-7 Discount (1-y)
5.0e-7
Figure 2: GSM8K accuracy (blue, left) and tokens
(orange, right) discount (1 - -)).
Main results. Tables and show that, on average over seeds, discounted models match the accu- racy of undiscounted ones while producing shorter responses. For example, on GSM8K, discounting reduces mean response length by 22% for Qwen2.5 7B Instruct and by 13% for Llama 3 Instruttttt with an insignificant change in Pass 1. This aligns with Theorem 3.6 which predicts shortest path behavior at fixed success probability. The trend holds for the larger models evaluated on datasets distinct from their training set. Specifically, the DeepScaleR math dataset does not contain problems from OLYMPIAD, MINERVA, or AIME 2025; however, does include problems from AMC prior to 2023. Across architectures and datasets, we consistently observe that discounting enforces length minimization subject to maintaining accuracy.
Effect of the discount factor: We run additional experiments with Qwen3 1.7B Yang al. 2025)
on GSM8K to examine performance as function of 7. For these runs, we increase the completion length limit 1536 because outputs were frequently clipped for being too long. As shown in Figure _ varying y confirms the predicted tradeoff: smaller reliably shortens responses but can reduce accuracy. Theory explains this: for y close to policies first maximize correctness: overly aggressive discounting shifts probability toward shorter trajectories even when that harms success.

---

Published as a conference paper at ICLR 2026
Model
Dataset
Undisc. Pass@1
Undisc. Len
Disc. Pass@1
Disc. Len
Phi-4
AMC 2023
51.00
1134.30
61 .1.
716.29
AIME 2025
14.00
1263.87
19.33
800.09
MINERVA
28.46
553.74
29.85
318. 10
OLYMPIAD
36.91
1059.92
35.67
707.64
Qwen2.5 Inssrustttt
AMC 2023
50.00
737.47
59,50
582.31
AIME 2025
10.00
891.43
10.67
699.56
MINERVA
27.21
522.14
27.43
437.31
OLYMPIAD
35.13
797.57
34.76
684.02
Table 2: @ass@1 1 and mean response length (tokens) for undiscounted discounted GRPO. Averages over 5
evaluation sceds per model.
CONCLUSIONS AND FUTURE WORK
We studied efficient reasoning in verifier based MDPs through the lens of Blackwell optimality (Blackwell 1962;
& Petrik (023). Within restricted policy classes, we showed that y sufficiently close to there exists Blackwell optimal policy that maximizes undiscounted suc- cess and, among accuracy maximizers, minimizes expected trajectory length. For softmax training with greedy deployment, the induced deterministic deployment class is finite and admits a bounded Blackwell discount factor; we provide an explicit upper bound on how close to 1 the discount must be. Guided by this theory, we proposed a practical recipe: discount only the environment reward as function of reasoning tokens, keep intrinsic formatting rewards undiscounted, add KL regularization a moving reference policy (Peters al.) 2010) and ensure comparable token budgets. Empiri- cally, discounted GRPO matches @ass@1 accuracy while substantially shortening responses across math benchmarks. Our theoretical results extend to methods that introduce small per token penalties in finite horizon MDPs with binary rewards (verifiers) (Bertsekas) 2012). suggesting that several length penalty methods (Arora & Zanette 2025 Su & Cardie 2025 Xiang et al. 2025) recover the same accuracy then length ordering in the near undiscounted regime when properly implemented. This further sheds light on adapting to the inherent token complexity of a given question (Lee et al.) 2025): choosing y within the Blackwell region steers the learner toward the shortest successful tra- jectories allowed by the class without sacrificing accuracy. Some of our empirical results suggest that discounted methods can achieve higher accuracy with shorter reasoning traces. An interesting avenue for future work is to investigate whether shorter, more compressed reasoming improves gen- eralization on reasoning tasks. As argued in Hutter (2007), compression (or prediction) is linked to improved generalization; whether this extends to compressed reasoning traces remains open, An- other direction to study whether methods that promote longer reasoning (Liu al.) (0055) can be combined with methods that shorten reasoning: longer reasoning promotes path finding, while shorter reasoning promotes path compression. A pipeline that first uses longer traces to discover strategies and then compresses them (akin to distillation (Hinton et al. 2015) ) may yield stronger reasoning policies.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08900 2024.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint 2tt:::5033046977 2025.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pictquin, Ahmet Üstün, and Sara Hooker, Back to basics: Revisiting reinforce-style optimization for learn ing from human feedback in llms. In Annual Meeting of the Association for Computational Linguistics, 2024.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502. 04463, 2025.
10

---

Published as a conference paper at ICLR 2026
Andrew G Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learn- ing in natural and artificial systems. Springer, 2012.
Dimitri Bertsekas. Dynamic programming and optimal control: Volume I. Athena scientific, edition, 2012.
Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.
David Blackwell. Discrete dynamic programming. The Annals of Mathematiical Statistics, 1962.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.141 2021.
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.0700 076666 2025.
Razvan-Gabriel Dumitru, Darius Peteleaza, Vikas Yadav, and Liangming Pan. Conciserl: Conciseness-guided reinforcement learning for efficient reasoning models. arXiv preprint arXiv:2505.17250 2025.
Mehdi Fatemi, Banafsheh Rafice, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504. 05185, 2025.
Alexey Gorbatovski, Boris Shaposhnikov. Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. In International Conference on Learning Representations, 2025.
Julien Grand-Clément and Marek Petrik. Reducing Blackwell and average optimality to discounted Mdps via the Blackwell discount factor. Advances in Neural Information Processing Systems, 36: 52628-52647 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, 2018.
Hasan Abed Al Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long, think short: Curriculum learning for efficient rea- soning. arXiv preprint arXiv:2508.088 089440 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Associa- fion for Computational Linguistics, 2024.
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm inference. Thinking Machines Lab: Connectionism, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 1XXv:2103.03874 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Mareus Hutter. Universal algorithmic intelligence: mathematical top-> down approach. In Arti- ficiil general intelligence. Springer, 2007.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722 2024.
11

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint
(a::::241..5222 2024.
Ayeong Lee, Ethan Che, and Tianyi Peng. How well do Ilms compress their own chain-of-thoughte a token complexity approach. arXiv preprint arXiv:2503.00 01 141, 2025.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 2022.
Zehui Ling, Deshu Chen, Hongwei Zhang, Yifeng Jiao, Xin Guo, and Yuan Cheng- Fast on the casy, deep on the hard: Efficient reasoning via powered length penalty. arXiv preprint arXiv:2506. 10446, 2025.
Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White. Adapting behavior via intrinsic reward: A survey and empirical study. Journal of artificial intelligence research, 2020.
Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking di- verse reasoning in Ilms via prolonged training. arXiv preprint arXiv:2507. 12507, 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.2488 24864, 2025b.
Jinghui Lu, Haiyang Yu, Siliang Xu, Shiwei Ran, Guozhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, et al. Prolonged reasoning is not all you need: Certainty- adaptive routing for efficient Ilm/mlim reasoning. arXiv preprint arXiv:2505.1 15154, 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffirey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning (ICML), 2020.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in rein- forcement learning. Journal of Machine Learning Research, 2024.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI Con- ference on Artificial Intelligence, 2010.
Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang. Lai Wei, Guanbo Wang, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: Confidence-guidee compression in step-by-step efficient reasoning. arXiv preprint 2xXiv:2505.04881, 2025.
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-LLuufr, Sándor Toth, and Sam Work. Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for 1ms.. arXiv preprint 1rX2 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint axXiv:2402.03300, 2024.
12

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. European Conference Computer Systems, 2025.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample concise reasoning. arXiv preprint arXiv:2508.099 09726, 2025.
think less: Group filtered policy optimization for
Satinder Singh, Richard Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2010.
Jinyan Su and Claire Cardie. Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards. arXiv preprint 2x.x:22005.18296 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419 2025.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. The MIT Press, 2018.
Csaba Szepesvári. Algorithms for reinforcement learning. Springer Nature, 2022.
Sharan Vaswani, Olivier Bachem, Simone Totaro, Robert Müller, Shivam Garg, Matthieu Geist, Marlos C Machado, Pablo Samuel Castro, and Nicolas Le Roux. A general class of surrogate functions for stable and efficient reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 2022.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Remi Munos, and Matthieu Geist. Leverage the average: an analysis of KL regularization in reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforce- 2020.
ment learning. htttps://githiii tiggggthu..... hwwpo//hhgggnggaaccg////aamaa//e
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in Neural Information Processing Systems, 2022.
Ronald J Williams. Simple statistical gradient-followngg algorithms for connectionist reinforcement learning. Machine learning, 1992.
Tong Wu, Chong Xiang, Jiachen T Wang, G Edward Suh, and Prateek Mittal. Effectively controlling reasoning models through thinking intervention. arXiv preprint 2rXhv:2503.24370, 2025.
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning, arXiv preprint trXiv:2506.05256, 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Kai Zhao, Yanjun Zhao, Jiaming Song, Shien He, Lusheng Zhang, Qiang Zhang, and Tian- jiao Li. Saber: Switchable and balanced training for efficient Il reasoning. arXiv preprint 10020 2025.
Uri Zwick and Mike Paterson. The complexity of mean payoff games on graphs. Theoretical Computer Science, 1996.
13

---

Published as a conference paper at ICLR 2026
A OMITTED PROOFS
Proposition A.1. Fix E (0,1) and 0 < 91 < 92 < 1, and consider the MDP in Figure 7 with horizon H 2 4 and deterministic initial state so. Let the restricted policy class be 11 { 22, T3 where at so; for i E 21, n; selects with probability and 02 with probability - qi and TT selects a1 with probability p and a₁ with probability 1 p. Let T()) denote the time step which Gend is taken under policy n. Then
(((())) = (1 =1,
B[T(T,)] = 3 1,2), (((TT) = =p, T(Ts)= = =2
For all y E 10.
(x: = = -:) + 07 (1= 1,2), J,( Py
Thus there exists threshold 7 E (p. 1) such that for every y > y/, Tu is both an optimal policy in and a shortest path policy.
Proof. Under n; for i E [1,2], the process terminates successfully with reward at time 7 3f d2 is chosen (probability 1 - q:) and at time = 4 if a3 is chosen (probability 4). Hence Ji JT,T,) 1.
[[r();)] = 3(1 q:) + 4g: = 3 + qs and
(n,) = (1 q:)n² + q:7"
Under T3, Gend is taken at T(Ts) 2 and yields reward 1 iff a1 was chosen f 1, which occurs with probability p. hence (Ta) = py and (TT)
For 7 < 1,
(1) - (T(22) [ 92 + 9 - [(1 g2) 927 =42 - = (1 7) 0,
so discounting always prefers 71 over T2. Now consider
((7) = -+((1)) - (T)) = (1 - 92)²² 9272 -py p 7(1 917 917 -p)
Let f(1) = (1-11) $07?
Then
(7) = (1 - +1) 2917 2 1 - 91 11>0 >0,
so f is strictly increasing on [0, 1]. Moreover.
(()) = (1 - ạ1 )p qıp² - = 1pp(1 - <0, [(1) = (1)= 0,
so there exists unique 7th (p.1) with f((h)) 0, i.e. ((Tt)) 0. For 7 Thh have
f(7) > 0 and hence J, > (Ta), while (TT) ) > Jr holds for all 1. Thus for every
7. > 7hh कi 7-optimal in II; since (TI) J1 (mT) = 1 p Ji (TT) and B[T(TI)) 3 9 <
3 - 02 = B[T(T2)), Ti is also a shortest path policy among accuracy maximizers.
We adapt classical Blackwell arguments (Wwick& & Paterson, 1996 Puterman 2014 Grand-Clément
& Petrik 2023) to the case where the admissible class is restricted. Throughout this section assume finite horizon < oc, finite state and action sets and bounded rewards. For policy π and y [0, 1), define the (discounted) value
2 (8)
ΣT'1RR
Rd
J+(T)= = (s)
We first handle finite admissible class and then specialize greedy deployment policies induced by a softmax training class.
Assumption A.2 (Finite policy class). The admissible class II is finite: [II| < oc.
Definition A.3. Given 7 E [0, 1), policy क E is 7 discount optimal if JT, 2 (T('') for all T' E HI. We call II; C II the set of discount optimal policies.
Definition A.4. A policy is Blackwell optimal if there exists y E [0, 1) such that II, for all 7 E h; 1). We call Th the set of Blackwell optimal policies.
14

---

Published as a conference paper at ICLR 2026
Lemma A.5. For any n, II, the difference (n) == - (="(")) is polynomial in r of degree at most 1. Consequently has finitely many roots in [0, unless it identically zero.
Proof. Linearity of expectation yields (T) Σ 7t - (T) with ct(TT) = L [Re], which is independent of Subtracting (= and applying the fundamental theorem of algebra to E (x((=)))=)))))))))))(()))))) (c(()) yields the result.
Theorem A.6. Under Assumption that for all y E (1',
A.2 there exists 7 E [0, 1) and a nonempty set lo S Il such
argmax [T(T) -T TEll
Proof. By Lemma A.5 each pairwise difference )n,x is a polynomial of degree most H 1. For each ordered pair IT, x²) with A * O, let ZT.=' {n E 1), Ar, (7) = 0). which is finite. Define
U (x,7"): #0 ZT,''' which is finite. Set 7 max I (or if 0). For any
(y', 1) and any pair TT', either Ax,s' 0 or it has zeros in (7': 1). hence it has constant sign on that interval. Therefore all pairwise comparisons between J, and (x') are fixed on (),, 1). It follows that II;; is constant (r', 1); denote this common set by Iw Nonemptiness follows from finiteness of II.
Definition A.7. The Blackwell discount factor is
Thw inf {7 [0, IT, Ilw VY (2:1)).
where 11; arg ma J, (T).
Lemma A.8. Under Assumption A.2 the Blackwell factor Yow exists and satisfies "/o < 1.
Proof. Theorem A.6 ensures that IT; is constant for all y sufficiently close to 1, so the infimum in Definition A.7 is well defined and strictly less than 1.
A.1 SOFTMAX TRAINING, GREEDY DEPLOYMENT
Let II, be the (possibly infinite) class of softmax policies used during training. We use the stan- dard time-augmented, stationary, infinite-horizon representation of the finite-horizon problem with horizon H. Define the augmented state space:
$ = {(8 () S S, E ......,H) U {absorb),
and the stationary transition kernel P and rewards by
P((s', (((ss,++1)11 (s,(),a) P(s' 8, (L< H),
P(absorb (s, H). 1, P(absorb absorb, a) = 1,
F(ss,t). a(s,a) (tS i(absorb, a) 0.
The initial distribution on augmented states á with (((s,,) 1)) pls) and zero elsewhere. A
(possibly nonstationary) finite-horizon policy = (T.) induces the stationary policy
(a (s, ()) = (a (LS H),
# abborbb arbitrary.
We fix a deterministic tie breaking rule on A and define the greedification map on augmented states
Greed(z, (s, ()) E arg max (a (s) aeA
V(s,1) E S (I....,H..
The deployment class is the image Σ = (Groed(a,
E II,), subset of the deterministic corresponds one-to-
stationary policies the augmented MDP M P, f.). Each one to a deterministic nonstationary policy on the original depth-H decision tree.
Lemma A.9. The set Σ is finite. In particular, if Nnodes is the number of reachable decision nodes up depth H in the original tree, then [ < |A| Nuodas.
15

---

Published as a conference paper at ICLR 2026
Proof. Finite states and finite horizon imply finite reachable decision tree. A greedy policy assigns exactly one action to each reachable node (equivalently, to each reachable augmented state (s, t) with t S H), so the number of labelings is at most [A] A
For any policy class II, let "ow (IT) denote the Blackwell discount factor given that class in the (augmented) stationary MDP. By Theorem A.6 Σ, obtain:
Corollary A.10. There exists Tow (E) < 1 and nonempty set Ibw S Σ such that arg max (T) = for all r E (7hw (2)w(2),1) (2),
Setup For stationary deterministic policy on (S, let Pa and be the induced transition matrix and reward vector on S. Define the y-weighted discounted return through the augmented value equation
ex = ra + P.
(T) = =
so that, for any finite-horizon policy and its image under time-augmentation, the objectives coincide: J. (finite-hoiizon л) = (stationary स) for all 7 E [0, For polynomial p(X) Σ ag X write the coefficient extractor [X*]p- = Og-
For π, x' in the admissible class II (we will take II = Σ), set
(x(T, x') == max E [0,1): A (", 1 = 00.
with the convention (T,, x²) = 0 the above set is empty or JT - (+²) = 0 on [0, 1). We
aim upper bound
7
max [M( m'). x
(If one restricts to a subclass IT' C replace by IT' everywhere; the bound below only becomes easier.)
Assumption A.11. There exists m e N such that for any (s, s') E S Axs,
a(s'|s,a)= n(s, a,
m
with n() E Zzo, n(s,a, 8') S m and
w(s a)
r(s, a)
m
with w(s a) E and [w[s a)l S roo-
The augmented kernel P and rewards F inherit this structure. Let Dj. minft Noo t Z3)
be the least positive integer such that t j is integer-valued.
Theorem A.12. Under Assumption A. for any rational μ E A(S) define
N 2| -
2 (S) roo in 2|5 41
Tp 2N N/2+2 (L,, +lu+0 1)
Then, with 7 maxx, (TT,
7 np
Proof. All objects are on the augmented state space S. By Cramer's rule (Lemma A.1 of Grand- Clément Petrik (,023), for any deterministic æ we have
8.
(57(s) d((,T)
d(T, det(1 7P7),
707, det (M(7,,, s)),
where M M 8, π) is formed by replacing the s-th column of 1 PPa by ra- Writing ū(),)) E.es (()) n17,,777 8, T), we get
(T) = 8(7, π. d(7, m)
p(7)
J.(T) Jx(T') = d(7, क x)((d)))))
16

---

Published as a conference paper at ICLR 2026
with
p()) i(7, d(7,, x') n(9, x²) d(y,
By Lemma A.2 of Grand-C Clément & Petrik 2023 d(7, T) > 0 on [o. 1) and by Lemma A.3, p(1) = O. Since deg v < |S| 1 and deg < S|. we have degp S = 2 -
By Proposition A.6 of Grand-Clément & Petrik (003)). m n(, π has integer coefficients and
81,,8, - |S] no m 1 21
Thus ml D2 n TT) E. (Daã(s)) ml n(:, s, has integer coefficients and coefficient-sum
most Dú r] m 2| By Proposition A.S of Grand-Clément & Petrik 22023), m ml)id(s π)) has integer coefficients and
N
(m
( d(.,x)) x)) < 2
Applying Proposition A.7 of Grand-Clément & Petrik 2023 to the two products defining p(7) and summing, we obtain that
p()) m Da p(7)
has integer coefficients and
| Σ
(Dx r ro m111 (ml512 51) Ly-
The degree of ý is at most and ÿ shares roots with p. By Theorem A.8 in Grand-Clément & Petrik 2023), any two distinct roots of integer-coefficient degree-N polynomial with coefficient-sum
< Ln are at distance least 7/2 [2 N N/2+2 (L,, +1)+] -1 the set in the definition of of (T,, x') empty, then (T, x²) = 0 and the claim holds trivially. Otherwise. 1 and N (T, x') ) [0. 1) are
distinct roots, hence 7/ (TT T') S Ty. Maximizing over म, x' E Σ gives 7 < 1 Mp.
Corollary A.13. For any E' Σ, the same bound holds with 7 replaced by max (T, xt).
Assumption A.14. There exists termination action Oterm E A (e.g., eos), an absorbing state Sabs E S, and a goal set G C S such that for all E
r(8, a) =
a + atermi
2. taking Oterm transitions to the absorbing state, i.c. P(Sabs 8, (term) 1;
3. the terminal reward is deterministic and binary, r(s, Oterm) I{s G} E {0, 1}. More- over, the absorbing state yields no further reward and transitions to itself: for all A,
r (abba a) O, P(sabe
Sabe. a) = 1.
Let < H be the (first) absorption time. Define the success probability and (conditional) successful- path length
p(T) = Pr,, (success within H),
L(T) = En,ulT success],
with the convention that L(Tr) is only evaluated when p(T)) 0. Call shortest-path policy if it maximizes р(τ) and, among all maximizers p, minimizes L(x). If p. = max, p(T) 0, the shortest-path condition reduces to the first criterion.
Lemma A.15. Let 1 7. For every policy
(L(T) Er [7 1 [ssccess)] = (1 (Z(x) -1) + R*(E),
with remainder satisfying the uniform bound [R((e)] S Cy e², where Cи (H - 1) -2).
17

---

Published as a conference paper at ICLR 2026
Proof. Since the reward is 1 only upon successful termination at time T,
J, = B [-- [success]] = = E - (1-=)--11 success]
where 8 = 7. For any integer n E {O...., - 1}. Taylor's theorem gives that for some E (0,e),
(1-8)==1- ne + nn(n 1)(1 - m-
which, since & E [0, 1), implies
((1 - [1" (1 [1=ne)| $ i/ 1)
Setting n = E {o,................. - 1} and conditioning on success yields
[1 - -) |success] = - (L(T) - 1) + -(5,-1(()) |success],
with ()-1((=) < (T 1) 2)=² < CHE². Define R2 (E) p(x) E conclude.
success] to
Theorem A.16. In finite finite-horizon MDPs with deterministic binary terminal verifier reward (As- sumption A.14). every Blackwell optimal policy is shortest path policy:
1l S argmin L(T), where TEllmaxy
11 p = argmax P(T).
xell
Proof. Let nT" bw For any II and E -1-
7, Lemma A.15 gives
Jn(=*) - (z( p(TT') p(T) (L(T') 1) ((T) 1)) + R.) R. (s),
(A)
(8)
(C)
with (C) S 2CHe². ((T) > p(=*), then for sufficiently small e > 0 the RHS negative, contradicting optimality of x* for arbitrarily close to 1. Hence p(r*)) 2 p(x) for all x, i.c.,
rt E m_
Now fix any E 1lmax that p(x) p(m*)= = p.. If L(T) < L(x') then (B) (L(Tt))
L(T))= 0 and for small enough the negative term -s (B) dominates the Oss²) remainder, again contradicting optimality. Therefore L(x*)= S ((T) for क E Imax
The same argument applies with II replaced by any subclass (e.g., the finite deployment class E)-
Corollary A.17. In the time-augmented reasoning MDP, the Blackwell-optimal deployed policies satisfy
E arg min L(o), aE F
Emaxp = arg augmaxp(o). aE..
Equivalently, for Y sufficiently close 1, the ryddisoonttdd optimal policies in E. are exactly the shortest successful-ppn policies.
18


Reviews:
[{"review_id": "e31TKP9B8v", "reviewer": "Reviewer_id2e", "summary": "LLM reasoning models generate verbose intermediate reasoning to solve complex problems, incurring high computational costs. While reinforcement learning improves accuracy, it often lengthens responses. This paper challenges the assumption that this accuracy-length trade-off is unavoidable (as discussed in prior works).\n\nThe authors apply discounting (γ < 1) to the reward function, motivated by Blackwell optimality from decision theory. They prove that within any fixed policy class, such a policy exists, establishing that no inherent trade-off between accuracy and reasoning length occurs up to an instance-dependent regime. The key practical insight seems to be to discount only the environment (correctness) reward while leaving intrinsic (formatting) rewards undiscounted. Combined with KL regularization and discounting only reasoning tokens, this approach substantially reduces response length without sacrificing accuracy.\n\nTo back up the theoretical work, the authors conduct experiments on math benchmarks (GSM8K, MATH, AMC, AIME, MINERVA, OLYMPIAD) and demonstrate 22% length reduction for Qwen2.5 7B and 13% for Llama 3 8B while maintaining pass@1 accuracy, even when controlling for total tokens processed during training.", "questions": "* How dependent is the \"no trade-off\" result on the quality of the base model and the difficulty of the problem? Could this approach only be effective for sufficiently large models on sufficiently simple problems and fail to scale as a general training paradigm?\n\n* Why has no one already tested this approach empirically given that discounting seems elegant and theoretically motivated? I am mainly asking as you mention ByteDance's existing implementation (Sheng et al. 2025) that already seems to do something similar, and if so, what exactly is novel here?\n\n* Beyond maintaining pass@1 accuracy, what is the impact on reasoning quality (coherence, correctness of intermediate steps)?\n\n* When problem difficulty increases and discounted RL is applied, does training become less effective or does the mean chain-of-thought length simply increase again, negating efficiency gains?\n\n* In practical deployment across multiple tasks, should one use a universal discount factor or does γ need to be tuned per dataset or per problem?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper makes a principled design choice to discount only extrinsic (correctness) rewards while leaving intrinsic (formatting) rewards undiscounted, which is well-justified both theoretically and practically.\n\n* The theoretical exposition clearly communicates Blackwell optimality concepts and the main results, making the mathematical framework accessible (I am not a hard RL person and found it pleasant to read). The paper provides thorough theoretical analysis that comprehensively covers finite policy classes, softmax training, and greedy deployment scenarios.\n\n* The experimental methodology is good (e.g., model selection), with results averaged over multiple seeds to properly control for variance inherent in RL-style post-training. The efficiency claims are validated even when matching token budgets across methods, ruling out the confound that discounted methods simply see less data. Ultimatively, the results consistently support the theoretical predictions across multiple benchmarks and model architectures.", "weaknesses": "The paper \n* evaluation is limited to math benchmarks (GSM8K, MATH, AMC, AIME, MINERVA, OLYMPIAD); it remains unclear whether these findings generalize to coding tasks or other reasoning domains.\n\n* theory section relies on the deterministic transitions assumption, but the paper does not specify whether training uses stochastic sampling (T>0) or greedy decoding (T=0), creating a potential theory-practice gap.\n\n* assumes the policy class is finite, which is needed for the existence of the Blackwell discount factor, but it is unclear how realistic this assumption is or whether violations would significantly impact the results.\n\n* has no systematic analysis showing where the difficulty cutoff lies, i.e., at what problem hardness the efficiency gains from discounting diminish or disappear.\n\n* applies a single discount factor γ per dataset but does not justify whether this is appropriate or whether different problems within a benchmark should use problem-specific discount factors.\n\n* does not discuss why discounting intrinsic rewards would be problematic, missing an opportunity to strengthen the design justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ss2yQqcjk9", "reviewer": "Reviewer_mEWe", "summary": "The  problem addressed in the paper  is that RL post training improves\naccuracy but lengthens responses, raising computational cost and latency. It\nsays that the response can be reduced to a threshold without reducing the\naccuracy.\n\nThe authors have modeled a verifier-based reasoning as a finite horizon MDP\nwith determinist transitions and binary terminal reward. And then they train\nwith discount factor (γ < 1). Here only the environment reward is discounted,\nleaving   the   formatting/shaping   reward   undiscounted.   Also,   only   the\nreasoning tokens are discounted, regularized with a KL penalty to a moving\nreference policy. This ensured that the token budgets comparable.\n\nThey show that in a fixed policy class Blackwell optimality policies maximizes\nundiscounted   success.   And   among   accuracy   maximizers,   minimize   the\nexpected trajectory length. They reduce average response length on GSM8K,\nMATH and other benchmarks while matching pass@1 base line Using GRPO\nwith discounted objective with (Group Relative Policy Optimization) with\ndiscounted objective. \n\nWhen tested on Quen 2.5 7B-Instruct and Lama 3 8B-Instruct models, the\nresponse length was reduced in discounted method. When compared to\nundiscounted method the GSM8K dataset showed 22% reduction in response\nlength while MATH dataset showed 13% reduction. The Phi4 and Qwen 2.5\n14B-Instruct models with AMC 2023, AIME 2025, MINERVA and OLYMPIAD\ndatasets also showed reduction in response length.", "questions": "1)\tIn addition to binary verifier rewards, have you tried Dense reward models or Hybrid rewards?\n2)\tInstead of a bisection search for the selecting γ, will a predictive hybrid approach provide a better result?\n3)\tBesides reduction in response length, did the latency and computational cost get reduced?\n4)\tFor GSM8K and MATH datasets you limited the completion length to 786 tokens & 2048 tokens. What happens when the runs hit the limit? Have you tried with larger limits?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe four design components used here are Discounting only the environment reward, Regularizing KL to a changing reference, Discounting only reasoning tokens and Comparable token budgets across methods. These components are simple to adopt and are also defended in the theory.\n2)\tIn the finite restricted policy classes, for γ close to 1 the Blackwell optimal policies are accuracy maximizing and have shortest mean response length. This is proved in the theorems 3.4, 3.7 and 3.10 which strengthens the claim.\n3)\tThe GSM8K dataset showing a 22% reduction and MATH datasets showing a 13% reduction in response lengths is significant when considering computational cost and latency.", "weaknesses": "1)\tModeling as a finite horizon with deterministic transitions and binary terminal reward worked theoretically. But many real-world reasoning workflows violate these assumptions.\n2)\tIn Blackwell optimality analysis, the γ is selected as for from 1 as possible. However, this is done by a simple bisection search.\n3)\tDiscounting is only applied to reasoning tokens. The authors say that discounting entire response slightly hurt the accuracy. This suggest there may be some issues that prevents errors frond being detected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qOTZh13dYf", "reviewer": "Reviewer_GRgL", "summary": "The paper deals with efficient reasoning language models, specifically solving problems with the minimum number of required reasoning tokens. The key idea is that using discounting, i.e. the standard technique of discounting the reward in time by a multiplicative factor, can result in an efficient policy. The authors give theoretical results using the idea of a Blackwell optimal policy. They show that in deterministic verifier problems with finite horizon and binary rewards, Blackwell optimal policies are shortest path policies. Experimentally, they show that discounting can maintain pass@1 while reducing length in their tested settings.", "questions": "Please respond to the points contained in the review above, including these questions:\n\n1. Can you provide experimental comparisons with other efficient reasoning methods to help judge the relative benefits of your approach?\n2. How does the method perform in settings where the original model produces much longer reasoning traces?\n3. Can you add discussion of the key intuition for why Theorem 3.10 holds and why discounting leads to an efficient policy?\n4. Do the theoretical results extend to stochastic sampling, multi-step agents, or real-valued rewards?\n5. Can you clarify the training data used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper makes a nice connection between discounting and reducing trajectory length in the context of reasoning language models.\n- The authors provide a formal underpinning to the connection.\n- Their recipe for training a model seems reasonable, and leads to a reduction in reasoning token usage in their tested settings.", "weaknesses": "- The experimental results do not seem to address the standard long chain-of-thought setting. For instance, in Table 1 the lengths are in the hundreds, and in table 2 the lengths are typically 1000 or less. These inference budgets are lower than those typically used to evaluate long chain-of-thought models. Based on the results it's unclear whether the proposed methods work for these settings. \n- There are many related methods for efficient reasoning models; the area is a very active research area. However, no comparison with other methods is done in the experiments. \n- The theoretical results are presented in a way that did not give me much insight. For instance, the main result Theorem 3.10 is stated but there is not a discussion of the key intuition for why it is true. In addition to the formalized results, it would be helpful to extract the intuition for why discounting leads to an efficient policy, e.g. through a less dense argument.\n- Based on the assumptions in 3.10 it is unclear whether the results would hold in settings like multi-step agent or tool calling settings that may have noise in environment transitions or rewards, or settings with real-valued rewards.\n- The theoretical results and the experiments use greedy decoding. However using stochastic sampling is a standard approach when evaluating reasoning language models and their efficiency techniques.\n- The training data is not clear.\n- The paper briefly mentions that \"similar conclusions can be drawn for methods that assign a reward proportional to the response length\". It implies that prior length penalty methods have the same properties, meaning that the primary contribution is the theoretical contributions in this paper rather than practical methods. In that case, it would be beneficial to expand the results and experimental analysis to the length penalty methods in addition to the discounting methods.\n\nOverall there is an interesting idea here and I like the simplicity of the method, but the experimental validation is below the bar for ICLR for reasons described above. I also think the scope could be improved by elaborating on the length penalty methods, and the accessibility could be improved by providing insight into the key ideas for why the theoretical results are true.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HbPYwA9uEC", "reviewer": "Reviewer_Dg1G", "summary": "The paper investigates reasoning efficiency in verifier-guided reinforcement learning by casting reasoning as a discounted MDP. The key theoretical result (Theorem 3.10) states that Blackwell-optimal policies not only maximise expected reward but also minimise the expected time until the reward is obtained. This motivates a practical training scheme where only reasoning tokens are discounted, implemented with GRPO and KL regularisation. Empirically, the approach reduces average reasoning length while maintaining Pass@1 across several mathematical reasoning benchmarks.", "questions": "1. Could the main result (Theorem 3.10) and associated guarantees extend to general finite MDPs with binary rewards, and can the need for the contrived finite “deployment” policy class be eliminated?\n2. Did the authors attempt to evaluate the tightness or practical relevance of the theoretical bound on the Blackwell discount factor?\n3. How does the proposed method compare empirically to other approaches aiming to produce shorter responses?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The problem---achieving concise yet correct reasoning---is well motivated and practically relevant.\n* Theorem 3.10 is a clear and intuitively appealing formalisation of how discounting encourages shorter successful trajectories.\n* The proposed method is simple to implement and shows consistent empirical effects across model scales and datasets.\n* The paper is easy to follow, and the theoretical exposition appears rigorous and technically careful.", "weaknesses": "* *Limited theoretical novelty.* Section 3.1 essentially restates known results on Blackwell optimality and the connection between discounted and average-reward MDPs (e.g. Grand-Clément & Petrik 2023). Although Theorem 3.10 might not have been stated explicitly before, follows straightforwardly from the classical theory.\n* *Restrictive and unnecessarily complex assumptions.* The deterministic-MDP setting appears more restrictive than necessary: the result should extend to general finite MDPs with binary rewards (i.e. reachability problems). The definition of a finite “deployment” policy class via greedified softmax policies also seems artificial and more complicated than required for the theoretical guarantees.\n* *Unused and uninformative bound.* It is quite clear from standard convergence arguments that *some* bound on the Blackwell discount factor must exist. The explicit bound in Theorem 3.9 is never used or evaluated, yet it makes the exposition noticeably more cumbersome. Since the experiments determine $\\gamma$ by naive search, it would have been more interesting to assess how tight or informative this theoretical bound actually is.\n* *Missing empirical comparison.* The experiments do not compare to other approaches designed to shorten model outputs. Without such baselines, it is difficult to gauge the practical benefit of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)