Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs
Authors: Jingyao Wang
Wenwen Qiang
Zeen Song
Changwen Zheng
Hui Xiong
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
Reinforcement Fine-Tuning
Process Reward Function
Primary area: deep_learning

Abstract: Large language models (LLMs) excel at complex tasks thanks to advances in their reasoning abilities. However, existing methods overlook the trade-off between reasoning effectiveness and efficiency, often encouraging unnecessarily long reasoning chains and wasting tokens. To address this, we propose Learning to Think (L2T), an information-theoretic reinforcement fine-tuning framework for LLMs to make the models achieve optimal reasoning with fewer tokens. Specifically, L2T treats each query-response interaction as a hierarchical session of multiple episodes and proposes a universal dense process reward, i.e., quantifies the episode-wise information gain in parameters, requiring no extra annotations or task-specific evaluators. We propose a method to quickly estimate this reward based on PAC-Bayes bounds and the Fisher information matrix. Theoretical analyses show that it significantly reduces computational complexity with high estimation accuracy. By immediately rewarding each episode's contribution and penalizing excessive updates, L2T optimizes the model via reinforcement learning to maximize the use of each episode and achieve effective updates. Empirical results on various reasoning benchmarks and base models demonstrate the advantage of L2T across different tasks, boosting both reasoning effectiveness and efficiency.

Learning to Think: Information-Theoree Reinforcement Fine-Tuning for LLMs
Jingyao Wang Wenwen Qiang!.2 Zeen Song Changwen Zheng! Hui Xiong
IIstitute of Software Chinese Academy of Sciences, University of Chinese Academy of Sciences,
'The Hong Kong University of Science and Technology (Guangzhou).
"The Hong Kong University of Science and Technology
(vangjingyao20233, qiangwenwen, songzeen (rngzeen)@iscas... @iscas ac cn, xionghui @ust hk
Abstract
Large language models (LLMs) excel at complex tasks thanks to advances in their reasoning abilities. However, existing methods overlook the trade-off between reasoning effectiveness and efficiency, often encouraging unnecessarily long rea- soning chains and wasting tokens. To address this, we propose Learning Think ( an anformation-theoretic reinforcement fine-tuning framework for LLMs make the models achieve optimal reasoning with fewer tokens. Specifically, L2T treats each query-response interaction as hierarchical session of multiple episodes and proposes a universal dense process reward, i.e., quantifies the episode-win wise information gain in parameters, requiring no extra annotations or task-specific evaluators. We propose method to quickly estimate this reward based on PAC- Bayes bounds and the Fisher information matrix. Theoretical analyses show that it significantly reduces computational complexity with high estimation accuracy. By immediately rewarding each episode's 's contribution and penalizing excessive updates, I.2T optimizes model via reinforcement learning maximize the use of each episode and achieve effective updates. Empirical results various reasoning benchmarks and base models demonstrate the advantage of L2T across different tasks, boosting both reasoning effectiveness and efficiency.
1 Introduction
Large Language Models (LLMs) have progressed from handling basic natural language processing tasks tackling complex problems, such writing and maintaining code bases 23
67
38
48
navigating the web and controling devices 666
[44] 17 and acting as personal assistants
21
30
22 7). thanks the advances in their reasoning abilities. Recent results 224
19
56
4 in L.M
reasoning show that scaling test-time compute can substantially improve reasoning capabilities, e.g..
gains. Based on this, a new class of reasoning models 43 53 49) has coupled test-time compute scaling with reinforcement learning (RL), achieving state-of-the-art (SOTA) results various challenging benchmarks [15 61 16]. These models employ -hain-of-thouglee (CoT) tokens to guide multi-step
35 5) demonstrated that generating more tokens during inference yields logarithmic-
reasoning and maintain logical consistency throughout the solution process [50] 52 58 by extending and optimizing CoT paths to produce trajectories longer than typical correct solutions, they more thoroughly explore the solution space and thereby boost final answer accuracy
Despite existing methods having demonstrated great performance, they still struggle to balance reasoning effectiveness and efficiency. Specifically, existing approaches typically rely final
"Equal contribution.
I Corresppnndig author.
'Project page: http:///aaaginggggo07.iiittu
io/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

outcome rewards for policy optimization, providing no feedback on intermediate reasoning steps. Under such delayed feedback, extending the reasoning chain does not incur any cost, and even a tiny accuracy gain from large amount of extra reasoning steps is treated as positive signal [59 55
Consequently, the models favor a "one more thought" and continually lengthen their CoTs, resulting in redundant computation and thus reducing reasoning efficiency. Our experiments in Subsection 3.2
further demonstrate this (Figure[]) []: existing uutcome-reward-bare based RL methods often lead LLMs to consume more than twice the tokens actually needed for the correct answer. Furthermore, by evaluating across different reasoning tasks, we find that this redundancy not only wastes resources but sometimes degrades reasoning effectiveness. For example, multi-stage math questions [33). moderate chain extensions improve coverage of critical steps; whereas simple tasks (e.g.. Tier 1 question "12 + 5"), overly long reasoning chains may reduce overall accuracy. Since real-world tasks vary, no fixed chain length is optimal for all cases. Therefore, designing effective dense process rewards to assess the contribution of each reasoning step is both necessary and valuable. Such rewards help the model to generate tokens that most benefit the answer, ensuring reasoning effectiveness with minimal token budget and efficient learning.
difficult questions (e.g.. Tier 4
To this end, we propose Learning to Think (L2T), an informatioonthhoretti reinforcement fine-tuning framework for LLMs. At its core, L2T proposes universal cettmmttionnthhorete dense process reward, which quantifies the information gain in model parameters. The proposed reward consists of (i) fitting information gain term that drives the model to capture prrecttess-crttica information in each update; and (ii) a compression penalty that discourages overly optimization. further preserving efficiency. By treating each question-answe
pair session of multiple episodes and immediately rewarding each episode, it makes the model focuses on the process progress, thus curbing redundant reasoning steps and the resulting computational waste. This reward is independent of input format, label type, or task domain, and no extra annotations are needed. We leverage this reward to train the LLM (also the policy) via reinforcement learning to make generate the tokens that best contribute to the answer correctness at each reasoning step. Specifically, L2T includes three stages: (i) Problem reformulation (Subsection :.II: we treat each question-answe of multiple episodes, where each episode represents a segment of the reasoning chain that underpins dense reward calculation and optimization; (ii) Reward design (Subsection 4.2 upon episode completion, we calculate the information-theoom
interaction
hierarchical session
reward via PAC-Bayer bounds and the Fisher information matrix. Based on this, we halt unproductive reasoning and thus balance effectiveness and efficiency; (iii) LLM fine-tuning (Subsection 4.3): we optimize the LLMs by maximizing cumulative reward across tasks via reinforcement learning, ensuring reasoning effectiveness and efficiency.
Empirically, across challenging reasoning benchmarks (c.g., AIME, AMC, HumanEval) and base models (e.g., DeepScaleR-I.ST --.5lleRR--.---eeeeeee -.SBB-rreveew -erpssee-RR--Dsiill----e-ee 1.5B, DeepSeekR1 I-Distill- Qwen-7B), L2T consistently achieves comparable performance and stable improvements (Section S Compared to standard outcome-reware approaches (e.g.. GRPO), boosts performance by about 3.7% and doubles token efficiency; compared to process-rewan baselines (c.g.. ReST-MCTS, MRT), raises accuracy by about 2% and increases efficiency by about 1.3x. In multi-task evaluations. L2T delivers an average accuracy gain of nearly 3% across tasks of varying difficulty and maintains stable improvements under different token budgets. These results demonstrate the advantages of L.2T, which effectively balances reasoning effectiveness and efficiency across diverse reasoning scenarios.
The main contributions are as follows: (i) We explore the trade-off between reasoning effectiveness and efficiency and propose Learning to Think (L2T), an information-theorelt reinforcement fine- tuning framework for LLMs. L2T decomposes each interaction into successive episodes and proposes a dense process reward to quantify each episode's 's performance gain; by optimizing LLM via episodic RL, it adaptively allocates reasoning depth across different tasks, enabling effective reasoning with limited but sufficient token budgets. (ii) We propose a universal information -theoretic process reward based on internal model signals, i.e., the information gain in model parameters, eliminating the need for external annotations or specialized evaluators. Leveraging PAC-Bayes bounds and the Fisher information matrix, we derive a scalable approximation of the intractable information gain with theoretical guarantees. (iii) Across diverse complex reasoning benchmarks and base models, L2T consistently achieves great performance, delivering boosts both effectiveness and efficiency.
2

---

2 Related Work
Reasoning of LLMs Complex reasoning has long been recognized as one of the most challenging 56 43 To enhance inference, several works [43] 49 [7] have incorporated outcome-reward RL during fine-tuning. RL paradigm mainly extends and optimizes CoT paths based on test-time compute scaling to more thoroughly explore the solution space [5 S
capabilities for LLMs [23 67 19
However, recent studies [59 9 55 S show that reasoning length and accuracy are not strictly positively correlated. Excessively long CoT's not only consume undue tokens but also degrade performance (also demonstrated in Subsection (.22). The wasting token budget reduces the efficiency of unit tokens under a limited budget due to problems such as attention dilution and context truncation, so that the final accuracy may decline [31 Some methods attempt to reduce reasoning depth via process rewards 663 36), heuristic scoring
20 or length penalties [57] [3]. These approaches, however, require task-specific evaluators, incurring prohibitive annotation costs with poor cross-tadd task reuse (with more comparison in Appendices D and G). To address these limitations,
propose a universal information-theoretic dense process reward and leverage reinforcement fine-tuning to adaptively recognize reasoning depth across diverse tasks. This design achieves the trade-off between reasoning effectiveness and efficiency without additional annotations or specialized evaluators.
Process Reward Models Unlike outcome rewards that assess the final answer, a PRM evaluates the correctness of each intermediate reasoning step [29 47). PRMs can be trained via automated supervision without costly coman-annotated process labels [4 39 60). Once learned, a PRM can
both guide test-time search by allocating additional compute and accelerate exploration in RL using
64 have applied process rewards to fine-tune L.LMs with RL, for example, (65] uses relative progress estimation to generate high-quality intermediate supervision labels; [57] derives exploration bonuses from length penalties or an LLM-based judge: and (36) introduces a minnmiimniiiittionn process reward to optimize test-time compute. However, these methods depend on external annotations or task-specific evaluators, which are expensive to produce and difficult to reuse when task requirements change. In contrast, our approach uses internal model signals, i.c., information gain, as intermediate rewards to optimize the policy without additional interaction data. By constructing a universal dense process reward, our framework applies scamlessly across diverse tasks and promotes efficient, step-wiss wise reasoning.
the policy's 's own trajectories 43 35 60] Recent works 42
65
63
10
3 Problem Settings and Analysis
3.1 Problem Settings
Our goal is to fine-tune an LLM so that can answer arbitrary questions æ within a fixed token budget Broken: enforcing efficient reasoning. We treat the I.IM as stochastic policy Ta(- x) parameterized by 0. Each question x is drawn from an underlying distribution Pa. Under the token budget Broken. the model may generate up to T Tmaz tokens at test time, and yields output sequence Z0:T = (zo, .1, for each æ. The generation process can be formulated as a finite-horizon Markov decision process (MDP): each time step f, the "state" st consists of the question together with the partial prefix (žu...., 2t-1), and the "action" a, is the choice of next token 2t. To guide learning toward correct answers, the MDP is equipped with reward function r(s,, a,). It is typically defined as a binary outcome reward 16. 20 to determine whether the generated answer is correct.
According to existing RL-based fine-tuning paradigm 41
we are given training dataset Durain = (a uz ) where each ye an oracle reasoning trace that leads to the correct answer. During fine-tuning, we use these traces both to calculate the reward and to guide learning. For each question 24, we first generate candidate token sequences 20:T ~ Tt(- xz). Then, we compute the reward r(z,, =0.T), which equals 1 if zo:T matches the oracle trace v," By maximizing the empirical sum of these rewards with the constraint of test-time budget Broken we update ng. The objective is:
43 16
max EnwDrra Ezgr 20:)) s.t. E E |2| < Btoken
Through D we train the L.LM my to capture both the need to produce correct answers (maximize the rewards) and the requirement of using limited token budgets (under fixed compute budget).
3

---

40
Tler2
16000
Tier
35
Tier?
Token Concumation
11400
Tirrk
Tiae4
Accuracy (%)
12000
10000
6000
4000
2000
10
20
25
10
15
20
Episode
Episade
(a) Reasoning accuracy episode number
(b) Token consumption episode number
Figure 1: Results of DeepScaleR- 1.50- 1.5B-Preview across different tasks Omni-MATH. We partition the generated reasoning chain into episodes, measuring accuracy Acc(k) and average token consump- tion (1)) at different episode depths. More details and results are shown in Appendix G.1
3.2 Empirical Evidence
To ensure reasoning effectiveness, existing methods 43 9 53
15 49 mainly leverage test-time compute scaling to lengthen reasoning chains beyond what is minimally required for a correct answer, therehy thoroughly exploring the solution space and improving final answer accuracy. These approaches typically use a sparse outcome reward for policy optimization, without feedback on intermediate reasoning steps [59] (55). Using this framework, extending the chain carries no penalty, and even minimal accuracy gains from extra steps yield a positive signal. Thus, models resort to consuming additional tokens secure the correctness of the final answer 33 However, under the fixed token budget Btoken considered in this paper, this extension may deplete the budget prematurely, undermining efficient reasoning in subsequent steps. To validate this argument, this subsection, we conduct a series of experiments to analyze the token- utilizaaionn abilities of existing methods.
use the Omni-MATH benchmark 13 for evaluation, which comprises 4,428 questions across more than 33 domains, including algebra, discrete mathematics, geometry, number theory, etc. The questions are organized by human experts into four difficulty tiers (Tier 1 1-4) to assess model performance on tasks of varying complexity. We evaluate two base --eeee-----iitil---e--eeee
Specifically, in this experiment,
running on the A100 GPU clusters with greedy decoding (temperature 0) and a maximum generation length of 16,384 tokens. To probe different reasoning depths, we configure the prompt to wrap each logical step with '<think>'... "</think>'' tags, segmenting the generated chain into up to K = 30 episodes. For each test question and each truncation point 1..... ....,,3 .0,0 we force-stop generation at the k-th '</think>' tag, append the prompt "Please output the final answer directly based on the above steps", and then use Omni-Judge determine correctness. This yields the accuracy Aco(k) at episode depth k and the average token consumption ((k) from the initial prompt to the truncation. Considering the impact of randomness, we introduce maj@4(k) baseline: for each truncated context, we sample four continuations, take the majority-vor result as the final answer, and record its accuracy maj@4(k) and token cost. By plotting "accuracy vs. episode number" and "token consumption vs. episode number", we can visualize how performance evolves with reasoning depth across tasks of differing difficulty and compare sequential generation against majority-vote under the same compute budget.
models [16]. i.e., DeepScaleR- 1.5-Prrvie and
From Figures we can observe that: (i) Existing methods may fail to use test-time compute budgets efficiently, leading to wasted resources: both models have on average used more than twice the minimum tokens required. For example, & 16 achieves accuracy comparable or exceeding sequential generation at l 24 with fewer tokens. (ii) The additional episodes add no new information and instead degrade performance due context redundancy: for both models, Acc(k) peaks around k = 16 20 and then declines as k increases. (iii) The questions of different difficulty tiers prefer different chain lengths: Tier questions tend to benefit from longer chains, whereas Tier questions can achieve correct results with short chains, where excessive reasoning depth may causes a marked accuracy drop (e.g., falls by over 5% at = 20). These findings underscore the limitations of existing methods, which ignore the balance between reasoning effectiveness and efficiency.

---

3.3 Motivation Analysis
For obtaining a powerful LLM to address the above limitations under the settings in Subsection 3.1 in this subsection, we discuss the solutions that need to be incorporated to address the realistic challenges of existing methods. Based on these analyses, we design our framework (Section 4
As illustrated in Section and Subsection 3.2, we discuss and demonstrate that, although outcome rewards can boost reasoning effectiveness by increasing the token budget, sacrifices efficiency, i.e., many reasoning steps are invalid for the answer correctness; much longer CoT's are used than the correct answer really needed. To balance reasoning effectiveness and efficiency, we aim for an algorithm that yields positive gains at every reasoning step, enabling the model to achieve comparable reasoning performance within a constrained yet sufficient token budget. This requires augmenting the learning objective with dense, step-wise process rewards that immediately quantify each reasoning step's s contribution to overall performance. By maximizing the cumulative reward across all reasoning steps, we encourage the model to generate tokens that most benefit the answer correctness. Therefore, the algorithm requires an effective dense process reward to ensure both effectiveness and efficiency: by maximizing total reward across all reasoning steps, we guarantee the reasoning effectiveness; by maximizing each token's ' contribution and preventing useless tokens, we ensure efficiency. Notably, some concurrent approaches attempt to reduce reasoning depth using task-specific process rewards 67] and length penalties (31; however, they require manually designing high-que high-quality process labels and task-specific evaluators, which is an expensive endeavor that may not generalize across tasks. Therefore, we must also address the second challenge of algorithmic generality.
Generality demands that the algorithm adapt to varied task requirements and remain effective. To achieve this, one would ideally define evaluation metrics that apply uniformly across all scenarios to construct reward functions. However, the heterogeneity of tasks makes it impractical to identify single, fixed metric suitable for every case. Accordingly, we propose leveraging an internal model signal, e.g., the change in parameters, to quantify the contribution of intermediate reasoning steps. This measure directly reflects the amount of new knowledge the model acquires on the current task and is agnostic to input formats, task types, etc. Consequently, it enables sustained, reliable reward feedback without requiring additional annotations or retraining when new tasks are introduced.
In summary, the above analyses motivate us to design dense process rewards derived from internal model signals to jointly address both algorithmic efficiency and generality. By optimizing LLMs with these rewards, we consider both reasoning effectiveness and efficiency across different tasks.
4 Learning To Think
Based on the above analyses, we propose Learning Think (L2T), an information-thhoretie rein- forcement fine-tuning framework for LLMs (with pseudo-code in Appendix The key is proposing a universal dense process reward for LLM optimization to adaptively allocate reasoning depth across tasks and prevent token waste. It recasts LLM optimization as an episodic learning problem to support episode-wise wise process reward calculation and optimizes LLM to maximize the contribution of tokens generated in each episode, ensuring reasoning effectiveness and efficiency. Specifically, in L2T, each query-response pair segmented into successive reasoning episodes (Subsection (4.1). within which the model performs an adaptation update to increase the likelihood of a correct answer. To ensure the effectiveness of each adaptation and curb excessive reasoning, we then propose an information-theorrtiic dense process reward that immediately quantifies the progress of each episode with universal information gain to support policy optimization (Subsection 4.2). It can be efficiently estimated leveraging PAC-Bayes bounds and the Fisher information matrix with theoretical guar- antees (Theorem 4.2). Finally, we optimize the LLMs by maximizing cumulative reward via RL (Subsection 4.3) ensuring both high reasoning effectiveness and efficiency across different tasks.
4.1 Problem Reformulation
In practice, reasoning questions vary widely, e.g., from simple arithmetic to complex mathematical
mainly treats the reasoning process of each question single episode and updates the policy via the final outcome reward. Under this setting, is difficult to implement dense process rewards. To address this issue, we reformulate the problem mentioned in Subsection 3.1 an episodic RL problem. We treat each question as a task sampled
proofs. Existing RL--based fine-tuning paradigm [51] 45
5

---

from broader distribution and decompose the reasoning process into successive episodes. This decomposition allows us assign process rewards at every intermediate episode, with each episode's reward reflecting its contribution to the answer correctness. Based on this dense reward, we can optimize the policy incrementally to maximize the performance gain of each episode.
Specifically, each question x is viewed as a reasoning task x ~ Prask. For a given œ, the LLM carries out an internal inference procedure and emits a sequence of tokens 20.T (20.... (......, 2T), as its final answer. To inject dense feedback, we insert '<think>... </think>' markers to break the full token stream into K consecutive reasoning episodes. Each episode shares the accumulated context but serves as a natural checkpoint at which we can evaluate intermediate progress and assign episode- wise rewards. We treat the entire reasoning process as a length- K MDP: at episode k. the state is 8i:- = (x, 21:k-1); )) the action 2 ~ TT(: 8t)) produces a token sequence 2 = (zt
2x ). The reward consists of (i) dense process reward r which measures the increase in the answer correctness probability after episode k; and (ii) a sparse outcome reward your = r(x, =1.k) E {0, } that reflects the final answer's correctness (Bq I. Note that we evaluate r ,.p. at the episode level rather than per token to (i) reduce variance: individual tokens rarely determine final correctness and are easily corrupted by stopwords, sampling noise, etc.; (ii) lower cost: episode-level evaluation reduces expensive calls to the reward function. During fine-tuning, given the training set Drrain = {( yt ))
we optimize the policy Ty by maximizing the rewards across all the Z4. with objective:
max Egw-Derann B por [romt
K Lk=ITh
s.t. (a |2| S Broken:
where Bioken is the fixed test-time compute budget. Through Eq the learned policy allocates its limited token budget (Broken) where it yields the greatest incremental benefit, expanding promising lines with higher process reward yP and maximize overall task success your, In practice, however, the crux is the design of the newly introduced dense process reward rP*K, which need to satisfy following desiderata: (i) Relevance: faithfully measure the progress that episode contributes toward correct solution; (ii) Efficiency: be cheap to compute; (iii) Generality: apply uniformly across tasks without bespoke engineering for each new domain. Recently proposed process-reward models 663 (36 62 677 predominantly focus on (i); however, they remain task-specific and depend on high- quality annotations, failing to satisfy (ii) and (iii.. Therefore, designing such dense, per-episode reward remains open challenge, also the key to unlocking truly efficient reasoning in LLMs.
4.2 Learning CnpemaationThheorede Dense Process Reward
To address the above challenge, we propose a novel information -hoomatton-thhoorticc dense process reward. is inspired by the information theory B 25 and consists of two constraints for reasoning effectiveness and efficiency: (i) a fitting information gain, which encourages the model to acquire key information about correctness in each episode; and (ii) a parameter compression penalty, which penalizes redun- dant information absorbed at each episode to maintain efficiency. This reward is agnostic to input format task domain, which can be applied in various scenarios. In this subsection, we begin by explaining the meaning of the above two constraints with the proposed reward. Then, we provide the formal definition of this reward (Definition 4.1p. Next, explain how to efficiently compute it in practice: to handle the large parameter scale of Ls, we develop an efficient approximation of this reward using PAC-Bayes theory and Fisher information matrix (Theorem 4.2). Finally, we illustrate why the proposed reward is effective, i.c., satisfying the three criteria mentioned in Subsection 4.1
Firstly, we explain what the two components of the proposed reward are. Specifically, the fitting information gain measures the reduction in uncertainty about Y given X provided by parameters 0 after each episode; formally, it is defined as the conditional mutual information I(e: Y|X) X - H(Y| 6). where (Y|])) = - Ey P(v) X) log p(y) (X) denotes the uncertainty of Y given X alone and H HY 0) the residual uncertainty after observing 0. The fitting gain for episode k. in which parameters update from Oz-1 1 to 0z. Ale 1 (0g; Y XX) I (0x-1; Y [X). Since direct computation is costly in LLMs, we approximate A by the increase in the predicted correctness which aligns with the direction of Ale (Appendix D.I.. It entails just two forward passes of Ta without the need of gradient updates to estimate A reducing computational overhead. The parameter compression penalty constrains redundant information captured from each episode, preventing excessive updates. It is defined as the mutual information between 0 and the context 8t, denoted as T(0: $+) = [.s(KL.((00{s))) pl))) quantifies the task task-specific idiosynerasies stored in 0 [14], where larger mutual information implies
probability of the models, i.e., AI x J. -x))-)(((

---

greater overfitting risk and unnecessary computational overhead. Thus, these terms align with our objective of evaluating reasoning effectiveness and efficiency in dense process reward design. More discussions, theoretical analyses, and the intuition behind are further illustrated in Appendix D
Based the above analyses, then present the formal definition of the proposed reward.
Definition 4.1 Let the context before episode k be = (x, 21:k-1), and the model generate the token sequence zk TT(. |s)). The dense process reward for episode can be expressed as:
rk Jr (To Sk: 2x)) J. sk)) -8 [(k:: 8k) 1(0k-1;8k--
Fitting Information Gain
Purameter Compression Penalty
where J,(.) denotes the correctness probability and B > 0 is a hyperparameter.
Eq indicates that the larger k the update of this episode is effective: the larger the first term, the greater improvement to predict the correct answer, increasing effectiveness; the smaller the second term, the less redundant information is absorbed, ensuring efficient and sufficient updates.
Obtaining Definition 4.1, we illustrate how to calculate it in practice. In the LLM setting, fitting information gain measures the contribution of each optimization episode to reasoning capability by tracking the change in the predicted correctness probability, i.e., the output distributions of TT. In contrast, computing the mutual information increment between the parameters and the historical context 8k, where direct estimation in the large parameter space of LLMs is intractable. To address this, we introduce efficient approximation that the low-rank parameter proxy 0 with singular value decomposition 37 to estimate the penalty term. We get:
penalty is more involved: it requires quantifying the
(SVD) 6 and the Fisher information matrix
Theorem 4.2 Given the low-rank parameter proxy 0g and - for parameters 0g and fg-1, assume that 0g and 0 follow Gaussian distribution, e.g., p(0x) = N (0xlu,, Ex) where is the mean vector of the parameters and Σg is the covariance matrix of the parameters, we get:
100g su) 1000-1888 1784 (0, 0n-1)T (Va. log (=(22 Vá, log =o((uls)))) (0n 02_1) (4)
Theorem 4.2 presents a method to estimate the intractable compression penalty (with proof in Appendix By. simplifies the parameter space using SVD and assumes 0 follows Gaussian distribution based on [6]. common and mild assumption. Note that the low-rank approximation of 0 E Rd. i.c. obtaining 0 R² (r < d), is avoid direct computation of the Fisher matrix in high-dimensional space. We use SVD to extract the principal directions variation in 0 and retain the top components (with r/d 1%-10% for 1.5B models and r/d 0.1% 0.1%-11% % for 7B models), resulting in low-rank surrogate Ô. Then, by computing the second derivative of the log-likelihood of 0, we obtain the Fisher information matrix, which captures the effect of 0k updates on the output and approximates the covariance calculation (Lemma B.1) The mutual information increment is then approximated using the second-order term of the Taylor expansion. This method significantly reduces the computational complexity (Theorem D.2) with limited approximation error (Theorem D.3) to support the computation of our proposed reward Definition 4.1 in practice.
Finally, we explain why the proposed reward effective. satisfies the three criteria mentioned in Subsection 4.1 (i) for relevance: the fitting gain term measures how much episode & improves the reasoning correctness, which is tightly aligned with task progress; (ii) for efficiency: it requires only one call for estimation per episode with Theorem 4.2) and the cost scales linearly with the number of episodes; (iii) for generality: neither term depends on task-specific models, just on the model's 's own correctness scores and parameter information gain, so the reward applies uniformly across tasks.
4.3 Optimizing LLM with Reinforcement Fine-Tuning
Based on the above-defined problem settings (Subsection 4.1p and proposed reward (Subsection 4.2). in this subsection, we introduce the optimization process of policy ve (i.e., the LLM).
Specifically, based on the reformulation Subsection 4.1 under the RI. framework relied upon by L2T, the optimization objective of the LLM can be decomposed into two parts: (i) for answer
7

---

correctness: maximizing the cumulative outcome reward your obtained after sequence of reasoning episodes to ensure the correctness of the answer; and (ii) for process progress: using the dense process reward to evaluate the improvement in correctness probability and the increase in model parameter information gain after each reasoning episode. This term is designed to capture the progress made at each episode of reasoning and optimize the model to maximize the incremental gain at each reasoning step. Among them, (i) corresponds to the standard fine-tuning objective in Eq while (ii) depends on the process reward defined in Subsection 4.2 Therefore, the objective can be expressed as:
mAx Bez-Deeoin R (-w) [r( 21:K)
>x= k= 1 Pr8
(5)
s.t. Prg = Jr. (TO( Sk: =x)) J. (T sk)) - 3 [(0): - 1(0k-1:8K=)]]
where a' is the importance weight (set to for simplicity), and the -p is calculated through Theorem Through Eq [] L2T optimizes the policy тe to achieve our goal of boosting effectivenesss and elficiency in two parts: (i) The first part ensures the correctness of the final answer through the outcome reward r(z, 20:k). (ii) The second part introduces the dense process reward r leveraging information tetirorttttthottiiccco internal signals to assess the progress of each episode update. It encourages the model to maximize correctness at each step while avoiding redundant information accumulation. Thus, this optimization enables the model to efficiently utilize the limited token budget, progressively improving reasoning effectiveness, and ultimately achieving high-accuracy reasoning outputs.
4.2
Practical Implementation with GRPO In practice, L2T is instantiated on top of GRPO to realize stable and efficient reinforcement optimization. Specifically, for each sampled question æ, the old policy Tem generates N reasoning rollouts, where each is automatically split into consecutive episodes based on the prompt designed with '<think>'.. ... '</think>' delimiters (Appendix G.44 provides an exampele). This segmentation enables us to assign both the sparse outcome reward and the dense process reward the episode level. Concretely, for episode k in rollout the reward is defined as RA, = i k + i,k' where the outcome reward is yuu = [ leads to correct y"], and the dense process reward is 7 Al _Ckk. with fitting information gain A (mo( Sk. 2x)) - J, (- (- sx)) and compression penalty C2 [(,; sx) 1(0x~1: sk-1) (see Section 4.2). Following GRPO, the episodic reward Rs, is further distributed to tokens using log- bog-prbbability surprise as weights, i.e., Wi,t x x-logg log (Zi, st,e). giving per-token rewards Ts,t- The truncated mean of these token-leve rewards (95%) yields f;. which is then normalized into group-level advantage, i.c., A, f==F, This group-level advantage is rescaled to tokens according to their relative contribution, i.e., Ar, A. TL. Finally, the advantages At, are used in the clipped policy gradient objective of GRPO with an additional KL penalty to stabilize training. Through this implementation, L2T maintains the advantages of GRPO while extending with explicit episodic decomposition and the integration of the proposed infornation-theoreiie dense process reward, thereby achieving both effective and efficient reasoning optimization.
prz
op
5
Experiments
In this section, we conduct extensive experiments on multiple reasoning benchmarks to verify the effectiveness and efficiency of L.TT. More details and experiments are provided in Appendix G
5.1 Experimental Settings
We evaluate on multiple reasoning benchmarks, including AIME24-25, AMC, MATH500 [8].
MinervaMATH (27). and Omni-MATH [13 (see Appendices and G for more benchmarks, c.g..
and DeepSeck- RI- Rel--islll--Wee----- models, which already generate reasoning traces marked with '<think>' We compare L2T against (i) outcome reward-basse RL methods (e.g., GRPO [1 for meepseek-model family, more in appendices) and (ii) test-time-compee wwwee--ommptte----eeeeeee focused methods, e.g., length penalty [2 and process-reward approaches such as ReST-MCTS [62] and MRT [36]. Since DeepScaleR-11558 1.5--rrvvv has already undergone one round of fine-tuning on 40k math muestion-answer pairs, we fine-tune it on the 919 AIME questions (from 1989 to 2023); for DeepSeek-R 1 ---istill--went 1.5B, we fine-tune
code generation). We use DecpScaleR-1.5B-Peeviie
as base
random sample of 4,000 question-answer pairs from NutinaMath (28). Both fine-tuning and evaluation use maximum token budget of 16,384. For optimization, set the learning rate to le weight decay to 0.01, and batch
8

---

Table 1: Pass performance on various math reasoning benchmarks. We compare base models trained with different fine-tuning approaches. The best results are highlighted in bold.
Base model + Method
AIME 2024
AIME 2025
AMC 2023
MATH500
MinervaMATH
Avg.
DeepScaleR-1 1-...8--P511iiii
42.8
36.7
83.0
85.2
24.6
54.5
+outcome-rewane RL (GRPO)
40.3 (+1.7+
(42.6)
(-1.5)
84.9 (-0.3)
24.7 (+0.1)
55.0 (+0.5)
+length penalty
30.31-644)
(-5.7)
83.2 (-2.0)
(-1.6)
50.8 (-3.7)
+ReST-MCTS
(+2.7)
39.5 (42.8)
83.4 (40.4)
84.8 (-0.4)
(-0.7)
55.4 (+0,9)
+MRT
47.2 (+4.4)
39.7 (+3.0)
83.1 (+0.1)
85.1 (40.1)
24.2 (-0.4)
$5.9 (+1.4)
+Ours
48.5 (+5.7)
40.2 (43.5)
(42.4)
88.1 (+2.9)
(41.9)
57.8 (+3.3)
DeepSeek-RR Baiecck----iiittii Distill-Qeen-L.5B
28.7
26.0
69.9
80.1
19.8
44.9
+outcome-rewal RL (GRPO)
(+1.(+
27.3 (+1.3)
(+0.6)
80.3 (+0.2)
(+2.3)
46.0 (+1.)
+length penalty
(-1.2)
22.6 (-3.4)
(-.4
77.1 (-3.0)
(-1.0)
(-209)
+ReST-MCTS
30.5 (+1.8)
28.6 (+2.6)
72.1 (+1.2)
80.4 (+0.3)
20.3 (40.5)
46.4 (+1.5)
+MRT
30.3
29.3 (+3.3)
72.9 (+3.0)
80.4 (+0.3)
(+2.7)
47.1 (+2.2)
+Ours
32,9 (+4.2)
30.1 (+4.1)
73.5 (+3.6)
84.7 (+4.6)
24.5 (44.7)
49.2 (+4.3)
1.5
Gscoma-RRard BIGT-NTS
FASTANCT
MAT Repesket
H4T Repordar
BIST-MCTS
1.0
1.2
MATWeursket
AASTOCTT
Mext Reprodyt
68
6.6
Teken 3.4
Mothods
Nothods
Methods
Motheds
(a) AIME
(b) AMC
(C) MATH500
(d) MinervaMATH
Figure 2: Efficiency comparison across different benchmarks. We compute the token budget required for each benchmark and treat the budget base model w/o fine-tuning as reference (1x).
size to 256. To approximate the parameter-compressine penalty, we employ a one-layer MLP with a Fisher information-matri. damping coefficient of 1 10-5 The hyperparameters a (Eq and ( are set to 0.8 and 0.6, respectively. All experiments are run on the A A100 GPU clusters. More details of implementation and hyperparameters are provided in Appendix
5.2 Effectiveness and Efficiency Analysis
Achieve better reasoning with higher efficiency We compare L.2T with the baselines across all the benchmarks and base models, recording both pass@1 accuracy and the required token budget. To reduce variance from limited samples, we use 20 outputs per question. Table and Figure that L2T attains SOTA performance, achieving the highest reasoning effectiveness with the smallest token budget. For example, compared to outcome-reward based methods, L2T delivers over a 3.7% gain in pass@1 and roughly doubles token efficiency; compared to methods focused on test-time compute, achieves more than 2% accuracy improvement while reducing the token budget by 20%. Moreover, L2T consistently outperforms baselines on multiple datasets with distributions different from the training data, further demonstrating its effectiveness across diverse tasks. Besides, we also assess the performance of L.2T on more base models with different scales, e.g., DeepSeek-R11 and more reasoning tasks, e.g., code generation tasks. Notably, mathematical
and Powen2-7B-Insteeen reasoning and code generation serve as classic benchmarks for testing an LLM's complex reasoning ability 43 [6 32] The results in Appendix G.2 demonstrate the advantage of L2T. These results confirm the superiority of our approach, which achieves effective reasoning with higher efficiency.
More efficient use of test-time compute Based on Subsection 5.1 we sample reasoning trajectories across various benchmarks with fixed token context window. We truncate these trajectories at different token budgets and evaluate performance. Figure shows the success rate against token consumption. We observe that (i) under the same token budget, L2T achieves higher reasoning accuracy; (ii) L.T consumes only 18% of the tokens required by the base model, 50% of those used by outcome-reward fine-tuning, and approximately 20% fewer tokens than process-reward models. These results demonstrate that L2T more effectively leverages test-time compute.

---

50-
ReST-METS
ReST-METS
40
MRT-Reprodact
30
MRT-Ruprodluct
30
20
15
4k
2k
30k
16k
Token Budget
Token Budget
(a) Bee-caleRR-.S--Peeee
(b) Dieisill-----sssill--e-ee--
Figure 3: @ass@1 vs. token budget of different methods on AIME We record the model reasoning accuracy under different maximum token budgets to evaluate the ability of using test-time compute.
50
G, (U2T)
48
(12T WT Config 11
(L2T Conlig2)
PHVVEE
47.5
(L2T Config
47.5
35
Q.en
41.2
50
Configurations
Figure 4: Effect of L2T components.
Figure 5: Parameter sensitivity of and B
5.3
Ablation study
We conduct series of ablation studies to evaluate the contribution of each component within L2T, the best parameterization and implementation choices, etc. See Appendix G for more results.
The effect of different components. We evaluate three alternative configurations: (i) replacing the fitting information gain with task-specific reward model; (ii) removing the parameter-comprrssion penalty; and (iii) substituting the low-rank approximation with random sampling of 30% layers. Notably, the overall contribution of our reward has already been demonstrated in Subsection 5.2 From Figure Hauure//andd Appendix G.3 we observe that both the fitting information gain and the parameter- compression penalty are critical for LLM reasoning; although random sampling faster than low-rank approximation, it introduces additional error. These findings underscore the advantages of our design.
G.3
Parameter sensitivity. We determine the hyperparameters of L2T by evaluating reasoning perfor- mance across benchmarks. Both a and B are swept over [0.3, 0.9). We first use grid search to screen the parameters with a difference of 0.05, then refine with 0.01, recording the average outcome. As shown in Figure the optimal setting is = 0.8 and 8 = 0.6, also our choices.
6 Conclusion
In this paper, we propose Learning to Think (L2T), an -nformation-hheortticc reinforcement fine- tuning framework for LLMs. reformulates LLM optimization as an episodic RL problem and proposes a universal information-theorees ie.. incentivizing the model focus on progress in each episode, thus achieving great reasoning performance under minimal token budget. Extensive experiments on multiple complex reasoning benchmarks demonstrate the advantages L2T in both reasoning effectiveness and efficiency.
dense process reward to support policy optimization,
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work is supported by the National Natural Science Foundation of China (No. 62506355).
References
[1] Pranjal Aggarwal and Sean Welleck, L1: Controlling how long reasoning model thinks with reinforcement learning, arXiv preprint arXiv:2503.0469) 2025.
10

---

[2] Daman Arora and Andrea Zanette. Training language models to reason efficiently, arXiv preprint arXiv:2502.0446.. 2025.
[3] Robert B Ash. Information theory. Courier Corporation, 2012.
[4] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. 2024.
[5] Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and performance in large language models-o3 (mini) thinks harder, not longer. arXiv preprint MrXiv:2502.15631 2025.
[6] Matthew Brand. Fast low-rank modifications of the thin singular value decomposition. Linear algebra and its applications, 415(1):20-30) 2006.
[7] Soumyabrata Chaudhuri, Pranav Purkar, Ritwik Raghav, Shubhojit Mallick, Manish Gupta, Abhik Jana, and Shreya Ghosh. Tripcraft: A benchmark for spatio-temporally fine grained travel planning. 2025.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint SIIV:22007.03374, 2021.
[9] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025.
[10] Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei-Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for reasoning, 2025.
[11] Xidong Feng, Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search guide large language model decoding and training. 2024.
[12] Roy Frieden. Science from Fisher information, volume 974. Citeseer, 2004.
[13] Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024.
[14] Robert M Gray. Entropy and information theory. Springer Science & Business Media, 2011.
[15] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang. Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small lims can master math reasoning with self-evolved deep thinking, 2025.
[16] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.12945; 2025.
[17] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis, 2024.
[18) Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. 2021.
arXiv preprint
[19] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
11

---

[20] Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Kai Yao, Jia Xu, Linjian Mo, and Min Zhang. Test-time compute: from system-1 I thinking to system-2 thinking, 2025.
[21] Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses scale, 2025.
[22] Song Jiang, Da JU, Andrew Cohen, Sasha Mitts, Aaron Foss, Justine T Kao, Xian Li, and Yuandong Tian. Towards full delegation: Designing ideal agentic behaviors for travel planning, 2024.
[23] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024,
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[25] A Ya Khinchin. Mathematical foundations of information theory. Courier Corporation, 2013.
[26] Sang Gyu Kwak and Jong Hae Kim. Central limit theorem: the cornerstone of modern statistics. Korean journal of anesthesiology. 70(2):144, 2017.
[27] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[28] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[29] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike. John Schulman, Ilya Sutskever. and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[30] Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang, and Irwin King. A survey of personalized large language models: Progress and future directions, 2025.
[31] Ryan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L. Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse, 2024.
[32] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint (iX.x:2212.04660 2021.
[33] Haotian Luo, Li Shen, Haiying He, Yibo Wang. Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ó1 Orrunerr: pruning. arXiv preprint arXiv:2501. 12570, 2025.
fine-tuning for ol-like reasoning
[34] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0m 06592, 2024.
[35] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
[36] Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, arXiv preprint ar 21X1::2503.05572,200 2025.
12

---

[37] Fazlollah M Reza. An introduction to information theory. Courier Corporation, 1994.
[38] Ahmed R. Sadik and Siddhata Govind. Benchmarking Ilm for code smells detection: Openai gpt-4.0 vs v33, 2025.
[39] Amrith Sethur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for IIm reasoning. arXiv preprint arXiv:2410.08146, 2024.
[40] Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. Reasonir:
Training retrievers for reasoning tasks, 2025.
[41] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[42] Shuaijie She. Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling, 2025.
[43] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
[44] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness? an analysis of cot in planning, 2025.
[45] Csaba Szepesvári. Algorithms for reinforcement learning. Springer nature, 2022.
[46] Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and TrendsO in Machine Learning, 8(1-2):1-230 2015.
[47] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang. Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
[48] Ning Wang, Bingkun Yao, Jie Zhou, Yuchen Hu, Xi Wang, Nan Guan, and Zhe Jiang. Insights from verification: Training verilog generation Ilm with reinforcement learning with testbench feedback, 2025.
[49] Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol ol-like Ilms, 2025.
[50] Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[51] Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
[52] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem- solving with language models, 2025.
[53] Zixuan Wu, Francesca Lucchetti, Aleksander Anderson, Joydeep Biswas, Federico Cassano, Molly Q Feldman, and Arjun Guha. Phd knowledge not required: A reasoning challenge for large language models, 2025.
Jingmiao Zhao, Carolyn Jane
[54] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024.
[55] Junjie Yang, Ke Lin, and Xing Yu. Think when you need: Self-adaptive chain-of-thought learning, 2025.
13

---

[56] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. 2023.
[57] Guanghao Ye, Khiem Duc Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, and Huseyin A Inan. On the emergence of thinking in llms i: Searching for the right intuition. arXiv preprint arXiv:2502.06773, 2025.
[58] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning, 2025.
[59] Edward Yeo. Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in Ilms, 2025.
[60] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:24/2.0198/ 2024.
[61] Sumeth Yuenyong, Thodsaporn Chay-intr, and Kobkrit Viriyayudhakomn Openthaigpt 1.6 and rl: Thai-centric open source and reasoning large language models, 2025.
[62] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772, 2024.
[63] Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang. Entropy-regularizee process reward model, 2024.
[64] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning, 2025.
[65] Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafci Lyu. Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. Genprm: Scaling test-time compute of process reward models via generative reasoning, 2025.
[66] Shuyan Zhou, Frank E. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Weharena: A realistic web environment for building autonomous agents, 2024.
[67] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-tun turn Ilm agents on collaborative reasoning tasks, 2025.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main contribution of this paper to propose an iiformationnthhoretfie reinforcement fine-tuning framework boost both reasoning effectiveness and efficiency. This has been discussed in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

We've stated the limitations and future directions to improve this work in the discussion section.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We state the assumption of each theorem and proposition in the corresponding location. All the proofs are provided in Appendix We also highlight the locations of corresponding proofs in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
16

---

Justification: We have provided the code, data, and instructions in the supplemental material. We have also provided all the implementation details, the type of resources, and all the results in Section Appendix F Appendix F. and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have provided the main code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides
conddeuubmissioooooccoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips..ccppbblll/ httpa///ggiiieeeeeeeuuubiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
17

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: We have provided the experimental setting/details Section Appendix and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The details and results are listed in the Section and Appendices, where all experimental results obtained on the basis of five rounds of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
18

---

Justification:
They are specified in Section and Appendix
F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: This work conforms, in every respect, with the NeurIPS Code of Ethics, e.g.. anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in the discussion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
19

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes)
Justification: We will provide the usage guidelines or restrictions to access the model or implementing safety filters when make our model public.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we ve properly cite all used data and provided the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version URL.
asset used and, possible, include
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we' 'v've properly eite all used data in Appendix. We've also submitted our own code in supplemental material.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
20

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for editing or formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy httt::::/eerrips.cc/ooffeeeeeee htt.://///cc//oeeeee////////2/2 for what should or should not be described.
21

---

Appendix
The appendix is organized follows:
Appendix
provides the list of notations.
Appendix
provides proofs of the theorem in the main text.
Appendix
provides the pseudo-code of our method.
Appendix
provides more discussions and further theoretical analysis about work.
Appendix provides the additional details of the benchmark datasets.
Appendix provides the additional details of the implementation details.
Appendix provides the full results and additional experiments.
A List of Notations
We list the definitions of all notations from the main text follows:
Symbols of Problem Settings
D train { 3f ) training dataset.
Iy i-th question.
y the oracle reasoning trace that leads to the correct answer.
the LLM (treated as stochastic policy).
20:r x :t):: the output sequence for dn.
Bioken the token budget.
r(z,, Ro:r): the reward, which equals 2p:T matches the oracle trace y!
Symbols of Problem Reformulation
D Purinn {(T,, vt )) training dataset.
No: the LLM (treated stochastic policy).
20:r (- 2x):: the output sequence for Zz-
Bioken the token budget.
K: the number of episodes.
34 = (z, 21: the state of k-th episode.
Fk ~ m 8g): the token sequence Fς = (z)....,
;k Ne
the k-th episode of ay
No: the number of tokens in 2k--
puut = rxz, 21:K) {0, 1}: the outcome reward indicating final correctness.
k prg; the dense process reward at episode & (Eq.).
J,((): the predicted correctness probability.
Information-TTeeorttee Quantities
(Y)))): conditional entropy of Y given X (uncertainty without parameters).
H [X),, 9): residual uncertainty of Y given X and parameters 6.
[(0; Y|X): conditional mutual information between 0 and Y given X.
AT: fitting information gain contributed by episode k.
I(0; sæ): mutual information between parameters and context.
Parameters, Proxies, and Updates
0, 0 1. 0g: model parameters before/after episode k.
0, 0 0y: low-rank proxies via SVD (retain top r components).
p(O,) -0, Pr: Ex): Gaussian assumption for the low-rank proxy.
r/d: retained-rank ratio (e.g., 1% 10% for 1.5B; 0.1% 1% for 7B).
GRPO Integration and Token-Level Credit
R4, R + a .T. per-episode reward for rollout i (with process- a:).
22

---

Wi,t & - Pe.a (2,, st,x): token weights by log probability surprise.
rie token-level rewards obtained by distributing Ri, using Wi,t-
F; 95% truncated mean of token rewards for rollout i.
A = (f-- f)/a group-level advantage (standardized).
A A. Tie token-level advantage rescaled by relative contribution.
B Proofs
In this section, we provide the proofs of Theorem 4.2 the main text. The theoretical analyses of the proposed dense process reward are provided in Appendix[D.]
Proof. Our proof comprises three parts: (i) Part Information-gain derivation: Beginning with the fundamental definition of mutual information and assuming a Gaussian form, we derive the information gain A = I(0x; ss) [(0 (10k-1:s-11) which under the Gaussian assumption admits -0)) (ii) Part II, Laplace approximation for variance estimation: We then employ the Laplace approximation to estimate the intractable posterior covariance in A/ by way of the Fisher information matrix. (iii) Part III, Derivation of the compression penalty
the approximation and get A/ Re (0n-0x-1) T Σg
penalty: Finally, we use the Fisher information matrix to derive the that regularizes redundant information accumulation.
Part I
We begin by giving the concept of the mutual information in which the information is stored in the weights. It can be expressed as:
[(; s) = EEs [uKz (p(0) s) I| p())]
(6)
Assume that both the prior and the posterior are Gaussian distributions, get:
p(0)= = (² 6o Eo), p(0 s) = NN(0 0g. E.),
then the mutual information in which the information is stored in the weights becomes:
1(0;)) - (p( s) p()) In det Σg
d (0.-0)) (0,-0) (tr 22) (8)
det Eo
where det(-) and tr(-) are the determinant and trace, d is the dimension of parameter 0 and is a constant for specific NN architecture. If we assume Σ, Eg following [. then the logarithmic determinant and the trace term are constants, and the mutual information simplifies to:
I(0:s) = E.
(0. (o) T Σn 10, Bn)
where N represents the number of trajectories for LLM optimization.
Why a Gaussian distribution is a common and mild assumption under this context. During LLM training, parameter updates 0 can be viewed as the accumulation of numerous small stochastic gradient steps. Each step introduces small, random perturbation in parameter space, effectively acting the sum of many independent random variables. When these perturbations are sufficiently numerous and diverse, the overall distribution of parameter changes tends toward Gaussian distribution, implied by the Central Limit Theorem. Therefore, when estimating the compression term, we model the low-rank surrogate / (obtained via SVD) using a Gaussian distribution. This assumption aligns with covariance approximation techniques the PAC-Bayes framework and Fisher information-bust methods, allowing us to derive closed-form expression for mutual information and significantly reduce computational cost.
Next, recalling our proposed parameter compression penalty, we care about the increase in mutual information between two adjacent proxy parameters, that is:
AI I(0k: sk)
(10)
Consider that the agents at step l and step (A: 1) be centered on the same prior Oo. then we get:
1 sk) a/4 (0. 0o) 20 (0x 0o)
(11)
23

---

and
((0--1:8-11 x (02-1 - T -0p Bn).
(12)
Based on the above formula, we can directly subtract and obtain:
AT
( 0)) To (0, 0o) (0x-1 x 1000
80) -
(13)
Using the quadratic increment identity, i.e., x² Ax
(x-pu --y) AAx + y) and for small step TTATrrAA y Ay = (x
updates 02 0 discarding the constant factor, we have AT ze (0x- 0a-1) 2n -0. 0x-1)
Part II However, it is difficult to calculate the covariance matrix for AT. To address this, we use the Laplace approximation En² z F3 and the matrix representation of Fisher information. We get:
Lemma B.1 Under the Gaussian-constant covariance assumption, the prior covariance matrix Co can be approximated by the inverse of the Fisher information matrix:
Eo ~ Fo1 s.t. Fe (Eswro(-lex)
Vā 02 log TT(2
8%) 0z log To 8x) st)
(14)
Proof. For given state Sks the posterior distribution p(0e sk) p(s* 0x) p(O)). take the logarithm to get the unstandardized posterior L(Ox) = log p(sk 0sx + logp(On). Among them,
log p(s (b))
Σ log (z sk), log - (0x To) (0p 00). (15)
Take the partial derivative of the mth component of 0s. we get:
N
3 log T 9x)
log ((sk (e)
Σ
Pi.ms
(16)
00 20
20
where Ii.m = 2m log (z sx). Continue to take the partial derivative of the nihh component. Taking the partial derivative of the component, we get:
0²
2
log p(kk 0x))
gmmm]
log To Te 2 sx),
(17)
00,,n 00
20
00k. 20
hemmn
Therefore, the Hessian matrix of the log-likelihood is:
N
H(rr) Vi ((0x)
V2
log T0(2; (sk) + (- 00) T
10p 00)
log p(02)
p(sk (k))
V2 log T0(2 sk) +
(0x 0o) T 20 (0.- 00)
(18)
0
Hprior
Huk
N
8²
d (k)
log Ne (26
+ 2p1
00 00
m,n=1
Let 0 = arg max; L(0) be the MAP (maximum posteriori) estimate point of the posterior, and perform second-order Taylor expansion of L(Ø) at Ō to obtain:
((θx) ~ ((O.) + (0. 0%) (θx) (0s - 0z)
(19)
then the Gaussian approximation is obtained as p(Ox sk) xe N Ōn. (-H(0)))-1) and the prior covariance should be Σn Cov [p(e) sx)] ~ (-H(0&))

---

Next, we discuss the relationship between Hessian and Fisher information. First, the observed Fisher of log-likelihood is defined as E, V log (2(2; (sk) = Σ: Va, log To (2,[s)))) Vā, log no (2ilsss)
When the third-order and higher derivatives are ignored, can be approximated by:
Hulk (fx)
=
0.. log na (2:
sk) V log mo(; sk)
(20)
Further take the expectation of the policy distribution and define:
Foz = Bswa (-10a) V log Ta(= sk)) Vā, Te 188) ".
(21)
For flat prior or oracle prior of Ep >> Hij we have priorl < || HHkk] so the effect of H prior on the population Hessian can ignored in the large sample limit. Combining the above, we get:
(Ox)) =- (Hukk + Hprior) = Fo..
Eo (- (-H(0x))
(22)
62
Thus, we complete the proof the Lemma B.1
Part III Bring the above results back to Theorem 4.2 AT zi (fp- 0n-1) "En (0p 0 can be derived as:
AI (Ōn-0x-1) Õn-1) -T_I) log To(a 8k) Vā, log (z( sk) ss)))) (0x 0x-1), (23)
Thus, we get:
1(0x: su) 1(0k-1isk (0x 1) (Vo, log (o(lss))) Va. log Ta(=1(5k) 7) (0x Ōx-1)
(24)
We complete the proof of Theorem 4.2
C Pseudo-Code
The pseudo-code L2T is shown Algorithm]] I
providing the main steps of L2T with GRPO.
Algorithm Pseudo-Code of L2T (GRPO Version)
Require: Initial policy Te: prompt distribution D; hyperparameters. and
1: for step = 1 to do
2: Sample batch D, from D
3: Set old policy meaia f- mi
4: for each query 2; D3 do
5: Sample rollouts {yu. "1.". (NN11 ~ To xx.))
6: for each rollout ys do
7:
Compute r prg via Definition 4.1 and Theorem 4.2 for each episode Compute rout following [16
i,
8:
9:
Compute per-episode reward R K. r put + tarf pre
10:
Set surprise weights Wi,t - log paay (2i,e sii)) Assign per-token rewards T4,t < Eret "a,t! Rick
11:
12:
end for
13:
Compute truncated mean F +- TruncMeangss (fradt) Compute group-level advantage Å; F=-
14:
0+
15:
Rescale to token-level advantages Ai, = A; Fst
16:
end for
17:
Update Ne via the GRPO objective
18:
end for
19:
return .
25

---

D More Discussion
D.I Theoretical Analyses about the Proposed Reward
In this subsection, we present the theoretical analysis for the proposed informmtiionthhorrete dense process reward. Specifically,, we first explain why the increment in model prediction accuracy can be used to approximate Als (Proposition D.I. Next, we provide a theoretical analysis demonstrating
that, under the current approximation, the computation method in Theorem 4.2 significantly reduces computational complexity (Theorem D.2,, while the approximation error (Theorem D.3) remains
bounded, thereby supporting the practical computation of the proposed reward.
As mentioned in Subsection 4.2 the fitting information gain quantifies the reduction in uncertainty about Y provided by the model parameters after each episode. Formally, defined as the conditional mutual information T(0:Y X) H(Y X) H(Y X.0), where H(Y X)
-y p(y X) log X) represents the uncertainty of Y given X alone, and H(Y x,0) the residual uncertainty after observing 8. The fitting gain for episode k. during which the parameters update from 0z-1 to Os. is given by A [(Os; Y X) - 1(0k-1: -; X). Given the computational expense of directly calculating mutual information in large models, we approximate Ale by the increase in the model's predicted correctness probability. Specifically, we use the approximation
Alę Rs (To Sk. zx)) (Te( sk)). which captures the direction of Ale and requires only two evaluations of the distribution per episode. Then, we get:
Proposition D.1 Given an episode k, where the model parameters are updated from 0k-1 to tk. the fit gain for this episode is defined as Aly 1(0u;Y X) - 100k-1; X), where [(0; Y X) represents the mutual information between the model parameters 0 and the labels Y, conditioned on the input X. We have:
AI Jr (TO( Sk: 2k)) - Jr (TT( [k))
(25)
where (mo) denotes the reward function under the policy ne, and Ne(- I sk, 2k)2k) and Tel. sk) represent the updated policy and the policy prior to the update, respectively. This approximation aligns with the direction of the mutual information increment.
Proof. We start by expressing the mutual information [(; Y X) between model parameters O and output labels Y conditioned on input X. This is formally defined as:
1(0:Y Y XX) H{Y X) H(Y ()
- Σ plu X) p(u X) + Σp(U) log x,
(26)
Ep(y p(y X) log ply X)
ply
X,0)
where H(Y X) is the entropy of Y given X (uncertainty of the labels given the input), H(Y X, X,0) the conditional entropy of Y given both X and 0. p(y X) is the conditional probability distribution of label Y given input X, and p(y Ix 0) is the conditional probability distribution of label Y given input X and model parameters 0.
The change in mutual information between two episodes (from 0--1 to 02)) is given by A/e [Bp:: Y X) 1( X). This represents the gain in the model's ability predict Y given X, as the parameters are updated from 0x-1 to 6j.
In RL-based LLM optimization, the reward function J,(z)) can be interpreted as the expected accuracy of the model, which evaluates how well the model's predictions align with the correct answer. For given policy no, the reward function is defined Jr-o (egeso((lez)) [r(=)] where denotes the model's output and r(-) measures how correctly the model generate the answer. In this case, the difference between the model's 's output probability (i.e., ply X, f)) and the label Y directly affects the model's reasoning accuracy.
From the perspective of information theory 3 255, the model's fitting information gain A/ reflects the change in the difference between the model's inference answer and the standard answer (label) before and after the parameter update. High mutual information means that the relationship between the model's is inference and the standard answer is stronger, in other words, given X and 0, the model
26

---

is able to predict Y more accurately. The reward function also reflects this by measuring the accuracy of the model in its predictions. In other words, increasing mutual information actually improves the accuracy of the model, and accuracy can be quantified by the reward function. Therefore, the reward function can reflect the information gain obtained by the model in label prediction, or in other words, the reward function can reflect the improvement in the accuracy of the model's predictions. The accuracy improvement and mutual information increment have similar directions: both reflect the improvement in the ability to capture label information. Recall the problem settings, the model updates the policy FH through the parameter 0, which affects the prediction accuracy of the model. The updated policy To(- Sk, 24.) and the pre-update policy Te(- sk) will lead to changes in prediction accuracy, thereby affecting the value of the reward function. Since accuracy is related to mutual information, we can approximate the mutual information increment by the difference in reward function. Therefore, we have:
(27) This shows that the difference in reward function is consistent with the direction of the increase in mutual information, both reflecting the increase in the amount of information when the model predicts the label Y (correct answer).
A Jr (TT((- 8K; =k)) - Jr (TO(- s:))
In the context of LLMs, the fitting information gain quantifies the contribution of each optimization episode to the reasoning ability by tracking changes in the predicted correctness probability, i.e., penalty is more
the output distributions of :Tg. In contrast, calculating the cerramtter-coppesssiii complex: it involves estimating the mutual information increment between the model parameters 0 and the historical context Sg. Direct computation of this increment is intractable in the large parameter space of LLMs. To overcome this challenge, we propose an efficient approximation in Theorem 4.2 to estimate the penalty term. Next, we demonstrate that this method significantly reduces the computational complexity (Theorem D.2) with limited approximation error (Theorem D.3) to support the computation of our proposed reward.
First, we prove that in Theorem 4.2 computing the prrameter-compesssin penalty via low-rank approximation and Fisher matrix estimation achieves speedups of several orders of magnitude.
Theorem D.2 Let the parameter dimension be and the rank cutoff value be r (r < d). Compared to the original full non-approx computation, estimating the compressionooo penalty via Theorem 4.2 reduces the complexity of quadratic-form evaluations by a factor of ((r/d)))
Proof. First, we construct and store the complete Fisher matrix F(0) E Rxdd which itself needs to store d2 scalars, so it is (d²)). Then, evaluate the quadratic form A F, 10, which can be completed in two steps: first, calculate totaling O(d2), and second, calculate the scalar AOTT u. which requires an additional (d), and the total is still (dd). If it is further necessary to solve F-1 or perform eigendecomposittin, the time complexity of the corresponding classic algorithm is ((d)).
F, A0, involving d inner product operations of length d,
Next, in the low-rank method, the original vector 0 E Rdd is first mapped to an r-dimensional subspace using truncated SVD (or randomized SVD). The main computation comes from the multiplication and addition of the p X r matrix, so the complexity is r2) Then, in this subspace, the gradient Jacobian V log Fe calculated and an approximate Fisher matrix F JJ is formed. Each entry requires d multiplications and additions, and the overall complexity is still ((d, r²). Finally, evaluating the quadratic form (A0) T F. (A0) only requires multiplication and addition of the r x r matrix and the length vector, with a complexity of ((r2); if F needs to be further decomposed, it will be ())3). Therefore, the overall complexity of the approximate method can be expressed as
Capprox
CSVD + Cgrad + ((2) Approx O(pr" r"))
(28)
Dividing the complexity of the above two methods can get the speedup ratio. On the one hand, the original method ((2) = (d)),, on the other hand, the approximate method (2) we get:
Capprox (1r2). Thus,
(eig)
O(p²) Oppr +33 ((p/))//( 0(//)))))))
orig
(29)
Cappeox
Next, we prove that the approximation error caused by the above approximation is limited and controllable.
27

---

Theorem D.3 Assume that in each episode update, the parameter increment Adj = 6k- 0k-1 satisfies || S B, and has uniform bound M on the third-order derivative of any 0. Let SI& (A F(()) 28, and Alę = I[0g; st) - [[0k-1;8k11 and estimate the Fisher matrix through N, independent sampling trajectories, and then take K episodes for joint statistics. Then for any confidence level ò E (0, 1), with probability at least 1 we have:
M 6
8 (n - -21/6
max ISACK K
A4
B"
+1
(30)
N, K
where d = dim is the parameter dimension, the first term M. B comes from the third-order remainder of the second-order Taylor expansion, and the second term is derived from the matrix Hoeffding inequality or Bernstein inequality following Ctarri-Conncetraalie
theory [46].
Proof. We decompose the potential error into two parts: (i) Taylor expansion remainder: approximate the true mutual information increment with second-order Taylor expansion, and the remaining third- order term gives the B upper bound. (ii) Sampling/statistiaal error: use the matrix condensation inequality to give the spectral norm level upper bound on the deviation between the empirical Fisher and the true Fisher, and then get the second term from the quadratic property. Next, we discuss and analyze these two items in turn.
For the function f(0) = log To(2k | 8k), at point 0g. do a second-order Taylor expansion along the direction h- = A6k, and we have
f(ox-1) f(oc) ff ) h + h (² h R3,
(31)
Among the remainders R3, for a certain & is between θ and og.. From || | s M and ||||| s B, we get
M
M
Rs h [(] h S
||Al
B
(32)
6
6
Thus, we have:
M
A - (20)) F (A
Ba.
(33)
Next, we turn to discuss the empirical Fisher's s condensation error. Assume there are K episodes in total, and each episode collects N, independent trajectories. Let Øj.k =V,, 0, log puo (= (3) 88)) E
Rdd j= l...... Nr. k 1...., K., then the true Fisher information matrix can be expressed as
1 F E[ggg]] 1, g Lid. 19,1). and the empirical Fisher is F N. k=1 Eド=1 Ij.k -3 note -.
Let the matrix corresponding to the ith sample be (flatten the double subscript to a single subscript) X, = 9 Jh., - F' where i = I,..., n, obviously B[X,] = 0 and F. - F = + Σm X, Applying matrix Hoeffding inequality, we obtain that: If { {X,} is an independent symmetric matrix and E[X,] = 0. ||x||| S R. then for all u 2 0 we have Pr Σ" x.I| 2 u) s 2dex R
Apply this to Σ; X, = nf(F= F), we have:
u²
[r F)1l 2 u) S 2dexp
(34)
8n Ri2
Let nt, we get
n²²t
nt²
Pr Pr( [[ (
F)1l 2 nt) S 2d exp(
= 2d exp
(35)
8 RD
8
If we assume that each gradient norm is restricted: |9 ll S 1, then I S 1 and ||]]| ||&|ll < 119...97..1 + ||||| < 2, where R In order to be effective for both positive and negative sides, it is often multiplied by 2 before the above formula, and we get:
< 1. so
Pr Fll 2 t)
(1 F)ll 2 nt)
n²t²
2t
< 2deepp
2 exe 122
8n 123
(36)
n12
8 ln
4d
< 4dexp(
8
t= R
8R2
8
28

---

Replace n2 = N,K, R 2, and remember d + 2d (according to the parameter dimension, we get
Iu (-2*/6)
1 = 2 N,K
4.2
Under the above event (with probability 2 1 1-6), 8), for
N,K
F) hl - |M | Fll $ ||b] which is the typical property of any vector h, have 1h | controlling quadratic forms using the spectral norm.
Superimposing the Taylor remainder with the empirical Fisher statistical error, we get for each episode:
M
A4 - 11 Fh <
B
+
B' campling/statisties
Tay remainder
(37)
M 6
8 11((2/6))
B
B
+
B2
N,K
If the update amount is normalized (or assumed B < 1), it can be simplified to c M B
6
8 18)24/8) Thus, complete the proof of Theorem D.3 N.K
Therefore, we can conclude that the computation method in Theorem 4.2 significantly reduces computational complexity (Theorem (D.2), while the approximation error (Theorem D.3) remains bounded, thereby supporting the practical computation of the proposed reward.
D.2
Intuition behind the Proposed Reward
D.2.1 Intuition behind the Fitting Information Gain
How to interpret LLM reasoning from information-theoreii perspective (fitting gain vs. uncertainty) The reasoning process of an LLM can be viewed as inferring the correct answer Y from input X. The more certain the model's 's prediction is, the better it "understands" the answer. Based on classical information theory [37 3 14], we can use conditional entropy XY) to characterize the model's uncertainty about the output Y. If the model is sufficiently confident, its XX) should be low: conversely, if it is uncertain or making a blind guess, H HY should be high. Within this framework, the goal of inference is gradually reduce H(Y[X) until the correct answer is output. Importantly, while HY) represents the relationship between the input and output, this process is controlled by the LLM (determined by the parameter 0). Therefore, more reasonable metric is: Given 0, what is the model's uncertainty about Y? That is, we want 0 only capture the input X but also provide strong discrimination of the correct answer Y. Therefore, we use conditional mutual information 7 [0; Y|X) X - (X||X, 0) to measure how much the known model parameters 0 reduce the uncertainty about Y given X. This ties rewards to meaningful reasoning progress, i.c., the reward for each episode depends on how much it helps the model understand the correct answer. Therefore, we define "fitting information gain the reduction in uncertainty".
How to characterize the gradual improvement of inference (introducing episode gain with Og and 0x-1) each episode h, the model updates its parameter state from 0y-1 to 0g by observing the new reasoning step 2g. The key question is: Does this episode help the model better "understand" the answer? Therefore, we define fitting information gain of this episode as A = [(O;; Y ()) T(0--1:Y| If this incremental gain is significant, it indicates that this episode has helped the model become more certain and effective. Here, 0₂ and f--1 represent the posterior model parameters before and after learning the knowledge from episode k, estimated by the change in log-probability during the forward prediction process.
Why the proposed metric corresponds to reducing uncertainty (approximating information gain with J,) Because directly calculating mutual information is too complex, we introduce an approximate metric J.(()) (-) to represent the model's predicted probability of the correct answer. This metric improves as the model's "confidence" increases. Therefore, the fitting gain approximated as Aly se Jr (T
(- (-) 54)), with thcoretical guarantees in Appendix D.I This sk, =k))
29

---

difference measures whether episode & improves the model's ' ability to predict the correct answer, reflecting the reduction process, i.e., the increasing reliability of reasoning.
D.2.2 Intuition behind the Compression Penalty
Why introducing compression penalty In LLM reasoning, if each episode's information causes significant change, while that episode may provide information gain, it may also capture unnecessary details within that episode, leading to overfitting or redundant computation. We want the model to only learn information that contributes to the answer within the episode. Therefore, inspired by the information bottleneck theory. we introduce a compression penalty based on the fitting term to further improve efficiency. It measures the "information overhead" incurred by each episode from an Coormationnthhorette perspective.
How is it measured (intuition behind the design) If 8g differs significantly from 8x-1, but the model's prediction performance (i.e., fitting gain) improves only slightly, this step may "absorb redun- dant information". Therefore, we use the mutual information increment (0s; sv.) 7(0,-1: 8k-1) between them with the fitting term measure whether the information in episode k introduces unnecessary overhead.
Why this is called compression The term compression comes from the idea that model should retain only the minimal amount of information sufficient to perform accurate reasoning. In our setting, the mutual information [(0; s&) quantifies how much the information of this episode is encoded into the model parameters 0. A larger value implies that the model has to "store" more bits to fit that episode, akin to using more storage in a compressed file. By penalizing the increase [(0g: s&) 1(0%=1: 1:: 5k-1), we explicitly discourage storing redundant information, promoting a more compact (i.e., compressed) internal representation. This aligns with principles from MDL and PAC-Bayes, where generalization favored when the hypothesis (here, 0) is simple and concise.
D.3 More Comparison
Our framework advances prior and concurrent works in three key dimensions, i.e., efficiency, general- ity, and robustness, which we briefly illustrate below.
mainly optimize via outcome rewards, which drives models to over-extend reasoning chains and waste test-time compute. In contrast, our dense process rewards immediately quantify each episode's contribution to performance, enabling the model to learn when to stop reasoning and thus achieve equal or better accuracy with a minimal token budget.
Firstly, previous methods [3 16
53
9
40
Secondly, some concurrent works [57 33 36 20 propose task-specific process rewards, heuristic scorers, and length penaltics to reduce the length of the reasoning chains, helping save test-time compute. However, these methods require costly manual labeling and do not transfer across tasks since they rely on task-specific settings, and there is no one-size-fits-all solution. We instead measure the parameter-updaa update signal inspired by information theory, i.e., the intrinsic change in the model's weights after each optimization, as a tank-agn proxy for learning progress. This internal metric requires no additional annotations or retraining and applies uniformly across diverse reasoning tasks.
Thirdly, existing reward- based updates to optimize test-time compute 36 57
64 may embed noise or task-specific artifacts into model weights, leading to overfitting and drastic performance drops under slight input shifts. contrast, we introduce a parameter compression penalty that quantifies and negatively rewards the absorption of redundant information at each update. Only updates yielding true information gain are amplified; small or harmful directions are suppressed, ensuring stability under noisy, ambiguous, adversarial inputs. Moreover, by recasting LLM fine-tuning episodic RL problem, combining a task distribution with dense per-step feedback, we enable the policy to maintain robust performance from simple arithmetic to complex proofs, without redesigning rewards or retuning hyperparameters for each new task.
Thus, by combining these three advantages into a unified RL objective, our L2T simultaneously maximizes reasoning effectiveness and computational efficiency across tasks of varying complexity, unlike prior approaches that trade off one for the other rely on bespoke reward designs.
30

---

D.4 Broader Impacts and Limitations
In this subsection, we briefly illustrate the broader impacts and limitations of this work.
Broader Impacts. This work explores how to simultaneously maximize inference effectiveness and efficiency across tasks of varying complexity meet real-world demands. It reformulates L.LM reasoning through episodic reinforeement-lcrrinng and information -formatton-thhooetic lens, and introduces general dense process reward to track reasoning progress, enabling optimal performance under minimal token budget. Extensive theoretical and empirical analyses validate its effectiveness and robustness. This work also opens up exciting new avenues for future research, e.g., provides way for more explicit and automated dynamic budget allocation in the future, especially in scenarios sensitive token costs (e.g., mobile deployment or real-time QA).
Limitations. This study evaluates general LLM reasoning tasks, such as mathematical proofs and code generation, which are commonly used to verify the reasoning capability of LLMM;; some newly proposed benchmarks, such as web design, were not used. Our current experiments use mainly the open-source DeepSeek base models, including the scales of 1.5B, 3B, and 7B, while the scale of 72B and even above 100B is not used due to resource limitations and not being open source. We will investigate additional case studies and more base models extend this work in the future.
E Benchmark Datasets
In this section, we briefly introduce all datasets used in our experiments. In summary, the benchmark datasets can be divided into two categories: (i) reasoning tasks for mathematical derivation, including
AIME24-25, AMC, MATH500 [8] MinervaMATH [27]. and Omni-MATH [3] and (ii) reasoning
tasks for code generation via HumanEval 8 The compositions of these benchmarks are as follows:
AIME24-25 comprises 30 questions from the 2024 and 2025 American Invitational Mathe- matics Examination, with 15 fill-in-the-blane questions per exam. These questions more difficult than AMC, spanning number theory, combinatorics, geometry, and algebra.
AMC10/12 consists of 25 multiple-choice questions each for the AMC10 (up to 10th grade) and AMC12 (up to 12th grade). Each competition consists of 25 multiple-choice questions, totaling 975 questions across 39 tests. Questions progress from basic algebra and geometry introductory probability and counting, covering various tasks for LLM reasoning evaluation.
MATH500 is a 500-question subset randomly sampled from MATH, covering seven top- ics-prealge,ra, algebra, number theory, geometry, intermediate algebra, precaleulus, etc. Each question includes a step-by-step solution and difficulty label from 1 to 5, enabling evaluation of an LLM's mathematical question-solving across diverse domains.
MinervaMATH comprises 12,500 high-school-level contest questions. Each includes de- tailed solution steps and spans prealgebra through precalculus.
Omni-MATH is an Olympiad-level benchmark of 4,428 competition questions across 33 subdomains (c.g., number theory, combinatorics. geometry. algebra), stratified into over 10 difficulty levels (divided into 4 tiers following [5]).
HumanEval consists of 164 Python programming tasks designed to evaluate the correctness of code generated by models. Each task includes a standard function signature, and the model must generate the corresponding code implementation based on the description. The evaluation metric primarily Pass@ k, which measures the proportion of times the generated code passes the test cases at least once within k attempts.
F Implementation Details
For model training, we directly load the base models from Hugging Face, including 1.5 Preview,
and Qwen2-7B--Inttmne For different reasoning tasks, we introduce the experimental settings in the corresponding sections of
-iit----iiilli--e---.- 1.5B., DeepSeekR --DistillQQee-----
Section and Appendix G Unless otherwise specified, we follow the protocol of each benchmark and record the maj @ results across different models. The training configuration is: the learning
31

---

40
Tlrr1
35
Tier?
Tier2
Tlera
14000
Tier
Accuracy (%)
30
Tierd
12000
25
10000
20
Token
6000
4000
10
15
20
25
10
15
20
25
Episode
Episode
(a) Reasoning accuracy vs. episode number
(b) Token consumption vs. episode number
Tler1
12000
Tier1
30
Tler2
10000
Accuracy (%)
25
Tler4
Tierá
6000
Taken
4000
10
15
Episode
Episode
Reasoning accuracy episode number
(d) Token consumption vs. episode number
Figure 6: Results of DeepScaleR- 1...eeR--------eeeeee -----reviiww (a,b) and DeepSeck-Rl-Distile Swweill--wee------ 1.5B (c.d) across different tasks on Omni-MATH. We partition the generated reasoning chain into episodes, measuring accuracy Acc(k) and average token consumption T(k) at different episode depths.
rate is set to 1.0 x 10-6, with a cosine learning rate scheduler, and a warm-up ratio of 0.1. We use batch size of 256, with maximum prompt length of 4,096 tokens and maximum completion length of 16,384 tokens. The model is trained for 1 epoch, up to 10 epochs. Additionally, we set the "use vllm' flag to True enable vLLM acceleration, with a GPU memory utilization of 0.8. We also utilize mixed precision training with BF16 enabled. The parameters for compression penalty approximation are handled by single-layer MIP,, with a Fisher information matrix damping factor set to 10-5, The regularization hyperparameters and B are set to 0.8 and 0.6 according to grid search results, respectively. Also, a can be set for simplicity, with the performance drop less than 1%. More evaluation of implementation is provided in Appendix G e.g., parameter sensitivity, prompt configuration, etc. The entire training is conducted on A100 GPU clusters, ensuring scalability and high computational efficiency.
G Additional Experiments and Full Results
In this section, we present the full results and additional experiments of this work, including extended settings, datasets, and base models, which are provided in the appendix due to space limitations.
G.1 More Details and Results of the Motivating Experiments
we evaluate how efficiently existing methods use tokens. We benchmark two base models, DcepScaleR-L.5B- -.reeeer and
In Subsection 3.2.
-is--isill-QQee--1.BBB on Omni-MATH (4,428 questions across 33+ subfields, split into Tiers 1-4 by expert difficulty labels). Both models have been fine-tuned with salcome-based RL. To study performance at varying reasoning depths, we split each generated reasoning chain into up to K 20 episodes using '<think>... </think>' then record the sequential-genneation accuracy Acc(k) and the average token usage T(k) at each episode k. For comparison, we also include a Maj4 baseline: under the same truncated context, we sample four answers and take majority vote, measuring Maj@4(k) and its token cost. Plotting "accuracy
32

---

Table 2: Pass performance on various math reasoning benchmarks. We compare base models trained with different fine-tuning approaches. The best results are highlighted in bold.
Base model + Method
AIME 2024
AIME 2025
AMC 2023
MATH500
MinervaMATH
Avg.
DeepScaleR-1 1-..----1111-iieeee
42.8
36.7
83.0
85.2
24.6
54.5
+outcome-rewane RL (GRPO)
40.3 (+1.7+
(42.6)
(-1.5)
84.9 (40.3)
24.7 (+0.1)
55.0 (+0.5)
+length penalty
30.31-644)
(-5.7)
83.2 (-2.0
(-166)
50.8 (-3.7)
+ReST-MCTS
(+2.7)
39.5 (42.8)
83.4 (40.4)
84.8 (-0.4)
(-0.7)
55.4 (+0,9)
+MRT
47.2 (+4.4)
39.7 (+3.0)
83.1 (40.1)
85.1 (40.1)
24.2 (-0.4)
$5.9 (+1.4)
+Ours
48.5 (45.7)
40.2 (43.5)
(42.4)
88.1 (+2.9)
(41.9)
57.8 (43.3)
DeepSeek-RR Rii---tiDD----O-O Distil--Qeen-L.BB
28.7
26.0
69.9
80.1
19.8
44.9
+outcome-rewald RL (GRPO)
(+1.1)
27.3 (+1.3)
(+0.6)
80.3 (+0.2)
22.1 (+2.3)
46.0
+length penalty
(-1.2)
22.6 (-3.4)
(-.45))
77.1 (-3.0)
(-1.0)
(-209)
+ReST-MCTS
30.5 30.5 (+1.8)
28.6 (+2.6)
72.1 (+1.2)
80.4 (+0.3)
20.3 (40.5)
46.4 (+1.5)
+MRT
30.3 (+1.6)
29.3 (+3.3)
72,9
80.4
22.5 (+2.7)
47.1 (+2.2)
+Ours
32,9 (+4.2)
30.1 (+4.1)
73.5 (+3.6)
84.7 (+4.6)
24.5 (+4.7)
49.2 (+4.3)
DeepSeck-R Biieck-----iin
55.5
50.2
85.1
87.4
42.1
64.1
+outcome-reward RL (GRPO)
(+1.4)
(41.5)
85.5 (40.4)
87.7 (40.3)
(41.4)
(+1.0)
+length penalty
53.8 (-1.7)
46.9 (43.3)
81.2 (-3.9)
83.7 (-3.7)
39.5 (-2.6)
(-.1)
+MRT Reproduct
57.0 (+1.5)
52.4 (+2.2)
86.0 (+0.9)
88.4 (+1.0)
(+2.2)
(+1.5)
+Ours
58.4 (+2.9)
53.6 (43.4)
87.5 (+2.4)
89.2 (+1.8
(4209)
66.8 (+2.7)
colcome FRAUE 1L
RATACCT
NETS
MCTS
MKT Reproiet
MAT Roardad
HRT Repiviat
Taken Budiee
Reprodett
Oam
Ou
Our
25
E.2
0.3
Mathods
20
Methods
0.2
Methode
80
Methode
AIME
AMC
(c) MATH500
MinervaMATH
lergh perally
Brgh Damaly
lergh paeaty
lengh parelty
BAST-EES
BAST-MCTS
REST-MTTS
0.
Dart
COMAT 30
OLIS
82
Methods
Metheds
Methods
Nethods
(e) AIME
(I) AMC
(g) MATH500
(h) MinervaMATH
ELtcorsa AL
cascome enward
Busgass
Dum
Reprodutt
10
OLES
Repreduct
Token Buddges
1.0
0.2h
Token Busgee:
Cuts
HITT PeT--Reprduct
4
Mothods
Mathode
Mathods
Methods
AIME
AMC
(k) MATH500
MinervaMATH
Figure 7: Efficiency comparison on DeepScaleR-1.5B-Preeiie (a-d), DeepSeek-RI -I----RtiD--ii-eeesre-------- 1.5 (e-h), and DeepSeek-RR R----still--wen---7 (i-l). We compute the token budget required for each benchmark and treat the budget of the base model w/o fine-tuning as reference (1x).
vs. episodes" and "token cost vs. episodes" reveals how model performance varies with question difficulty and compute budget, and highlights the relative merits of sequential decoding versus voting.
Due space constraints, we previously showed only DeepScaleR-1 -.B..----..B--Peieee. Figure 6 now presents both base models, demonstrating the same trends: (i) Acc(k) peaks and then declines as k increases, indicating extra episodes add no new information and may hurt performance via context redundancy: (ii) token usage rises rapidly with k exceeding twice the minimal budget before peak accuracy, underscoring that existing methods may not efficiently use test-time compute; and (iii) the optimal k depends on difficulty-Tier 4 questions benefit from longer chains, whereas Tier questions achieve strong results with very few episodes. These findings motivate 1.2T's dense process reward, which adaptively adjusts reasoning depth.
33

---

72
HumanEval 0-shot Pass@1
HumannEvl 5-shot Pass
W/o L2T
70
L2T
Pass@1
62
0.76
1.0
Token budget
Figure 8: Effectiveness and efficiency analysis on code-related reasoning tasks.
Takes Bcip 00
Teken Badkert
Perte
Tokas Gurgpe
Spech
Rpach
Spech
Rpnck
(a) AIME
(b) AMC
(c)
MATH500
(d) MinervaMATH
Figure The effects of different components within L2T across different tasks.
G.2 Full Results of Effectiveness and Efficiency Analysis
To evaluate the proposed L2T, we conduct experiments various benchmarks, including mathemati- cal and code-related tasks, and across base models of different scales. In the main text, we report the performance of L2T mathematical reasoning tasks of varying complexity. Table Figure 2 and Figure 3 show that, compared to prior outcome-reward or process-reward methods, L2T delivers superior reasoning with less test-time compute. In this subsection, assess its performance more base models and more reasoning tasks, e.g.. code generation tasks. Notably, mathematical reasoning and code generation serve as classic benchmarks for testing LLM's complex reasoning ability 43 16 32]. First, we report performance across additional model scales. The results for inference accuracy and compute efficiency are shown in 2 and Figure 7 We observe the same conclusion: L2T achieves state-of-the-al performance, attaining the highest inference accuracy with the smallest token budget. These findings demonstrate the broad effectiveness of our approach across models of varying scales. Secondly, we provide the performance of the proposed framework on the code generation task, and we evaluate the improvement of L2T on the LLM reasoning performance on the code generation task. Specifically, we run the GRPO/L2T fine-tuning pipeline on Swen2-7B-Innsrrun and evaluate using the standard HumanEval protocol. We set the temperature 0.6 and top-p to 0.95 and generate 64 solutions per question, with Is timeout per attempt. We report the proportion of questions that pass all unit tests at least once. From the results in Figure g we can observe that compared with the nutcome-reward-basse RL method, L2T achieves better reasoning performance with less token budget. This proves the superiority of our proposed L.2T and dense process rewards.
To further validate the versatility of our approach, we conduct two complementary studies. First, since L2T improves reasoning efficiency by adaptively allocating token budgets, we examine whether additional test-time search (e.g.. MCTS) provides further gains. We apply MCTS to models fine- tuned with GRPO and L2T, and evaluate pass@) I accuracy on AIME and MinervaMATE. The results in Table 3 show that GRPO benefits noticeably from MCTS, while L2T already achieves strong reasoning performance and gains only marginally, confirming that L2T effectively reduces reliance on large-scale search. Second, as our dense process reward is defined in a task aggossic form, we also apply to inference-only methods as reranking signal. We evaluate 7B under a best-of-16 16 setting on MinervaMATH, comparing log-likelihood reranking. PRM-based reranking, and I.2T reranking. The results in Table 4 demonstrate that L.2T achieves the highest accuracy, outperforming both likelihood-baat and PRM baselines. Together, these findings indicate
34

---

Question: What
Question: Campute
0.9
a 0.9
12 plas 5 equais 17. Auswer:
The comuoon denomiinator is 12. 3/4 9112, S16 18:12. Adding gives 1911. Answer: 18112
To selve chis, first recall the addition operation aperatioon
We are asked to add 344 and Sio. To this, we need common decominatoor The decorminaters are and 6, and their least common mulliple is 12. Cunvert: X44 9/12, 56 1811., Now add: 9112 10/12 - 1911.
Addiiion is combining two numberrs into tecal. Let's start with 12 and add 5. 12 plas equai IT. I doubblechhccc: 10 15, The finel result should be 17. Anewer: 17
c==022
a=0.22
We can cheek: 19/12 - and 7/12, Looks concistent.
Anower: 1813
(a) Simple Arithmetic
(b) Fraction Addition
triangle lege length Find hypotemuse.
Quariou:: la right trianali, lop ard Wast
c=0.9
0=0.9
squared plus 12 squared 25 plas 144 169, The square reot of 169 is Answer: 13
Hoprotenanc
Arcs seniperinetter (6-tterrr------00 8- 10y2
Redies erce scrtperiiactr 24/12--
Anmwr
inpedenuse right triangle with By the Pythagorean typatenuse 144 the
We sakel for the inrriloo of uriasale with lop und Fins, computi the typolomss: 5 ard ss hypetemas souiperineter
0c-0.2
ar-0.2
2412
Doable-chrck:
Let's cantira: arm my dr thes 26, musiching srea. Anewen:
The inradies formala is
12-
Answer:
(c) Geometry
(d) Geometry
Figure 10: Examples of qualitative analysis about a.
Method
AIME 2024
MinervaMATE
Method
MinervaMATH
GRPO
44.5
24.7
Best-of- 1 log-likelihood
42.6
GRPO+MCTS
46.8
25.3
Best-of-16 Owen2.5-Math-PPP
43.3
L2T
48.5
26.5
Best-of-16 Skywork-PRM
43.0
L2T+MCTS
48.9
26.7
Best-of- 16 L2T rerank
43.9
Table 3: Effect of combining with MCTS.
Table 4: Applying to inference-only reranking.
that L2T not only enables efficient reasoning during fine-tuning but also serves as an effective scoring mechanism in inference-time pipelines.
G.3 Full Results of Ablation Studies
conduct ablation studies to evaluate the optimal configuration and parameter settings. Considering questions of varying complexity, we perform evaluation on multiple bench- marks. We conduct an ablation study on the three core components of L.T,, evaluating the contribution of each one by constructing alternative configurations: (i) Replacing information gain (config I): replacing the proposed process reward with a task-aligned pre-trained reward model; (ii) Removing parameter compression penalty (config 2): completely removing the parameter compression penalty driven by the Fisher information matrix; (iii) Replacing low-rank approximation (config 3): using random sampling of 30% of network layers to approximate the Fisher information matrix, instead of the original low-rank approximation method. All ablation configurations follow the same hyperpa- rameters and test protocols as the main experiment, each Tier is repeated five times to report average accuracy and average token consumption. The results in Figure 9 confirm similar conclusions as in the main text: replacing information gain with task specific reward leads to an average accuracy drop of about 1.9%, with a slight increase token consumption; removing the parameter compression penalty results in about a 12% increase in consumption and drop in accuracy; while random layer sampling reduces approximation overhead, the accuracy drops significantly, and the fluctuations increase substantially. These results validate the crucial role of each proposed component within L2T.
In Subsection 5.3
35

---

G.4 Visualization
G.4.1 Qualitative Analysis of
In this subsection, we construct qualitative experiments to illustrate how eliminate redundant inferences at low and high alpha values. The coefficient a denotes the weight of our proposed process reward (Eq.5). When rewards only when the correct answer is found. Without guidance and given that correct answers are sparse the output space, the model may consume a large number of tokens exploration, reducing efficiency. In contrast, when is large, the model is driven by the process reward, which assigns high rewards only if the current reasoning step has high contribution to the accuracy the answer, i.e., the correct answer is reached within short token sequence. This encourages the model to generate informative tokens at each step, thereby improving efficiency. The qualitative results are shown in Figure 10. which demonstrate the above analyses. Take the Tier-l Omni-MATH problem "What is 12 + 5?" as an example, the qualitative analysis shows that with a 0.9, the model may answer within 2-3 steps, whereas with 0.2, it may take more than steps.
small, the model relies more on the outcome reward, which provides high
G.4.2 Prompt Configuration for Episode Segmentation
For episode segmentation, we automatically segment the chains by designing specific prompts.
Taking mathematical reasoning tasks example, to segment the reasoning chain into fixed episodes (e.g., "segment up to 30"), we add the following instruction to the prompt file:
<think>
In this section, show your detailed reasoning process. Break down your reasoning into at most 30 logically coherent segments.
Each segment must clearly marked with numbered tags the format <episode_ //episode_21,
</episode_1>, <episode_2
to <episode_30>.
Ensure each <episode_i> should contain only single complete logical move, such definition, formula derivation, transformation,
split.
</think>
For adaptive segmentation (c.g., "break the reasoning chains into K segments"), we add:
<think>
In this section, show your detailed reasoning process. Break down your reasoning into logically coheret segments.
Each segment should be enclosed in <episode_k> </episode_Kt k> tags, such as: <episode_1>... .t..d_./d//eeeeedeee_. <episode 2>. 2
<episodekk...<<ppiiddd...
_>.. </episode_K>
Ensure each <episode_i> should contain only single complete logical move, such as a definition, formula derivation, transformation, or case split. Avoid grouping multiple logical stepe into one.
</think>
G.4.3 Visualization of Episodes and Dense Reward
In practice, episode segmentation guided by semantic completeness and logical separability. is automatically handled by the model during generation and sampling via prompt instructions, rather
36

---

than based on fixed token count. As result. episode lengths are variable. For example, each episode may correspond to a distinct reasoning action, such "defining variables", "substituting into equations", or "structural inference". We provide brief example below:
Q: Let x + 100, where y, 2 are positive integers such that ged(x, 2) 1. How many such ordered triples (x, y, z) are there?
<think>
<episode_1>
We given a positive integer equation + 100 with the constraint that ged(x, y, 2) 1. Our goal is to count the number of positive integer triples satisfying both conditions. [Dense Reward: 0.03]
</episode_1>
<episode_2> 2>
To simplify the counting, observe that the total number of positive integer solutions formula C(99, 2). [Dense Reward: 0.08] </episode_2>
100 is given by the standard stars-and- sorr-andd-arr bars
<episode_3:
Now we need to subtract from this count number triples where ged(x, y, > 1. To do this, we use the Mobius inversion principle to count only the primitive triples with ged = 1.
[Dense Regard: 0.15]
</episode_3>
<episode_
Let d (xd z). Then we write = da, y db, de that + 100/d, and ged(a, b, c)
[Dense Reward: 0.17]
</episode_4>
<episode_5>
We compute number auch primitive triples using the formul::
all that divide of md) C(100 / 1, 2), where mu) the
Mobius function. This ensures that we only count the triples with ged = 1. [Dense Reward: 0.23]
</episode_5>
<episode_8>
Therefore, the total number ordered triples (x, y, z) such that y 2960.
100 and ged(x, y, [Dense Reward: 0.12]
</episode_8>
<episode_9>
We briefly confirm the result by noting that the Mobius-based inclusion- exclusion method is standard and sound for counting primitive compositions.
[Dense Reward: 0.02]
</episode_99 </think>
To accommodate tasks with varying reasoning granularity: for problems with long reasoning chains (c.g., mathematical proof's code generation), single episode may span multiple tokens; in
37

---

contrast, for shorter or more fragmented tasks (e.g., factual questions), episodes may be automatically compressed into short logical units without forced segmentation. Below, we provide two examples. question with short reasoning chain:
q: Lily has pencils. She buye 5 more and gives 3 to her friend. How many pencils does she have now?
<episode_1s Lily starts with pencils. </episode_it <episode_2> She buys 5 more, so now she has 7 + 5 12 pencils. </episode_2> <episode_3> She gives away 3 pencils, so 12 3 = 9 pencils remain. </episode_3 3>
A question with (relatively) long reasoning chains:
Q: Let b, and be real numbers such that 6,
ab ca = 9, abc = 2.
Find ar3 + b-3 + c-
<episode_1x are given episode_1>
6, ab ca 9, and abc 2. </
<episode_2> 2> Recall the identity: a b-3 c3 (a + c) 3(a + b c)
ab bc + ca) + 3abc. </episode_ 2>
<episode_3> Compute (a b c) 6-3 216. </episode_3: 3>
<episode_4> Compute 3(a
(ab
ca)
9 162. </episode_4>
<episode_5> Compute 3abc = 3 6. </episode_5 5>>
<episode_6 6> Substitute into the identity: 216 162 6 60. </episode_6>
38


Reviews:
[{"review_id": "61rMuWunsw", "reviewer": "Reviewer_fdAZ", "summary": "This paper proposes L2T, which uses an information-theoretic dense reward for RL training of reasoning models. The core idea is to estimate information gain by comparing the probability score of the answer with and without the corresponding reasoning episode.", "questions": "See the weakness for more questions\n\n- How are the generated reasoning chains segmented, and how do you control the number of episodes (up to 30)?\n- How do you break the reasoning chains into *K* segments?\n- Why do you think outcome-reward RL does not lead to a significant accuracy improvement?\n- What would happen if L2T were applied to non-reasoning models like Qwen1.5?", "limitations": "The authors stated that they discussed the limitation in the checklist, but I couldn’t find it.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The proposed solution is interesting. Directly using the informativeness of each reasoning step as a process reward makes sense.\n- The method is general and could be applied to various forms.\n- The paper is written with a clear narrative and is easy to follow, although some parts feel unnecessarily long.\n\n**Weaknesses:**\n\n- There are many existing works addressing overthinking and the elimination of redundant steps. The paper should include this line of research on concise reasoning and consider adding them as baselines. [1]\n- Continuing from that, the motivation is already well-addressed in prior work. Since the paper makes a solid contribution, I suggest trimming the motivation and instead adding more extensive evaluations, such as on coding tasks.\n- The exact implementation details for the low-rank approximation are unclear; the explanation is too verbose and lacks clarity. Efficiency aspects are also not well discussed.\n- The baseline scores are much lower than those reported in the original papers (DeepSeek and DeepScaler). Is there a reason for this discrepancy?\n- There is no qualitative analysis. I’m curious how the method eliminates redundant reasoning at low vs. high alpha values.\n- There are several typos (e.g., effectiiveness) and unclear phrases (e.g., “both models” in line 163, while only one result is shown in the main paper).\n\n**Reference:**\n\n[1] Chen et al., *Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs*, 2024", "was_revised": true, "final_justification": "As outlined in my review, the method makes sense and also the author rebuttal addresses most concerns, although there are some presentation issues. Therefore, I’ve raised my score to 5.", "rebuttal": "# Responses for Reviewer fdAZ\n\nWe sincerely appreciate the reviewer fdAZ's constructive suggestions and the time and effort dedicated to the review process. We are also grateful for the recognition of our work and sincerely hope the following responses can help address the concerns.\n\n## **Response to Weaknesses 1 & 2**\n\n- We would like to kindly clarify that in the original paper, we have cited [1] as [9]. As suggested, we further added more discussion and comparison with [1] in Sections 2 and 5: \n  - While both our method and [1] aim to reduce unnecessary reasoning in LLMs, our goals and mechanisms differ. [1] mainly focuses on early stopping effects, aiming to detect when the model has already produced the correct answer and can halt generation. It may still allow inefficient reasoning traces before the correct answer appears. In contrast, our method proactively maximizes per-step information gain during training, encouraging the model to only generate reasoning steps that are most useful, without relying on any external stopping criterion. \n  - We are supplementing the comparison with the baseline [1] using the same settings (currently running). Due to time constraints for the rebuttal, we will add the comparative experimental results in the final version.\n- We have compared L2T with multiple baselines targeting \"overthinking\" and \"efficiency\" in Sections 5, e.g., [2, 5,36,62]. As shown in Tables 1–2, L2T achieves nearly 2% performance gain while using less than 70% of the tokens, outperforming prior SOTA methods.\n- We sincerely appreciate the reviewer's constructive suggestions and affirmation of our work. According to the comments, we further refined the motivation section and moved the experimental results on coding tasks from Appendix F.2 to the main text (Figure 8).\n\n\n\n## **Response to Weakness 3**\n\n**Implementation for low-rank approximation**: The goal is to obtain a low-rank approximation $\\tilde{\\theta} \\in \\mathbb{R}^r$ of the model parameters $\\theta \\in \\mathbb{R}^d$ ($r \\ll d$), avoiding direct computation of the Fisher matrix in high-dimensional space (Appendix C.1 and L288–299). Specifically, we apply SVD to extract the principal directions of variation in $\\theta$ and retain the top $r$ components (with $r/d \\approx 1%$–10% for 1.5B and $r/d \\approx 0.1%$–1% for 7B), resulting in a low-rank surrogate $\\tilde{\\theta}$.\n\n**Improvement**: Building on Appendix C.1, we further incorporated the above explanation into the main text and revised the writing to improve clarity, following the valuable suggestions.\n\n**Efficiency**: \n\n* **For the mentioned low-rank approximation:** \n  * **Method:** This mechanism avoids directly calculating the Fisher matrix in high-dimensional space, and instead uses low-rank proxy calculations to reduce the amount of computation and improve efficiency. \n  * **Evidence:** Appendix C.1 theoretically shows that the method reduces complexity from $\\mathcal{O}(d^2)$ to $\\mathcal{O}(dr + r^2)$; Section 5.3 and Appendix F.3 (configurations ii and iii) empirically demonstrate that it introduces minimal computational overhead while improving performance (Figures 4 and 9).\n* **For L2T:**\n  * **Method:** L2T introduces a general information-theoretic dense process reward to guide the model toward reasoning steps that yield the most performance gains. By assigning higher rewards to informative steps and down-weighting redundant ones, the model learns to recognize effective reasoning. Then, the policy optimization based on this reward encourages the model to make the most of each step, generating accurate answers with minimal token usage.\n  * **Evidence:** The comparison results in Section 5.2 and Appendix F.2 show that L2T triples LLM's token efficiency (Figure 2-3) and improves accuracy by more than 2% (Table 1-2) across various tasks.\n\nWe appreciate the constructive suggestions and have further polished the text and emphasized the above analysis in the main text as suggested.\n\n\n## **Response to Weakness 4**\n\nThis discrepancy may stem from differences in experimental settings and evaluation protocols. We follow the setup of [36] for our evaluations, and the scores of all baselines are consistent with those reported in [36] (Section 5.1 and Appendix E).\n\n\n\n## **Response to Weakness 5**\n\nFollowing the valuable suggestions, we further added a qualitative analysis for $\\alpha$. \n\nThe coefficient $\\alpha$ denotes the weight of our proposed process reward (Eq.5). When $\\alpha$ is small, the model relies more on the outcome reward, which provides high rewards only when the correct answer is found. Without guidance and given that correct answers are sparse in the output space, the model may consume a large number of tokens in exploration, reducing efficiency. In contrast, when $\\alpha$ is large, the model is driven by the process reward, which assigns high rewards only if the current reasoning step has a high contribution to the accuracy of the answer, i.e., the correct answer is reached within a short token sequence. This encourages the model to generate informative tokens at each step, thereby improving efficiency.\n\nTake the Tier-1 Omni-MATH problem “What is 12 + 5?” as an example, the following qualitative analysis shows that with $\\alpha = 0.9$, the model may answer within 2–3 steps, whereas with $\\alpha = 0.2$, it may take more than 7 steps. More cases have been included in Appendix F.4.\n\n```\n- alpha=0.9:\n12 plus 5 equals 17.\nAnswer: 17\n\n\n- alpha=0.2：\nTo solve this, we first recall the addition operation.\nAddition is combining two numbers into a total.\nLet’s start with 12 and add 5.\n12 plus 5 equals 17.\nWe double-check: 10 + 5 = 15, so 12 + 5 = 17.\nThe final result should be 17.\nAnswer: 17\n```\n\n## **Response to Weakness 6**\n\nWe sincerely appreciate the reviewer's suggestion and have made the corresponding revisions, including:\n- correcting “effectiiveness” to “effectiveness” in L186;\n- changing “Figure 1” to “Figures 1 and 6” in L162.\n\n\n\n## **Response to Questions 1 & 2**\n\nWe automatically segment the chains by designing specific prompts. Taking mathematical reasoning tasks as an example:\n\nFor the motivation experiment (\"segment up to 30\" in Question 1), we added the following instruction to the prompt file:\n\n```\n<think>\nIn this section, show your detailed reasoning process.  Break down your reasoning into at most 30 logically coherent segments.\n\nEach segment must be clearly marked with numbered tags in the format <episode_1> ... </episode_1>, <episode_2> ... </episode_2>, ..., up to <episode_30>.\n\nEnsure each <episode_i> should contain only a single complete logical move, such as a definition, a formula derivation, a transformation, or a case split.\n...\n</think>\n```\n\nFor performance analysis (\"break the reasoning chains into K segments\" in Question 2), we added:\n\n```\n<think>\nIn this section, show your detailed reasoning process.  Break down your reasoning into logically coheret segments.\n\nEach segment should be enclosed in <episode_k> ... </episode_k> tags, such as:\n<episode_1>...</episode_1>\n<episode_2>...</episode_2>\n...\n<episode_K>...</episode_K>\n\nEnsure each  <episode_i> should contain only a single complete logical move, such as a definition, a formula derivation, a transformation, or a case split. Avoid grouping multiple logical steps into one.\n\n...\n</think>\n```\n\n\n\n## **Response to Question 3**\n\nWe would like to kindly clarify that while we acknowledge the effectiveness and potential of outcome-reward RL, its improvements may be limited *under constrained token budgets*, and in some cases, performance degradation could occur. \n\nSpecifically, when the model relies solely on outcome rewards, high rewards are only given upon reaching the correct answer. Without intermediate guidance and given the sparsity of correct answers in the output space, the model may consume a substantial number of tokens during exploration, potentially affecting efficiency (L31–47). \n\nThe motivation experiment in Section 3.2 and performance analyses in Section 5 further demonstrates this. Under limited budgets, outcome-reward RL may introduce unnecessary reasoning in simpler tasks, which could hinder decision-making, or prematurely exhaust tokens in complex tasks, risking the truncation of critical steps and a drop in overall performance.\n\n\n\n## **Response to Question 4**\n\nWhen applied to the mentioned \"non-reasoning models like Qwen1.5\", L2T can still yield performance improvements, as supported by the following rationale and evidence:\n\n1. **Methodology**: L2T does not rely on strong inherent reasoning abilities of the base models; even for models with limited reasoning capabilities, L2T can still be effective by using heuristically segmented episodes and learning from dense feedback during training. Moreover, L2T operates based on the model’s internal probability outputs and parameter dynamics, without requiring external evaluators, making it applicable to a wide range of LLMs.\n\n2. **Empirical Evidence**: Appendix F.2 presents the experimental results on Qwen2-7B-Instruct. Similar to the mentioned Qwen1.5, Qwen2-7B-Instruct has not been fine-tuned specifically for reasoning tasks; its reasoning ability largely stems from broad coverage in pretraining data [1,2]. As shown in Figure 8, compared to GRPO, applying L2T leads to over a 4% improvement on code reasoning tasks.\n\n[1] Qwen Technical Report.\n\n[2] Qwen2 Technical Report.\n\n\n\n## **Response to Limitations**\n\nWe would like to kindly clarify that we have discussed the limitation in Appendix C.3, and sincerely apologize for any concern it may have caused by the location. We have now moved the corresponding discussion in Appendix C.3 to the end of Section 6 in the main text."}, {"review_id": "CzvJURLE02", "reviewer": "Reviewer_Sp46", "summary": "The paper introduces a general-purpose dense reward framework for Reinforcement Learning in LLMs to improve their reasoning abilities. The purpose of their framework is to avoid wasteful tokens that do not add value to the final prediction, improving not only efficiency but also performance by removing clutter from the context of the model. They show superior performance and efficiency compared to other contemporary approaches.", "questions": "Addressing the weaknesses mentioned above should be feasible in the rebuttal, so I expect to see the new descriptions (or an explanation as to what is needed to address my confusion) that will appear in the paper.", "limitations": "The authors have not adequately discussed the limitation of having to select a token budget during training. Ideally, the token budget would emerge organically from training or from the rewards themselves, and would be different for problems of different difficulty. While not a weakness per se, it would be good to have a discussion of the issues and potential avenues for future research.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is simple to follow in retrospect and intuitively makes sense as to why anyone would want to apply similar methods, which is a major strength of the paper.\n- It studies an important problem and proposes an efficient solution that has a good chance of having an impact in the community.\n\nWeaknesses:\n- Confusion with current notation and terminology. Examples: L120 $B_{token}$ and $T_{max}$, some terms defined in page 3 and appear again in page 6, making reading the paper a bit more difficult.\n- L157: where does randomness arise if temperature is set to 0 (L150)\n- Intuition lacking behind some design choices:\n    - L266: what is the intuition behind defining the fitting information gain wrt as the reduction in uncertainty arising from $\\theta$? In L269, why is the $\\theta_k$ and $\\theta_{k-1}$ used, and what do these mean? In L271, why is $J_r$ not defined mathematically in the approximation?\n    - L275: what is the intuition again behind the definition of the compression penalty using $\\theta_k$ and $\\theta_{k-1}$?\n- Weird citation: [26] is a generic citation that does little to explain how a Gaussian distribution is a mild assumption. You should explain how the CLT applies to model parameters.\n- Please fix the legends in figure 2. (you can keep only one and put it next to the final figure to make the bars visible)\n- Please fix Figure 4, it look like it should be a barplot. Otherwise, I would appreciate the intuition behind having a regular line plot.", "was_revised": true, "final_justification": "There were not major points of contention for me in the paper, just some clarifications required to improve readability. The authors have adequately addressed any concerns, and I did not find any major weaknesses unaddressed from other reviewers, so the score can remain unchanged.", "rebuttal": "# Responses for Reviewer Sp46\n\nWe sincerely appreciate the reviewer Sp46's constructive suggestions and the time and effort dedicated to the review process. We are also grateful for the recognition of our work, which is a great encouragement to us, and sincerely hope the following responses can help address the concerns.\n\n## **Response to W1**\n\nWe sincerely appreciate the valuable suggestions and have carefully polished the paper, including:\n\n* Removed redundant explanations of previously defined symbols (e.g., $B_{token}$ and $T_{max}$), defining them in detail only where they first appear.\n* Added a symbol table before Appendix A, listing all symbols and their corresponding meanings.\n\n\n\n## **Response to W2**\n\n* In the motivation experiment, we use greedy decoding to focus on analyzing the impact of outcome reward RL on the trade-off between efficiency and effectiveness (L150), avoiding the bias and randomness introduced by sampling. However, considering that there may exist other sources of randomness, such as the model’s adaptive episode segmentation guided by prompts, where different truncation points may slightly affect the context and outputs, we still adopt the commonly used evaluation metric maj@4 (L157).\n\n* In contrast, for performance analyses, we do not use greedy decoding; instead, we set the temperature to 0.6 (L887), following the same experimental setup and evaluation protocol with baselines for comparison.\n\n* We appreciate for pointing out this insightful question and further emphasized the above clarification at L157.\n\n\n\n## **Response to W3**\n\nWe would like to provide an outline to convey the intuition behind our design:\n\n**For the fitting information gain:**\n\n- **How to interpret LLM reasoning from an information-theoretic perspective (fitting gain vs. uncertainty):** The reasoning process of an LLM can be viewed as inferring the correct answer $Y$ from input $X$. The more certain the model's prediction is, the better it \"understands\" the answer. Based on classical information theory [3, 25], we can use conditional entropy $H(Y|X)$ to characterize the model's uncertainty about the output $Y$. If the model is sufficiently confident, its $H(Y|X)$ should be low; conversely, if it is uncertain or making a blind guess, $H(Y|X)$ should be high. Within this framework, the goal of inference is to gradually reduce $H(Y|X)$ until the correct answer is output. Importantly, while $H(Y|X)$ represents the relationship between the input and output, this process is controlled by the LLM (determined by the parameter $\\theta$). **Therefore, a more reasonable metric is: Given $\\theta$, what is the model's uncertainty about $Y$?** That is, we want $\\theta$ to not only capture the input $X$ but also provide strong discrimination of the correct answer $Y$. Therefore, we use conditional mutual information $I(\\theta; Y | X) = H(Y|X) - H(Y|X, \\theta)$ to measure how much the known model parameters $\\theta$ reduce the uncertainty about $Y$ given $X$. This ties rewards to meaningful reasoning progress, i.e., the reward for each episode depends on how much it helps the model understand the correct answer. Therefore, we define \"fitting information gain as the reduction in uncertainty\".\n\n- **How to characterize the gradual improvement of inference (introducing episode gain with $\\theta_k$ and $\\theta_{k-1}$):** In each episode $k$, the model updates its parameter state from $\\theta_{k-1}$ to $\\theta_k$ by observing the new reasoning step. **The key question is: Does this episode help the model better \"understand\" the answer?** Therefore, we define fitting information gain of this episode as $\\Delta I_k = I(\\theta_k; Y | X) - I(\\theta_{k-1}; Y | X)$. If this incremental gain is significant, it indicates that this episode has helped the model become more certain and effective. Here, $\\theta_k$ and $\\theta_{k-1}$ represent the posterior model parameters before and after learning the knowledge from episode $k$, estimated by the change in log-probability during the forward prediction process.\n\n- **Why the proposed metric corresponds to \"reducing uncertainty\" (approximating information gain with $J_r$):** Because directly calculating mutual information is too complex, we introduce an approximate metric $J_r(\\cdot)$ to represent the model's predicted probability of the correct answer. This metric improves as the model's \"confidence\" increases. Therefore, the fitting gain is approximated as $\\Delta I_k \\approx J_r(\\pi_\\theta(\\cdot \\mid s_k, z_k)) - J_r(\\pi_\\theta(\\cdot \\mid s_k))$, with theoretical guarantees in Appendix C.1 and Proposition C.1. This difference measures whether episode $k$ improves the model's ability to predict the correct answer, reflecting the reduction process, i.e., the increasing reliability of reasoning. **As for the definition of $J_r$**, we briefly described it in L270 and detailed it in Appendix C.1. We sincerely apologize for the confusion caused by the location and have now added the precise formula and definition to the main text.\n\n**For the compression penalty:**\n\n- **Why introducing compression penalty:** In LLM reasoning, if each episode's information causes a significant change, while that episode may provide information gain, it may also capture unnecessary details within that episode, leading to overfitting or redundant computation. We want the model to only learn information that contributes to the answer within the episode. Therefore, inspired by the information bottleneck theory, we introduce a compression penalty based on the fitting term to further improve efficiency. It measures the \"information overhead\" incurred by each episode from an information-theoretic perspective. \n- **How is it measured (intuition behind the design):** If $\\theta_k$ differs significantly from $\\theta_{k-1}$, but the model's prediction performance (i.e., fitting gain) improves only slightly, this step may \"absorb redundant information\". Therefore, we use the mutual information increment $I(\\theta_k; s_k) - I(\\theta_{k-1}; s_{k-1})$ between them *with the fitting term* to measure whether the information in episode $k$ introduces unnecessary overhead.\n- **Why this is called compression:** The term *compression* comes from the idea that a model should retain only the **minimal amount of information sufficient** to perform accurate reasoning. In our setting, the mutual information $I(\\theta; s_k)$ quantifies how much the information of this episode is encoded into the model parameters $\\theta$. A larger value implies that the model has to “store” more bits to fit that episode, akin to using more storage in a compressed file. By penalizing the increase $I(\\theta_k; s_k) - I(\\theta_{k-1}; s_{k-1})$, we explicitly discourage storing redundant information, promoting a more compact (i.e., compressed) internal representation. This aligns with principles from MDL and PAC-Bayes, where generalization is favored when the hypothesis (here, $\\theta$) is simple and concise.\n\nWe have further added the above illustrations in Section 4.2 according to the valuable suggestions.\n\n\n\n## **Response to W4**\n\nWe appreciate the suggestions and have further added a justification for the Gaussian assumption in Appendix C.1. \n\nIn brief, during LLM training, parameter updates $\\theta$ can be viewed as the accumulation of numerous small stochastic gradient steps. Each step introduces a small, random perturbation in parameter space, effectively acting as the sum of many independent random variables. When these perturbations are sufficiently numerous and diverse, the overall distribution of parameter changes tends toward a Gaussian distribution, as implied by the Central Limit Theorem [26]. \n\nTherefore, when estimating the compression term, we model the low-rank surrogate $\\tilde{\\theta}$ (obtained via SVD) using a Gaussian distribution. This assumption aligns with covariance approximation techniques in the PAC-Bayes framework and Fisher information-based methods, allowing us to derive a closed-form expression for mutual information and significantly reduce computational cost. Relevant details have been added to Appendix C.2, with a brief explanation in the main text.\n\n\n\n## **Response to W5 & 6**\n\nWe sincerely thank the constructive suggestions and have improved the figures accordingly: \n\n- The legend in Figure 2 has been moved below to ensure each bar is clearly visible.\n-  Figure 4 has been redrawn as a bar chart to better illustrate the ablation results.\n\n\n\n## **Response to Question 1**\n\nWe sincerely appreciate the reviewer for the valuable suggestions and are grateful for the time and effort devoted to the review. Based on this constructive feedback, we have improved the paper accordingly (as detailed above). We promise that new descriptions will appear in the paper as expectations (\"as to what is needed to address the mentioned confusion\").\n\n\n\n## **Response to Limitations**\n\n\nWe sincerely thank the valuable suggestions and further added the following discussion in Appendix C.3:\n\n- The token budget serves as an important constraint on test-time compute, helping to assess a model’s reasoning ability under limited compute. In our experiments, we adopt the same budget settings and evaluation protocols as the baselines to ensure fair and reproducible comparisons.\n- L2T does not rely on hard budget truncation. Instead, it uses a universal dense reward to guide the model to adaptively select the most informative tokens at each step, enabling dynamic adjustment of reasoning length. The model can flexibly determine the number of steps based on task complexity, converging early on simpler tasks or extending reasoning on more complex ones.\n- This provides a way for more explicit and automated dynamic budget allocation in the future, especially in scenarios sensitive to token costs (e.g., mobile deployment or real-time QA). We sincerely appreciate the constructive comments, where integrating reward signals with adaptive budget control is meaningful."}, {"review_id": "KoR9fPOFQd", "reviewer": "Reviewer_NSuU", "summary": "The authors propose a reinforcement learning framework for training LLMs to achieve optimal reasoning using a dense process reward designed based on an information theoretic approach. Treating each query-response interaction as a series of multiple episodes, the authors define an episode-wise information gain to estimate the contribution of each reasoning step of LLMs in achieving the desired output. This dense reward model based on PAC-Bayes bounds and the Fisher information is then used to optimize LLMs to encourage efficient reasoning. Experiments demonstrate that fine-tuning LLMs with the proposed dense reward model achieves superior results than standard outcome rewards.", "questions": "- Q. How is the length of each episode determined in practice? Does it make sense for tasks in which reasoning steps vary in length?\n- Q. Can the proposed dense reward model used for inference-only methods (best-of-N, beam search, etc.)? How does that perform compared to using other process reward models?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed dense reward is theoretically well-motivated.\n- It has the practical advantages of requiring no human annotations compared to other process reward models.\n- Uses both fine-tuning approaches (outcome reward, length penalty) and inference methods (MCTS) as baselines.\n\nWeaknesses\n- More ablations that would be useful: 1) experiments with models of sizes other than 1.5B -- this would help better understand how the method scales, 2) combining fine-tuning with test-time approaches, e.g., MCTS -- this would help understand whether reasoning efficient fine-tuning avoids the need for test-time compute, or further gains can be achieved.\n- Some qualitative analysis on the dense reward would have been nice to get a better sense of whether the reward model assigns sensible scores to episodes based on the episodes' contributions.", "was_revised": true, "final_justification": "I appreciate the authors' responses with additional experimental results and discussions. It is useful to see that the proposed method combined with test-time inference techniques results in further gains compared to baselines such as GRPO. The additional qualitative examples also better illustrate how models trained with the method perform in terms the length and type of output reasoning traces. While the method is theoretically justified and supported with empirical results, more analysis on why RL with an outcome-based reward model (GRPO) often significantly underperform compared to the proposed method would enhance empirical validation. I'm also a bit concerned with the results in Table 1 where the models become considerably worse after training with GRPO (e.g., DeepScaleR-1.5B-Preview on AMC 2023) and wonder whether a sufficient hyperparameter has been conducted for the baselines -- and if so, what are some plausible explanations of these negative results. Overall, I'm quite happy with research in this direction of designing dense reward signals without explicit training of process reward models and feel the original rating is still fair (but I've increased my confidence score).", "rebuttal": "# Responses for Reviewer NSuU\n\nWe sincerely appreciate the reviewer NSuU's constructive suggestions and the time and effort dedicated to the review process. We are also grateful for the recognition of our work and sincerely hope the following responses can help address the concerns.\n\n\n\n## **Response to Weakness 1**\n\n\n- We would like to kindly clarify that Appendix F.2 presents experiments with models other than 1.5B, e.g., DeepSeek-R1-Distill-Qwen-7B and Qwen2-7B-Instruct. The results in Table 2, Figure 7, and Figure 8 demonstrate the superiority of L2T, achieving over 2% performance improvement while using less than 60% of the tokens compared to SOTA methods. \n- We sincerely thank the reviewer for the valuable suggestions and further conducted toy experiments for evaluation. The following results and more details are also added in Appendix F.5 according to the constructive comments. Briefly, we integrated MCTS into models trained with L2T and GRPO, and evaluated their performance. The results below show that L2T already exhibits strong reasoning capabilities; MCTS brings only marginal improvement for L2T, while GRPO benefits more from MCTS. This further indicates that L2T effectively reduces reliance on large-scale search by guiding the model to generate the most informative tokens.\n\n| Method    | AIME | MinervaMATE |\n| --------- | --------- | ----------- |\n| GRPO      | 44.5      | 24.7        |\n| GRPO+MCTS | 46.8      | 25.3        |\n| L2T       | 48.5      | 26.5        |\n| L2T+MCTS  | 48.9      | 26.7        |\n\n\n\n## **Response to Weakness 2**\n\nWe sincerely appreciate the constructive suggestions and have further added more qualitative analyses in Appendix F.3 and Appendix F.4 as suggested. \n\nSpecifically, we randomly sampled problems from different datasets and visualized the rewards along their reasoning trajectories. An example is shown below. The results indicate that L2T assigns relatively low rewards to non-essential steps (e.g., background information) that may not influence the model’s decision, while assigning higher rewards to critical steps such as final equations, structural counting, and Möbius techniques. \n\nCombined with the quantitative results in Tables 1–2, this further demonstrates the effectiveness of L2T.\n\n```\nQ: Let x + y + z = 100, where x, y, z are positive integers such that gcd(x, y, z) = 1. How many such ordered triples (x, y, z) are there?\n\n<think>\nIn this section, we analyze the problem step-by-step using 9 logically structured episodes.\n\n<episode_1>\nWe are given a positive integer equation x + y + z = 100 with the constraint that gcd(x, y, z) = 1. Our goal is to count the number of positive integer triples satisfying both conditions.\n[Dense Reward: 0.03]\n</episode_1>\n\n<episode_2>\nTo simplify the counting, we observe that the total number of positive integer solutions to x + y + z = 100 is given by the standard stars-and-bars formula C(99, 2).\n[Dense Reward: 0.08]\n</episode_2>\n\n<episode_3>\nNow we need to subtract from this count the number of triples where gcd(x, y, z) > 1 . To do this, we use the Möbius inversion principle to count only the primitive triples with gcd = 1.\n[Dense Reward: 0.15]\n</episode_3>\n\n<episode_4>\nLet d = gcd(x, y, z). Then we write x = da, y = db, z = dc so that a + b + c = 100/d, and gcd(a, b, c) = 1.\n[Dense Reward: 0.17]\n</episode_4>\n\n<episode_5>\nWe now compute the number of such primitive triples using the formula: sum over all d that divide 100 of mu(d) * C(100 / d - 1, 2), where mu(d) is the Möbius function. This ensures that we only count the triples with gcd = 1.\n[Dense Reward: 0.23]\n</episode_5>\n\n...\n\n<episode_8>\nTherefore, the total number of ordered triples (x, y, z) such that x + y + z = 100 and gcd(x, y, z) = 1 is 2960.\n[Dense Reward: 0.12]\n</episode_8>\n\n<episode_9>\nWe briefly confirm the result by noting that the Möbius-based inclusion-exclusion method is standard and sound for counting primitive compositions.\n[Dense Reward: 0.02]\n</episode_9> </think>\n\n...\n```\n\n\n\n## **Response to Question 1**\n\n\n* In practice, episode segmentation is guided by semantic completeness and logical separability. It is automatically handled by the model during generation and sampling via prompt instructions, rather than based on a fixed token count. As a result, episode lengths are variable. For example, each episode may correspond to a distinct reasoning action, such as \"defining variables\", \"substituting into equations\", or \"structural inference\". We provide a brief example in Response to Weakness 2 (the above response).\n\n* To accommodate tasks with varying reasoning granularity: for problems with long reasoning chains (e.g., mathematical proofs or code generation), a single episode may span multiple tokens; in contrast, for shorter or more fragmented tasks (e.g., factual questions), episodes may be automatically compressed into short logical units without forced segmentation. Below, we provide two examples. More examples and visualization results have been added in Appendix F.4, according to the constructive comments.\n\nA question with a short reasoning chain:\n\n```\nQ: Lily has 7 pencils. She buys 5 more and gives 3 to her friend. How many pencils does she have now?\n...\n<episode_1> Lily starts with 7 pencils. </episode_1>  \n<episode_2> She buys 5 more, so now she has 7 + 5 = 12 pencils. </episode_2>  \n<episode_3> She gives away 3 pencils, so 12 - 3 = 9 pencils remain. </episode_3>\n```\n\nA question with (relatively) long reasoning chains:\n\n```\nQ: Let a, b, and c be real numbers such that\na + b + c = 6,\nab + bc + ca = 9,\nabc = 2.\nFind a^3 + b^3 + c^3.\n...\n<episode_1> We are given a + b + c = 6, ab + bc + ca = 9, and abc = 2. </episode_1>  \n<episode_2> Recall the identity: a^3 + b^3 + c^3 = (a + b + c)^3 - 3(a + b + c)(ab + bc + ca) + 3abc. </episode_2>  \n<episode_3> Compute (a + b + c)^3 = 6^3 = 216. </episode_3>  \n<episode_4> Compute 3(a + b + c)(ab + bc + ca) = 3 * 6 * 9 = 162. </episode_4>  \n<episode_5> Compute 3abc = 3 * 2 = 6. </episode_5>  \n<episode_6> Substitute into the identity: 216 - 162 + 6 = 60. </episode_6>\n```\n\n\n\n## **Response to Question 2**\n\nYes, our proposed dense reward can be applied in inference-only settings to score and rerank candidate reasoning paths. \n\nBased on the model DeepSeek-R1-Distill-Qwen-7B (also used in our paper, as shown in Appendix E and F.2), we conducted a toy experiment to rerank multiple reasoning paths. The results in the table below demonstrate the effectiveness of the proposed L2T, i.e., L2T reranking achieves the highest accuracy compared to other methods and PRMs. We sincerely thank the reviewers for the valuable suggestions, and all results are now added to the appendix of the paper.\n\nFurthermore, we have compared L2T with other process-reward-based methods such as MRT and ReST-MCTS in Section 5. The results (Table 1-2, Figures 2, 3, and 7) show that L2T consistently outperforms them, achieving better performance with fewer tokens across multiple datasets.\n\n| Model                       | Method                          | MinervaMATH |\n| --------------------------- | ------------------------------- | ----------- |\n| DeepSeek-R1-Distill-Qwen-7B | Best-of‑N@16 + log-likelihood   | 42.6        |\n| DeepSeek-R1-Distill-Qwen-7B | Best-of‑N@16 + Qwen2.5-Math-PRM | 43.3        |\n| DeepSeek-R1-Distill-Qwen-7B | Best-of‑N@16 + Skywork-PRM      | 43.0        |\n| DeepSeek-R1-Distill-Qwen-7B | Best-of‑N@16 + L2T rerank       | 43.9        |"}, {"review_id": "GYcRsAuOnf", "reviewer": "Reviewer_R3Rw", "summary": "This paper presents Learning to Think (L2T), an information-theoretic reinforcement fine-tuning framework designed to enhance the reasoning effectiveness and efficiency of LLMs.\n L2T addresses the issue of unnecessary long reasoning chains in existing methods by introducing a universal dense process reward that uantifies episode-wise information gain, requiring no extra annotations or task-specific evaluators. The framework estimates this reward based on PAC-Bayes bounds and Fisher information matrix, theoretically reducing computational complexity while maintaining estimation accuracy. \nThrough experiments on various reasoning benchmarks and base models, L2T demonstrates advantages in both reasoning effectiveness and efficiency, adaptively allocating reasoning depth across different tasks to achieve optimal results with minimal token budgets.", "questions": "The paper utilizes two types of rewards: output reward for correctness and process reward aimed at efficiency. \nCould the authors clarify how these two rewards are combined in the advantage estimation step? What specific advantage estimation technique is employed to jointly account for both the correctness and efficiency rewards when optimizing the policy?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. This paper propose a novel framework for efficient chain-of-thought reasoning from the perspectives of information theory based reinforcement learning.\n\n2. This paper provides theoretical basis for the proposed dense process rewards, using PAC-Bayes bounds and Fisher information matrix for estimation.\n\n3. This paper conducts extensive experiments across multiple reasoning benchmarks, and the experimental results show that L2T improves the efficiency and performance.\n\n\n# Weaknesses\n1. The paper focuses on efficient chain-of-thought reasoning. \nTo enhance the comprehensiveness of the experimental validation, the authors are encouraged to compare their approach with more related works that leverage reinforcement learning to improve reasoning efficiency.\n\n2. The writing in Sections 3 and 4 is somewhat disorganized, with extensive use of in-line formulas making it difficult to read. It is recommended to clarify these sections. Moreover, presenting the overall methodological framework using a schematic diagram would help improve clarity and reader understanding.", "was_revised": true, "final_justification": "4: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.\n\nThe author has addressed my concerns.", "rebuttal": "# Responses for Reviewer R3Rw\n\nWe sincerely appreciate the reviewer R3Rw's constructive suggestions and the time and effort dedicated to the review process. We are also grateful for the recognition of our work and sincerely hope the following responses can help address the concerns.\n\n\n\n## **Response to Weakness 1**\n\n\n- We have compared L2T with *several representative RL-based methods that focused on efficient reasoning and were released before the NeurIPS 2025 deadline*, such as ReST-MCTS [62] (Nov 2024), length penalty [2] (Feb 2025), and MRT [36] (Mar 2025). All comparisons were conducted under the same experimental setup and evaluation protocol. The results in Table 1–2 and Figures 2, 3, and 7 demonstrate the superiority of L2T, i.e., compared to the SOTA baselines, it achieves over 2% performance improvement while using less than 70% of the tokens.\n- According to the reviewer's valuable suggestions, we further added more comparisons with recent works that focus on \"efficiency\" (publicly released or reported results under the same evaluation protocol). The table below shows results against methods published after early 2025, showing that L2T achieves best performance with a smaller token budget. All the results are added in the revised version. In addition, we have also compared L2T combined with PRM on inference-only baselines. The results further confirm the advantage of L2T (please see Response to Q2 for Reviewer NSuU). We are also comparing with some contemporaneous work (published after Jun 2025). Due to the time constraints of the rebuttal, the results on all the base models will be supplemented in the final version. \n\n| Model                         | Method         | Date | AMC  | Avg. Token Cost |\n| ----------------------------- | -------------- | ---------------------- | ---- | --------------- |\n| DeepSeek-R1-Distill-Qwen-1.5B | length penalty | Feb 2025               | 64.4 | 1x              |\n| DeepSeek-R1-Distill-Qwen-1.5B | TERL           | Apr 2025               | 69.8 | 1.4x            |\n| DeepSeek-R1-Distill-Qwen-1.5B | VRDecouple     | Apr 2025               | 72.4 | 1.7x            |\n| DeepSeek-R1-Distill-Qwen-1.5B | RL-STAR        | Apr 2025               | 71.2 | 2.1x            |\n| DeepSeek-R1-Distill-Qwen-1.5B | MRT            | Mar 2025               | 72.9 | 1.6x            |\n| DeepSeek-R1-Distill-Qwen-1.5B | L2T            | Ours                   | 73.5 | 1.1x            |\n\n\n\n## **Response to Weakness 2**\n\n\n- According to the reviewer's valuable suggestions, we have carefully revised and restructured the in-line formulas in Sections 3 and 4. Specifically, the previously in-line formulas (L267, L269, L271) have been reformatted as standalone numbered equations, each accompanied by concise contextual comments for clarity.\n- Following the constructive comments, we have also added a framework diagram to illustrate the actual workflow of L2T (added in Appendix B; the section title has been updated from \"Pseudo-Code\" to \"Pseudo-Code and Overall Procedure\"). Since visual figures cannot be included during the rebuttal phase, we provide the following outline for reference:\n  - **Sampling & Generation**: We randomly sample a question $x$ from the training set and use the old policy $\\pi_{\\theta_{\\text{old}}}$ to generate $G$ reasoning rollouts. Each rollout is automatically segmented into episodes using the `<think>` delimiter defined in the system prompt, allowing different logical fragments to be evaluated independently.\n  - **Reward Computation**: For each rollout, the reward $R_{i,k}$ consists of two parts. The first is a sparse outcome reward, i.e., $r^{\\text{out}} = \\mathbb{1}[z_{1:K} \\text{ leads to correct } y^*]$. The second is the proposed dense process reward, i.e., $r_k^{\\text{prg}} = \\Delta I_k - \\beta C_k$, where the information gain term is $\\Delta I_k = J_r(\\pi_\\theta(\\cdot \\mid s_k, z_k)) - J_r(\\pi_\\theta(\\cdot \\mid s_k))$, and the compression penalty is $C_k = I(\\theta_k; s_k) - I(\\theta_{k-1}; s_{k-1})$  (Please see Section 4.2 and Appendix C.1 for details). Then, the reward for episode $k$ can be expressed as $R_{i,k} = r_i^{\\text{out}} / K_i + \\alpha r^{\\text{prg}}_{i,k}$.\n  - **Advantage Estimation**: Following GRPO, we use token-level log-probability surprise as the weighting signal. Each $R_{i,k}$ is distributed over tokens using weights $w_{i,t} \\propto -\\log p_{\\theta_{\\text{old}}}(z_{i,t} \\mid s_{i,t})$, resulting in per-token reward $r_{i,t}$. The truncated mean of these rewards (95%) gives $\\tilde{r}\\_i$. We then compute the group-level advantage as a z-score, i.e., $\\hat{A}\\_i = (\\tilde{r}\\_i - \\bar{\\tilde{r}}) / \\sigma\\_{\\tilde{r}}$,\n    and rescale it to each token using its relative contribution, i.e., $A_{i,t} = \\hat{A}\\_i \\cdot (r\\_{i,t} / \\tilde{r}\\_i)$.\n  - **Optimization**: The final advantage $A_{i,t}$ is used in GRPO’s clipped policy gradient, with an additional KL penalty term to stabilize training.\n\nIn summary, L2T is implemented on top of GRPO, with the key differences being the explicit construction of episodes and the integration of the proposed universal information-theoretic dense process reward.\n\n\n\n## **Response to Question 1**\n\nIn the above **Response to Weakness 2**, we provide a formal illustration about the reward composition and advantage estimation of L2T. Building upon the above response, we would like to provide an outline to further answer each point raised in the \"Questions\":\n\n1. **The combination of the rewards for the following advantage estimation:**\n   - **Implementation:** The two rewards are first combined at the episode level, i.e., the outcome reward $r^{\\text{out}}$ is uniformly distributed across $K_i$ episodes, while the process reward $r^{\\text{prg}}\\_k$, which measures information gain and includes a compression penalty, is scaled by a weight $\\alpha$. The total episode reward is $R_{i,k} = r_i^{\\text{out}} / K_i + \\alpha r^{\\text{prg}}\\_{i,k}$. This reward is then distributed to each token within the episode, weighted by the token’s surprise.\n   - **Benefit to Effectiveness and Efficiency:** (i) This design considers both dense progress and global correctness, encourages the model to learn the most efficient and effective steps; (ii) The process reward encourages the model to focus on episodes that provide the most useful information while penalizing redundant ones; (iii) The reward relies on internal model signals (information gain), making it broadly applicable without external annotations.\n\n2. **The specific advantage estimation for policy optimization:**\n   - **Implementation:** L2T adopts a GRPO-style advantage estimator that combines normalization and surprise-based token weighting. Episode rewards $R_{i,k}$ are first allocated to tokens based on their negative log-probabilities $w_{i,t} \\propto -\\log p_{\\theta_{\\text{old}}}(z_{i,t} \\mid s_{i,t})$, forming token-level rewards $r_{i,t}$. A 95% truncated mean across rollouts is then z-normalized to obtain a group-level advantage $\\hat{A}\\_i = (\\tilde{r}\\_i - \\bar{\\tilde{r}}) / \\sigma\\_{\\tilde{r}}$, which is redistributed to each token proportionally to its contribution, i.e., $A_{i,t}$.\n   - **Benefit to Effectiveness and Efficiency:**  This method ensures that each token receives a reward signal reflecting both its local importance and overall trajectory quality, guiding the model to prioritize tokens with the highest expected performance gain.\n\n3. **Empirical Evidence:** Experiments in Sections 5.2 and 5.3 show that the token efficiency of L2T can be nearly doubled, while still outperforming other methods in accuracy (Table 1-2, Figures 2, 3, and 7); the proposed dense reward and advantage estimation together lead to more efficient and accurate reasoning under limited compute budgets (Figures 3, 4, and 9).\n\nIn summary, based on the above process, the model can adaptively adjust reasoning depth within a limited budget, generating tokens with the highest contribution, thus achieving higher accuracy with fewer tokens.\n\nWe have supplemented the above illustrations in the appendix and further summarized them in the main text based on the valuable suggestions."}]

Author rebuttal:

Meta-review: This submission proposes Learning to Think (L2T). It is an information-theoretic reinforcement fine-tuning framework with the goal of improving the reasoning efficiency and effectiveness of llms. Specifically, in L2T the authors introduce a dense process reward that estimates episode-wise information gain based on PAC-Bayes bounds and the Fisher information matrix. In doing so, it requires no additional annotations or task-specific evaluators. Experiments on reasoning benchmarks show the proposed method offers improvements over standard outcome-based RL methods.

Strengths:  

1. dense reward design: the information-theoretic approach to defining process rewards is well-motivated and avoids the need for annotated reasoning traces, making the method general and broadly applicable.
2. improved reasoning efficiency: experiments demonstrate that L2T boosts both reasoning accuracy and efficiency across multiple benchmarks, outperforming existing outcome-reward RL baselines.
3. theoretical grounding: the use of PAC-Bayes bounds and Fisher information provides a sound theoretical basis for the reward estimation. 

Weaknesses that are raised by the reviewers: 

1. incomplete evaluation: while the results are promising, several reviewers noted the lack of ablation studies, scaling experiments on larger models, and comparisons with more related works addressing concise reasoning and RL-based efficiency improvements.
2. clarity and presentation: sections with dense formulas are difficult to follow, and the lack of a clear schematic diagram for the overall framework makes it harder for readers to grasp the main ideas.
3. limited qualitative insights: the submission lacks qualitative analyses demonstrating how redundant reasoning is reduced and how reasoning depth adapts under different reward weights.

overall, this submission presents a technically sound and well-motivated method that addresses an important challenge in improving LLM reasoning efficiency. While some evaluations and presentation aspects could be strengthened, the proposed approach is theoretically sound, practical, and demonstrates consistent empirical gains. Most importantly, the authors managed to address many of the issues during the rebuttal phase, thus two reviewers did raised their ratings. Given the four positive ratings from four reviewers and the addressed concerns during rebuttal, an acceptance recommendation is given to this submission.
Decision: Accept (poster)