Title: Optimizing Anytime Reasoning via Budget Relative Policy Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Anytime Reasoning

Reinforcement Learning

LLM Reasoning

GRPO
Abstract: Scaling test-time compute is crucial for enhancing the reasoning capabilities of large language models (LLMs). Existing approaches typically employ reinforcement learning (RL) to maximize a verifiable reward obtained at the end of reasoning traces. However, such methods optimize only the final performance under a large and fixed token budget, which hinders efficiency in both training and deployment. In this work, we present **AnytimeReasoner**, a novel framework for optimizing reasoning performance under varying thinking budget constraints. To achieve this, we truncate the complete thinking process to fit within sampled token budgets from a prior distribution, compelling the model to summarize the optimal answer for each truncated thinking for verification. This introduces verifiable dense rewards into the reasoning process, facilitating more effective credit assignment in RL optimization. We then optimize the thinking and summary policies in a decoupled manner to maximize the cumulative reward. Additionally, we introduce a novel variance reduction technique, **Budget Relative Policy Optimization (BRPO)**, to enhance the robustness and efficiency of the learning process when reinforcing the thinking policy. Empirical results in mathematical reasoning tasks demonstrate that our method consistently outperforms GRPO across all thinking budgets under various prior distributions, enhancing both training and token efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an innovative reinforcement learning framework, AnytimeReasoner, which optimizes the real-time reasoning ability of LLM through budget sampling and dense reward mechanism. The core contributions include: budget sampling mechanism, BRPO variance reduction technology, and policy decoupling optimization. The paper verifies the superiority of the method on 6 mathematical reasoning datasets (AMC/AIME, etc.).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1)  Dense Reward Mechanism: Generating verifiable intermediate rewards through mind truncation to solve the credit allocation problem caused by sparse rewards.\n\n2) ​Performance Boundary Proof​​: Deriving the performance relationship between real-time reasoning and standard reasoning to provide a theoretical guarantee for the framework.\n\n3) Tree-like attention mask (Figure 9): Solve the computational redundancy of traditional sequence training.\n\n\nWeaknesses:\n\n1) Convergence proof missing: BRPO's upper bound on gradient variance is not quantified (Formula 6)\n\n2) Reward function assumptions: r(x,y)≥0 is required (Formula 12), and negative reward scenarios are not discussed\n\n3) Limitation of baseline comparison: lack of comparison with classic RL algorithms such as PPO/REINFORCE\n\n4) Insufficient domain generalization: only validates mathematical reasoning (algebra/geometry), does not cover scientific reasoning/decision making\n\n5) Theoretical supplement: Add BRPO convergence proof (upper bound of gradient variance)\n\nQUESTIONS:\n1) Does the gradient derivation of formula 6 imply a KL constraint? If V(x,z_{<t}) is removed, will the strategy collapse?\n\n2) Does the negative reward scenario (such as wrong answer penalty) destroy the inequality chain of formula 13? Additional theoretical analysis is needed\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AnytimeReasoner, a novel framework designed to optimize the \"anytime reasoning\" capabilities of LLMs under varying computational budgets. The core technique involves sampling thinking budgets from a prior distribution, truncating the reasoning trace at these budgets, and compelling the model to generate a summary. Overall this work proposes three key contributions: \n\n- **Verifiable Dense Rewards in Anytime Reasoning**: This work introduces verifiable dense rewards throughout the anytime reasoning trajectory, which provides richer training signals and enables more effective credit assignment compared to the sparse rewards of conventional methods.\n\n- **Budget Relative Policy Optimization (BRPO)**: A novel variance reduction technique that leverages these dense rewards to stabilize and improve the learning process.\n\n- **Decoupled Optimization**: A strategy for separately optimizing the thinking and summary policies to ensure high-quality output from incomplete thoughts, resulting in a better summary policy for anytime reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n1. This paper introduces a novel and practical concept of \"anytime reasoning\" to LLMs, addressing a practical problem where computational resources are limited or users are cost-sensitive, especially in online services.\n\n2. This paper proposes a novel mechanism for introducing verifiable dense reward. By truncating the reasoning process and generating summaries at various budgets, the framework artfully transforms the sparse reward problem common in RL into a dense reward one. This provides a powerful signal for credit assignment and a richer way to analyze the model's reasoning process.\n3. Through solid ablation studies, the paper validates the effectiveness of its three core components: verifiable dense rewards, decoupled optimization, and BRPO variance reduction. This clearly illustrates each component's contribution to the performance gains.\n\n\n### Weakness\n\n1. The study's findings are based on a single model architecture (DeepSeek-R1-Distill-Qwen-1.5B) and are validated exclusively on mathematical reasoning tasks. This narrow scope makes it unclear if the AnytimeReasoner framework would be equally effective for other foundational model series, reasoning domains, or for tasks requiring response lengths beyond the 8,000-token budget used in the experiments.\n\n2. The framework introduces several new hyperparameters, such as the set of budget points $\\{b_1,\\cdots,b_B\\}$ used in training , the discount factor $\\lambda$ in BRPO , and the choice of prior distribution $p_B$. Although the paper tests a few distributions, it lacks an in-depth discussion of how these hyperparameters were chosen and how sensitive the model's performance is to them.\n\nQUESTIONS:\n1. What is the computational overhead of the AnytimeReasoner framework during training compared to a vanilla GRPO implementation, especially considering the tree-like generation and multiple summary evaluations?\n2. Have you considered evaluating the AnytimeReasoner framework on other foundational models like Llama, across different model scales, or extending to longer response length (>8k) ? These experiments may further validate the effectiveness of your method.\n3. What is the intuition or theoretical explanation for using a linear prior distribution for budget sampling?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper suggest anytime reasoning, a way to predefine the thinking budget and reason based on the budget. To implement this effectively, the author proposes to truncate the reasoning process based on the given budget and further propose budget relative policy optimization to reduce the variance.  The experimental result show that the propose method largely improve the efficiency and effectiveness over GPRO on anytime reasoning scenario.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1) I believe this topic is important as overthinking (and thinking complexity) is one major problem of current reasoning models. In this sense, anytime reasoning is a novel topic to tackle this issue.\n\n2) Also, the method is very well-designed. All components are reasonable and also somewhat simple.\n\n3) The experiments are very well conducted. I like the fact that the authors have considered multiple benchmarks, including easy to hard datasets. Also, I like the analysis. All components are verified well.\n\n4) Overall, the paper is well written and easy to follow\n\n**Weakness**\n\nI do not find a major weakness in this paper. I think the overall paper is very well written and should be accepted to NeurIPS.\\\nThe following are some minor suggestions, and I hope these are well resolved throughout the rebuttal.\n\n1) I can see that the proposed method is more effective on challenging datasets such as AMC or AIME. Does the author have any intuition behind this improvement? To validate this, I think it would be great if the author can provide the performance on easy dataset (e.g., GSM8K) and also include some challenging ones such as AMC 2023, AIME 2025 and GPQA diamond as well (I think it would have similar trends as AIME 2024 and AMC 2023, but will maybe worth it).\n\n2) While I do agree that math is one reasoning-intensive domain, but also considering coding domains will benefit the readers, e.g., MBPP, Humaneval\n\n3) Can the author provide the final accuracy based on no budget (i.e., generate until the model outputs the answer)? I think it is also good to know the upper bound performance without the constraints.\n\n4) One concern is that the budget should be pre-defined before solving the problem. If there is a simple way to estimate the budget (i.e., complexity of the problem) before thinking, it would be great.\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nSee the weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to improve the token efficiency of large language models by optimizing the accuracy of summarization across different token budgets with reinforcement learning. The authors introduce dense rewards uniformly spaced in the reasoning process. To ensure stable optimization with dense rewards using GRPO, the authors propose two variance reduction baselines, one considering rewards before some certain token budget, and one computing the average reward of all trajectories. The correlation analysis along with the variance analysis shows that the proposed techniques effectively reduces variances.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The writing is clear and easy to follow.\n- This work introduces a natural online extension to the prior work, Meta Reinforced Fine-Tuning, by introducing dense rewards.\n- The proposed variance reduction techniques have shown to be able to reduce the variance by a clear margin. \n\nWeakness:\n- The major issue is lack of sufficient baselines. There are many prior works on improving the token efficiency through training, e.g. MRT [1], Efficient Reasoning [2], and O1-Pruner [3]. \n- There exists a convergence issue regarding the experiment setup. The GRPO baseline has not converged according the benchmark results and training curves. If the DeepSeek-R1-Distill-Qwen-1.5B model is well trained, it should achieve at least 40+ score on the AIME24 benchmark.\n- Lack of evaluation on other LLMs. The experiments are all taken on the DeepSeek-R1-Distill-Qwen-1.5B model, limiting the scope of evaluation.\n\n[1] Qu, Yuxiao, et al. \"Optimizing test-time compute via meta reinforcement fine-tuning.\" arXiv preprint arXiv:2503.07572 (2025).\n[2] Arora, Daman, and Andrea Zanette. \"Training Language Models to Reason Efficiently.\" arXiv preprint arXiv:2502.04463 (2025).\n[3] Luo, Haotian, et al. \"O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning\" arXiv preprint arXiv:2501.12570v2 (2025).\n\nQUESTIONS:\n1. In line 113 - 120, the claim \"a larger thinking budget is supposed to yield better performance in expectation\" is questionable. It is possible that an LLM could have higher accuracy under a restricted thinking budget than the full thinking budget. Therefore Eq. 4 and Eq. 5 may not hold in practice.\n\n2. A natural counterpart for the introduced variance reduction techniques is using value models with PPO. How does GRPO with variance reduction compare with PPO w. value model?\n\n3. In Figure 5, is \"GRPO+Linear\" using the variance reduction techniques? How does \"GRPO+Uniform\" that uses uniform distribution perform compared with \"GRPO\" and \"GRPO+Linear\" in this case?\n\n4. In this paper, the authors use a summary policy to summarize a partial reasoning trajectory within 128 tokens. This design would allow the LLM learning to perform additional reasoning within the 128 tokens, contradicting the argument of \"Budget-agnostic LLMs\" in the MRT paper [1]. Could the authors provide a discussion on any negative effects of using and optimizing a summary policy?\n\n[1] Qu, Yuxiao, et al. \"Optimizing test-time compute via meta reinforcement fine-tuning.\" arXiv preprint arXiv:2503.07572 (2025).\n\nLIMITATIONS:\nAs I discussed in the Weakness, the major limitation of this work is lack of meaningful baselines, making it unable to make a fair comparison with prior works. It would be great if the authors could include more baselines.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}