Title: Efficiently Scaling LLM Reasoning Programs with Certaindex
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

LLM Reasoning

LLM Efficiency
Abstract: Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50\% compute savings and 3.3$\times$ higher throughput in real workloads with no accuracy drop. Our code is available at https://github.com/hao-ai-lab/Dynasor.git


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Certaindex, a lightweight, algorithm-agnostic metric designed to measure the stability of intermediate answers in LLM reasoning tasks. The authors observe that many test-time reasoning methods (e.g., CoT, MCTS) generate excessive tokens even after reaching a stable answer. To address this inefficiency, they introduce a probing technique (\"Probe-in-the-Middle\") that periodically prompts the model to give an intermediate answer. If these answers stabilize, the model likely will not change its final answer, allowing for early termination of generation. They generalize this insight into Certaindex, which can be used across multiple reasoning algorithms and integrated into a scheduling system (Dynasor) to dynamically allocate compute.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper identifies and addresses a compelling inefficiency in modern LLM reasoning—token overuse even when the answer is already internally determined, especially in CoT and similar paradigms.\n\n- Across multiple reasoning methods and datasets, the proposed method achieves substantial token savings and throughput improvements without hurting accuracy.\n\n- The implementation of Dynasor within an LLM serving system (SGLang) with minimal code change shows the practical viability of the approach.\n\nWeaknesses:\n\n- The probe prompt (\"Oh, I suddenly got the answer to the whole problem, Final Answer: boxed\") seems overly dramatic and could potentially bias the model's behavior. The paper claims the exact phrasing is not critical, but does not provide quantitative or ablation studies on alternative, less stylized prompts.\n\n- Certainty metric is primarily based on answer repetition: While effective, it may not capture deeper uncertainty or error modes beyond output consistency, especially when models are confidently wrong. The authors attempt to filter out hesitations (e.g., “wait”), but this is still heuristic.\n\n- Limited analysis on incorrect stabilization: While the paper emphasizes when to stop generating because the answer is stable, it does not deeply discuss when this leads to confidently wrong outputs and how often that happens, which could impact the overall utility of early stopping.\n\n- Truncation fidelity: The underlying assumption is that truncated CoTs still contain all necessary information to reach the correct answer. While there's theoretical justification and empirical support, this aspect could benefit from more intuitive or diagnostic discussion.\n\nQUESTIONS:\n1. Probe Prompt Design: Have you tried alternative, less dramatic probe prompts, such as \"Based on the reasoning so far, the answer is: boxed\"? If so, how sensitive is performance to this choice?\n\n2. Information Completeness: Do you have quantitative or intuitive analysis on whether the truncated CoT contains all the information necessary to output the correct final answer? Are there counterexamples where early answers are stable but ultimately incorrect due to missing later reasoning steps?\n\n3. Incorrect Stabilization: Could you provide more insight into how often the model stabilizes to an incorrect answer, and whether Certaindex can differentiate between \"correct and stable\" vs. \"wrong and stable\" situations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using Certaindex to address the issue of redundant tokens in large model reasoning tasks. The motivation is clear, and both experimental and theoretical analyses demonstrate that Certaindex can effectively eliminate redundant reasoning branches. Moreover, the approach is supported by a well-developed reasoning framework, Dynasor, built on SGLang, which significantly improves the efficiency of online batch inference systems.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper introduces Certaindex as a novel metric to assess the stability of intermediate reasoning outputs, aiming to determine whether the model has already converged on a correct answer. This is a timely and meaningful contribution, as reducing token usage during inference without compromising answer quality is a pressing challenge in LLM deployment.\n\n2. The authors provide a complete and reusable implementation of their method through the Dynasor framework, built on SGlang. This not only validates the practicality of their approach but also offers valuable tooling for the research community, promoting reproducibility and further exploration.\n\n3. The paper presents a well-rounded analysis, combining theoretical insights with empirical evaluations across multiple reasoning paradigms (e.g., Chain-of-Thought, Self-Consistency, MCTS, and REBASE). This strengthens the case for Certaindex as a general-purpose signal for early stopping in LLM reasoning.\n\n**Weaknesses:**\n\n1. While the method effectively reduces redundant tokens in high-confidence scenarios, it may also prematurely terminate reasoning when the model has not yet arrived at a correct answer. This could limit the model’s opportunity for self-correction and increase the risk of incorrect outputs. The paper would benefit from a more detailed discussion of such failure cases and how Certaindex behaves under different reasoning dynamics.\n\n2. The evaluation is primarily focused on logic-intensive tasks such as mathematical reasoning, where the model’s confidence is easier to quantify. However, it remains unclear how well Certaindex generalizes to broader domains, such as open-ended or commonsense questions, where the model’s internal certainty may not align with actual correctness.\n\nQUESTIONS:\n1. While thresholding-based allocation is acceptable, could it become difficult to manage as different data distributions and reasoning flows vary across tasks? If so, how is this issue addressed in the current design?\n\n2. The “Probe-In-The-Middle” analysis in Figure 2 is insightful, but the visualization itself is not very intuitive. Is there a more straightforward or visually clear way to present this idea?\n\n3. As mentioned in the weaknesses, can Certaindex reliably handle all types of reasoning paths when judging confidence? Is there a risk that it might interfere with certain reasoning strategies that the model would otherwise naturally follow?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a solution to solve the trade-off between increasing the token budget for sampling long chain of thoughts to improve accuracy vs. the compute inefficiency of overly long thoughts. They propose using uncertainty estimate based on the model's several intermediate answers in the long CoT to determine an early exit criteria, with the uncertainty calculated using discrete answer frequencies, or incorporating external reward models depending on the task. The paper then pairs this an adaptive scheduling algorithm (Dynasor) that allocates the token budget for batch and online workload. Empirical results show the effectiveness of Certaindex to estimate the correctness or number of steps before termination as well as the prowess/efficiency of their scheduling algorithm.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well-written, easy to follow and addresses a useful problem addressing balancing the length of CoT and its inefficiencies.\n- Convincing empirical results that demonstrate the effectiveness of Dynasor (the scheduling algorithm) over other baselines for batch and online workloads, demonstrating its utility to the community at large.\n\n**Weaknesses**\n- Certainty computation itself is not novel and quite similar to ideas used in prior works such as adaptive self-consistency (https://arxiv.org/abs/2305.11860). The adaptive scheduler basically determines the token budget in blocks/steps which is similar to automatically selecting the value of k (number of samples) and early stopping of the sampling process in adaptive SC. The authors should compare with this baseline in addition to self-consistency.\n- Missing related works on using confidence estimates for inference-time decisions:\n    - https://arxiv.org/abs/2305.11860\n    - https://arxiv.org/abs/2311.09553\n    - https://arxiv.org/abs/2411.04109\n    - https://arxiv.org/abs/2504.05419, https://arxiv.org/abs/2504.15895 (concurrent)\n\nQUESTIONS:\n- The \"settling concept\" mentioned in Line 31 has also been described in prior works: https://arxiv.org/abs/2504.07128\n- Sec 2 para 2, the \"probe-in-the-middle\" is a standard test-time scaling approach as used in the S1 paper and should be cited as such.\n- Certaindex using answer frequency to measure entropy for code, how does it compare to CodeT, another cross-consistency approach with unit test outputs and other methods relying on unit test feedback https://arxiv.org/abs/2207.10397?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}