Efficiently Scaling LLM Reasoning Programs with Certaindex
Authors: Yichao Fu
Junda Chen
Siqi Zhu
Zheyu Fu
Zhongdongming Dai
Yonghao Zhuang
Yian Ma
Aurick Qiao
Tajana Rosing
Ion Stoica
Hao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
LLM Reasoning
LLM Efficiency
Primary area: deep_learning

Abstract: Test-time reasoning algorithms such as chain-of-thought, self-consistency, and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after a certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnostic metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex as a scheduler into Dynasor, our reasoning-aware LLM serving system, and demonstrate up to 50\% compute savings and 3.3$\times$ higher throughput in real workloads with no accuracy drop. Our code is available at https://github.com/hao-ai-lab/Dynasor.git

Efficiently Scaling LLM Reasoning with Certaindex
Yichao Fu"
Junda Chen"l
Siqi Zhu
Zheyu Fu' Tajana Rosing'
Zhongdongming Dai' Ion Stoica" *UC Berkeley
Yonghao Zhuang²
Yian Ma
Aurick Qiao
Hao Zhang'
UCCD
Cammegie Mellon University
Snowflake
Abstract
Test-time reasoning algorithms such chain-of-thouge self-consistency. and MCTS enhance LLM problem-solving but can wastefully generate many tokens without improving accuracy. At the same time, we observe that these algorithms exhibit answer stabilization: their intermediate solutions often cease to change after certain point, and further investment of compute does not change their final answer. To quantify this phenomenon, we introduce Certaindex, an algorithm-agnont metric measuring this evolving stability, signaling when further computation is unlikely to alter the final result. Certaindex is lightweight, can accelerate reasoning program inference via early exit, and further enables dynamic token allocation, gang scheduling, and many opportunities when integrated with real-world LLM serving systems. To quantify real-world benefits, we built Certaindex scheduler into Dynasor, our reasoning-awand aware LLM serving system, and demonstrate up to 50% compute savings and 3.3x higher throughput in real workloads with no accuracy drop. Our code is available at httts::/ // rithub..oo com/hao ai aiblab/ Dynasor.git
Introduction
Large language models (LLMs) have recently demonstrated remarkable ability in solving complex problems. Central to these advances are lest-time scaling algorithms [
2 3 which allocate additional inference resources progressively enhance model accuracy. Whether the reasoning algorithm is externally programmed (e.g. Self-Consistency [4, MCTS [5]
6, etc.) or internalized
in the model (Chain-of-Thought based models such Deepseek-RI 77). combination both
(as used in OpenAl-o3 (8). Grok3 [9]. Gemini 2.5 Pro [I0),, these methods fundamentally empower LLMs tackle challenging problems more effectively.
However, we observe that LLM reasoning is highly ooken-inefficient i.e., often leads to token overuse without further accuracy gains. This phenomenon is most observable in reasoning models: models like Deepseek-R. R1 can generate excessive amount of tokens, resulting in substantial 3x more tokens than it actually needs (see Figure 2} Similar token overuse behavior is also observed in recent studies 11 12 This inefficiency arises because existing reasoning algorithms lack mechanisms to detect diminishing returns in accuracy and terminate inference early, thereby wasting resources without gains in solution quality.
To address this inefficiency, we leverage a key observation about LLM reasoning: LLMs often signal when they's ve "settled" on an answer during reasoning. In particular, we study reasoning models with Chain-of-Thought (CoT, $2.1): we force the model periodically output result in the middle of its reasoning process, and examine how these intermediate answers evolve across successive steps. We find these intermediate answers frequently stabilize; that is, they rarely change after certain number of reasoning steps, regardless whether the answer is ultimately correct or not. This highly indicates that the model itself has reached point of "high certainty" that further computation is unlikely to alter the final result, justifying safe termination. Conversely, significant variance in the
'qqual contribution.
Work done while interning at UCSD
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Token Deprivation: Pressure Testing DeepSeek AMC23/AIME
Accuracy Usage
Direct
Probe-In- Ihee------e--ddlee
Accuracy on NATH500 (9%
78% fewer tokens
60
Score
40
Qiren2.5 78
20
Instruct Model
AIME24
Owen22.
Reasoning Model
Average Tokens Used (k)
Taken Budget (R)
Token Budget (k)
Figure I: The token efficiency curve for the traditional model is (lower) at varying token budgets (scoring lowest to highest over 10 attempts). much steeper than reasoning model.
Figure 2:
DeepSeek RI performance on AMC23 (upper) and AIME24
(Left) Standard reasoning with late answer outputs. (Right) Farly answer extraction using Probe-In-Ther Thhh-Middde technique.
outputs suggests the model remains uncertain and is still exploring different solution paths, warranting continued reasoning. By monitoring these signals, early, saving computational resources without token waste.
decide if we can terminate LLM reasoning
Taking the certainty signal, we further explore advanced algorithms such as Self-Consisteng (SC), Monte Carlo Tree Search (MCTS), and Rebase [3] all of which have seen empirical success in improving CoT-based reasoning in practice. While these algorithms have distinct mechanisms, we found they share fundamental behavior: as more computational resources (e.g., tokens, reasoning steps) invested. their answers tend stabilize. This common observation indicated that general measure of this evolving confidence is possible. To formalize this as a unified metric, we introduce certaindex ($3.1 universal metric designed to capture this developing certainty. Certaindex generalizes certainty into a normalized confidence score that serves as proxy to measure reasoning progress. Regardless of what the reasoning algorithm is, high certaindex values indicate that the model's s answer unlikely to change significantly with further computation.
Even better, we find that certaindex offers a narrorw interface to real-world LLM serving engines. Because certaindex light-weight, enables opportunities including early exit, dynamic token allocation across reasoning queries, and gang scheduling in multi-stage reasoning programs, all with almost no scheduling overhead ($3.2). To this end, we develop Dynasor - reasoning-awwan end-to-end serving system that leverage certaindex to optimize compute usage in both batch and online serving ($3.)). Dynasor employs certaindex to adaptively allocate token budgets to reduce the compute cost, and enables gang scheduling increase request SLO attainment and throughput. Our evaluations on various datasets, LLMs, and reasoning algorithms show that batch inference, saves up 50% compute to reach the same overall accuracy, and in online serving, it sustains up to 3.3x more queries or achieves 4.7 x tighter latency SLOs at the same attainment rates.
In summary, this paper makes the following contributions:
1. We identify significant token overuse in concurrent reasoning models and introduce Probe-In- ThedMiddle, method that extracts intermediate answers during CoT to detect convergence and enable carly stopping validated both empirically and theoretically.
2. We develop Certaindex, a unified metric that quantifies reasoning progress across diverse strategies (CoT, SC, MCTS, Rebase), enabling adaptive compute allocation at test time.
3. We build Dynasor as reasoning aware serving system that leverages certaindex to dynam- ically allocate tokens and co-schedule multi-stage reasoning steps, achieving up 50% compute savings in batch serving and 3.3x throughput gains in online serving.
2 Systematic Token Overuse A Case Study in CoT
Recent work has shown that allocating more compute at inference time, so-called "test-time scaling", consistently boosts performance on hard reasoning tasks 8 14 [5). Methods such as Chain-of-
Thought (CoT) [16), Best-of-N sampling [17] [8), Self- Self-Consistency (SC) [4, and search-based

---

Deepooak
5-78
Ovend
Evaluale (1-2)6-3)
triak udoud appraach the, samander when doaling complix
zuke to 2oth tere inrie parvehhous
numbbrl, is dippccation finn somne bebore adtilin and dereati inouid
reprrsiio
coprossion to mall sars unddassard parectyy (1 20 2ilmulllleed by 5. and
Lat' start canibudte (1:2)5 p- wiwd
tane wuluiias hec onige rocrmsiio (ive 17
multllccutin frus. krow that othan ruullphying complee number mot riter coan damitus the ru ruripppction somes toth and inagihery parts.
tmegg partipd
the fiurt sosee &
peseeinneo uupuraidyy Latts Butt shoz to shap Mu yany fiay gino re is Thans ree kop anr 88 Sigredy me "12 Su padl losoer Letute arti tho dower di Scberactin 2f from 125 chould te ceeepttowent 128 3 is min the expree diss aimgs line
Lig secondd deck duderdd
makeraan did't rale vedaas So. 23 6o coarre becuraee you
Than eetencting
leck
lullarr andyiing whe insodd co head' problem and soloo aouendaaa the coupepinn seadi (1lak (aans dani Lat jost make
sudtruct Aiberaliiyyy acteeh
hepeot E11 20-3 which
So ciling
reaull 6 12)-
Wesl which i. don' torn rewe mn anathing hacs con eec montinee hul drmplfee sox So, siis fnd mestios abouee
12irinee deriaday Woot,
fire because Phere no ofeer real barn to combene with Co. s a final aroneciati queae ancthr way to check a su bu the by pivas 120 Thin
Resmmonee
Yeaks that commatent think confident frut evakutted eeprossion don't oon malaanoe my manniing betimes foor the connect anower fiooe Aoo the value <hnaaso
Parpactiv &i liginng
Re-eritodrr
rumber Fast, fullle
Confitance Culddig
tern the complex number 2 - 2 1 21 120 Need ue purnec 30 hom the meu + Contriingg the
iregieay parts wi gul 112 3) Thxn wales oxprrssionn 1
Conticad
Figure 3: An Example of Self-Doubt Comparing a Reasoning Model Seepseek-distille Qwen-2.5 7B) vs. Traditional Model (Qwen-2.5 7B) on a Problem from the MATH500 Dataset
algorithms (e.g., MCTS G guided beam search 19 or Rebase 13 all exploit more token usage to trade compute for accuracy.
While advanced reasoning algorithms boost accuracy on complex tasks, they often suffer from severe
token inefficiency. To quantify this, we compare a Qwen2.5 7B (Non-Reaoonigg [20] instruct model against Qwen2.5 7B reasoning model [14] on the MATH500 dataset, gradually increase the maximum token budget for each question, and track the accuracy change for the dataset. Figure reveals that the reasoning model, despite eventually achieving higher peak accuracy, required up to 4.5% more tokens to reach the same accuracy as the instruct model.
Why do reasoning models overuse so many tokens? To answer this question, we conduct a case study focused on CoT reasoning models in math datasets (i.e., AMC23, AME24) to understand where this overuse occurs and why. We develop and use Probe-In-The-Midee prompt such as "Oh, suddenly got the answer to the whole problem, Final Answer: boxed(" to force the model to output its answer mid-reasoning, and record whether this answer is correct. Figure 2 presents the LLM's accuracy for each question at various token budget allocations with (left) and without (right) Probe-In-The-Miie number tokens spent (left) vs actually needed (right) to arrive at the correct answer. For example, on AMC23, the model produces median of 2.7K tokens, but typically output the correct answer by a median of 830 tokens.
($2.1), periodically inserting
Each row in this figure gives us an approximation of the
We further zoom in to one case, and realize the model the major source of inefficiency stem from what we called "self-doubt". Figure 3 shows an annotated output where a reasoning model (Deepseek- distilled Qwen2.5-7B) answers a simple math question. Even as the reasoning model reached the correct answer very early (~~ 300 tokens), it continues the reasoning process, spending excessive tokens re-evaluating already correct answers, checking completeness, re-verifying assumptions, building confidence, etc. Several recent studies I| [2] have also found similar phenomenon, indicating that this pattern of "self-doubt" is not an isolated issue specific to single model or dataset but rather reflects a systematic tendency in current reasoning models.
2.1 Addressing Token Overuse In Long CoT Reasoning by Probing-In-The- Mhe-Midllee
How do we address token overuse? We conjecture that LLMs, during their reasoning, will emit measurable signals to their internal reasoning state. Indeed, much prior work indicates that LLMs might possess an internal awareness of their answer confidence, akin to 'LLMs knows when they know' [1 If these signals can be accurately detected, we can use them to stop the reasoning process without sacrificing accuracy, thus reducing token usage.
Our high-level idea is that if an LLM consistently produces the same intermediate answer over a sequence of reasoning steps, it indicates a high degree of certainty or stability for that answer, whether not that answer itself is correct not. Once such stability is reached, additional reasoning
3

---

What is the value of 4 87.88 17?
Case Early Exit
<lhinic-Aiight, so have this math problem here: S- 2*- 4 6 -1111-1
Himm, okay [a[31s9] Lot me try to figure our what's going on here. looks like series where each form is aube, add number.
the cube number subtractnng the [3(3158 (Early Exit: because we get the same answer all time)
attemating 6ia betweon
Result
<thinioAArigh,, have this meth problem here: - 1 3" 5 ...... -18- 177
Himm, okay, [3 Let to
what's altarnating (B(4435) between an even number and odd number. The palkern adds the an aven number subtracting the (b(6732) (Inconsistent Result: continue generation because we did not get the same answer)
looks series where each term cube,
Case 3: Hesitate
<thinic-Alrgh,,
problem
Himm, okay (3324 Waitfj try figure what's going hare. looks cube. alternating (44335, Hmm...] between even number and an odd number. The pattern adds the cube of an even subtracting E3(8245, But...] (Hesitant: continue generation because hesitant...)
d probe the middlee.... Oh: suddony got the answor to whole problem "Fnai Arswer"/ locacd force solution
Figure 4: Illustration of Dynasor CoT: (1) Probe-In- Trobe-In-Thh-Mire Midde for answer extraction, (2) early exit based on certainty (case 1), (3) post- post-generation validation for hesitation words (e.g., "wait") (case 3), and (4) continue if not certain enough (case 2)
unlikely alter the outcome, presenting opportunity for early termination. Conversely, if intermediate answers remain highly varied or fail to converge after significant effort, it may signal the problem intractable for the model, also justifying early termination. Given this insight, we use "Probe-In In-Tin-Meirr The-Middle to first capture intermediate answers (e.g. every 64 tokens) as the model reasons, and then assess the certainty based on the consistency of these probed answers decide whether to stop generation. By monitoring this certainty, we can truncate the reasoning chain, thus reclaiming wasted compute without sacrificing accuracy.
Trobe-ln-The-Midlle. Figure shows the high-level process. To capture whether the reasoning LLM has settled on its final answer, interleave periodic "probes" into its reasoning process. Specifically, after set number of generated tokens, we append the prompt: "Oh, 1 suddenly got the answer to the whole problem. Final Answer: boxed(", and record the output answer. This prompt forces the model to take a guess for the final answer at this specific point in its reasoning. Note that the exact phrasing of the extraction prompt is not critical. What matters is that effectively guides the model to produce an answer immediately.
Concretely, split the full reasoning chain into m steps, Y1, Y2, ....... Y.,. where each Yg corresponds to a fixed token interval (e.g., 64 tokens). At each step k we sample Y2 ~ Ps (Ya xx Yi...., -)- All probe tokens and their responses are then discarded before resuming the original decoding path. This lightweight probing mechanism allows us to pinpoint exactly when the model's answer convergss-enabling systematic comparison of token efficiency across models on large datasets.
Certainty Assessment via Answer Consistency. Using the answers captured through Probe-In In-The- Middle, we perform immediate consistency check to assess the model's 's certainty at each probing step. Concretely, let {y...... ym be the sequence of probed answers from step 1 to step m. We measure consistency over a sliding window of width w steps: C
Zj=k- Bejk-woh w+ [ = yu]. (0, 1J. deem the
where I[-] is the indicator function. Once Ck > for some threshold model sufficiently certain and terminate generation at step k (case in Figure Otherwise, we will continue reasoning (case 2 in Figure 4 until the budget is drained some other stopping criteria is matched.
Gost-Generation Validation. In addition to the answer consistency, we found that some linguistics markers like "wait" or "hmm" also indicates uncertainty. If we find these uncertainty indicators the probed answers (case 3 in Figure 4. we will mark this answer as unconfident and omit this responses from the consistency test. This validation mechanism works synergistically with the consistency assessment to create a robust certainty metric.

---

Theoretical Grounding and Efficacy for COT. The rationale for early terminating based on this observed stability is not merely empirical. In $ $3.4, we provide a theoretical analysis. This analysis demonstrates that if a reasoning chain consistently yields the same answer across several consecutive reasoning steps, then as the number of these consistent steps increases, the observed answer provably converges to the final answer the CoT would have produced with full, unconstrained generation.
Applying "Probe-In- Prooe-In-The-Middee The-Middle" to CoT reasoning on datasets like MATH500 ($4.1) demonstrates significant token efficiency gains. Identifying answer convergence allows early termination, which reduces computational overhead while maintaining improving accuracy. This success with CoT motivates certaindex, our universal metric extending the certainty-driven early exit concept to broader spectrum of reasoning algorithms including Self- ColfiConsistency. MCTS, and Rebase.
3 Certaindex in General Reasoning Algorithms
3.1 Generalizing Certainty to a Broad Spectrum of Reasoning Algorithms
Beyond just CoT, a wide spectrum of LLM reasoning algorithms also exhibit (or can be equipped with) measures that reflect their progress towards a stable answer. For instance, iterative refinement methods (e.g.. CoT) may show convergence in output, while search-basee algorithms (e.g., MCTS) might reach a state where further exploration yields diminishing returns. Recognizing this common signal of "certainty" or solution stability across different algorithms, propose certaindex- unified, algorithm-agnen agnostic confidence metric. High certaindex indicates close proximity to a solution or that additional computation is unlikely to improve the outcome.
While previous research has explored uncertainty estimation through various approaches (semantic metrics [22] 23 rog-robabbility entropy [21 24], and hidden state analysis (25 26 27 ). certaindex
offers a practical, integrative measure of model confidence suitable for direct implementation in LLM serving engines (e.g.. SGLang and vllm). This facilitates a scheduling layer optimized for LLM reasoning workloads. In the following sections, we will define specific instantiations of certaindex for two common reasoning algorithm archetypes.
Certaindex in reasoning algorithms with multiple reasoning paths. Reasoning algorithms like Self- Self-Consistency (SC), MCTS, and Rebase expand multiple reasoning paths to derive aggregated final answers (Appendix Bj. To quantify L.LM's certainty among these paths, we employ semantic entropy (22). Given a question. n reasoning paths are generated and clustered into m groups based on their answers: C1, Cz...., Cm. where |C:| denotes the number of paths in answer group C, The semantic entropy is calculated as H. = Σm= [Cul [Cal. log
n We normalize H by its maximum log [0, 1]. For closed-form tasks (e.g., arithmetic or multiple- choice), we group outputs by exact string matching; for open-ended generation (e.g., code (28) or flexible mathematical expressions (29)), we compute pairwise similarities with small embedding model 30 and cluster. Both approaches incur little to no overhead compared to LLM inference.
to obtain certaindex: H log n-
log n
Certaindex in reasoning with a reward model. For reasoning algorithms that incorporate a reward model (e.g., MCTS, Rebase), we simply use the reward model's normalized output R E [0, 1] as a measure of certainty. This approach builds on prior research demonstrating that reward signals can effectively guide resource allocation in program execution 31 We collect the terminal reward scores from each reasoning path and aggregate them to compute certaindex: for MCTS, we take the average reward: for Rebase, the maximum reward. A higher aggregated reward indicates stronger certainty in the reasoning paths" validity. These reward scores are obtained during normal execution and therefore incur no extra overhead during LLM inference.
3.2 Effectiveness of Certaindex for Dynamic Token Budgeting in Reasoning Optimization
Since certaindex tracks the model's self-assessed proximity to a final solution, it can effectively guide token budgeting to prevent unnecessary token expenditure. In this section, we empirically validate this core hypothesis: that higher certaindex values consistently predict fewer remaining token needs to reach a correct solution. This relationship holds across various models, reasoning algorithms, and datasets, forming the basis for smart token allocation.
5

---

Correlation between Certaindex and Remaining Computational Effort. Our hypothesis is that as a model becomes more "certain" (as measured by certaindex), should be closer to outputting its final, stable answer. To test this, we compare these certaindex values against the "oracle" number of additional tokens the model actually required from that point to arrive at the correct final answer (for solvable queries). This analysis spanned 4 different representative examples of these correlations shown in Figure 5 (further examples are in Appendix (C). Across all combinations, for solvable queries, we found Pearson correlation coefficients ranging from O. 77 to 0.75, with a strong mean of 0.52. This consistently positive correlation indicates that a higher certaindex value is indeed reliable predictor of solution proximity, requiring fewer additional tokens.
combinations, with
wwvesholldig-bbseet allocation. An immediate application of certaindex for token budgeting is thresholding- -based allocation policy. If query reaches a high certaindex value early in its reasoning, it suggests the model is confident in an answer. We select certaindex cutoff threshold at a chosen reasoning step (e.g., after fixed number of reasoning steps, visualized as the horizontal orange line in Figure 5 Any query whose certaindex surpasses this threshold at this step is immediately halted. This approach has two benefits: (1) it efficiently reduces token usage on straightforward cases that quickly achieve high certainty, and (2) it halts unproductive paths early, preventing wasted resources on unsolvable or confidently incorrect answers. As our empirical results show, this approach often achieves little to drop in accuracy on the broader set of solvable queries that do not cross this early exit threshold. Thresholding-bree -bsed allocation is not strictly optimal, as we show in the pareto-frontier frontier optimal strategy in Appendix C and Figure [2] Nevertheless, thresholding-be based allocation is simple and efficient in practice, making it a compelling choice for real-world deployments.
C-HCR> R.)
1.0
Rcboso/554/umma> 78
NCTS/GSMVL amo
1.0
ICoT/SSM/QWQ 328
1.0
SCIGSMLisma 708
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Step
Step
Step
Step To Collect Certaindex
Threshold
Fitted Curve
Salvable Problems
Unsolvable Prablems
Figure 5: Correlations between certaindex strength (y-axis) and ground truth steps to solution (x- axis) with (algorithm, LLM) settings where algorithm E (SC, Rebase, MCTS, CоT) with LLM (Llama (32) and QWQ 33 on GSM8K. Certaindex is measured at the reasoning step marked by allocation. The green line shows a more
the red line. The orange line shows the thresholding- fine-grained approach through curve fitting. All plots except MCTS have both certaindex values and oracle steps averaged across multiple runs combat randomness. See Figure 12 the full result.
3.3 Dynasor: A Eettaindex-Driveed Adaptive Compute Scheduler
The previous section demonstrated how Certaindex effectively quantifies certainty during LLM reasoning, enabling dynamic token allocation and reducing waste. Building this, we introduce Dynasor- a reasoning-awarc end-to-end serving system that leverages Certaindex for adaptive scheduling. In real-world deployments for LLM reasoning, single user query often spawns multiple related sub-queries, which we refer to as "reasoning programss" These programs consist of logically connected requests that share context and computation. Traditional schedulers treat these sub sub-queries independently. missing opportunities for optimization. Dynasor addresses this by introducing a lightweight, Certaindex-driven scheduler that optimally manages both individual requests and entire reasoning programs. Beyond simple early exits, Dynasor implements two other key mechanisms to accelerate inference:
Cortanddxx-rriver Adaptive Compute Allocation. Dynasor dynamically adjusts token budgets for individual requests based their real-time Certaindex values. As validated in $3.2 a high Certaindex indicates that the answer has stabilized, allowing the scheduler to reduce the token budget and resources. This fine-grn mnaigraieed control minimizes waste queries that have converged or unproductive, enhancing overall efficiency.
Gang Scheduling. Reasoning programs usually generate multiple related requests in parallel or at differet stages. Gang scheduling batches or prioritizes these requests together to maximize shared computation (c.g., KV-cache utilization) or carly-exit high-certaindex programs to free resources.

---

This mechanism reduces latency and improves throughput by aligning resource allocation with real-time reasoning progress.
Implementation. Dynasor is implemented as a lightweight scheduling component in SGLang [34 For each decoding step, Dynasor monitor certaindex reallocate resources terminate requests, and use gang scheduling to prioritize requests within a reasoning program. Dynasor only introduces ~500 lines of code into the core system, and requires no change to the model weights, reasoning algorithms. or the core execution backend. It acts as a thin layer around the standard decoding loop, yielding substantial gains latency, throughput, and overall token efficiency. We describe the detailed design and implementation of Dynasor in Appendix
3.4 Theoretical Basis for Early Exiting in CoT without Accuracy Loss
This section gives sketch of the theoretical foundation for our early exiting strategy based on We show how can terminate CoT reasoning without accuracy degradation, and we believe this theoretical grounding can be generalized to other reasoning algorithms. See Eapendix[Ffoo a riguous proof.
Sketch of Proof. Our method assumes that the next-token distributions Pr(Y |r, YI..) in a reasoning chain eventually converge stationary distribution (.(Y)=). Once P3 = P., further computational steps are redundant. The "Probe-In-Tiie In-The---ddddee technique empirically detects this convergence. If observed answers (and thus their empirical distributions P) stabilize across several probing steps, it indicates the underlying true distributions Pt (and their mixtures P) have likely converged. To analyze this, we define mixture distributions:
Definition 1 (Mixture Distributions). Consider n samples, each corresponds Pt.... ...,. Petn. Denote the mixture distribution of samples + i....,i+ + [I.....n] to be pi+k = Σ}= Pa+s-
Our formal justification shows that if our empirical stopping criterion is met, the empirical mixture distributions P are e/3-close (in Total Variation distance) the true mixture distributions P. This requires a sufficient number of probes k, as established by the following concentration bound:
Lemma If l: = 52 ( A (4+log(1/))
). then TV pltt) S e/3. for all I = i i,....itt k, and t= k 1, k. with 1 5 probability, assuming have M disjoint groups of the outputs.
The empirical stopping criterion triggered when the TV distance between consecutively estimated mixture distributions, pi+* and pit)+k (and similarly for mixtures of size 1),, remains below a small threshold = e/3 for all S S k (or 1 S j S k - respectively). When this empirical stability is observed, and k is sufficiently large as per Lemma (ensuring TV(P, P) < </3), the triangle inequality implies that the true underlying mixture distributions Pi+k and pitjth are also e-close to each other (i.e., (V Pitsth) S c).
The connection between the stability of these true mixture distributions and the stability of individual distributions Pt (which ultimately implies convergence to P. given Assumption [ is provided by:
Lemma 2. TV p(ts+k) = O. for any S j S k, and TV P p
= 0, for
i4g then Eq. (2) holds for any i 2 i, and T < 1.
any 1 < j <
Therefore, observing empirical stability with a sufficiently large & indicates that the true individual distributions P. have stabilized and are e-close the stationary distribution P. PY(1)). This justifies that early termination preserves accuracy up to a tolerance e. The effective & must satisfy both the
concentration requirement (Lemma) 1 and be large enough relative k* from Assumption I
4 Evaluation
To show the effectiveness of certaindex in real-world workloads, we evaluate certaindex and the end-to-end serving system Dynasor in CoT [16] and three other reasoning algorithms (SC [4], Rebase
[3] MCTS (6)3]) on diverse datasets [35: 36 37
28 20 38). For brevity, we provide detailed
experimental setup in Appendix G
Our experiments try to answer two main questions: (1) Batch workload: How effectively does certaindex reduce token consumption while preserving accuracy in real-world batch inference work-
7

---

(2) Online workload: To what extent does Dynasor improve end-to-ent fo-end performance in online serving scenarios? ($4.2). In addition, we perform ablation studies in ($4.3) to compare certaindex against other signals and selection of certaindex thresholds at runtime. We also perform a few more Dynasor system ablation studies in Appendix H.
loads? ($4.1
4.1 Batch Inference with Certaindex
To evaluate the effectiveness of certaindex in batch inference, we evaluate Dynasor on CoT with reasoning models, and compare the tokens-accuracy tradeoff curve for different setups. We also evaluate certaindex on other reasoning programs (SC, MCTS, Rebase) to see if the effect generalizes. See detailed experiment setup in Appendix G. for CoT, and Appendix G.2 for SC, MCTS and Rebase.
COT.. Figure our approach achieve significant token savings across all configurations, reducing token usage by 11-29% while maintaining same accuracy as the baseline. For the 10% of problems where our method achieves the highest token reduction, we observe savings of 34% on AIME and 53% on MATH500. reductions of 53% on AIME and 81 % on MATH500. A similar result is shown in Deepseek-RI (See Appendix H.T. The primary gains of the token savings comes from the effective probing mechanism to identify delbt in CoT reasoning, and using certaindex to early- stoppig...
particular, for the top 1% of problems, we achieve even more substantial
Baseline
Ours 132)
Ours (64)
Ours (128)
Ours (256)
Durs 1320)
78
14B
328
Accuracy Nh: Accurayy (96) Accurso) (16)
AIME24
45
60
65
10
8
10
20%
MathS0n AMC23
-25%
-24%
80
3.5
4.0
5.5
4.5
3.5
4.5
90-
-2%%
90
18%
90
10%
2.0
2.5
3.0
3.5
2.5
3.0
3.5
2.5
Tokens (K)
Tokens
Tokens
Figure 6: Comparing Dynasor Token-Accuracy Curve Across Deepseek Qwen-Distilled Model Scales
(7B, 14B, 32B) and Datasets (AIME24, AMC23, Math500)
Baseline-length MCTS ASDiv
Baseline-even
SC LiveCodeBench
Rebase Math
50.0
74
35
73
Accuracy (%)
45.0
30
72
0.5
1.5
2.5
2.5
3.5
2.0
MCT5 GSMBK
G5MBK
92
71
88
70
90
2.0
6.0 Number of Tokens (1e6)
4.0
4.0
6.0
8.0
0.2
0.5
0.8
1.0
Number of Tekens (1e5)
Number of Takens (1e7)
Figure 7: laken-to-accuraee metric on batch processing workloads. Mean performance and std (error bars) of 10 runs are reported. Baseline-even allocates resource uniformally across all reasoning program. Baseline-length uses Detect knob (Table !T as the program's 's process signal.
To show our method generalizes to other reasoning algorithms, we also compare Dynasor against modified SGLang intra-program scheduler using the following policies (1) baseline- even, which allocates resources uniformly: and (2) baseline-length which uses Detect@ knob, the cumulative tokens generated at specific step (Table 3 the proxy.
SC, MCTS, Rebase.
Figure 7 show certaindex consistently reduces token usage by 9-52% across workloads compared to both basclines, without loss in accuracy. Notably, achieve over 47% savings SC-GSM8K and
8

---

over 50% on primarily come from the intra-pri program scheduling algorithm( J. which accurately identifies high-certaindex programs and terminates them early without accuracy loss. This early termination strategy significantly reduces resource consumption by eliminating unnecessary sampling com- pared with baseline-even. In contrast, baseline-length leads to accuracy degradation even with less aggressive compute pruning, highlighting the effectiveness of certaindex- berasddex-bbsed resource allocation.
highlighting Dynasor's 's efficiency across diverse benchmarks. These gains
4.2 Online Serving with Certaindex
We evaluate Dynasor against SGL.ang 34 and Parrot [39 with different inter-program schedulers. We use P90 deadline attainment as the SLO attainment (detailed setup in Appendix G.3) Each row in Figure paurrerteeeenttt
key performance trade-off's in online serving: (a) Program arrival rate vs SLO attainment: what percentage of programs can meet the SLO as the arrival rate increases; (b) SLO scale vs SLO attainment: how tightly service provider can configure the SLO while maintaining reliable attainment. (c) Tradeoff between Accuracy and SLO attainment.
Ours
SGLang
Parrot
Self-Consistency Math
MCTS ASDiv
Rebase GSMBK
100
100
100
80
Attalmment (%)
10
0.5
10
1.5
Program Rate (program/s)
Program Rate (programis)
Program Rate (programis)
100
100
100
25
2.0
1.5
1.0
0.5
1.5
10
0.5
10.0
7.5
5.0
2.5
100
100
100
80
53
54
55
56
72.5
73.0
73.5
74.0
07.4
87.8
87.8
88.0
Accuracy (96)
Accuracy (%))
Accuracy (%6)
Figure
Evaluation on 3 online workloads on Dynasor against basclines. Rows from top to
bottom: (a) Program rate vs SLO attainment, (b) SLO scale vs SLO attainment, (c) Accuracy vs SLO Attainment.
(a) Rate vs. SLO attainment. Figure S shows that Dynasor achieves much higher sustainable request rates under P90 deadline attainment: 1.6 - 3.3x and 1.6 3.2x compared to SGLang and Parrot, resp. These gains stem from two key factors. First, our intrramm scheduler identifies and early-terminates programs with high confidence, freeing resources for pending requests. Second, inter-program scheduler prioritizes requests from the same program, increasing turnover rate.
(b) SLO scale vs. SLO attainment. Figure ()) shows deadline attainment under a fixed request rate for different systems. At these rates, Dynasor allows tighter SLO scales: 1.3 4.7x tighter than LGLangg and 1.7 3.3x tighter than Parrot. The source of gain is similar to (a): intra-program scheduler early-terminates requests and increase turnover rate, both increases effective request rate.
(c) Accuracy vs SLO attainment. Figure (c) (details in Appendix G.3) shows Dynasor achieving 0.7% 2% higher accuracy than SGLang and Parrot across all three workloads the same SLO attainment. This improvement results from Dynasor's 's ability to redistribute compute between simple and hard queries, enabling solve more queries while maintaining SLOs that baselines could only match with significantly more compute.
Throughput. Our system shows equal average throughput (token per second) in all online settings compared to the baselines. The is because that under the given request rates, all workloads saturate the GPU memory and making the system memory bound. This also validates the introduced scheduler has no overhead.
4.3 Ablation Studies
Choosing Different Thresholds. We show that certaindex threshold selection is vital. Fig. demonstrates the impact of different certaindex thresholds. For SC on GSM8k, we select entropy

---

Selt Consistency G3MBK
MCTS ASDIV
01.5
14.0
Pearson Correlation
Kendall's Tau
Aucorac
01.0
Oues ibroway-t
73.5
D.8
0.69
90.5
0.61
Entmepye
73.0
90.0
Enempy-1
0.4
0.35 Length
0.34
Extoppym2
72.5
2.0
6.0
1.8
2.5
2-2
Length
Leg P Entropy
irtuuss)
Linear Comb.
Log P Entropy Mibrs)
Linear Comb.
(1c6)
Number (1c5)
Figure 9: Performance comparison with different entropy threshold reward score threshold.
Figure 10: Correlation between certainty and steps required on solvable problems.
threshold 0.5, as higher thresholds led to overly aggressive termination and accuracy degradation. Similarly, for MCTS on ASDiv, choose score threshold of 0.4, as lower thresholds (<0.4) decreased accuracy while higher thresholds (>0.4) increased token consumption without proportional accuracy gains. These results highlight the importance of careful threshold selection in balancing compute efficiency and accuracy. In practice, Dynasor uses a profiler-guided approach to select these thresholds (see detailed discussion in Appendix E.2.2)
Choosing Different signals. We compare certaindex with two alternative metries for estimating resource needs. (1) Reasoning path length: Longer reasoning paths (more tokens) often indicate harder problems, suggesting a potential correlation between path length and compute needs (c.g., more reasoning paths). (2) Mean normalized log probability: Established as measure of LLM confidence [24], higher log probabilities may correlate fewer samples needed for correct answers in SC. Using the (SC, GSM8K, Llama3.1-8B-instruue 1-8B-instrucc) setup, evaluated certaindex alongside these metrics. Four metrics were tested: certaindex 's entropy measure H. mean output length [40]. mean normalized log probability [24, and their linear combinations.
As shown in Fig. 10 achieved the strongest correlation with ground truth compute requirements (Pearson Correlation of 0.68, Kendall's Tau of 0.61), outperforming other metrics and matching complex combinations. These results confirm that certaindex is a simple yet effective proxy for estimating inference computational demands, offering robust performance across tasks and models. Our end-to-end token-to-accuracy evaluations in Appendix compare different signals for resource allocation, confirming certaindex's 's superior performance.
Other ablations. We also ablate the contribution to scheduling algorithm H.2
fairness analysis
($ H.3) and comparing static thresholding vs fine-graines resource allocation ($ H.4)
5 Limitation and Broader Societal Impact Discussion
Limitation. Our study primarily focused on optimizing token allocation through certaindex. but did not explore its integration with advanced serving techniques like PD disaggregation or chunked prefill. The potential impact of these integrations on the latency-accuracy trade-off real-world workloads remains an open area for future research.
Broader Impact. Improvements of Dynasor in LLM inference efficiency can further democratize access to powerful models. By reducing computational costs and energy consumption without model changes, our method enables more sustainable and scalable LLM serving. However, we acknowledge potential risks, including bias in early exits, security vulnerabilities through side-channel attack for certaindex multi- tenant settings, and manipulation risks if malicious inputs exploit the mechanism.
Conclusion
In this paper, we show that LLM reasoning algorithms frequently exhibit answer stabilization, leading to unnecessary token generation. We developed certaindex, an sggorith--agnosti metric, to detect this stability and enable early exit from the reasoning process. When we implement Certaindex as a scheduler into Dynasor, our reasoning-aware serving system, this simple yet effective mechanism results in significant practical benefits, including up to 50% compute savings and 3.3x higher throughput in production systems, without accuracy drop. Certaindex thus represents a key advancement in making sophisticated LLM reasoning more efficient and scalable.
10

---

Acknowledgment
We sincerely thank the anonymous reviewers for their valuable feedback. We gratefully acknowledge NVIDIA for providing computational resources for this project, and Snowflake for their contributions and support. This work was supported part by PRISM, and by the National Science Foundation (NSF) under grants 21 2112665, 2112167, 2003279, 2120019, 221 2211386, 2052809, and 1911095.
References
[1] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025.
[2] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding meta-generation Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
[3] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can more effective than scaling model parameters. arXiv preprint 1eXiv:2408.031/4 2024.
[4] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
[5] Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Aiphazero-like tree-search can guide large language model decoding and training. arXiv preprint 2rXiv:2309.171777 2023.
[6] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992, 2023.
[7] Deepspeed model implementations for inference (mii), 2023.
[8] OpenAl. Openai o3-mini system card. January 2025.
[9] xAI. Grok beta the age of reasoning agents, 2025.
[10] Google DecpMind. Gemini 2.5 pro, 2025.
[11] Xingyu Chen, Jiahao Xu, Tian Liang. Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. not think that much for 2+ 3=? on the overthinking of ol ol-like 1lms. arXiv preprint arXiv:2412.21187, 2024.
[12] Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296 2025.
[13] Yangzhen Wu, Zhiqing Sun. Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for probemm-sllvigg with language models. arXiv preprint arXiv:2408.0077 2024.
[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294: 2025.
[15] OpenAl. Learning to reason with 1lm,, 2024.
[16] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2485 2022.
11

---

[17] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprini urXiv:240..21787, 2024.
[18] Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, et al. Rewarding chatbots for real- world engagement with millions of users. arXiv preprint arXiv:2333.006. 06115,, 2023.
[19] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36:41618-4160 2023.
[20] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
[21] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522), 2022.
[22] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv: 2304096644 2023.
[23] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, 2024.
[24] Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07656, 2020.
[25] Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3607-3625, 2023.
[26] Hanyu Duan, Yi Yang, and Kar Yan Tam. Do llms know about hallucination? an empirical investigation of Ilm's 's hidden states. arXiv preprint arXiv:2402.09733: 2024.
[27] Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin Edelman. Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.03566. 2024.
[28] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint WxXiv:2403.07974 2024.
[29] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
[30] N Reimers. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
[31] Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20299. 2024.
[32] Al Meta. Introducing llama 3.1: Our most capable models to date. Meta Al Blog, 2024.
[33] Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024.
[34] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs, Advances Neural Information Processing Systems, 37:62557-62583, 2024.
12

---

[35] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[36] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXi::2110. 2021.
[37] Shen- Yun Miao, Chao-Chun Liang, and Kch- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprint arXiv:2106.15777 2021.
[38] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305..00050 2023.
[39] Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. Parrot: Efficient serving of (LLM-based) applications with semantic variable. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pages 929- 905,9445.22 2024.
[40] Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, lon Stoica, and Hao Zhang. Efficient Ilm scheduling by learning to rank. arXiv preprint arXiv:2408. 15792, 2024.
[41] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36:1 1809-11822 2023.
[42] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
[43] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao. Min- Yen Kan, Junxian He, and Qizhe Xie. Self-evaluation guided beam search for reasoning. In Advances in Neural Information Processing Systems, volume 36, pages 41618-41650, 2023.
[44] Qiyuan Zhang. Fuyuan Lyu, Zexu Sun, Lei Wang. Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. CoRR, 2025.
[45] Jiefeng Chen, Jie Ren, Xinyun Chen, Chengrun Yang, Ruoxi Sun, Jinsung Yoon, and Sercan Ö Ank. Sets: Leveraging velf-verifiaatio and self-correction for improved test-time scaling. arXiv preprint arXiv:2501. 250009906,.. 2025.
[46] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ili Kulikov, and Zaid Harchaoui. From decoding meta- gestaaeneration: Inference-time algorithms for large language models. arXiv preprint arXiv:2406.1666 16838, 2024.
[47] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprini arXiv:2410. 12832 2024.
[48] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
[49] Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, and Mingxuan Yuan. Unlocking efficient long-to-short 1lm reasoning with model merging. arXiv preprint arXiv: 20641, 2025.
[50] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, and Hao Ding. Kimi k1.5: Scaling reinforcement learning with lms. arXiv preprint 155.19991.1199,, 2025.
13

---

[51] Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference- compute: Llms can predict if they can do better, even mid- aid-gennration. arXiv preprint arXiv:2410.02725, 2024.
[52] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms, arXiv preprint arXiv:2502. 12067, 2025.
[53] Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang. Yujin Kim, and Se- Young Yun. Self-training elicits concise reasoning in large language models. arXiv preprint arXiv:2502.20122, 2025.
[54] Haotian Luo, Li Shen, Haiying He, Yiho Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. 01-pruner: ength-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501. 12570, 2025.
[55] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-compressbbe chain-of-thought tuning. arXiv preprint arXiv:2502.0900 09601, 2025.
[56] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aawee Ilm reasoning. arXiv preprint arXiv:2412.18547, 2024.
[57] Sara Vera Marjanovic, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lå, et al. Deepseek-rl thoughtology: Let's think about 11m reasoning. arXiv preprint arXiv:2504.07128, 2025.
[58] Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. Let's sample step by step: Adaptive-consisteney for efficient reasoning and coding with llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12375 12396, Singapore, 2023. Association for Computational Linguistics.
[59] Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. Escape sky-high cost: sarly-stopping self-consistency for multi-step reasoning. arXiv preprint arXiv:2401. 24000 2024.
[60] Guangya Wan, Yuqi Wu, Jie Chen, and Sheng Li. Reasoning-aware self selfsconnssteney: Leveraging reasoning paths for efficient I1m sampling. arXiv preprint arXiv:2408. 2024.
[61] Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin. Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. arXiv preprint arXiv:2504. 15895, 2025.
[62] Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mo- hit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization. arXiv preprint urXiv:241 2.00:049 2024.
[63] Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: ceaiddnne-guidee compression in step-by-step efficient reasoning. XrXiv preprint arXiv:2505.0488), 2025.
[64] Xixian Yong, Xiao Zhou, Yingying Zhang, Jinlin Li, Yefeng Zheng, and Xian Wu. Think or not? exploring thinking efficiency in large reasoning models via an coormatton-thhorete lens. arXiv preprini arXiv:2505. 182. 2025.
[65] Jinghui Lu, Haiyang Yu, Siliang Xu, Shiwei Ran, Guozhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, al. Prolonged reasoning is not all you need: Certainty-based adaptive routing for efficient 11m/mllm reasoning. arXiv preprint 2xXi:25005.5554 2025.
[66] Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. Program- aided reasoners (better) know what they know. arXiv preprint 1XXi::33110055.. 2023.
[67] Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they'r re right: Probing hidden states for self-verification arXiv preprint 2541v:2504.0541, 2025.
14

---

[68] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen, Codet: Code generation with generated tests. arXiv preprint arXiv:2207.13397 2022.
[69] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05 0228, 2023.
[70] Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, 2023.
[71] Haau-Sing Li, Patrick Fernandes, Iryna Gurevych, and André FT Martins. Doce: Finding the sweet spot for execution- based code generation. arXiv preprint arXiv:2408. 137455, 2024.
[72] Gyeong-In Yu, Joo Seong Jeong, Geon- Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A distributed serving system for ("ransformer-Basee) generative models. In osdi, 2022.
[73] Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. Sarathi: Efficient I1m inference by piggybacking decodes with chunked prefills. arXiv preprint arXiv:2308. 16369, 2023.
[74] Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. Deepspeed-fastee fastte::: High-throughput text generation for Ilms via mii and deepspeed-inference. arXiv preprint urXiv:2401.086777 2024.
[75] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663-679, 2023.
[76] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pages 61 6-66, 2023.
[77] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. {DistServe): Disaggregating prefill and decoding for model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). pages 990-210,220 2024.
large language
[78] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative I1m inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118-132. IEEE, 2024.
[79] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXin:2408.00//8 2024.
[80] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[81] Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkataraman, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. Themis: Fair and efficient (GPU) cluster scheduling. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 289-304, 2020.
[82] Skywork ol Team. Skywork-ol open series, November 2024.
[83] Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.0592) 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitation in
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The full proof of the theoretical result in $3.4 is provided in Appendix F
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Our code is available at https///gghhh...oom/haaaaala/a//
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli///uuii gwwpc//ggus////eee CoodeSubeiiieeee ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are provided in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are suitably and correctly shown in Figures in $
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix G detailed the experimentation setting, including the GPU type, relevant memory and storage used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research strictly adheres to the NeurIPS Code of Ethics outlined in the official guidelines. Our work focuses exclusively on algorithmic improvements for token allocation and reasoning efficiency in LLM-based serving, with collection processing of sensitive personal data. We also ensure full transparency and reproducibility of results, aligning with ethical standards for open scientific research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the social impact in
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper has no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our paper utilizes various LLM serving engines (SGLang), LLMs (Llama, Qwen, Deepseek and its distilled models, QwQ), algorithms (Self-Consistency (SC), Rebase, MCTS), and datasets (MATH, AMC23, AIME24, LiveCodeBench. GSM8k. ASDiv). All assets are properly cited, and their corresponding copyright information is acknowledged. For each asset, have ensured that the appropriate licenses and terms of use are respected and explicitly mentioned. Additionally, we have referenced the original paper, creators and specified the versions used where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Research does not involve component with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No core methodology is LLM-generated. LLM is only used for writing. editing, and formatting that does not impact the core methodology, scientific rigorousness, or originality of the research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

A Related Works
LLM Reasoning with Test-Time Scaling. Large language models (LLMs) are increasingly being augmented with explicit step-by-step reasoning programs to tackle complex tasks. Chain-of-Thought (CoT) prompting [16] clicits step-by-step natural- language rationales, raising task accuracy with simple trigger phrases. Self-Consistency 4 enhances CoT by sampling multiple reasoning paths and selecting the most frequent answer, while Best-of-N sampling [7 generates multiple candi- dates and selects the highest-scoring solution. Tree-of- Thoughts (41 casts reasoning as tree search, where nodes represent intermediate states and search algorithms explore promising branches, enabling backtracking and global evaluation. The ReAct framework structures timesteps as (Thought, Action, Observation) triplets, allowing LLMS to reason internally and invoke external tools dynamically. More sophisticated approaches leverage Monte Carlo Tree Search (MCTS) (5, guided beam search [43], and Rebase
[K] to systematically explore reasoning spaces. More recent work has established Test-Time- Test-Time-Scalingg as key paradigm for eliciting deeper reasoning from fixed- parammeeee LLMs without retraining [3. The sl framework introduces budget-forcing rules by appending "Wait" tokens or terminating early to dynamically manage compute allocation [.. Further work shows that compute-optimal allocation favors parallel sampling over sequential deepening [44], that methods like SETS interleave sampling with self eeff-verficationn (45)), and that advances in meta- meta-generation algorithms [46] and generative reward models [47] can further enhance inference-ting scaling. These methods exploit multi-step reasoning, representing fundamental shift toward inference-tinee scaling in modern LLM applications. However, the gains in accuracy often come at the cost of efficiency: multi-step reasoning produces far more tokens, leading higher computational cost and latency. This trade-off sets the stage for Dynasor, which is a system designed to address these efficiency challenges by intelligently managing token usage and early exits.
Efficient CoT Reasoning. Efficient LLM CoT reasoning targets reducing excessive token usage in
long reasoning chains 48 Model merging blends fast shallow and thorough deep reasoning models
for CoT inference 49 50 Mid-generation self-evaluation prunes unpromising continuations on
the fly [5.] Specialized fine-tuning compresses rationales by skipping irrelevant tokens [52] 53
[54 55 IT, while reinforcement-be based methods learn to trim overlong thought sequences [2]
Parameter-space controls such as Co'T Valve adjust chain length based on learned cues [56]. Token-
budget-aware frameworks estimate question difficulty to allocate computation dynamically [56
and verification- -gidded systems like FlashThink identify when reasoning can safely terminate [57
While each approach improves the trade-off between compute cost and reasoning quality, most depend on weight modifications, additional modules, extensive retraining, complicating real-world deployment. By contrast, Dynasor adds only a lightweight proxy scheduler that manages reasoning depth within standard inference pipelines, guided by certaindex for efficient reasoning and consistent with observations that CoT reasoning often settles early (57.
Certainty-Drive Early Stopping for Efficient LLM Reasoning. Recent work has focused on
cutting reasoning costs by leveraging model certainty signals. Adaptive-Connissenccy [58 introduces
a lightweight, model-agnostic stopping rule. Early-Stopping cost by halting once answer distributions converge, while Geasoning-Aware Self-Consistency [60]
(59] reduces sampling
adjusts the number of samples required based on the quality and agreement of reasoning paths.
Concurrent work Dynamic Early Exit [6]] detects high-confidence points during Chain-of-Thought reasoning to stop early. Self-Consistency Preference Optimization (62 improves reasoning by
training the model favor consistent answers. COnCISE (63) shortens reasoning steps by reinforcing teken-level confidence and reducing unnecessary reflections. An information-fhherrtic method [64] introduces two metrics to measure how far the model's reasoning is from ideal and how much each step contributes, stopping once confidence (measured via entropy) is high enough. Certainty- based adaptive routing [65] and program-aides reasoning [66] confirm that model contidence often correlates strongly with correctness. Another method 67 checks hidden states to predict whether the answer is correct. However, these methods focus primarily on single reasoning paradigms (self-consio geeff-consistency or CoT), neglecting other reasoning algorithms and parallelism opportunities. In contrast, Certaindex provides a theoretically grounded stopping criterion that generalizes beyond self-consistency or CoT to multiple reasoning programs, including CoT with majority voting, MCTS, and reward-ge reward-guided tree search, while adding only a lightweight proxy scheduler that preserves both simplicity and deployment compatibility.
23

---

Tool- Validatdd Reasoning. Several approaches have been proposed to enhance LLM generation by injecting external validation via test execution or unit-test generation to improve answer quality. CodeT [88] automatically generates unit tests with the same I.I,, then applies dual execution consensus among code candidates to select the most reliable solution, Self-Debugging [69] prompts the model to detect when generated code fails, let it rubber-duck the error, and autonomously patch the program based on runtime feedback. LEVER 70) trains a verifier that jointly considers the input task, code, and its execution outcomes to rerank and filter generated programs more accurately, DOCE [71] combines sample reranking, irecuution-based scoring, minimum-Bayen and self-debugging into single pipeline for execution aware decoding optimization. While these methods rely on external tools, such as runtime environments, test rumners, or auxiliary verifiers. Dynasor takes different path: it only leverages the LLM itself, harnessing internal uncertainty and cross-consistency signals to drive early stopping, without executing any code or requiring additional tooling.
decoding.
LLM Inference and Serving Systems. Existing LLM serving systems improve throughput and latency via batching [72] 73 74 75), memory paging (76). and disaggregation 78), and many other techniques. Recent frameworks such as ParrotServe [9] and SGLang [4 co-design frontend and backend for multi-query workflows and KV-cache reuse, substantially enhancing efficiency. However, these systems treat requests as independent and do not target reasoning algorithms, which exhibit dynamic scaling and within-request branching. Dynasor is the first system to exploit adaptive compute-accurace trade-off's in reasoning workloads through a Certaindex-guie scheduler that dynamically allocates compute based on query difficulty.
B Example of LLM Reasoning Algorithms
We describe four widely used reasoning algorithms. While their specifics differ, they share two core operations: (1) expansion: generating tokens to expand solution trajectories, and (2) aggregation: combining results from trajectories to derive a final answer. Increasing compute for expansion generally improves the quality of answers during aggregation.
Atouc lead soe
Internacble
Cunditate Arswer
Dact arcoqpation
Figure 11: Illustration of the workflow of different LLM reasoning algorithms discussed in Ap- pendix B
Self-consistency (SC). Fig. al]a) depicts the computational process of SC (4. Starting with a prompt P. SC expands n trajectories Ti, .,...,,,. ..,"...,,, Ti by sampling different outputs from the LLM (e.g.. varying random seeds or temperature). Each trajectory represents reasoning path that terminates in candidate answer. During aggregation, SC applies majority voting across answers in Ti., and selects the one that appears most frequently. The key compute-comee contute-control parameter is n, which dictates the number of generated trajectories.
Rebase. Rebase 113, shown in Fig. 11 lb.,,, also begins by generating independent outputs 7, J2,,.., J, from the input prompt P. Unlike SC, these outputs represent intermediate steps, which may may not contain the final solution. They are ranked and assigned with scores 81, . 8n. typically by a learned reward model. Rebase selects nodes with higher normalized exponential scores / e as parent nodes, from which the next n reasoning steps are branched out.
This process is repeated until candidate solutions are returned. The final result is aggregated by (weighted) majority voting across all n candidate answers or selecting the top-scored answer (a.k.a. best-of-n). Rebase structures the reasoning process solution tree, where tree's 's depth (i.c., solution
24

---

RebaselG3MUeema 78
MCTS//9N/uomaa 78
KOT//SM/SO 328
SC/O5M/Liama 708
C= RR4 C=MRR RR)
CR N.
1.0
0.5
0.5
0.0
0.0
0.0
0.0
Rebasc/GSM/ulemms 348
NCTSJASDMVLiama
SCMATH/LLama
SC/GSM/Phi
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
Rebase/AATH
MCTS/Game24LLmm
SC/LiveCode chyl
SC/CSM/Gemma
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
10
Step
Stop
Step
Step
Step Collect Certaindex
Threshold
Fitted Curve
Solvable Problems
Unsolvable Problems
Figure 12: Correlations between certaindex strength (y-axis) and ground truth steps solution (x-
axis) on 12 (algorithm, task dataset, LLM) settings where algorithm {SC, Rebase, MCTS, ICoT), dataset [iveeCoeeBench [28]. GSM8K, ASDiv [37] GAME24 [29 ), and LLM E [Llama [32] Gemma [79]. Phi [80]. QWQ [3]] J. How certaindex is measured in each setting is shown in the y label of each plot. Certaindex measured at the reasoning step marked by the red line. The orange line indicates the thresholding- based allocation. The green line illustrates more fine-grained approach through curve fitting. For all plots (except MCTS), both certaindex values and oracle steps were averaged across multiple runs to combat randomness.
steps to reach an answer) depends on LLM outputs and its width is exactly which controls its inference-time compute.
Monte Carlo Tree Search (MCTS). Starting from the prompt P. MCTS 6 5 iteratively builds a solution tree (Fig. ()))). It expands nodes by sampling continuations from the LLM step-by-step. until reaching leaf node containing a candidate solution. Each solution is scored via reward model or simulation, and the score is pack-propagated to its ancestral nodes to update their reward values. This completes one iteration (or rollout). MCTS iteratively performs n iterations, starting each from a node with the currently highest reward. n the key knob that controls the inference-time compute increasing allows deeper and broader exploration of potential solutions.
Internalized Chain-of-thought (ICoT). Lastest LLMs, such OpenAl o1 155 and Qwen- QWQ 33 can internalize reasoning behaviors at training, without needing for explicit reasoning algorithms. These models generate extended chain-of-thought [16) sequences to derive candidate answers to reasoning queries. They iteratively refine these answers by reflecting on prior outputs and continuing sequential decoding until termination, yielding the final solution. The number of refinement rounds n, which corresponds to total decoded tokens, directly determines inference-time compute.
C Certaindex Based Resource Allocation
Large language models (LLMs) have an inherent ability to "know when they know", meaning they can sclf-assess their confidence in their answers. This ability, discovered in prior work such as [21], allows LLMs to indicate their certainty while generating tokens. In this section, We introduce certaindex, a measure of this confidence, as a proxy for reasoning progress: high certainty suggests the LLM is nearing final answer directly indicates lower absolute compute needed to reach final answer, whether correct or not. Our goal is to measure and track certaindex during inference, enabling dynamic resource allocation, prioritization of complex queries, scaling back simpler queries, and terminating unpromising queries. Some of the contents will overlap with $ 3.1 as we want to provide more complete narrative of the certaindex definition.
Various methods have been proposed estimate L.M uncertainty, including semantic mea- sures 22 23), log probability entropy 21144 and hidden state-based indicators [25] 26 27 While
25

---

certaindex's 's mathematical formulation varies across reasoning algorithms, its core interpretation remains the same: the LLM's certainty in its reasoning paths.
C.1 Measuring Certaindex
This section presents two basic formulations of certaindex.
Certaindex in typical reasoning algorithms. Reasoning algorithms like SC. MCTS, and Rebase expand multiple reasoning paths to derive aggregated final answers (Appendix B). To quantify LLM's certainty among these paths, we employ semantic entropy [22]] which is derived from empirical entroy. Given a question P, n reasoning paths are generated and clustered into m groups based on their answers: C1, C22 .... Cm, where Cjl denotes the number of paths in answer group C. The
empirical entropy is calculated as: H. = - C.I log [C:l
H. intuitively measures the diversity of answer distributions. A higher H. indicates greater uncertainty, where reasoning paths diverge into many different answers. Conversely, lower H suggests higher certainty, typically when most paths converge to the same answer (large [Ca] for some group i). The maximum entropy max(H)) log occurs when each path yields unique answer (m n, |C;| = 1 for all i). We normalize H. by max(H) to obtain certaindex:
1 max(H) H max(H() E [0,
For SC/Rebase/MCTS on exact-answer tasks such as arithmetic and multiple-choiee, those exact answers can extracted by applying string-matching on the generated outputs, and grouped based on their equality. In more open-ended generation tasks such as code (e.g., LiveCodeBencch (28] or flexible mathematical expressions [29]. answer extraction becomes non-trivial. We employ small embedding models [30] (c.g.. 100M parameters) compute textual similarities between final outputs, and cluster reasoning paths based on semantic proximity, which enables efficient answer grouping across varying problem formats.
Both clustering methods are computationally lightweight: string matching fast and efficient; running the embedding model and calculating similarity remains computationally insignificant compared to LLM prefill and decode operations.
Certaindex in reasoning algorithms with a reward system. For reasoning algorithms that incor- porate a reward model (e-g., MCTS, Rebase), we simply the reward model's normalized output R [0, 1] as a measure of certainty. This approach builds on prior research demonstrating that reward signals can effectively guide resource allocation in program execution [3]]. We collect the terminal reward scores from each reasoning path and aggregate them to compute certaindex. The aggregation method varies by algorithm: for MCTS, we use the mean reward across its different paths, while for Rebase, we take the maximum reward value. A higher aggregated reward indicates stronger certainty the reasoning paths' validity, while a lower score suggests uncertainty. These reward scores are collected during the normal execution of the reasoning algorithms and thus do not incur additional overhead.
Combining multiple certaindex indicators. Certaindex can also be composed by multiple signals, provided they effectively capture certainty during the LLM reasoning process. When multiple signals are available, they can be combined by applying individual threshoids to each metric. For instance, in our MCTS/GSM8K experiments (Fig, [2]. we use distinct metrics to measure certaindex: the reward score R and the entropy-based measurement H. Each metric is compared against its respective threshold, R, and H., respectively. A program considered meet the certainty requirement only if all metrics exceed their thresholds.
C.2 Effectiveness of Certaindex
This section empirically demonstrates that certaindex correlates strongly with the computational resources required reach correct solutions across diverse models, reasoning algorithms, and task datasets. Higher certaindex values consistently indicate lower total compute needs.
26

---

Correlation. We measure certaindex at intermediate reasoning steps and analyze its correlation with the ground-truth number of steps required to yield correct answers, as determined by an oracle. Queries are categorized as solvable or unsolvable, where unsolvable queries cannot produce correct answers even with near-infinite compute. Figure [2 illustraaees this correlation across 12 (model, algorithm, task) settings. On solvable queries, Pearson Correlation values between certaindex and required compute range from 0.17 to 0.75 (mean 0.52), indicating a strong correlation between high certaindex and fewer steps needed to solve a query. We next demonstrate two potential resource allocation strategies using certaindex.
Thresholding-bate baade based allocation. We measure certaindex at specific reasoning step, shown as a red vertical line in each plot of Fig. [2] A straightforward allocation strategy sets threshold value (orange horizontal lines) and halts inference for any query with certaindex exceeding t. In all plots, nearly no solvable queries fall the upper-right area beyond the red and orange lines, indicating that an appropriately chosen i can effectively carly-terminate queries once their certaindex is greater than For these queries, this reduces resource usage for solvable queries and prevents over-allocation to unsolvable ones (red dots above the orange line), without sacrificing accuracy.
Pareto-frontier allocation. While thresholding offers a coarse-grained control, a more nuanced approach is a Pareto-frontics allocation. Instead of a binary stop/go decision based on a single early checkpoint, this policy aims to assign dynamically tailored computational budget to each query based on its evolving certaindex throughout the reasoning process. The underlying principle is to leverage the observed correlation to inform a more continuous resource allocation. Rather than simply "fitting curve", we establish an empirically-dere derrived relationship that maps different certaindex values to an estimated optimal remaining token budget for that level of certainty. This mapping (conceptually represented by the green curve in Figure [2) reflects a trade-off: for higher certaindex values, the optimal additional budget is small, while for lower certaindex values, more computation might be warranted, up to a certain point. This allows for a dynamic allocation scheme where the maximum additional tokens for a query are continuously adjusted or capped based on its current certaindex. Queries exhibiting high certaindex (signaling proximity to a stable solution) are allocated fewer future tokens, while those still showing lower certaindex (indicating ongoing exploration or difficulty) might be allowed more, guided by this learned relationship. This strategy seeks to optimize resource use more globally across all queries, distributing the computational budget more effectively, as further elaborated in our "smart scheduler" design (.H.4).
C.3 Comparing Certaindex with Other Signals
One may ask if there exist other signals better than certaindex. We compare certaindex with two alternative metrics for estimating resource needs. (1) Reasoning path length: Longer reasoning paths (more tokens) often indicate harder problems, suggesting a potential correlation between path length and compute needs (e.g., more reasoning paths). (2) Mean normalized log probability: Established as a measure LLM confidence (24), higher log probabilities may correlate fewer samples needed for correct answers in SC.
Using the (SC, GSM8K, Llama3. 1-8--..s-------cec setup, we evaluated certaindex alongside these metrics. Four metrics were tested: certaindex's 's entropy measure H. mean output length [40]. mean normalized log probability (24), and their linear combinations. As shown in Fig. [3 H achieved the strongest correlation with ground truth compute requirements (Pearson Correlation of 0.68, Kendall's Tau of 0.61), outperforming other metrics and matching complex combinations. These results confirm that certaindex is simple yet effective proxy for estimating inference computational demands, offering robust performance across tasks and models. Our end-to-end token-to-ace tacecrra- evaluations in Appendix H compare different signals for resource allocation, confirming certaindex's 's superior performance.
D Characterizing the Relationship Between Certaindex and Detection Steps
Fig. 14 demonstrates the relationship between certaindex values and program steps across different detection points in a single run, using SC on the GSM8K dataset with Llama3. 1 8B Instruct model. The figure consists of six subplots, each representing a different detection step ranging from 5 to 30, indicated by "Detect @ @knob" values.
27

---

Pearson Correlation
Kendall's Tau
0.8
0.68
0.69
0.61
0.59
0.6
0.40
0,40
0.4
0.35
0:34
Length
Log P Entropy H(Ours))
Linear Comb.
Length
Log P Entropy H(Ours)
Linear Comb.
Figure 13: Correlation between certainty measurements and mean steps required to solve problems on solvable problems. We obtain the ground-truth mean steps by solving the queries using the LLM multiple times and counting the average steps.
Detect gieneb
Giknbb 10
gginob
Giknob
(oknob
Glenh
0.6
0.6
0.6
0.6
0.6
0.6
10m
0.2
0.2
02
02
D2
02
0.0
20
0.0
Early Stogpable Prozlems
Salvabre
Probiems
Threshold
Figure 14: Certaindex Values Across Different Detection Steps in Seff-Consistency Reasoning
Each point in the plots represents an individual problem, categorized into three types: Early Stoppable Problems (pink), Solvable Problems (blue), and Unsolvable Problems (gray). The x-axis shows number steps taken, while the y-axis displays the certaindex value (C = HD). A horizontal orange dashed line indicates the certaindex threshold value, and vertical red dashed line marks the step at which certaindex is collected.
The consistent pattern across all detection points demonstrates a strong correlation between certain- dex and required reasoning steps, with Pearson correlation coefficients exceeding 0.5 for solvable problems. As detection steps increase from 5 to 30, we observe a 30% increase in early-stopped problems, indicating that higher certaindex values reliably predict when the LLM is converging toward a final answer. Certaindex works reliable proxy for reasoning progress. However, this improved accuracy trades off against potential compute savings, as later detection points leave less opportunity for early termination. Despite this tradeoff, certaindex proves to be reliable predictor of required reasoning steps across all detection timings, maintaining its effectiveness regardless of measurement timing.
E
Dynasor: System Design
Dynasor is straightforward to use. Developers define reasoning programs through the provided abstraction, implementing certaindex and scaling knob control functions. These programs submitted to the application runtime, which monitors certaindex values, dynamically adjusts resource allocation, and forwards requests the system runtime for exccution. Programs either scale up for further computation or terminate early when resources are no longer allocated.
Figure 15a illustrates Dynasor's three-componee architecture: (a) a Reasoning Program Abstraction (Program) offering standardized interface for diverse reasoning algorithms, (b) an Application Runtime that dynamically allocates computational resources based on Certalndex measurements, and (c) System Runtime managing request-leve scheduling on the backend infrastructure.
This architecture enables Dynasor to adapt computation dynamically each query's 's difficulty level, allocating resources proportionally to reasoning complexity rather than using static, one-size-fits-al token budgets.
28

---

class BearondrgProgrea and def -.Innt... (061))
GolE kndb self, stuto self, certitud
OKC LESITA CEPEEN CETTIIL DETELEE
Apolioriis Rurthn
Ropening Phoguan Acatracton
corstnate
Local
Sebatractiethhod
Server
Ra ettime
det update, centstudeetteet
001 resoene sIL cortcsude rasredical bod. (911)
dubstroctnethiod
Stati
def
raturs aajority sttrggratt
Figure 15: Left(a): Dynasor Architecture. Middle(b): Reasoning Program Interface. Right(c): Example Program (SC),
E.1 Reasoning Program Abstraction
reasoning program maintains three runtime properties: (1) certaindex, the certainty measure the reasoning progress; (2) knob, the intrinsic scaling factor of the program; and (3) state, which stores the intermediate variables and results in previous steps.
The Reasoning Program Abstraction (or Program) provides a unified framework for developers to define variety of reasoning algorithms. It introduces narrow interface for the program interact with the application runtime. Fig. 15a shows the structure of a reasoning program. Developers only implement (I) update_ certaindex O, which calculates and updates the certaindex at particular inference step, and (2) execute (), which runs the reasoning algorithm (Fig. 15b).
Fig. 15c illustrate simple example implementation of SC: the update_ cetrreedertaiddeeeeee () function computes the entropy across different branches, and the execute () function iteratively expands the generation. After each iteration, it aggregates the result and update certaindex. This process iterates until the program depletes its resources or exits.
E,2
Application Runtime
E.2.1
Certaindex-Basee Intra-Program Scheduler
The certaindex-based intra-progit proa-program scheduler controls the resource allocation of the program at runtime. The scheduler lifetime is described as follows:
Initialization. When a new program arrives, the scheduler initializes it with predefined maximum resource cap and allocates resources for its first run. also establishes resource scheduling policy to guide future allocations.
Resource Allocation. The scheduler continuously monitors each program's certaindex and uses it to determine resource allocation. As programs run, they repeatedly request resources from the scheduler while updating their certaindex reflect the progress of reasoning. When multiple programs run concurrently, the scheduler can prioritize resource allocation between them via certaindex- -based intra-program allocation policy.
Termination. When a program's 's certaindex exceeds a limit based on its allocation policy or reaches the maximum resource cap, the scheduler denies further resources allocation for this program. The program then receives a termination signal and aggregates results based on its generation history.
We use the SC in Fig. 15c to illustrate the scheduler behavior. The program is first submitted to the application runtime, where the intra-progran scheduler is initialized and assigned the initial resource (5 branches) to the program. SC sets simple certaindex threshold to determine whether the program should terminate. As the program expand, it updates certaindex, and request resource from the scheduler for next iteration. The scheduler checks the certaindex against the policy, and decide if should allocate resource for the program to continue running.
Resource allocation policy. A resource allocation policy determines compute allocation (e.g.. branches in SC and iterations MCTS) for each program based on its certaindex. $C.2 discussed two certaindex-based allocation policies: simple thresholding and pareto-frontier, both implemented in the system. Additional alternatives are analyzed in $H.4) Developers can easily configure the antra-program scheduler use other certaindex-base policies.
29

---

batch avg latency = (4 + 9 9 6.5ms
batch
avg latency (8 10 9 ms prog2. 5*2=10ms -
progg. 4+5=9ms--- -
-prog11, 4ms
-poogl 4 4*2==ms----
prog1, req1
prog2,
prog1, req1
prog1, req2
prog1, req2
prog2, req2
prog2, req1
prog2, req2
time(w/ gang)
time gang)
Figure 16: Illustration of Gang Scheduling
E.2.2 Profiler-Guided Policy Calibration
We explain how to determine (or calibrate) a resource allocation policy, focusing on certaindex threshold policy as our example. Determining an effective resource allocation policy is critical for reasoning program performance ($H.4). Program submitters often lack insight into runtime patterns, and program characteristics may shift due to algorithmic changes or data distribution shifts,
Dynasor provides optional profile- profile-based based hyperparameter tuner to help users identify optimal scheduling policies for certaindex-bet based resource allocation. Users submit a batch of programs with labeled data such as verified answers to questions (e.g., math problem solutions), response rankings, or reward model scores. The profiler collects runtime metrics, including certaindex and token usage, to determine optimal resource allocation across different certaindex ranges while maintaining accuracy requirements. For threshold-based allocation, the profiler stops allocating resources when program's 's certaindex exceeds the threshold, and otherwise sets resources to maximum cap. The threshold calibrated meet the accuracy requirements (e.g., not hurt accuracy in all calibration data points). This calibration process can be periodically executed in real-world serving scenarios to meet the dynamics of data distribution shifts.
E.3
System Runtime
E.3.1
Program-Aware Inter-Program Scheduler
The Program-Aware Inter-Progrant Scheduler optimizes scheduling and memory management the program level, reducing per-proge program latency through two key strategies: (1) gang scheduling algorithm to prioritize requests originated from the same program, and (2) approximate Shortest-Job- First (SJF) scheduling algorithm to reduce head-of-the-line (HoL) blocking and improve per-request latency.
Gang Scheduling. Gang scheduling groups requests from the same program together to minimize stragglers and reduce overall completion time. Fig. [ddemonstrass the advantage of Gang scheduling over sequential scheduling using an example with two programs arriving at t=0, where the system has batch size of 2. Each program has two requests: program I's requests take 4 ms each, and program 2's take 5 ms each. By prioritizing one program at a time, Gang scheduling (left) reduces average latency from ms to 6.5 compared to sequential scheduling (right).
Approximating Shortest Job First (SJF). Our inter-program scheduler implements an approximating SJF scheduling algorithm to mitigate Hol. blocking and improve average per-program latency. A program's total exccution time depends on two key factors: total compute requirements (knobs, c.g., number of branches/itcrations) and token length per branch/iteration. While per branch/iteration exact LLM generation lengths cannot be known in advance [40], we can estimate token length per iteration by leveraging program locality and using historical averages from previous iterations. This estimation, combined with the compute requirements, helps predict total execution time. In Dynasor, certaindex controls the total compute requirements (deciding how many iterations to run), while SJF uses the estimated token length per iteration to optimize execution order.
Starvation Prevention. Dynasor promotes finish-time fuirness 81 which compares program's 's completion time in shared system (Tshared) to its estimated independent completion time (Tindependem). This metric naturally suits LLM serving it accounts for generation length. We define finish-time
30

---

fairness as ④ / #output tokens in our case, which is approximately proportional to the standard mmish-tima fairness since independent can be estimated as k #output tokens when decoding dominates the generation time, where k is constant. While Gang scheduling alone significantly improves performance without causing starvation, adding SJF optimization can potentially lead to starvation of programs with longer estimated lengths. Gang scheduling alone already provides substantial performance benefits, making a viable standalone option. For deployments using SJF, we implement priority escalation mechanism where programs that have been waiting too long receive elevated priority, effectively preventing starvation. As demonstrated HH.3 Dynasor promotes fairness compared to other baseline scheduling strategies.
E.3.2 Other System Runtime Components
Prefix Cache Manager. Dynasor leverages prefix cache sharing. Independent branches of program share a unified prompt KV-cache, while dependent reasoning chains reuse their longest common prefixes. For algorithms with reward models. Dynasor reuses prefixes during reward evaluation, reducing latency. The manager also handles automatic cache eviction. When memory is constrained, the KV cache of the programs with no active requests is assigned lower priority and evicted first.
Program Context Manager is a thin wrapper that tracks registered programs and their runtime characteristics. It provides cache eviction hints to the prefix cache manager based program behaviors. Unlike the static program DAGs used in flexibility by supporting dynamic generation patterns required by algorithms like MCTS and Rehase.
this component offers more
Execution Backend manages the execution of LLM requests, optimizing model performance through techniques like CUDAGraph. This layer is designed to be adaptable to various execution engines including vLLM, TensorRT, and SGLang.
E.4 System Implementation
We build Dynasor on top of SGLang (version 0.3.3 postl). The intra-program scheduler is imple- mented as Python library on the client side, while the inter-program scheduler is integrated into the server-side scheduler.
Using Dynasor, we adapt various reasoning algorithms (Appendix B) to the Program interface, including their custom certaindex implementations. Our system comprises approximately 1.5k lines of Python code, covering the Program interface, application runtime, and system runtime. Notably, the core system runtime only comprises around 500 lines of code change, and the changes are modular and non-invasive, making it a very clean implementation into the core serving system logic.
Adapting each reasoning program requires an additional 40 150 lines of code define certaindex logic and integrate with the Program interface. This implementation overhead is minimal compared to the original implementations of these algorithms, which spans up 4,000 lines of code.
F Proof of Stopping Criteria for CoT and Probing
Given the input prompt x, we can stop the reasoning chain Yi.... Y,, Yet1 at time if the following holds for any T t.
P.(YotIZe, Y..... Y,) L (Y Y++1) 41 T Pr(Y YI...., Yr) = (Y(x)).
(2)
The above definition means that adding new steps in the reasoning chain does change the distribution of the generated answer anymore. As a result, the reasoning chain reaches the stationary distribution, P. (Y [z), which the desired output given the prompt x in the first place.
We assume that the language model will eventually converge in the chain of thought, if the answer distribution remains the same for long enough.
Formally it expressed as follows.
Assumption If equality Po(e+ala,, Yi,..., Yi) = Pe+a(Y Yi.... Ye4s) holds for any 1 < S K",
31

---

then P. (x (1.11/,,,Y.......... ,I...., ..). Pr(Y++1le ,I....,Yr) = P. holds for any T >> t. Further assume that k- (M). where M is the number of distinct output groups.
Based on the above assumption, we propose the test criterion in Definition 2
Definition 1. Consider n samples, each corresponds to Pt..... Pitn- Denote the mixture distribution of samples 1,...,. i
& E ml. to be Pit E5= Pi+j.
Let the smallest where Eq. 2 holds be 1* Then for any i, 2 1", P. hand, if < 1*, then there exists j i, so that Pit + P This intuition is formally expressed in
P =
P.. On the other
1+1 Lemma 2 beloo.. We therefore propose test the difference among Pitk, pi+k+t ................................
pi+2k
i
i+h
Lemma 2. (P: P O, for any S S k, and (Pttt Pitsth-1) 0. for any 1 S j S 1, then Eq. 2 holds for any t 2 i, and T < - 1.
H4)
Proof. We first have from the definition of the TV distance that
p - |, 0. For general 1 j S k we have from the definition of P that
Σ
(P++ Po+1t*)
-||ppth Pit] 0.
Therefore, El Pi+1 Eim P.+++
Take j = 1, then
P++1 = P.+1+k. Plugging into the case where 2, then we have PH2 = P44244- Expanding the recursion for all S j S k, we obtain that
Pi+s
Petitk:
visis Ssk
plojtt-1) = 0, for any 1 1 S k- - 1, we have Similarly from the assumption that TV (pith-1, [+] that:
= pi+k-1 pi+i+k-1
Σ
(Pit P
vi
k-
and consequently that
vi
The equalities in Eqs. (3 and 4 form graph over the vertices that are P.tl...., P-+2k-1, where
the equalities are the edges. Note that all the nodes are edge connected by virtue of Eqs. (3) and Therefore, we have that
P- M+lsiie 2k
We therefore define the approximate stopping criteria as follows.
Definition 2. We define the e-accuracy of the stopping criteria:
TV Pititt < e, for any 1 Asis
TV(P P pitjttk < c, for any -sjsk-
We then ask: how many data samples do we need, to be able to test our stopping criteria up to e-accuracy? More concretely, we construct the estimator of pt+ for 1 = i 1,..., + k. and 1, k as follows. Assume we have M disjoint groups of the outputs. Denote each group of answer Cm. for I...,M.. Then we estimate the average probability mass
32

---

function p/+t by defining the Bernoulli random variable for each Cm: D. = 1[,, E Cm J. and P| (S) = - z
The next lemma establishes that with io 52 M A+ then TV(P/-', .(tt)) S c/3, for all I 1 1 i i k. and t= -ー k, with 8 probability, Then we just have to test that
TV(Ptt,,Pi - </3, for any S j S k; TV(P+th11, p( - e/3, for any 1 s
If the above inequalities stand with the established number of steps k. then by the triangle inequality of the total variation (TV) distance, we obtain the e-accuracy in the stopping criteria in Definition 2
TV P P & TV pitth) +V((P .P:3 TV(P: PP: S.
Lemma 1. kf = 52 ++ ). then TV ,(tt) S for all = i I,..., + k, and
t= k 1, ki with 1 5 probability.
Proof. To prove convergence in the total variation (TV) distance, we first express it as follows. Let S belong to the power set (denoted as expx) of U C.. Then
TV(, = sup
P(S) - P(S)
Seexp"
We then prove convergence for fixed set S, and then apply union bound to expc to obtain the final result.
Denote Bernoulli random variable 2 = 1[Y, E S}. Then P)+(()) & Ett4 2 On the other hand, Pl PS+ = - PY E S|x, VY..)..,y-I)..
Note that Y, |rr, YI,... Y P+( Y ,...., 1-) Therefore,
E | xr Yi.... Y--] = P+(Y, E Sla, YY-1).
This means that - - P,(Y, E S S ,,,..,,, Y1) is a martingale difference sequence. We also know that - PY, E S|x Yy....,,Y.... Y,-1) 2 1. Therefore, by the Azuma-Hooe coena-Hoef inequality, we have:
P
Σ
(z PP, S|x, Y...... ,Y,,)) 26 2exp
2t
1=1+1
or
1+t
1 1+t
TP -() P|) =0)=P/ =P
1
Σ
z
Σ
P,YY, E Y,-1) 2c
t
r=|+1
7=141
te2
S 2exp
2
Taking a union bound over S E expo, we have:
P ( pltt) 2 20) =P
p( (1)())((((s)) 2c
sup SEexp
< MP ( (1 ((tt(()) p() 2() S 24+ exp
tea
2
On top of that, taking union bound over / = i + 1,..., + K. and / = k 1, k. we have that
the probability of (V plttt 2 for one of the combination of ( = i + 1...., + k, and
t = 1, is less than:
P (T p(tt) 2) 22 exp (- (k: - 1)r
Σ f=
Σ
cexp
33

---

Table 1: Offline workload Configurations. LiveCodeBench (LCB), Llama2 7B [5], Skywork7B [82).
Llemma 7B and 34B [33] are fine-tur sune-tundd models used in different settings as LLM/reward model.
(Algo., Dataset, LLM)
Reward Model # Samples
Resource Cap
(SC. LCB, Llama3.1 8B)
/
400
5.10.15.20.25.30
(SC. GSM8K, Llama3.1 8B)
1000
5,10.15,20.25.30
(MCTS, ASDiv, Llama2 7B)
Skywork 7B
300
3,7,10,15,20
(MCTS, GSM8K, Llama2 7B)
Skywork 7B
300
3,7.10,15,20
(Rehase, MATH, Llemma 7B)
Llemma 34b
500
16,32,64,122
(Rehase, GSM8K, Llemma 34B)
Llemma 34b
500
16,32,64,128
Table 2: Online workload Configurations
Algorithm
Dataset
LLM
Reward Model Base Deadline (s)
SC
MATH Llama3.1 8B
240
MCTS
ASDiv
Llama2 7B
Skywork 7B
60
Rebase
GSM8K
Llemma
Llemma 34b
300
In other words, whenever
M + (gg(1/0) k = 52
2²
we have (V P(tt) s e/3, for all Here S2(-) means that we omit logarithmic order terms in M and e.
k. and k 1, k, with 1 1- probability.
G Detailed Setups for Offline and Online Experiments
Table 3: Hyperparameter configurations for certaindex. Rebase is evaluated on the MATH-OAI 38
subset of the MATH benchmark.
Algorithm
Dataset
Thres. (H.)) Thres. (R,) Detect @ knob
SC
MATH
0.7
/
S
SC
GSM8K
0.7
/
5
SC
LiveCodeBench
0.4
/
5
MCTS
GSM8K
0.99
0.4
3
MCTS
ASDiv
0.4
3
Rebase
GSM8K
0.85
0.99
16
Rebase
MATH
0.75
16
Compute. All experiments run on GPU cluster (Runpod) equipped with A100 (80GB) GPUs. GPU usage varies by method: Rebase and MCTS use two GPUs to separately serve their LLM and reward models, while SC requires only GPU for request processing, For final answer selection, SC uses simple majority voting, Rebase applies weighted majority voting, and MCTS selects the best path using a reward model via best-of-n.
General settings. Tables summarrie our offline and online evaluation workloads, respectively, Table lists the certaindex hyperparameters for each workload, including the thresholds at the detection step (detect @knob). program terminates at the detection step certaindex values meet all threshold conditions.
G.1 Batch Inference Setup: CoT
Metrics. For offline workloads, we measure tokens-to-can ac-cuurcc,, the total number of generated tokens needed to achieve specific accuracy levels across reasoning queries. This metric reflects accuracy-cost trade-off, critical for end users as LLM platforms charge based on token usage. Table [ details the offline experimental settings for SC/MCTS/Rebase. Each problem is allocated a maximum computation budget (parameterized by a Resource Cap) defining the sampling limit for SC and iteration limit for MCTS. Resource caps correspond to different points in Fig. 7 While
34

---

scheduling methods may terminate programs early, they cannot exceed these caps. Query deadlines are calculated as the product of three factors: the SLO scale, the query's 's difficulty factor, and base deadline, reflecting the additional time required for more challenging problems.
We evaluate the Certaindex- based early termination method ($2.1) against baseline uniform token allocation across multiple scales of distilled DeepSeek models (7B, 14B, and 32B) [4] on mathemat- ical reasoning benchmarks AIME24 and AMC23 [20], and MATH500 [38). Unlike the baselines that uniformly increases token budgets, our appraoch early terminates by monitoring Certaindex at various intervals (T 32, 64, 128. 256, and 320). All requests are given max token budget of 16K. For each interval, we vary the early termination parameter N (the required number of consecutive consistent answers), generating different points along each line. For fair comparison, appropriate accuracy thresholds were calibrated to model scale with 32B models evaluated against stricter thresholds above QwQ [33] levels and reduced thresholds for smaller models while setting higher targets for simpler tasks where greater accuracy is achievable.
G.2 Batch Inference Setup: SC, MCTS, Rebase
Metries. For online workloads, we simulate query arrivals using dataset queries and assign deadlines to each query. System performance evaluated by P90 deadline attainment, the percentage of queries completed within their deadlines. We vary arrival rates and SLO scales to assess system performance under different conditions. Tab. D ootllnnes the online experiment settings. Request arrivals follow Poisson process with varying rates. Deadlines are difficulty-aware. determined using a simple policy: oracle difficulty for each query is estimated through extensive trial runs (> 100) for each algorithm-dataset combination, classifying queries as always correct (difficulty factor = 1), always incorrect (= 3), or variable (= 2). Deadlines are calculated as the product of the SLO scale, the difficulty factor, and base deadline, accounting for the additional time required for more challenging problems.
Baselines. For offline evaluation, we compare Dynasor against modified SGLang intra-program scheduler using the following policies:
(1) baseline-even, allocates resources uniformly across all reasoning programs using the resource cap settings in Tab[] Different cap values result in different accuracy levels, as larger caps allow more computation for all problems. Specifically, for each algorithm, all queries receive identical resources: SC uses the same number of branches, Rebase uses the same width, and MCTS uses the same number of search iterations, as specified in Tab[] This baseline reflects the behavior of reasoning programs in current systems: user may allocate equal resources on all problems.
(2) baseline-length. This policy uses the cumulative token count generated at a specific step (specified as Detect @ knob in Table 3) the program's 's progress signal. The detection step matches the one used in the certaindex -baee approach. The scheduler either continues allocating resources or terminates the program based on a predefined token-length threshold.
Output token length is commonly used as a scheduling indicator in existing LLM serving systems 40 76
It also correlates strongly with the compute requirements (knob size) as we have discussed
G.3 Online Inference Detailed Setup
Baselines. For online evaluation, we evaluate Dynasor against modified SGLang with different inter-program schedulers on P90 deadline attainment.
SGLang (34]. SGLang represents a strong baseline as incorporates key optimizations in LLM serving: its longest-prefis matching (LPM) algorithm efficiently batches requests with common prefixes, while prefix caching reduces redundant computation. We enabled these optimizations and tuned for stable performance with 70% memory utilization.
Parrot [3.. Parrot uses gang scheduling (App-FIFO) to prioritize requests in the same program. We implemented its scheduler on top of SGLang for fair comaprison. By grouping requests from the same program together, it minimizes context switches and maximizes KV cache reuse across batches.
For experiments (a) and (b), we use fixed resource cap settings (SC: 20, MCTS: 15, Rebase: 128) to compare Dynasor with basclines, which lack adaptive compute scheduling for LLM reasoning
35

---

Uniform Allocation
Ours
Mean Labency
410s
Uniform Allocation
Ours
Rate: Program/s
2
2
1.9x
200
200
165s
159s
1346
1.0x
1.1x
1.3x
1-
LPM
Cang
Ours (UPM) Ous (Gangl Ours (SJ)
Figure 18:
Performance improvement break- down in online SC (GSM8K): Impact of gang scheduling, certaindex-basee allocation, and SJF on mean latency with fixed request rate (rps 16).
LPM
Gang
Ours (LPM) Ours (Gang) Ours (SJF)
Figure 19: Performance improvement break- down of online self-consistency (MATH) under fixed P90 SLO constraints.
programs. We measure SLO attainment while ensuring accuracy is maintained. Our method employs the same resource allocation policy validated in $4.1 $ which preserves accuracy. In experiment (c), we explore the relationship between achievable accuracy and SLO attainment by varying resource cap settings.
H
Other Ablation Studies
H.1 Effectiveness of CoT in Deepseek-R1
To validate scalability, we extended experiments the larger DeepSeck-RI model on AIME and AMC datasets (Figure [7]. The results align with findings from smaller distill mod- els, demonstrating consistent efficiency gains: DeepSeek-RI achieves 12% to- ken savings on AIME problems and 24% on AMC problems while maintaining baseline accuracy levels.
1.00
0.76
AIME Accuracy
Accuracy
AMC
9.96
0.72
12%
0.94
0.70
24%
90
Baseline
Ours (64)
Ours (256)
Ours (32)
Ours (1226)
Ours (320)
Figure 17: Apply Dynasor DeepSeck-
H.2
Ablation: Scheduling Component Contributions
Figures 18 and 19 show the contribution of different components using SC on GSM8K and MATH datasets, measuring mean latency and maximum sustainable rate under the same P90 attainment. Using LPM (SGLang) as the baseline, we analyze three factors: gang scheduling, SJF, and certaindex- based resource management. Gang scheduling improves latency and throughput in both uniform and certaindex-awar aware resource allocation by ensuring the requests in the same program are scheduled together, reducing resource fragmentation and context switching overhead. For GSM8K workload, certaindex-based resource management dominates the improvements, reducing mean latency by up to 60% through token savings of up to 50%, while Gang scheduling and SJF contributions are more modest. For MATH workload, certaindex-avee aware allocation achieves 1.2x peak rate through early termination of low-value computations (as only 10% tokens are saved on MATH), while SJF provides the most significant gain with a 1.9x peak rate by prioritizing shorter jobs and reducing head-of-ling blocking. These results validate our two-level scheduling approach: intra-program optimization through gang scheduling and certaindex-awee -ware allocation, combined with inter program optimization through SJF.
H.3 Fairness Analysis
We apply finish-time fairness ($ E as the fairness metric and use latency / number of tokens as proxy. Figure [agure[0]showw shoss the comparison on certaindex-bast resource scheduling, gang scheduling. and SJF on finish-time faimess on MATH using SC with a rate of 8 pps using Llama3.1 1 8B. Gang scheduling shows consistent fairness improvement compared to non-gang scheduling. This shows that prioritizing existing programs can improve finish-time fairness. Certaindex-basee resource allocation also consistently improve finish-time fairness due to resource cutting by the intra-program scheduler. Adding SJF shows faimesss improvement at the later 50% fraction of the job compared to without
36

---

Table 4: Token consumption comparison of different scheduling strategies while maintaining accuracy
Allocation Method
SC/MATH
MCTS/ASDiv
Baseline
1.180M
354K
Statie Thres. (Ours)
1.05M (-11.0%)
308K (-13.0%)
+ Initial Step Curve Fit.(Ours)
1.04M (-11.9%)
306K (-13.6%)
+ 5-Step Thres. (Ours)
1,03M (-12.7%)
307K (-13.3%)
+ Single-Step Thres. (Ours)
1.03M (-12.7%)
306K (-13.6%)
+ Dynamic Curve Fitting (Ours)
1.01M (-14.4%)
298K (-15,8%)
SJF, and at the later 35% fraction of job compared to LPM. In all case, SJF shows the fairness metric no worse than even resource allocation.
CDF of Finish Time Fairness (Latency / #Tokens)
1.0
Fraction of Jobs
Ours (SJF)
Ours (Gang)
0.5
Ours (LPM)
SGLang
Parrot
08.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Finish Time Fairness (Latency # #Tokens)
Figure 20: Finish-Tima Fairness.
H.4 Fine-grained Resource Allocation
We compare simple thresholding against fine-grained resource allocation in this section. $4.1 and $4.2 adopt a simple static threshold mechanism: extract certaindex at a fixed step and decide if to terminate program based of the value of certaindex. While effective, there is room for optimization through more sophisticated scheduling strategies. To explore this, we conducted experiments in offline batch processing using the setting (SC, MATH) and (MCTS, ASDiv), both with a maximum step limit of 20 as per-query resource cap, to evaluate more complex allocation strategies that enable early termination while maintaining accuracy. The results, measured in token savings, are presented in Tab. For the static threshold, collected certaindex and apply the threshold in Tab. 3
We first examine an Initial Step Curve Fitting approach, which fits a skyline curve (illustrated by the green lines in Fig. 12) using certaindex values collected at the same steps as the static threshold. Although this method enables finer grained resource allocation for varying certaindex values, relied on early certaindex values, which maybe inaccruate as reasoning progresses, resulting in marginal improvements (less than 1% compute savings) compared to simple static thresholding.
To enhance prediction accuracy, we test more frequent certaindex collection: every 5 steps (5-Step Thres.) and every step (Single-Step Thres.), based which similarly apply threshold filtering or skyline curve fitting at every step (Dynamic Curve Fitting). These approaches achieve higher token savings (up 3.4% over static threshold). However, implementing them introduces scheduling trade-off's in real-world deployment due to their impact on scheduling and parallelism. Specifically, frequent certaindex collection may disrupt the concurrent execution of reasoning programs. For instance, in SC, instead of running 20 samples concurrently, we must process them in sequential batches (e.g.. batches of 5 samples) to track the certaindex. This shift from parallel to sequential execution substantially increases latency. Our benchmark shows an increase from 289s to 366s in mean latency when serving 500 programs. Given these practical constraints, we opt to implement the simple static threshold in our end-to-end experiments, prioritizing system performance over marginal token savings; but note applications which prioritize cost over latency, such allocation strategies remain effective and are implemented in Dynasor.
37

---

I Toker-to-accuracy Performance of Dynasor on MATH using SC
Figure 21 presents the are allocated by Dynasor using the threshold and detect @ knoo configurations described in Tab. 3 Our proposed method achieves the same accuracy while reducing computational costs by 11%.
results using Llama3.1 8B Instruct model, where resources
Ours
Baseline- even Self-Consistency MATH
Accuracy (%)
0.56
0.54
0.5
0.8
1.0
Number of Tokens (1e7)
Figure 21: Token-to-accuracy Performance Using SC on MATH
Probe Prompt Sensitivity
We conducted comprehensive ablation studies using three different probe prompts to evaluate robust- ness of early-exit performance prompt phrasing in COT.
Pl:
Oh, suddenly got the answer to the whole problem, Final Answer\nAal \boxed{
P2: Based on the reasoning so far, the answer is: \boxed
P3: Final Answer\n'n[ \boxed(
Table annd demonstratee that different prompt styles achieve similar token savings comparable accuracy levels. For instance, on AMC-32B at 96% accuracy, all three prompts save 20-25% tokens (P1: 22.5%, P2: 19.8%, P3: 24.9%) Similarly, AIME-32B at 71 % accuracy, the token savings range from 11-14% across all three prompts (P1: 13.9%, P2: 14.4%, P3: 11.4%). This shows the method's effectiveness is robust to prompt phrasing, with consistent performance regardless of the specific guidance style used.
Table 5: Lower Target Accuracy Scenarios. Format: Accuracy Token Saving (%).
Model + Dataset
Early Exit (P1)
Early Exit (P2)
Early Exit (P3)
AIME 8B
49% / 30.4%
49% 32.2%
49% / 34.9%
AMC 8B
88% / 28.0%%
88% 25.1%
80% / 33.6%
AIME 32B
69% 22.4%
69% 17.8%
69% / 16.9%
AMC 32B
89% 36.6%
90% 36.9%
92% / 32.7%
Table 6: Higher Target Accuracy Scenarios. Format: Accuracy / Token Saving (%).
Model + Dataset
Early Exit (P1)
Early Exit (P2)
Early Exit (P3)
AIME 8B
52% 15.6%
52% 4.0%
52% / 27.7%
AMC
91% / 19.4%
90% 14.5%
83% / 30.9%.
AIME 32B
71% / 13.9%
71% 14.4%
71% / 11.4%
AMC 32B
96% / 22.5%
96% 19.8%
96% / 24.9%
38

---

K Information Completeness and Truncation Fidelity
To examine whether truncated CoT retains sufficient reasoning information, we varied the confidence threshold that governs early exits. We report the fraction of Right->Wrong cases, where standard decoding is correct but early exit not.
Table 7: Trade-off between accuracy, token saving, and information loss (Right-Wrrog
Model
Accuracy (%)
Token Saving (%)
Right Wrong (%)
AMC: 32B
89.0
36.6
8.2
96.2
22.5
1.0
97.3
16.1
AIME 32B
62.0
39.8
11.7
69.3
22.4
4.7
72.0
9.7
1.7
Table demonstrates that aggressive early exits show substantial Right-Wrrong rates of 8.2% and 1 11.7% % for AMC and AIME respectively. indicating significant information loss. However, as confidence thresholds become stricter, this drops dramatically from 8.2% to 1.0% 0.0% for AMC, and from 11.7% to 4.7% to 2.7% for AIME. With sufficiently strict thresholds, the Right->rrong rate becomes negligible, indicating that when the confidence threshold high enough, the truncated reasoning contains nearly all necessary information for correct answers. This validates that adjustable confidence mechanism effectively controls the trade-off between efficiency and reasoning completeness.
L Incorrect Stabilization
Sometimes, our method encounters incorrect stabilization in CoT, where the model converges to wrong answer with high confidence. However, our analysis shows that with sufficiently strict confidence thresholds, approach largely preserves the model's original reasoning capability while primarily serving as token-saving mechanism.
The key insight that when confidence thresholds are strict enough, our method essentially maintains the model's inherent accuracy: if the model would originally get an answer right, our method keeps right; if the model would originally get it wrong, our method doesn't 't fix that underlying limitation in most cases.
Table Shoow the result. With conservative thresholds, the (0.0% for AMC, 1.7% for AIME), while Wrong-+Righ remains minimal but consistent (0.7-1.0%). This demonstrates that our method primarily acts as an efficiency optimization rather than an accuracy enhancer. The small Wrong-Right improvements likely represent where early termination prevented overthinking, but this effect is marginal. Essentially, when thresholds are strict enough, we preserve the model's original problem-solving trajectory while achieving substantial token savings (9.7-16.1% % even with conservative settings).
rate drops to nearly zero
Table 8: Stabilization behavior under different thresholds.
Model
Accuracy (%)
Token Saving (%)
Wrong Right (%)
Right - Wrong
AMC 32B
89.0
96.2
22.5
0.8
1.0
97.3
16.1
0.8
0.0
AIME 32B
62.0
39.8
0.7
11.7
69.3
22.4
1.0
4.7
72.0
39


Reviews:
[{"review_id": "lDmjbuyO2M", "reviewer": "Reviewer_Jsvc", "summary": "This paper proposes a solution to solve the trade-off between increasing the token budget for sampling long chain of thoughts to improve accuracy vs. the compute inefficiency of overly long thoughts. They propose using uncertainty estimate based on the model's several intermediate answers in the long CoT to determine an early exit criteria, with the uncertainty calculated using discrete answer frequencies, or incorporating external reward models depending on the task. The paper then pairs this an adaptive scheduling algorithm (Dynasor) that allocates the token budget for batch and online workload. Empirical results show the effectiveness of Certaindex to estimate the correctness or number of steps before termination as well as the prowess/efficiency of their scheduling algorithm.", "questions": "- The \"settling concept\" mentioned in Line 31 has also been described in prior works: https://arxiv.org/abs/2504.07128\n- Sec 2 para 2, the \"probe-in-the-middle\" is a standard test-time scaling approach as used in the S1 paper and should be cited as such.\n- Certaindex using answer frequency to measure entropy for code, how does it compare to CodeT, another cross-consistency approach with unit test outputs and other methods relying on unit test feedback https://arxiv.org/abs/2207.10397?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-written, easy to follow and addresses a useful problem addressing balancing the length of CoT and its inefficiencies.\n- Convincing empirical results that demonstrate the effectiveness of Dynasor (the scheduling algorithm) over other baselines for batch and online workloads, demonstrating its utility to the community at large.\n\n**Weaknesses**\n- Certainty computation itself is not novel and quite similar to ideas used in prior works such as adaptive self-consistency (https://arxiv.org/abs/2305.11860). The adaptive scheduler basically determines the token budget in blocks/steps which is similar to automatically selecting the value of k (number of samples) and early stopping of the sampling process in adaptive SC. The authors should compare with this baseline in addition to self-consistency.\n- Missing related works on using confidence estimates for inference-time decisions:\n    - https://arxiv.org/abs/2305.11860\n    - https://arxiv.org/abs/2311.09553\n    - https://arxiv.org/abs/2411.04109\n    - https://arxiv.org/abs/2504.05419, https://arxiv.org/abs/2504.15895 (concurrent)", "was_revised": true, "final_justification": "I am satisfied with the effort authors have put into updating the related works which was my biggest issue. I think the papers contribution is reasonable.", "rebuttal": "Respond to Jsvc\n\nWe appreciate your time and suggestions in reviewing our work.  We take this opportunity to clarify our key contributions and address the concerns raised.\n\n# Weakness \n\n## W1(a): Novelty of certainty computation.\nThe reviewer suggests that *“certainty computation itself is not novel … such as adaptive self-consistency (Aggarwal et al., 2023).”* We stress that our contributions extend **far beyond** a single heuristic of certainty computation: we present a generalized metric (Certaindex), theoretical foundation, and system-level scheduler (Dynasor), making Dynasor both broadly applicable and practically efficient in serving real-world reasoning programs. Below, we clarify these contributions in detail.\n\n**(Generalizability of Certaindex.)** \nUnlike adaptive SC (Aggarwal et al.), Certaindex is a unified certainty metric that adapts to multiple reasoning paradigms beyond SC alone – including CoT, Monte Carlo Tree Search (MCTS), and Rebase – across a variety of workloads and models. Beyond introducing the metric, we conduct systematic ablation studies (Section 4.3) on different thresholding strategies for early stopping (Figures 9, 10), showing how these criteria affect both accuracy and token usage.\n\n\n**(Theoretical contribution.)** \nUnlike other works that rely on heuristics to perform early-stopping, we developed a theoretical framework (Section 3.4 & Appendix E) that provides provable guarantees for early exit and token allocation. This theoretical underpinning ensures that our certainty-based stopping criteria can be tuned for optimal trade-offs between performance and efficiency.\n\n**(System contribution.)** \nOur work is the first to apply Certaindex to build an end-to-end system (Dynasor) designed for optimizing token efficiency for LLM reasoning programs and reasoning models. Dynasor uses a system-level scheduler to optimize batch and online workloads with parallel decoding and dynamic token budgets allocation. This system design leads to substantial end-to-end latency and throughput improvements that prior approaches, such as Aggarwal et al., cannot achieve due to their sequential nature.\n\nTaken together, the generalizability of Certaindex, its theoretical guarantees, and its integration into a high-performance scheduler establish Dynasor as a comprehensive solution in efficiently scaling LLM reasoning, not just a repackaging of existing heuristics. \n\n\n## W1(b): Comparison to https://arxiv.org/abs/2305.11860 (Aggarwal et al.) . \n\nBesides previously mentioned, we emphasize that an empirical comparison with adaptive SC (Aggarwal et al., 2023) is not meaningful, given the inherent differences in their underlying execution paradigms.\n\n**(1) Sequential vs. Parallel Decoding in real-world deployment.**\nAggarwal et al.’s adaptive SC decodes one trajectory at a time, checking the stopping criterion after each generation. This sequential decoding results in latency proportional to the number of rounds × average tokens per round. In contrast, Dynasor exploits parallel decoding and dynamic token allocation, which significantly reduces latency in both batch and online workloads.\n\n**(2) Batch vs. Online Workload Performance.**\nWhile Aggarwal et al. can in theory reduce token counts in offline settings, it is not designed to optimize real-time throughput or latency. Dynasor’s scheduling explicitly addresses both token efficiency and online responsiveness, offering a better tradeoff between speed and accuracy.\n\n**(3) Conceptual Superset.**\nOur framework subsumes adaptive SC as a special case – in fact, an adaptive SC policy can be expressed as a particular instantiation of Dynasor with per-step thresholds. We provide a theoretical analysis of stopping criteria (Section 3.4) and explore multiple adaptive strategies (Figures 9–10), which covers the design space that adaptive SC inhabits.\n\nFor these reasons, we believe running experiments against adaptive SC would not provide additional insight and could be misleading in evaluating online systems. We will clarify the relationship in the revision and explain why Dynasor generalizes and improves upon adaptive SC.\n\n## W2: Related works\n\nWe will cite these works in the final version of the paper. \n\nhttps://arxiv.org/abs/2305.11860 - *included in discussion above, omit here due to limited input length*\n\nhttps://arxiv.org/abs/2311.09553 - Kabra et al. focus on Program-Aided reasoning: they invoke external tools during inference and measure how well the combined LLM-plus-tool pipeline is calibrated. Their contribution is a calibration study and metric suite; they do not define an early-exit strategy, a token-budgeting algorithm, or any scheduling mechanism. \n\nhttps://arxiv.org/abs/2411.04109  - Prasad et al. present SCPO, which leverages SC during training to generate preference data for RLHF. All compute happens offline instead of online.  \n\nhttps://arxiv.org/abs/2504.05419 - Zhang et al. (concurrent) learns a white-box linear probe on hidden states to predict correctness and exit early. Their method requires fine-tune the model and access to activations, and evaluated only on CoT. \n\nhttps://arxiv.org/abs/2504.15895 - Yang et al. (concurrent) propose a heuristic early-exit rule that monitors “wait” tokens in CoT and stops once a confidence threshold is met. The work provides neither theoretical guarantees nor a scheduler, and only target at CoT. \n\n\n# Questions\n\n## Q1: “Settling concept” is introduced in https://arxiv.org/abs/2504.07128. (Marjanović et al.)\n\nWe clarify that our contributions differ substantially from Marjanović et al. \n\n**Scope and Novelty**: Our contributions differ substantially from Marjanović et al. Their work is mainly a survey and analysis of Deepseek-R1 reasoning models, introducing a settling heuristic limited to that scope. In contrast, we show that this heuristic broadly exists in both reasoning models and non-reasoning models with reasoning programs (Section 3.2, Appendix C). We go further by developing a theoretical framework (Section 3.4, Appendix E) that rigorously bounds early exit without sacrificing accuracy – something not addressed by Marjanović et al. Beyond theory, our system-level contribution, Dynasor, extends this principle to real-world serving engines and delivers substantial efficiency gains on diverse workloads (GSM8K, AIME, LiveCodeBench), for both batch and online settings. In this sense, our work has a broader scope, a rigorous theoretical grounding, and system-level impact that clearly sets our work apart from Marjanović et al.\n\n\n**Timing and originality**: The work is arxiv-ed in May 2025, so the reviewer is correct that this work is concurrent to our work. \n\n## Q2: Comparing with S1 (https://arxiv.org/abs/2501.19393, Muennighoff et al.)\n\nWe respectfully disagree with the reviewer’s opinion.\n\nIn fact, reviewer's comment overlooks a critical distinction: **S1 requires fine-tuning** the model to obtain the ability to scaling, whereas **Dynasor works out-of-the-box** with any model and reasoning programs by exploiting intrinsic signals during the reasoning process. This makes Dynasor far more practical and widely applicable across models and workloads, avoiding the costly retraining step required by S1. Moreover, S1 does not address a central question that Dynasor tackles: when should the model stop thinking without sacrificing accuracy? This is a key contribution of our work, supported by both a theoretical framework and extensive empirical results.\n\nWe also disagree with the claim that “prompt in the middle” is a standard test-time scaling approach. To our knowledge, no closed-source models (e.g., OpenAI, Claude, Gemini) have publicly documented using this technique as a standard practice. We have also not found published works other than S1 that generalize this approach for test-time scaling without the need to fine-tune the model. Also note that this technique is limited to reasoning models and does not generalize to reasoning programs (e.g., Self-Consistency, Rebase, MCTS), which Dynasor supports but S1 does not.\n\nTaken together, our work does not require fine-tuning the model, provide theoretical grounding of the stopping criterion to save tokens without losing accuracy, and the generalizes to reasoning models and reasoning programs. Therefore, our contributions are substantially distinct from S1, both in goals and in method.\n\n\n## Q3: Comparison with CodeT (https://arxiv.org/abs/2207.10397)\n\nCodeT and our work address different goals and settings.\n\n**Summary of CodeT**: CodeT automatically generates test cases to verify solutions for coding tasks, checking (1) the consistency of outputs against these tests and (2) agreement with prior code samples.\n\n**Key differences**: \nCodeT’s “cross-consistency with unit test feedback” fundamentally differs from Certaindex. CodeT is limited to coding tasks with strong external verifiers, whereas Certaindex measures certainty through answer frequency and entropy in a purely black-box LLM setting. In this sense, Certaindex is applicable to both coding and non-coding reasoning tasks, making it more generalized than CodeT.\n\nMoreover, CodeT does not address inference-time efficiency. Its evaluation (Section 4.3) focuses on the tradeoff between sample count vs accuracy, but not on token budgets or latency vs accuracy. In contrast, our work targets inference-time efficiency, aiming to minimize token usage while preserving accuracy across arbitrary inference-time algorithms, which is a more relevant problem in real deployment setup (i.e. price-per-query that the user needs to pay for).\n\nWhile external rewards or verifiers are beyond this paper’s scope, Dynasor’s flexible interface (Section 3.3 & Appendix D.1) can incorporate such signals. CodeT’s approach could be implemented as a specific policy within Dynasor, but our focus is on generality and efficiency without relying on test execution. In that sense, we don’t think comparing with CodeT is necessary."}, {"review_id": "LlDXEtlIFY", "reviewer": "Reviewer_pntd", "summary": "This paper proposes using Certaindex to address the issue of redundant tokens in large model reasoning tasks. The motivation is clear, and both experimental and theoretical analyses demonstrate that Certaindex can effectively eliminate redundant reasoning branches. Moreover, the approach is supported by a well-developed reasoning framework, Dynasor, built on SGLang, which significantly improves the efficiency of online batch inference systems.", "questions": "1. While thresholding-based allocation is acceptable, could it become difficult to manage as different data distributions and reasoning flows vary across tasks? If so, how is this issue addressed in the current design?\n\n2. The “Probe-In-The-Middle” analysis in Figure 2 is insightful, but the visualization itself is not very intuitive. Is there a more straightforward or visually clear way to present this idea?\n\n3. As mentioned in the weaknesses, can Certaindex reliably handle all types of reasoning paths when judging confidence? Is there a risk that it might interfere with certain reasoning strategies that the model would otherwise naturally follow?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper introduces Certaindex as a novel metric to assess the stability of intermediate reasoning outputs, aiming to determine whether the model has already converged on a correct answer. This is a timely and meaningful contribution, as reducing token usage during inference without compromising answer quality is a pressing challenge in LLM deployment.\n\n2. The authors provide a complete and reusable implementation of their method through the Dynasor framework, built on SGlang. This not only validates the practicality of their approach but also offers valuable tooling for the research community, promoting reproducibility and further exploration.\n\n3. The paper presents a well-rounded analysis, combining theoretical insights with empirical evaluations across multiple reasoning paradigms (e.g., Chain-of-Thought, Self-Consistency, MCTS, and REBASE). This strengthens the case for Certaindex as a general-purpose signal for early stopping in LLM reasoning.\n\n**Weaknesses:**\n\n1. While the method effectively reduces redundant tokens in high-confidence scenarios, it may also prematurely terminate reasoning when the model has not yet arrived at a correct answer. This could limit the model’s opportunity for self-correction and increase the risk of incorrect outputs. The paper would benefit from a more detailed discussion of such failure cases and how Certaindex behaves under different reasoning dynamics.\n\n2. The evaluation is primarily focused on logic-intensive tasks such as mathematical reasoning, where the model’s confidence is easier to quantify. However, it remains unclear how well Certaindex generalizes to broader domains, such as open-ended or commonsense questions, where the model’s internal certainty may not align with actual correctness.", "was_revised": false, "final_justification": "", "rebuttal": "Respond to pntd\n\nWe thank Reviewer pntd for the thoughtful review and the positive feedback on Certaindex, Dynasor, and the breadth of our evaluation. We address the concerns and questions raised below.\n# Weaknesses\n## W1: Risk of premature termination and failure cases.\nWe acknowledged in the paper that early stopping introduces an accuracy-latency trade-off (Figure 8 row 3): if the threshold is set too aggressively, the model may terminate reasoning before self-correcting. Our theoretical analysis (Section 3.4 & Appendix E) specifically addresses this by providing a theoretical framework with a bound on the minimum number of reasoning steps required to preserve correctness with high probability. Moreover, our ablation on different thresholding methods (Figures 9 - 10) empirically explores this trade-off, demonstrating how different thresholds affect both accuracy and token savings. We will add an explicit discussion of failure cases in the revision, highlighting workload-specific behaviors and practical tuning strategies.\n\n## W2: Generalization beyond math/logical reasoning.\nWe acknowledge that our evaluation does not cover open-ended or commonsense questions. Our experiments focus on math and logic-heavy benchmarks because these tasks provide objective and easily verifiable correctness signals, which are critical for validating Certaindex. That said, Certaindex is inherently domain-agnostic, as it measures the convergence of reasoning patterns rather than relying on task-specific heuristics.\nWhile we have not included experiments on open-ended tasks, Dynasor’s framework can be readily extended to such domains, for example by incorporating an LLM-as-a-judge signal or other external rewards to approximate correctness. We will clarify this in the revision and discuss how threshold tuning can adapt to different workloads.\n\n\n# Questions\n## Q1: Managing thresholds across tasks.\nIt is true that the optimal threshold varies with data distribution and reasoning dynamics. Dynasor is designed for this flexibility: in practice, a service provider can collect statistics on model outputs for a given domain and tune thresholds using a small calibration set (similar to how temperature scaling is used for confidence calibration). We will clarify this point in the revision.\n## Q2: Visualization of “Probe-In-The-Middle” (Figure 2).\nWe appreciate the suggestion. While we aimed for completeness, we agree that the current visualization may appear dense. In a revised version, we will include a simplified version of the plot that highlights key trends (e.g., stability convergence) more clearly.\n## Q3: Applicability to all reasoning paths.\nCertaindex is not tied to a single reasoning strategy. By tracking stability signals across any sequence of intermediate steps (e.g., CoT, SC, MCTS), it acts as a lightweight meta-controller rather than interfering with reasoning. Our evaluations (Sections 4) show that Certaindex complements existing reasoning flows without reducing their inherent flexibility."}, {"review_id": "Fgu2LWXHBW", "reviewer": "Reviewer_nxvt", "summary": "This paper proposes Certaindex, a lightweight, algorithm-agnostic metric designed to measure the stability of intermediate answers in LLM reasoning tasks. The authors observe that many test-time reasoning methods (e.g., CoT, MCTS) generate excessive tokens even after reaching a stable answer. To address this inefficiency, they introduce a probing technique (\"Probe-in-the-Middle\") that periodically prompts the model to give an intermediate answer. If these answers stabilize, the model likely will not change its final answer, allowing for early termination of generation. They generalize this insight into Certaindex, which can be used across multiple reasoning algorithms and integrated into a scheduling system (Dynasor) to dynamically allocate compute.", "questions": "1. Probe Prompt Design: Have you tried alternative, less dramatic probe prompts, such as \"Based on the reasoning so far, the answer is: boxed\"? If so, how sensitive is performance to this choice?\n\n2. Information Completeness: Do you have quantitative or intuitive analysis on whether the truncated CoT contains all the information necessary to output the correct final answer? Are there counterexamples where early answers are stable but ultimately incorrect due to missing later reasoning steps?\n\n3. Incorrect Stabilization: Could you provide more insight into how often the model stabilizes to an incorrect answer, and whether Certaindex can differentiate between \"correct and stable\" vs. \"wrong and stable\" situations?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper identifies and addresses a compelling inefficiency in modern LLM reasoning—token overuse even when the answer is already internally determined, especially in CoT and similar paradigms.\n\n- Across multiple reasoning methods and datasets, the proposed method achieves substantial token savings and throughput improvements without hurting accuracy.\n\n- The implementation of Dynasor within an LLM serving system (SGLang) with minimal code change shows the practical viability of the approach.\n\nWeaknesses:\n\n- The probe prompt (\"Oh, I suddenly got the answer to the whole problem, Final Answer: boxed\") seems overly dramatic and could potentially bias the model's behavior. The paper claims the exact phrasing is not critical, but does not provide quantitative or ablation studies on alternative, less stylized prompts.\n\n- Certainty metric is primarily based on answer repetition: While effective, it may not capture deeper uncertainty or error modes beyond output consistency, especially when models are confidently wrong. The authors attempt to filter out hesitations (e.g., “wait”), but this is still heuristic.\n\n- Limited analysis on incorrect stabilization: While the paper emphasizes when to stop generating because the answer is stable, it does not deeply discuss when this leads to confidently wrong outputs and how often that happens, which could impact the overall utility of early stopping.\n\n- Truncation fidelity: The underlying assumption is that truncated CoTs still contain all necessary information to reach the correct answer. While there's theoretical justification and empirical support, this aspect could benefit from more intuitive or diagnostic discussion.", "was_revised": false, "final_justification": "", "rebuttal": "Respond to nxvt\n\nThank you for your thoughtful review and the encouraging feedback on Certaindex and Dynasor. Below, we address your questions in detail.\n\n# Questions\n\n## Q1 Probe Prompt Design\n\nThank you for this important question about probe prompt sensitivity. We conducted comprehensive ablation studies using three different probe prompts to evaluate their impact on performance and token efficiency across 10 problem traces:\n- P1: \"... Oh, I suddenly got the answer to the whole problem, Final Answer\\n\\n\\[ \\boxed{\"\n- P2: \"Based on the reasoning so far, the answer is: \\boxed{\"\n- P3: \"Final Answer\\n\\n\\[ \\boxed{\"\n\n### Results Summary\n\n**Lower Target Accuracy Scenarios**\n\n| Model + Dataset | Early Exit (P1)    | Early Exit (P2)    | Early Exit (P3)    |\n|-----------------|--------------------|--------------------|--------------------|\n| AIME 8B         | 0.49 / 30.4%       | 0.49 / 32.2%       | 0.49 / 34.9%       |\n| AMC 8B          | 0.88 / 28.0%       | 0.88 / 25.1%       | 0.80 / 33.6%       |\n| AIME 32B        | 0.69 / 22.4%       | 0.69 / 17.8%       | 0.69 / 16.9%       |\n| AMC 32B         | 0.89 / 36.6%       | 0.90 / 36.9%       | 0.92 / 32.7%       |\n\n**Higher Target Accuracy Scenarios**\n\n| Model + Dataset | Early Exit (P1)    | Early Exit (P2)    | Early Exit (P3)    |\n|-----------------|--------------------|--------------------|--------------------|\n| AIME 8B         | 0.52 / 15.6%       | 0.52 / 4.0%        | 0.52 / 27.7%       |\n| AMC 8B          | 0.91 / 19.4%       | 0.90 / 14.5%       | 0.83 / 30.9%       |\n| AIME 32B        | 0.71 / 13.9%       | 0.71 / 14.4%       | 0.71 / 11.4%       |\n| AMC 32B         | 0.96 / 22.5%       | 0.96 / 19.8%       | 0.96 / 24.9%       |\n\n*Format: Accuracy / Token Saving %*\n\nThe results demonstrate that different prompt styles achieve similar token savings at comparable accuracy levels. For instance, on AMC-32B at 96% accuracy, all three prompts save 20-25% tokens (P1: 22.5%, P2: 19.8%, P3: 24.9%). Similarly, on AIME-32B at 71% accuracy, the token savings range from 11-14% across all three prompts (P1: 13.9%, P2: 14.4%, P3: 11.4%). This shows the method's effectiveness is robust to prompt phrasing, with consistent performance regardless of the specific guidance style used.\n\n\n\n## Q2 Information Completeness and Truncation Fidelity (weakness 4 & question 2)\n\nYes, our method does encounter cases where early answers are stable but ultimately incorrect due to missing later reasoning steps. This is precisely why we implement adjustable confidence thresholds.\n\nOur quantitative analysis reveals a clear relationship between confidence thresholds and information completeness. The analysis shows that when confidence thresholds are more aggressive, there's significant information loss, but as thresholds become stricter, the Right→Wrong rate (cases where standard method was correct but early exit was wrong) decreases dramatically.\n\n**AMC 32B Results (Standard Baseline: 96.5% accuracy):**\n\n| Accuracy | Token Saving % | Right→Wrong % |\n|----------|----------------|---------------|\n| 0.8900   | 36.6%          | 8.2%          |\n| 0.9625   | 22.5%          | 1.0%          |\n| 0.9725   | 16.1%          | 0.0%          |\n\n**AIME 32B Results (Standard Baseline: 73.0% accuracy):**\n\n| Accuracy | Token Saving % | Right→Wrong % |\n|----------|----------------|---------------|\n| 0.6200   | 39.8%          | 11.7%         |\n| 0.6933   | 22.4%          | 4.7%          |\n| 0.7200   | 9.7%          | 1.7%          |\n\n\nThe data demonstrates that aggressive early exits show substantial Right→Wrong rates of 8.2% and 11.7% for AMC and AIME respectively, indicating significant information loss. However, as confidence thresholds become stricter, this drops dramatically - from 8.2% to 1.0% to 0.0% for AMC, and from 11.7% to 4.7% to 2.7% for AIME. With sufficiently strict thresholds, the Right→Wrong rate becomes negligible, indicating that when the confidence threshold is high enough, the truncated reasoning contains nearly all necessary information for correct answers. This validates that our adjustable confidence mechanism effectively controls the trade-off between efficiency and reasoning completeness.\n\n## Q3 Incorrect Stabilization\n\nOur method does encounter incorrect stabilization, where the model converges to a wrong answer with high confidence. However, our analysis shows that with sufficiently strict confidence thresholds, our approach largely preserves the model's original reasoning capability while primarily serving as a token-saving mechanism.\n\nThe key insight is that when confidence thresholds are strict enough, our method essentially maintains the model's inherent accuracy - if the model would originally get an answer right, our method keeps it right; if the model would originally get it wrong, our method doesn't fix that underlying limitation.\n\n**AMC 32B Results (Standard Baseline: 96.5% accuracy):**\n\n| Early Exit Accuracy | Token Saving % | Wrong→Right % | Right→Wrong % |\n|---------------------|----------------|---------------|---------------|\n| 0.8900              | 36.6%          | 0.8%          | 8.2%          |\n| 0.9625              | 22.5%          | 0.8%          | 1.0%          |\n| 0.9725              | 16.1%          | 0.8%          | 0.0%          |\n\n\n**AIME 32B Results (Standard Baseline: 73.0% accuracy):**\n\n| Early Exit Accuracy | Token Saving % | Wrong→Right % | Right→Wrong % |\n|---------------------|----------------|---------------|---------------|\n| 0.6200              | 39.8%          | 0.7%          | 11.7%         |\n| 0.6933              | 22.4%          | 1.0%          | 4.7%          |\n| 0.7200              | 9.7%           | 0.7%          | 1.7%          |\n\n\nWith conservative thresholds, the Right→Wrong rate drops to nearly zero (0.0% for AMC, 1.7% for AIME), while Wrong→Right remains minimal but consistent (~0.7-1.0%). This demonstrates that our method primarily acts as an efficiency optimization rather than an accuracy enhancer. The small Wrong→Right improvements likely represent cases where early termination prevented overthinking, but this effect is marginal. Essentially, when thresholds are strict enough, we preserve the model's original problem-solving trajectory while achieving substantial token savings (9.7-16.1% even with conservative settings).\n\n## Q4 (Weakness 2) Repetition-based Certainty Metric\n\n> Weakness 2: Certainty metric is primarily based on answer repetition: While effective, it may not capture deeper uncertainty or error modes beyond output consistency, especially when models are confidently wrong. The authors attempt to filter out hesitations (e.g., “wait”), but this is still heuristic.\n\nSurprisingly, early termination of a confidently wrong answer (request) is a feature, not a weakness when we apply Certaindex into real LLM inference engine. In the real-world deployment, a service provider wants to reduce the token expenditure to gain profit for completing a request correctly. If the model does not have the ability to answer the problem correctly, early terminating this request will help save tokens (thus costs). In our case, if a model makes a consistent answer to the question, even if it is wrong, it provides a strong signal that the model will not correct the answer in later iterations. Thus, early termination will benefit us from saving more tokens. Our method does not aim to change (increase) the model's accuracy - rather, we preserve the same accuracy distribution while achieving significant computational savings through early termination. The key value proposition is computational efficiency without sacrificing the model's inherent performance characteristics.\n\nRepetition-based certainty metric. We acknowledge this limitation of our repetition-based certainty metric. We note that our framework is flexible and allows for user-defined certainty metrics. In our experiments, we have also explored alternative approaches such as using reward model scores to define certainty (as shown in Section 3.1). This reward model-based approach can partially address the uncertainty and error modes beyond output consistency."}]

Author rebuttal:

Meta-review: This paper addresses the problem of excessively long latent reasoning chains in LLMs prompted to answer problems requiring multi-step thinking. A heuristic method is proposed to detect whether reasoning can be terminated early by injecting a prompt that probes for a guessed answer at intermediate steps and tracking the consistency of these guesses. The method is integrated into a dynamic compute allocation framework, which delivers a reduction in total compute cost when amortised over multiple queries under a fixed budget. Experiments on maths benchmarks show that this approach does not harm accuracy while significantly reducing average response length.

Strengths of the paper:
- Good motivation for and presentation of the algorithm.
- Strong empirical results and a system that makes the method readily deployable.
- Thorough empirical evaluation and analysis.

Main weaknesses:
- Comparison with related work (Jsvc). In the rebuttal, the authors provided an extensive related work discussion, which satisfied the reviewer who raised this concern, and this should be included in the revision.
- Scope of the tasks evaluated on is somewhat narrow (pntd). The authors agreed that the method is developed with a focus on maths / "logic-heavy" tasks. The reviewer who raised this concern was satisfied with the authors' responses.
- Some of the design choices and motivations could be better justified and failure modes better analysed (pntd, nxvt). This was answered in the rebuttal with additional discussion and experiments, which should be included in the revision.

Overall, the paper is just above the bar for acceptance, and the authors are encouraged to include the additional discussion and experiments from the rebuttal in the final version.
Decision: Accept (poster)