Zero-Overhead Introspection for Adaptive Test-Time Compute
Authors: Rohin Manvi
Joey Hong
Tim Seyde
Maxime Labonne
Mathias Lechner
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Test-time compute
Value Function
Sampling
Primary area: generative models

Abstract: Large language models excel at reasoning but lack key aspects of introspection, including the ability to anticipate their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when to signal success or failure. Without this ability, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods such as Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation, and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthiness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial inference cost by requiring extra models or forward passes. We present ZIP-RC, which equips models with zero-overhead introspective predictions of reward and cost. At every token during generation, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remaining length—no extra models, architecture change, or inference overhead. This full joint distribution is used to compute a sampling utility, which is the linear combination of the expected maximum reward, total compute, and latency of a set of samples if generated to completion. During inference, we maximize this utility with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficulty mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By providing real-time reward-cost introspection, ZIP-RC allows models to reason adaptively and more efficiently.

Published as a conference paper at ICLR 2026
ZERO-OVERHEAD INTROSPECTION FOR ADAPTIVE TEST- TIME COMPUTE
Rohin Manvi
Joey Hongt
Tim Seyde 18
Maxime Labonne
Mathias Lechner
Sergey Levinet
ABSTRACT
Large language models excel at reasoning but lack key aspects of introspection. including the ability to anticipate their own success and the computation required to achieve it. Humans use real-time introspection to decide how much effort to invest, when to make multiple attempts, when to stop, and when signal success or failure. Without this ability, LLMs struggle to make intelligent meta-cognition decisions. Test-time scaling methods such Best-of-N drive up cost and latency by using a fixed budget of samples regardless of the marginal benefit of each one at any point in generation. and the absence of confidence signals can mislead people, prevent appropriate escalation to better tools, and undermine trustworthi- ness. Learned verifiers or reward models can provide confidence estimates, but do not enable adaptive inference and add substantial inference cost by requiring extra models or forward passes. We present ZIP-RC, which equips models with zero-overhead introspective predictions of reward and cost. At every token during generation, ZIP-RC reuses reserved or unused logits in the same forward pass as next-token prediction to output a joint distribution over final reward and remain- ing length-no extra models, architecture change, or inference overhead. This full joint distribution is used compute sampling utility which is the linear combination of the expected maximum reward, total compute, and latency of set of samples if generated completion. During inference, we maximize this util- ity with meta-actions that determine which prefix of tokens to continue or initiate sampling from. On mixed-difficltyy mathematical benchmarks, ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost, and traces smooth Pareto frontiers between quality, compute, and latency. By provid- ing real-time reward-cost introspection, ZIP-RC allows models to reason adap- tively and more efficiently.
INTRODUCTION
The rapid evolution of large language models (LLMs) has enabled unprecedented capabilities in complex tasks ranging from general question-answering to automated coding and mathematical rea- soning (Brown et al. 2020 Kojima et al., 2022, Wei et al. 2022). To become truly reliable, however, LLMs must develop capacity for introspection: the ability to assess their own progress and antic- ipate the effort required to succeed. Humans can be introspective and can effectively act upon this information make better decisions. If model could predict its future success (reward) and the sources needed to achieve it (cost), it could allocate compute more effectively, expose likely failure modes before they occur, and provide transparent signals about confidence and anticipated "thinking time." A key obstacle has been that such introspection typically requires auxiliary mechanisms that add nontrivial computational overhead and complexity.
The need for introspection is growing more urgent reasoning traces continue to lengthen. Recent work shows that scaling test-time compute through reasoning often yields larger performance gains
"Conesponding author, rchi rchinn@@eereeeee..ee
UC Berkeley
I MITT CSAIL
1 Liquid AT

---

Published as a conference paper at ICLR 2026
Repurposes Unused or Reserved Logits
Real-Time Expacted Reward and Remainiing Length Predictions
P/Cernedi: High
Inclace
Lcaem in Like larettin
ZIP-RC
PiCumreti: Medium Tokers keft 4-8k
0
PCComent:: Low
Expected Reward
Tokene left:
Mapping
ZP Logis (e.g. 66 reserved unused sckeni))
PrComeet:: Low Tokers
K0g D IB (eed 1S8 (itgare set tite inacleet (nabee
ii Stavee SS (Si laree
Slice for Auxillary Prediction
PcComeeti: Medium
(thore
Culput Huad
P/Commeti High
Tomber
Language Modal
Camreh
Adaptive Parallel Test-time Compute that Optimizes Performance, Compuna and Latency
3t+
20-
Low Varlance 344
High Varianco
13
Lower Expected
10-
Higher Expected
46
Higher Cost
Lower
ZIP-RC
8
2PP-RC
2-
2IP-RC
ZIP-RC
One Sample
Sampling:
Enough
532
18
Sample More (Branch)
Sampling:
14
(Tentattvve Prune)
Sampling:
51
Sampling:
Sampling
Easy Promot
State: token prefix (partial generation)
Meta-state: prefix tree of all states Meta-action: multiset of prefixes to extend
ZIP-RC Sampling selects Meta-actions maximizing
Sampling Utility
E(Reward]
E[Compute]
E[Latencyl
Top left shows how ZIP repurposes reserved or unused logits in the output head of a lan- guage model to instantiate auxiliary predictions, such as the grid mapping for the joint reward-cost distribution that ZIP-RC uses. Top right demonstrates how 7IP-RC can provide real-time expected reward and remaining length predictions. Finally, the bottom shows the joint distributions from ZIP-RC and how they indicate optimal sampling strategies. ZIP-RC sampling uses these joint dis- tributions to calculate a sampling utility to autonomously select meta-actions for optimal test-time compute allocation.
Figure I:
than simply increasing model size (Wang et al. 2023b Yao et al. 2023 Jaech et al., 2024 Snell eet al., [222]] Guo al.) (025).. But performance has scaled only logartthmically with additional computation, forcing models to produce ever longer chains of thought-sometinng sometimes tens of thousands of tokens today and plausibly orders of magnitude more in the future (Wu et al., 2002)). With time as a fundamental limiting resource, a critical question is how to use a fixed wall-clock budget to achieve the highest performance possible.
A promising approach is the canonical test-time scaling method Best-of-N (BoN) sampling, which generates N candidates and selects the best using learned verifier, reward model, or majority vote (Cobbe et al. 2021 Zheng et al. 2023 Kwon et al. 2023 Lightman et al. 20236 Wang et al. 2023b). While appealing theory due to its purallelism, BoN not adaptive: every trajectory is carried to completion regardless of promise. On easy tasks this wastes computation, and on hard tasks inflates latency, since wall-clock time governed by the longest generation and both length and total compute grow with N (Leviathan et al. 2023). What is missing is way for models to anticipate which samples are worth continuing and which should be paused or abandoned, so that parallel effort is concentrated trajectories most likely succeed and fastest complete.
strly-stopping and pruning methods aim to reduce BoN's inefficiency by terminating unpromising samples mid-gen generation (Fu et al.) 2025 Huang al. 2025). These approaches valuable first steps toward adaptivity, but they typically rely on scalar signals- such as a confidence score from classifier-on on simple heuristics. This creates two limitations. First, scalar cannot capture
2

---

Published as a conference paper at ICLR 2026
the central reward-com trade-off: a low-confidence trajectory may be worthwhile if nearly finished, while a bigh-confidence one may be impractical if implies a long, costly continuation. Second, these methods do not quantify the marginal benefit of drawing more samples, which depends on the entire reward distribution rather than its expectation. As a result, such strategies can reduce compute in some cases but often fail improve wall-clock time, falling short of the broader goal of enabling models to allocate compute adaptively- eeaptively-exeedinng more effort on difficult queries and less on easy ones (Manvi al. 2024 Graves 2016).
We introduce ZIP-RC, which addresses these limitations by training language models to provide zero-overhead introspective predictions of the joint distribution over reward and cost. At each de- coding step, unused vocabulary logits parameterize a joint distribution over final reward and remain- ing generation length (see fig. [. Access to the full joint-not just a scalar- order-statistic calculations that quantify the marginal utility of continuing partial samples or spawning additional samples. For example, when the predicted reward distribution has high variance, allocating more samples can substantially increase the expected maximum reward. We maximize sampling utility that explicitly balances accuracy, compute, and latency through a linear combination of their expec- tations. The coefficients of the linear combination can be tuned to the desired balance of reward, compute, and latency. Optimizing this utility produces the behaviors observed in our experiments: when lateney is prioritized, ZIP-RC spawns larger pools of samples and schedules early pruning to chase an early finisher; when compute is prioritized, it deprioritizes low-value trajectories aggres- sively and allocates more samples only when they are likely to pay off.
Experiments on mixed-difficulty mathematical benchmarks show that ZIP-RC improves accuracy by up to 12% over majority voting while using less average cost. By adjusting the utility coefficients, it traces smooth Pareto frontiers between accuracy, compute, and latency. We contribute ZIP and ZIP-RC which enable models be introspective for more interpretable generations and maximize a sampling utility to improve performance with fixed compute and latency.
2 RELATED WORK
Improving the efficiency and reliability of LLM reasoning requires both new methods for guiding generation and principled strategies for allocating computational resources at inference time. Our work builds on three key areas of research: the use of verifiers for response selection, process-level rewards for fine- -rei-graied feedback, and adaptive inference strategies for efficient computation.
Verifiers and reward models for output selection. A common approach to enhancing LLM per- formance is to train an external verifier or reward model (RM) to assess the quality of complete responses. Such models provide outcome- based feedback, typically assigning a scalar score or probability of correctness to an entire output sequence. Outcome RMs have been widely used in reasoning and alignment works, from math problem solving to preference-based fine-tuning (Cobbe) et al. 202T, Yu et al., 2023 Stiennon et al. 2020). They can be integrated during training, as in reinforcement learning settings (Ouyang et al. 2022 Bai al. (022). or applied at inference time through selection strategies such as Best-of-N sampling (Cobbe et al.] 2021 Li et al. 2022). Re- cent work has explored unifying the generator and verifier, using the model's 's logits for certain tokens as a proxy for a reward model (Ren et al. 2023). Our work extends this introspective direc- tion, moving beyond scalar correctness prediction modeling a joint distribution over the expected future reward and computational cost at every token.
Process-based rewards for fine-grained feedback. A limitation of sutcome-supervision is its reliance on a sparse reward signal that makes credit assignment challenging, especially for long rea- soning chains. Process- based reward models (PRMs) instead score intermediate steps via human annotation (Lightman et al., 2023b). LLM-as-judge (Zheng et al.) 2023). or automated token-level value estimates. These automated estimates can be generated by propagating final outcome rewards back to individual tokens (Liu et al., 2024) or through other value estimation techniques (Uesato cet al. 2022 Luo et al., 2024). While most PRMs aim to improve the training signal, our goal is dis- tinct: we use predictive feedback in real time to guide inference itself. Closest to the calibration side of this literature, Damani et al. (2025) augment a binary correctness reward with a confidence score to improve model calibration. Our approach is complementary: rather than training for calibrated confidence, we predict a joint distribution over future reward and future cost, turning process-level signals into direct control knob for utility-aware inference.
3

---

Published as a conference paper at ICLR 2026
Adaptive inference and introspective models. Our work enables a form of adaptive inference,
long-standing goal in machine learning (Graves 2016; Bengio et al., 2015) that has become increas- multiple models or sequential sampling have been explored (Damani et al.] 2024 Wang et al. 2024). A more recent direction has involved parallel sampling that includes the pruning of unpromising generation paths.
ingly critical for large models (Snell al.] 20024) Adaptive methods that
confidence scores(Manvi 2024 Fu et al., 2025) or prune exploration based on step-wise consistency checks (Aggarwal et al. 2023 We advance this line of work with more general formulation: instead of relying on simple heuristics for pruning. we use our joint reward-cost predictions to explicitly optimize utility function. This enables a richer set of meta-actions, such as dynamically resizing the sample pool and reallocating budget across trajectories. Conceptually, our approach parallels the integration of value functions with search in reinforcement learning (Silver et al.) (016]. where predictive signals guide exploration. It is also complementary to inference optimization techniques like speculative decoding (Leviathan et al. 2023 which accelerate generation at the token level. By providing real-time estimates of success and cost, the predictions from ZIP-RC contribute to a broader vision of introspective models that report their internal states (Binder et al. 2024 Kadavath et al. 2022 enhancing efficiency and interpretability.
For instance, recent methods terminate samples based on mid-generatio
al.
3 PRELIMINARIES
Generation as token-level MDP. We formalize text generation as finite-horizon Markov De- cision Process (MDP), following Ramamurthy et al. (2022). The MDP is defined by the tuple M AS, R. y. H) over finite vocabulary V. where S is the state space, and A the ac- tion space, R the reward function, P the transition function, Y E [0. 1] the discount factor, and H the horizon. Given an input prompt x (Zg...... ...,..,, Fm,) consisting of tokens in the vocabulary J; E V, the initial state is 80 x. At timestep L, the LLM acts policy <a (&x that outputs the probability distribution over actions a, E V. The transition function P deterministically appends a to state 8, yielding next state Sc+1 (Zo. ..... Im, do..... Qt). The episode terminates when the model emits an -ed-un-eeueence token <EOS> or the length of the generated sequence reaches the horizon H. Upon termination at timestep T, the environment returns terminal reward R(sT) The discount factor defined as 7 = and the value of any state st under policy π is the expected terminal reward from that state onward Vs{)) E, [R(ST) st].
Best-of-N. Best-of-N (BoN) is an inference-time selection mechanism that decouples generation from evaluation to improve output quality. Given a prompt x and a generator policy the method
(1) draws N independent and identically distributed (i.id.) terminated states ST ..... ST
(N) from the
policy. A learned verifier V V* + R. typically a reward model, then assigns a scalar score to each terminated state. The final output is the state with the highest score, selected as
s/. E arg max (S (9)
(I)
CE[N]
The selection depends only on the relative ordering of scores from V-),, ties are broken arbitrarily.
ZERO-OVERHEAD INTROSPECTIVE PREDICTION OF REWARD & COST
We introduce Zero-overhead Introspective Prediction (ZIP), method for extracting auxiliary signals during inference without extra models, architectural changes, or forward passes. ZIP repurposes the logits of a small set of reserved tokens to parameterize these auxiliary predictions within the same forward pass that generates the next-token probabilities. We then instantiate 7IP for reward and cost prediction (ZIP-RC).
Zero-overhead introspective prediction (ZIP). Let V be the vocabulary and R. C V a fixed contiguous set of reserved tokens. At decoding step t, the model produces logits 2 E RM ZIP interprets logits over R as parameters of an auxiliary predictor (c.g. via a softmax). A rough visualization of this is shown in the top right of fig. T Before sampling, these logits masked to
4

---

Published as a conference paper at ICLR 2026
remove probability mass:
exp(zi(a.)) E EvERiRe exp exp (2+(u])) de E VV R,
aa(a | 8t)
(2)
at E R.
Thus, each forward pass yields both (i) the decoding distribution on V \ R and (ii) auxiliary predic- tions from 24 [R]. incurring zero additional cost at inference time.
During training. we supervise the auxiliary head via a task-specific loss L applied to 2, [ (e.g., cross-entropy for categorical targets, Bernoulli NLL for binary targets, MSE for continuous targets), while regularizing the policy toward a frozen copy of the original policy .:
((x+) = L (s+) + o'K.. (T (x) || .(- [8)))
ZIP is agmostic to the prediction target loss, it simply standardizes how auxiliary predictions produced during inference, with zero inference overhead. An alternative that keeps the model frozen discussed in section A.5
ZIP-RC: joint reward-cost distribution prediction. We use ZIP to predict a joint distribution over the (expected) reward and remaining length of rollout using æ starting from any prefix 8t- We define the random variables
= [R(s))]
=
(= = |sr| [se),
for the expected terminal reward and length. In practice, approximate the expected terminal reward with (ST) from a learned verifier. We discretize the range of values into By bins with boundaries {vo) By++ and lengths into Br bins with boundaries By+1 assigning one reserved token per (b, é) using index in the output vocabulary V given by is,e iR + (6 - B) + (e- -),, where iR the index of the first reserved token. Let =!tx (b, E) 2 [r,,). The joint distribution is exp(= (b, ())
po é st) E8 Eez exp( (fxxxxx ())
rough visual representation of the grid mapping and examples of this learned distribution is shown in fig. Given a completed trajectory sr, for each timestep we construct training targets for each prefix St by computing (b", (*) such that
((r) [ub., 1b++1), |sr| - |sel [te., t2.
Finally, we train with cross-entropy Laux(st) = - log (b e st), together with the policy- preserving KL above. Other practical implementation details are discussed in section
Why expected reward instead of realized reward? It may initially seem unnatural to use an es- timated value ((T) by trained critic rather than the realized reward. To explain this choice, let (1) (N) Lid. (T(. 8o) be completions for prompt, and V the estimated value function used in BoN selection. The chosen index st arg max, ((s(") yields score ( max (ss (0) ). Modeling the distribution of possible terminal values V*(sT) via (sr) rather than possible ter- minal rewards R(ST): (i) aligns with the actual selection objective, and (ii) admits closed-form order-statistic expectations since noisy environment rewards from R(ST) cannot be assumed to be independent but its expectation ((ST) can.
ST
ZIP-RC for sample selection and interpretability. also compute individual marginal distributions:
Using the learned joint distribution, we can
By
By
o6 (bl
8t)
Σ
po(b, st).
( 8t) = Σ (b (b, st),
which can be used to estimate the value and the expected remaining tokens to completion
By
Br
(*)) =(Z
2b ub+1
qø (b 8c),
4e-1
(e st).
2
(8)
5

---

Published as a conference paper at ICLR 2026
Here, the value estimation can be used for final sample selection and both the value and the expected remaining tokens to completion act as confidence and "thinking time" signals. A rough visualization of the interpretable signals is shown in the top right of fig.
TEST-TIME COMPUTE USING ZIP-RC (ZIP-RC SAMPLING)
While large language models are post-trained to maximize the likelihood of high-reward genera- tions, they remain imperfect policies due to finite data and compute. Low-reward completions are often sampled even when their deficiencies are apparent-eiteee or explicitly via external reward models. Even greedy decoding (temperature 0) does not guar- antee high-likelihood or high-reward outputs. Thus, one-shot sampling is insufficient for reliably accomplishing tasks. Test-time methods such as majority voting, BoN, Weighted BoN, and Pass @k show the alternative: by actively searching across multiple trajectories, they substantially outperform sinlle-saple decoding. The performance gap highlights that the gain comes from active search.
implicitly through low likelihood
Existing test-time methods, however, are heuristic and often inefficient. BoN can, in principle, explore as much all other approaches with large enough N, but this is impractical given compute and latency constraints. While more sophisticated search strategies like beam search try to explore more efficiently by allowing for intermediate branching and pruning at intervals, one could imagine removing constraints on search further. Though the goal of test-time search clear-maximize task success while minimizing generation cost post-prior methods do not achieve this in a principled way. Our goal is to propose method that does. In this section, we introduce ZIP-RC sampling, which leverages predictions from ZIP-RC to explicitly optimize generations for both success and cost. We provide a high-level overview of the framework here (visually summarized in fig. I and provide the full formalisms and derivations in section A. and section A.2
Test-time compute as Decision-Making Under Meta-MDP. We formalize the problem of test- time compute as decision-making under a high-level meta-MDP, detailed in section [A.1 The state of this MDP is the current prefix tree (the set of all partial generations). At each step, the "meta- action" determines which prefixes in the tree to extend or branch from. Prefixes that are not selected are effectively paused rather than discarded. The objective is to maximize a meta-reward defined as the final correctness of the best answer minus the generation cost incurred. Crucially, this cost function includes both total compute (sum of tokens generated) and latency (depth of the longest trajectory). balanced by coefficients a and B. This formalism allows us to treat inference not as static procedure, but as a dynamic resource allocation problem.
The Sampling Utility. Solving for the optimal policy in this meta-MDP intractable. Instead, we approximate the optimal value function using a quantity we call the sampling utility. As derived in section A.2 the sampling utility estimates the value of a specific, interpretable strategy: performing rollouts from the current set of candidates, but with the capability to pause them at optimized future horizons. Maximizing this utility allows the controller to explicitly balance the marginal benefit of adding more samples (higher probability of finding high-reward answer) against the marginal cost of computation and time. This sampling utility can computed tractably using the joint predictions described in Section 4 Because ZIP-RC predicts the joint distribution of reward and remaining length, we can compute required order statistics-sude as the expected maximum reward of a set of samples or the expected latency given specific pausing schedule-in in closed form. Note this requires lightweight CPU-based calculations that are negligible compared to the forward pass of an LLM.
Sampling Loop At inference time, ZIP-RC sampling operates as a meta-policy. At regular inter- vals, it evaluates the sampling utility of various candidate meta-actions (c.g., pausing weak samples, branching strong ones, continuing the current set). It selects the action that maximizes this util- ity (as visualized in the bottom panel of fig. D and executes it for the next decoding steps. This allows the model to adapt online: if trajectories the system redirects computation elsewhere. We discuss practical implementation details, such as normalizing cost coefficients and reducing the search space, in section A.4
projected to be low-value excessively costly,

---

Published as a conference paper at ICLR 2026
Serser
3000 503
W00
3030 3000
3000
Figure 2: Predictions and ground truth for the initial joint distributions of 10 questions randomly sampled from the AMC 2023 benchmark and 10 questions from the AIME 2024 benchmark. The ground truth for each prompt was estimated with 256 rollouts from Qwen3-1.7B, and predictions made using ZIP-RC trained with the same model. This shows that the joint distribution from ZIP-RC is calibrated and relatively accurate in forecasting the outcomes of its own rollouts.
Beginning (Reward+Cost)
End (Reward)
Model
Total Variation
Fl Score
Accuracy
Recall (Incorrect)
Qwen3-1.7B
0.46
0.91
0.88
0.82
LFM2-1.2B
0.45
0.91
0.87
0.69
LFM2-350M
0.48
0.80
0.82
0.87
Table 1: Prediction accuracy ZIP-RC at the beginning and end of generation. At the beginning, no ground-truth reward remaining-lengeen label exists due to stochastic decoding, so we evaluate the joint reward-cost prediction using Total Variation. At the end of generation, the ground-truth reward known, allowing us to report Fl score, accuracy, and incorrect-answer recall using a threshold of 0.5.
EXPERIMENTS
Our experiments aim to test the following hypotheses:
(1) ZIP-RC can accurately predict the joint reward-cost distribution.
(2) ZIP-RC sampling can be tuned to balance between output quality, and compute cost and latency, tracing Pareto frontier over the quantities over strong inference baselines.
(3) ZIP-RC sampling is adaptive and generalizes across tasks of varying difficulty and across models of varying size.
We will describe and present results that provide positive evidence for each hypothesis individually.
6.1 EXPERIMENTAL SETUP
Models. We use three open models spanning capability and scale: Qwen3-1.7B (Alibaba) in rea- soning mode (Yang et al. 2025); LFM2-1.2B Math (Liquid Al), compact mathematical--easonn model (LiquidAl 2025): and LFM2-350M Math, smaller variant targeting efficient math rea- soning. Unless stated otherwise, decoding is identical across methods; ZIP-RC modifies only the sampling policy at inference time.
Training data for ZIP-RC and baselines. We construct a mathematical training corpus by com- bining DeepScaleR (Luo et al., 22025), the MATH training split (Hendrycks et al. 202T). p. and the GSM8K training split (Cobbe al. [222I). For each prompt. model. yielding roughly 100k rollouts in total. We then label each rollout for correctness against the ground-truth answer. These labeled rollouts used to train model-specifie ZIP-RC predictors as well as any learned baselines.
generate two on- on-policy rollouts per
Baselines. We evaluate against the following baselines that consist of popular sampling strategies that fall under the parallel sampling paradigm where multiple candidate samples are generated in parallel and there is some selection method. Other notable paradigms include beam search self-

---

Published as a conference paper at ICLR 2026
Laterny Bound 6.11
135
150
315
130
150
335
280
125
300
175
260
308
125
150
335
308
125
590
Cerpate Sourdd Oe 1.40
Figure 3: Performance of ZIP-RC sampling and baselines across all models and benchmarks. The top half demonstrates the latency bound setting where 0.1, and the bottom half demonstrates the compute bound setting where a = 1.0. Adjusting 3 in ZIP-RC sampling allows it to trade generation cost for higher performance (similar to increasing in BoN) while adjusting allows adjust the prioritization of compute and latency. By navigating the Pareto frontier and allocating compute adaptively, ZIP-RC sampling significantly outperforms majority voting and other baselines.
refinement. However, reported as they do not suffer from collapsing diversity issues that arise from branching and gen- erating with similar prefixes the ballooning latency issues from methods that generate samples sequentially. We use stronger adaptations of Best-of-N (BoN), and an ablation of ZIP-RC that performs pruning without the sampling utility optimization and instead uses the expected reward directly.:
parallel sampling methods, which the most commonly used and
(1) Majority Voting (MV) (self-consistency), which selects the most frequent final answer, breaking ties uniformly at random (Wang et al.) (202a)). This is an extremely common method since it does not require any learned verifier.
(2) MV with length-based pruning, which discards very long, potentially looping samples (cut at 8k tokens). This baseline acts as sanity check to see our latency gains only come from preventing looping samples from generating to the maximum 32k generation length.
(3) Weighted BoN with external RM, which scores each sample with a separate reward model trained on the same math corpus; because the RM reprocesses the full sequence without KV cache, FLOPs roughly double relative to generation alone (Li et al. (023). This baseline demonstrates strong performance that goes beyond Best-of-N sampling.
8

---

Published as a conference paper at ICLR 2026
Model
Method
Gen. Cusi
AIME2004
AMC2023
MATI]-500
GSMRK
Mixed
ZIP-RC smpliing
1.43
65.8
90.9
94,1
92.2
92.2
Majority Noting
1.40
53.1
87.9
93.0
91.2
91.0
Owen3:
MV enggi--punne
Whigheed BoN ext.
1.46
1.43
25.1
54.7
59.5
36.5
84.7
92.6
91.6
91.0
Weighedd RoN Seif-eoal
1.40
59.4
89.1
93.5
91.6
91.6
ZIP RC rewanl prame
1.33
43.3
$6.0
90.3
89.6
88.9
ZIP RC sampling
1.35
60.9
93.3
93.4
83.6
86.0
Majority
1.60
19.5
90.6
91.8
81.4
83.8
1PM2-1
MV sength-prunee Weighted BoN ext. KM
1.70
51.3
89.8
91.6
83.0
1.53
20.3
89.0
91.1
79.8
82.5
Weighted BuN Self eval
1.00
55.1
91.8
92.6
82.5
84.9
ZIP-RC reward prase
1.49
57.5
ZIP-RC sippling
1.99
38.8
83.9
86.1
70.1
74.1
Majority Nating
1.70
LFM2-350M
MV length prune
Wrighted RoN ext. RM
1,66
1.50
28.5
28.3
74.8
73.4
83.6
81.9
63.2
67.8
70.6
Weighted RoN Seif-eval
1.70
31.4
77.6
84.4
66.8
71.1
ZIP-RC reward prame
1.27
21.7
69.7
83.2
Table 2: Performance and generation at or 0.1 under matched-cost configurations. ZIP-RC
sampling uses 3 = 0.01 and a maximum of eight samples. MV uses three samples; MV length- prune four; Weighted Bo Self-eval (GenRM) uses three; Weighted Bo with external RM uses two; and ZIP-RC reward prune uses 0.4 threshold with eight samples.
(4) Weighted BoN with self-evaluatioon (GenRM), which replaces the external RM with trained self-evaluations derived from the generator (Manvi al., 2024 Zhang al. 2025 Mahan
jet al., 2024). We specifically include this baseline as it is another method that uses less compute than external reward models for selection.
(5) ZIP-RC with reward-based pruming, which starts with a fixed pool and prunes any trajectory whose predicted expected reward falls below a threshold using ZIP-RC's real-time signal. This acts as natural and strong ablation to our sampling utility optimization as it directly prunes weak samples that have less promise than those with high expected reward.
Benchmarks. We report performance AIME 2024. AMC 2023, MATH-500 (Lightman et al. 2023a), and GSM8K. We additionally evaluate on a Concatenated Mixed- Diffiiiufyy Benchmark formed by concatenating the above, which probes adaptive allocation across difficulties.
Metrics. First and foremost we measure accuracy on each benchmark as it is an obvious and good measure for performance and high-quality responses. Beyond performance, measure efficiency and latency. Normalized compute reports total FLOPs per prompt normalized by the FLOPs of a single-sample generation for that prompt. We compute FLOPs with the standard 2N rule (propor- tional to the sum of input and generated tokens) and account for KV caching where applicable. Normalized best-case latency measures the lower bound on wall-clock time the maximum num- ber of sequential forward passes across the candidate set; with unconstrained data- tataa-paralll sampling, latency governed by the longest trajectory. Generation cost aggregates these via linear combi- nation, GenCost
NormCompute + (1 -a) NormLatency. Unless otherwise specified, we use a = 0.1, which roughly balances compute and latency in typical parallel regimes (e.g., eight paral- lel samples often behave like two to three serial generations in practice). For ZIP-RC sampling we sweep 8, which trades off expected quality against cost in the utility: when reporting matched-cosy comparisons set 0.005 and cap the pool at 8 samples for fair comparison to other baselines.
6.2 ACCURACY OF ZIP-RC'S REAL-TIME PREDICTIONS
ZIP provides auxiliary predictions with zero overhead, but for this be useful they must be reliable. We first visually validate whether the joint reward-cost distribution predictions from ZIP-RC are reasonable. To do so, we first obtain ground truth estimates of the joint distributions at the start of generation AMC 2023- AIME 2024, which exhibit nontrivial error rates and diverse reasoning trace lengths. The ground truth estimates are derived with 256 rollouts from Qwen3-1.7B and the predictions are made using ZIP-RC trained with the same model. From the 10 random examples from each benchmark in fig. 2 we can see that the predictions are calibrated and relatively accurate in forecasting the distribution of outcomes.
To quantitatively validate the accuracy of the predictions, we measure the total variation at the be- ginning of generation using the same ground truth joint distribution estimates, as well as standard classification metrics for reward prediction using a threshold of 0.5. As seen in table I the total variations from the ground truth confirm the visual validation that the predicted distributions are rel-

---

Published as a conference paper at ICLR 2026
atively close the ground truth, and the reward prediction at the end of generation further confirms this; it demonstrates high accuracy in terms of Fl Score, accuracy, and recall for incorrect answers (using threshold of 0.5). Overall, these results indicate that ZIP-RC and ZIP predictions can be calibrated and accurate despite being done in the same forward pass as next-token prediction.
6.3 TRACING THE COALITY- LATENCY FRONTIER
We next test whether maximizing the sampling utility with specific cost coefficients achieves con- trollable tradcoffss. At each decision point, ZIP-RC evaluates meta-actions that serve three com- plementary purposes. First, initiating new samples only when necessary and avoiding continuing to generate low-value trajectories saves compute, which is reflected in the compute bound setting in the bottom half of fig. f (a 1.0), where ZIP-RC achieves compute savings. Second, penalizing the continued sampling of long outliers avoids samples that would dominate latency. Third, expanding the initial pool of samples while planning to use a near-term maximum horizon enables the search pursue early finishers without paying the full wall-clock cost of long runs. These mechanisms together drive the latency savings observed in the top half of fig. B (a 0.0). In both settings / is successfully used similar to N in BoN in order increase performance for more generation cost. Parameters and 3 together thus provide simple control knobs over compute-lateng, emphasis and quality-cost trade-off.
Across both a regimes, ZIP-RC sampling traces smooth Pareto frontiers that strictly dominate MV across benchmarks and scales validating that single utility can jointly improve quality, compute, and latency. When a = 0.1 (latency-emphasis), it substantially reduces cost, with the largest relative reduction observed on LFM2-350M (up to roughly 40%). Because we cap at eight samples, the frontier saturates once pass @ 8 performance is reached for given B.
6.4 ADAPTIVE INFERENCE WITH ZIP-RC SAMPLING
Finally, we compare ZIP-RC sampling against all baselines at matched generation cost with a = 0.1 in table 2 Two patterns emerge: (i) at fixed cost, ZIP-RC improves accuracy relative to MV and weighted Bo.N baselines; (ii) it allocates more samples to harder instances (AIME/AMC) and to weaker models, while pruning aggressively on easier problems or stronger models.
At matched cost, ZIP-RC sampling improves accuracy over MV and weighted Bo all models and benchmarks. On harder subsets such as AIME 2024, gains reach up to 12% absolute while using less average cost. The adaptive policy naturally uses more samples when the predicted reward distribution is high-variance where the expected benefit of best-of-N is greatest and conserves compute when one trajectory is expected to be dominant. This pattern is evident the mixed- difficulty benchmark (left-most column in fig. 3) and across model scales: weaker models and harder tasks receive more samples, leading higher overall accuracy.
Takeaways. ZIP-RC's real-time predictions are accurate and reliable enough to enable principled search during decoding. This yields (i) reliable mid- did-ggeneration detection of weak or overlong tra- jectories, (ii) smooth and tunable Pareto frontiers between quality, compute, and latency, and (iii) adaptive allocation that consistently outperforms fixed- bedget Best-of-N the same or lower cost.
7 CONCLUSION
We introduced ZIP-RC, a zero-overhead framework for introspective inference that predicts future reward and cost by repurposing existing logits. This enables principled, real-time decoding search during inference, yielding up 12% absolute accuracy gains over strong Best-of-N baselines at lower average cost, while tracing smooth Pareto frontier between quality, compute, and latency. These findings open natural extensions, such applying to diverse domains and testing fully dynamic resource allocation across different models and reasoning modes. Ultimately, ZIP-RC marks a conceptual shift from rigid, heuristic-based scaling to principled, utility-aware inference. By empowering models to anticipate their success and computational cost, our work is a key step toward more autonomous, reliable, and efficient I.LMs. limitation of our method is that we rely on LLMs achieving sufficient diversity of samples during inference; namely, if we double the number of initial samples, but the new samples are not sufficiently different, then our method and any similar test-time compute method like BoN is unable to achieve higher performance. We believe an important direction of future work is investigating how to improve diversity of samples during inference, potentially by using a mixture of prompts or models. Overall, we believe ZIP-RC establishes a strong foundation for the next generation of introspective models and provides a timely, impactful contribution to adaptive test-time scaling.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pranjal Aggarwal, Aman Madaan, Yiming Yang, et al. Let's sample step by step: Adaptive- consistency for efficient reasoning and coding with 1lms, arXiv preprint arXiv:2305. 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv: /51 1.06297, 2015.
Felix J Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. arXiv preprint arXhv:2410.13788 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamner. Advances in neural information processing systems, 33:1877-1901, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of Im computation, 2024. URL http:://aaxii..orr org/ abs/ abs/2410 04707
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training Ims to reason about their uncertainty. arXiv preprint arXiv:2507.16806 2025.
Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint wwXiv:2508.15260 2025.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint
arXiv: 1603.08880 2016.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12290 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt, Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503. 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dee Dodds,, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.0522. 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 15:22199-22213 2022.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint (3X1:230330000) 2023.

---

Published as a conference paper at ICLR 2026
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286 PMLR, 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier, 2023. URL https://arxiv.
org/ ab///b///// 2206.222622 02336
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624) 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2003 2023a.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023b.
LIquidAI.
Introducing Ifm2:
The fastest on-device
foundation mod-
els on the market, 2025.
URL http::/www.iqu....a../m
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406. 06592, 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by scaling rl. DeepScaleR Surpassing 01 Preview with 1- Model by Scaling RL 1 2025. Notion Blog.
radio b75 notion.iioooooo site/
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models, 2024. URL
https: /ttp://aaavvvv/aaa//aa//aa.a..aa org/ abs/2410, 2410. 12832
Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Lims can predict they can do better, even mid-generation. arXiv preprint arXiv:2410.02725, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint 1rX0:22210.01241, 2022.
Jie Ren, Yao Zhao, Tu Vu, Peter Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. In Proceedings on, pp. 49-64. PMLR, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieseer, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7787):484
2016.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 1rXiv:2408.0333/ 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
12

---

Published as a conference paper at ICLR 2026
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arX1v:2211.1427.. 2022.
Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Yueqi Zhang, Boyuan Pan, Heda Wang. Yao Hu, and Kan Li. Make every penny count: Difficulty-adaptive self-consistency for cost- efficient reasoning. North American Chapter of the Association for Computational Linguistics, 2024. ot
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models, 2023a.
wttp:///rrr/bbb///22/// 1.1711
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- thouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0077 2024.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttp:ii//axxiv...// org/ abs/2500 09388
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances neural information processing systems, 18:1180111182, 2023.
Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, wwtcome--uupeerisseeeee value models for planning in mathematical reasoning. arXiv preprini arXiv:2311.09724 2023.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal.
preddcctionnoon 2025. URL httphttsss///xiitl Generative verifiers: Reward modeling as next-token prediction,
erq/abs/ 2408 15240
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-4661 2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 SAMPLING AS DECISION- MAASION-MAKING MAKING UNDER A META-MIP
We formalize the problem of test-time search decision decision-making under high-level MDP that dub a meta-MDP. We describe the components of the meta-MDP detail below:
Meta-states. At timestep t. the meta- is prefix tree (trie) St rooted at the prompt Formally, St = Ni,EE,) where & S Nr x V x Ni is the set of directed edges (s. a, s') labeled by tokens E V. is the root corresponding given by the concatenation of the root and the token labels along the path from r to 8. Each node also corresponds
the prompt, and each node E N, represents the prefix
state in the base MDP as it is sequence of tokens that the base policy has generated. A prefix is finished if its last edge is the special token <EOS>, corresponding to the terminal state in the base MDP. Initially, So (fr), 0, where r is the root containing the prompt. Conceptually, the meta-state therefore encodes all prefixes processed or generated by the policy. In practice, the trie only requires space on the order of the total tokens generated to represent all sequences, and their corresponding prefixes can be stored in the KV cache.
Meta-actions. At step f, the meta-action selects a finite multiset of prefixes (nodes) A, S (M)) to continue sampling from. Multiplicity encodes branching: if prefix appears times in A,, then is sampled from r times independently. This definition encodes any viable single-token sampling and any strategy one might want to perform. If A but none of its children appear in A,, this equivalent to pruning. If E A, already has children, this equivalent to backtracking.
Meta-transition function. Given (S,, At). for each occurrence of E A, we sample .T( and add the edge (s, a, and its child s' to the trie, yielding S For notational simplicity, we treat part of the transition dynamics of the meta-MDP the transition function P(S++1 Sc, Ar) implicitly includes sampling from x. The process terminates at horizon T, corresponding to the maximum allowed search steps.
Meta-reward function. At each step we incur a cost C(S, Ad) = (suppspp(A)))) (A))+ -1 (1-a)). where supp(Ar is the set of distinct prefixes chosen in A, (one forward pass per unique prefix) and the second term accounts for step latency. The parameter a [o, 1] balances compute versus latency, and /3 > sets the trade-off' between reward and cost. At the terminal timestep T, one completed generation si is selected, and the reward is the base MDP reward R(st). Including this cost term is essential, since otherwise one could trivially maximize reward by always branching.
Search strategies as mota-policies. A strategy f is policy in this meta-MDP: at each timestep it maps the current prefix tree S; to a multiset A, = ((S;)) of nodes to expand, and at horizon T selects finished f BoN corresponds to placing N copies of the root in Ao and thereafter always expanding every unfinished leaf until completion, finally selecting the highest-scoring candidate. Beam search with width B instead enforces AA,| B at all times: at most steps A, is just the B current leaves, but at pruning intervals of length it ranks leaves by a score, discards the weakest p, and duplicates stronger ones so that the frontier is refilled back to B, thereby pruning and branching in controlled manner before ultimately selecting the best finished prefix at T.
A.2 COMPUTING AN OPTIMAL ZERO-OVERHEAD SEARCH STRATEGY
clear from our formalism what an optimal search strategy should be: at every timestep the optimal strategy je should choose the meta-action that maximizes:
μ² (Sp) arg max Q" (S),A)) A.
(9)
where we define meta Q-function over the meta-MDP for strategy as,
T-1
Q" A,) F,, R(s c(S. Ac) Sr, A
(10)
P=t
However, computing Q" for arbitrary strategies μ is often intractable, primarily because we cannot generate on-policy trajectories from a without incurring much computational overhead.
14

---

Published as a conference paper at ICLR 2026
The sampling utility. To avoid having to generate rollouts, consider class of predefined strategies at any timestep t as follows:
M₁ = {p: : p1 Se) [ s:))e(EN))
A'' 2 f
(11)
where f denotes any function the set of next-token distributions for every prefix in the meta-state. Concretely, Mt consists of all strategies where future meta-actions are determined entirely from generation behavior at timestep This essentially means for μa E M,, we can compute its value Q" without explicitly executing y over future timesteps.
For any meta-state and meta-action at timestep i, we define the sampling utility to be the value of some strategy in the aforementioned class of strategies M,. Because each strategy performs worse than optimal strategy f due to the imposed constraint, we choose the best-performing strategy in Mt to act as the tightest possible fower-boun
U(S,, At) = max (S, Ar) S Q" (St,At). Ar).
(12)
W.M
We show later how this maximization M,, well computation of Q" for μ E Mr, can be done tractably using the quantities obtained via ZIP-RC, without any additional forward passes, auxiliary models, or architectural modifications beyond standard decoding.
Finally, ZIP-RC sampling is defined as the strategy that maximizes our proposed sampling utility:
ZIP RC (S.) arg max U(S, As).
(13)
Intuitively, we can derive the following property of our learned strategy:
Theorem A.1. At every timestep t, our strategy μ ZP--CC performs better than any predefined strategy Mr Namely, for any meta-state St, we have
QTIP-RC (O.-CC(S,, (1 (S.)) 2 ((S),, Vp E M..
(14)
Proof. We can prove this via induction on t. Naively, this holds for terminal timestep t = T. For any p E Mr, we let AQIP-RC = (IZIPRC((S)) (S:) and A," ((()). Then, we have
QZI.RC(S A A(IPRC) QZII (5411, C(S,, A RCP
(15)
Therefore, ZIP-RC sampling is a powerful test-time search strategy that explicitly optimizes for reward and generation cost.
Approximating the sampling utility. To approximate the sampling utility, we aim to answer two questions: (1) for every meta-state S₁ and action A,, how do we search for strategy μ E M, that achieves high value (S, Ar), and (2) how do we compute Q* Ar) tractably using only predictions by ZIP-RC.
First, let us consider the naive strategy y Rollouta of always selecting the unfinished leaf-node descen- dants of the prefixes in the current action A,, or in other words, obtaining rollouts or generations using starting from each selected prefix. At the end, fl Rollouts selects the generation with the highest value V (sT), similar to BoN. Its meta-MDP state-action value exactly given by:
T-1
Q Rollouts (St, Ad) = E Rostonne
R(S
Σ t'zt
C(Se, Ax) St
(16)
E
max V (8) -
Σ
L (8) + a) max (7(ss) sEA,
SEA,
BEA.
1)
max V (s) HEA,
- a Σ
B() + (1 a
max I (7(()) (17)
HEA.
sEAr
15

---

Published as a conference paper at ICLR 2026
We can observe that the expression QRollouts contains several interpretable quantities. Namely, the expected maximum value quantifies the marginal benefit of branching or pruning; the incremental gain from increasing N is large when the value distribution has high variance, and conversely, the marginal loss of pruning is small under low variance. Furthermore, the expected maximum remain- ing tokens and the expected total tokens capture the marginal cost of branching; increasing N always increases the expected total remaining tokens and the maximum remaining length, which drives up latency, and pruning will always reduce the cost.
While y Rolloouts has several nice properties, the strategy itself is naive as it assigns maximum cost for every new sample and does not consider that those samples can be pruned in the future. This is exacerbated further by the empirical correlation between the length of reasoning traces and the likelihood that they are incorrect. Being able to "het" on an early finishing sample that has high reward is crucial. To remedy this problem, we introduce an additional parameter into the meta-value that enables the y Rollouts strategy to prune each sample in the future at a predefined horizon.
Formally, let Q Rollouts (St, As; H,) be the value of executing μ Rollouts with the additional capability that each active prefix 8 E A, will stop generating upon reaching length ht E Hr, where H. is a set of lengths of size |Ag]. We can now define an improved lower bound by maximizing over Hy:
Rooloonts (St, H.? = max Q Rollouts (S,, H.) 2 Q Rollouts AI), Ht
(18)
where it easy to see that the our new meta-value is monotonically better than the value of the naive I Rollouts strategy without pruning capabilities. We use this to approximate the sampling utility:
u(S, A.) QRolloutssS. (St: H2))
(19)
Tractable computation of expectations. Next, we show how our sampling utility in Equation 19 can be computed tractably using only predictions by ZIP-RC. Predicting Qitollouts is straightfor- ward since we can estimate distributions over the value qb (-1s;) and remaining-tokee % (-(st) conditioned on each prefix s, using our previously proposed ZIP-RC. Hence, we can estimate by computing:
By
E
max V (s) A
Ub + 1b+1
(F) V.V.maxx(b At)) - F.V.axx (b At)).
(20)
2
b=1
Br
(()
to
+
te+1
40
L (e) s),
(21)
2
t=1
Br
te+1 (F) (ax.max( Ar) F.. (e- 1|AA)).
E
max LT(s) SEA:
(22)
2
where
F.V. V.mxx
(b (b1
II (b 18),
F p...ax (el (1) II (e 8);
(23)
SEA.
SEA.
Fý =b) I
Ft 8) = Σ
a 8).
(24)
35b
JSE
Now, to incorporate the predefined horizons H. over all prefixes A,, we modify the joint distribution (b, to a capped joint po (b, Els; h2) that collapses probability mass beyond the cap h2 into a designated "clipped" state (bo. h=):
Pa (b, (1)).
f 5 has
(b El s;h,)
1lb = bo. e= h-hhhh Σ
pol E s),
8> h2
(25)
This construction ensures that all probability mass corresponding to continuations exceeding the allowed horizon is reassigned to a truncated state at l= h., while the value component is collapsed to the designated base bin bo reflect the forfeited reward from pruning.
16

---

Published as a conference paper at ICLR 2026
From the capped joints, recover the corresponding marginal distributions:
Br
By
00 (b 8: h;)
Σ polb, 8 8; h,),
d 8; ha))
Σ po & h,))
(26)
These capped marginals directly encode the expected effect of planned pruning on both value and remaining length for each prefix. Notice that this is only possible by modeling the joint distribution as ZIP-RC is defined and is not possible with only the two marginal distributions. Thus, we demon- strate that we able compute our sampling utility in Equation T9 Jusiigg only our zero-overhead predictions from ZIP-RC.
Summary. ZIP-RC sampling defines a meta-policy that, at each meta-state S,, selects the meta- action A.-a multiset of prefixes to expand for one decoding step- that maximizes the sampling utility in eq. (19). This utility is the state-action value the best policy in the predefined strat- egy class MM., whose future behavior is fixed. Because ZIP-RC sampling re-optimizes A, at every timestep. it adapts online to the stochastic evolution of the prefix tree: if current trajectories are pro- jected to be costly low-value, it can immediately redirect computation elsewhere. As formalized in eq. (I4), this dynamic strategy is guaranteed to perform at least as well as any predefined policy in My.
To approximate the sampling utility tractably, we use the value of the best rollouts-with-pruning strategy Qulllouts (St, Aci Hz). in which each prefix may continue only up to an optimized (and possibly distinct) horizon. This value can be computed in closed form using ZIP-RC's joint reward- cost predictions. Concretely, for every prefix AA, we: (i) obtain its predicted joint distribution pb(b, E s); (ii) apply the prefix-specific horizon using the capped construction in eq. (26): and (iii) compute the expectations of the required order statistics using eq. (22). This yields a fully tractable, zero-overhead estimate of the sampling utility for any candidate meta-action.
A.3 ZIP-RC IMPLEMENTATION DETAILS
Demaining-token discretization. For the joint ZIP-RC head, we discretize the remaining-lengle
variable (s:) using logarithmic bins. Let (te)
[=1 Br++1 denote the bin boundaries, and define bin as
[te, 1x+1) with representative value (1x + (++1)/ //2 To obtain fine resolution for short continuations while keeping By small, we collapse all very short lengths into single initial "startup" bin and set the remaining boundaries to grow as powers of two. This construction preserves precision where
matters while limiting the number of reserved tokens required for ZIP-RC.
KL weight. In practice, the KL term is a very small component of the total loss because the policy remains close to the original policy as seen fig. Accordingly, we use relatively large coefficient oKL, typically in the range 10-100.
trainykl loss
without
0.8
0.6
0.2
500
1.4k
Figure 4: KL divergence from the original policy during training of ZIP-RC with and without the KL term. Using akl = 10 keeps the KL nearly zero throughout training, stabilizing around 0.005. Without the KL term, the policy eventually changes, emphasizing the importance and effectiveness of this component of the ZIP objective. We used the same training data as in our main experiments.
17

---

Published as a conference paper at ICLR 2026
Temporal smoothing. Token-level ZIP-RC predictions be noisy. inference time option- ally smooth the joint by averaging the last W predictions along the current trajectory: for a prefix st we set
W-1
po(b, (1b,els))
(b, st-w), Σ
(27)
where the sum runs over the previous non-terminal prefixes on the same path. We use po in place of Pg when computing the ZIP-RC marginals and the sampling utility.
A.4 ZIP-RC SAMPLING IMPLEMENTATION DETAILS
Normalization of cost term 3. Because rollout lengths can differ by orders of magnitude across prompts, rescale the cost coefficient 3 by per-prompt estimate of the typical total token count. At decision step t, we use the normalized coefficient
8 B
1
3
where
B.
Σ (1s++E(L7()))) SEA,
(28)
Atl
Here |s| is the current length of prefix 8, and (+)] is its predicted remaining tokens from ZIP- RC. This keeps the reward-cost tradeoff stable across prompts with very short very long genera- tions.
Practical reduction the search space. The full meta-action space over multisets of nodes in the prefix tree is extremely large, and jointly optimizing per-prefix horizons is combinatorial. In our implementation we therefore operate within a structured subelass of meta-actions and horizons. First, at timestep t restrict candidate meta-actions to multisets over the root and the unfinished leaves of S, with multiplicity only allowed at the root, and further downselect to a small set of prefixes prioritized by higher predicted value and lower predicted remaining length under ZIP-RC. Second, when computing the sampling utility we use single shared horizon h, for all active pre- fixes, rather than independent horizons per prefix, reducing the search over pruning schedules to a one-dimensional search over h4 Finally, instead of recomputing م ((I,)((S)) at every token, we update the meta-action only fixed intervals and simply continue all eurrently active prefixes in between. These design choices substantially reduce the search space while preserving an expressive and adaptive family strategies. This represents just one concrete instantiation of our framework; many other reductions are possible.
A.5 ZIP-RC-LITE
We add an ablation we refer to as ZIP-RC-Lite where we use the ZIP-RC objective described in eq. (3) with the KL term removed, keeping only the output head of the language model trainable while freezing the rest of the model. As shown in fig. 5 and table we find that ZIP-RC-Lite is able to non-trivially predict the joint distribution but unsurprisingly struggles to do as accurately as ZC.-RC. Its predictions are poorly calibrated and may not be suitable for interpretability of the generation process and output. Despite this, we find that ZIP-RC-Lite search provides substantial gains with respect to baselines in the latency-bound setting with a 0.1, suggesting that predicting the expected reward and remaining length to any degree is useful for pruning overly long trajectories as seen in the first row of fig- 6 However, we do find that ZIP-RC-Lite substantially over-allocates compute in the compute-boun bound setting with a = 1.0 due to overestimating the variance of the expected reward, resulting in poor efficiency seen in the second row of fig. 6 Overall, while ZIP-RC clearly more calibrated and accurate, ZIP-RC-Lite is a compelling alternative one does not want to keep the whole model trainable.
18

---

Published as a conference paper at ICLR 2026
WAN
300AM
WAW
"ARA MARAY
Figure 5: Similar to the demonstration of ZIP-RC's joint distribution prediction in fig. 2 we sualize the joint distribution predictions from ZIP-RC-Lite and compare them with ground truth estimates, While the predictions correlate with the ground truth, ZIP-RC-Lite tends to produce more similar-looking distributions across prompts and overestimates variance compared to ZIP-RC.
Beginning (Reward+Cost)
End (Reward)
Method
Total Variation
F1 Score
Accuracy
Recall (Incorrect)
ZIP-RC
0,46
0.91
0.88
0.82
ZIP-RC-Lite
0.63
0.82
0.71
0.12
Table 3: Similar to the evaluation of ZIP-RC in table we show the prediction accuracy of ZIP- RC-Lite at the beginning and end of generation. The results indicate that ZIP-RC-Lite predicts the joint distribution non-trivially, but less accurately than ZIP-RC.
Latency Bound (a 0.3)
GSMBK
WAT1500
AIMC2024
60-
02.0
5
36
90.0
Performance
1.25
1.25
2.5
150
1.5
1.25
175
Compute Bound
02.5
N1.
91.5
41.5
91.0
40.5
50
90.0
Gereeration Cost
normelized lized
normalized Intency
Sf We Sellow antor Conti
ZP-ac Sampiing zevaciit Samt
Midoenta Viiting kinghibaoed srandig
2P soard based prrinng nd sunolea)
Figure 6: Similar to the demonstration of RC sampling in fig. 3 present results with -IP- RC-Lite search in green, indicating it is able to provide significant gains, albeit lower than ZIP-RC sampling, especially in the compute-bound setting. This suggests that predicting the joint reward- cost distribution to any non-trivial degree helpful allocating test-time compute more optimally. However, the results in the compute-boune bound setting indicate that HIP-RC-Lite' overestimation of variation in the expected reward distribution results in over-allocation of compute.
19


Reviews:
[{"review_id": "5XK92SHxdF", "reviewer": "Reviewer_hUjT", "summary": "This paper proposes ZIP-RC, an adaptive inference method that equips model with zero-overhead inference-time predictions of reward and cost. ZIP-RC makes use of *reserved logits* in the forward pass to model a joint distribution over the reward as well as the remaining sequence. The authors show that ZIP-RC improves accuracy by up to 12% over majority voting at comparable computational cost.", "questions": "- Could you please give me a breakdown of your technical approach? What exactly does your approach do at training time? How do you modify autoregressive decoding to model the joint distribution of sequence continuations and their corresponding rewards? A diagram would've been extremely helpful.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The promise of the paper, \"Zero-overhead inference-time\" control is a very promising and timely research direction\n\n- The authors show that their approach is decently accurate at predicting the rewards during generation.\n\n- The proposed approach achieves improvements, often significant, on a suite of reasoning benchmarks at the same cost as the baselines.", "weaknesses": "- I found the paper extremely difficult to read, and beyond the promise of a \"zero-overhead inference-time prediction of reward\" found it very hard to glean much if any insight on the core contributions of the paper beyond the fact it makes use of extra logits at every step.\n\n- The related works section is really lacking giving how active an area of research inference-time control of LMs is.\n\n- line 85, broken figure reference. \n\n- Paragraph 074-085 of the introduction misses the mark when it comes to introducing the approach, in my opinion. Specifically, starting line 080, the paragraph makes mention of sampling utility and coefficients $\\alpha$ and $\\beta$ that supposedly balance reward and cost. This level of detail at this point of the paper left me more confused than informed, with no big picture idea of what the core contribution is.\n\n- I am unsure if the subsection on \"Generation as a token-level MDP\" offers much value. In fact, it seems that variables defined in that section were used differently in later section e.g. $P$.\n\n- Lines 170-171: \"ZIP interprets the slice on $\\mathcal{R}$\". I believe the previous sentence should be \"slice on $\\mathcal{V}\" instead?\n\n- It took me quite some time to realize that the approach is *not* inference time, but instead relies on some form of training. In my opinion, the authors should make sure this is clear very early on in the paper as well as potentially clarify the title which to me is currently a bit misleading.\n\n- The presented approach fails to compare against many of the latest inference-time approaches in their experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZzOUiGhZVA", "reviewer": "Reviewer_5iv6", "summary": "This paper proposes ZIP-RC, a zero-overhead inference-time mechanism that equips large language models with real-time predictions of task reward and remaining generation length. The key idea is to utilize a set of reserved vocabulary tokens as auxiliary prediction channels, allowing the model to output both next-token probabilities and reward/length estimates within the same forward pass. To prevent distribution shift, the method introduces a KL-regularization term that constrains the model’s behavior relative to a frozen reference policy.\nOn top of these predictions, the paper introduces an adaptive sampling and pruning strategy that dynamically reallocates compute across candidate trajectories, enabling the model to prune low-quality or unnecessarily long samples mid-generation. This yields controllable trade-offs between output quality, compute cost, and latency, improving inference-time efficiency.\nExperiments on mathematical reasoning benchmarks (GSM8K, AIME24, AMC23, MATH-500) and small-to-medium LLMs (≤1.7B) show that ZIP-RC consistently outperforms majority voting and best-of-N baselines under equal or lower generation cost, demonstrating more favorable quality-compute Pareto frontiers.", "questions": "Address the shortcomings identified in the Weaknesses section, and include the required supplementary experiments to adequately resolve these concerns and strengthen the study.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1.The paper proposes the ZIP-RC mechanism, which leverages reserved tokens in the vocabulary to enable real-time prediction of reward and remaining tokens without adding extra forward passes. This idea is highly innovative. Combined with adaptive pruning, the approach significantly improves inference efficiency and is of substantial practical importance.\n\n2.Experimental results show that ZIP-RC clearly outperforms the baseline methods without such modification, demonstrating the strong potential of the proposed approach.", "weaknesses": "1.Although reserved tokens are used to avoid additional computation overhead, this strategy may still introduce distribution shift to some extent. The paper would benefit from additional comparison or analysis on the extent of distribution shift before and after modifying the loss function.\n\n2.The experiments are primarily conducted on relatively small models (mostly under 2B). It would strengthen the work to extend evaluation to larger-scale models to verify scalability. Additionally, the experiments currently focus mainly on mathematical reasoning tasks; further results on other domains (e.g., instruction following, coding, and open-domain dialogue) are needed to validate generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "quRmPPfiB0", "reviewer": "Reviewer_VsZ8", "summary": "This paper tackles an important limitation of current test-time scaling methods for large language models: their inability to adaptively allocate compute based on problem difficulty. The authors propose ZIP-RC, a method that equips models with real-time predictions of both final reward and remaining generation cost without any inference overhead. The key innovation is repurposing reserved vocabulary logits to output a joint distribution over reward and cost at every decoding step. This enables optimizing a sampling utility that explicitly trades off accuracy against compute and latency through meta-actions like dynamic pruning and adaptive sample allocation. On mathematical reasoning benchmarks, ZIP-RC achieves up to 12% accuracy improvements over majority voting while using equal or lower cost, and traces smooth Pareto frontiers across quality, compute, and latency dimensions.", "questions": "1. How many positions are reserved for the reward-cost predictor? How does this choice affect both the predictor's expressiveness and the base model's generation quality? Have you experimented with different numbers?\n\n2. Can you provide any evidence that ZIP-RC works on non-mathematical tasks? Even preliminary results on a single coding or open-ended QA benchmark would significantly strengthen the generalization claims.\n\n3. What is the wall-clock time and compute cost to train ZIP-RC compared to training the baseline models? How much data is needed for the predictor to become reliable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using reserved vocabulary positions to produce auxiliary predictions with truly zero overhead is novel and elegant. Rather than requiring separate forward passes or additional models like most verifier approaches, ZIP-RC extracts rich signals from logits that would otherwise go unused. The joint modeling of reward and cost (rather than just scalar confidence) is a key insight that enables principled decision-making about the reward-cost tradeoff.\n\n2. The authors clearly articulate why existing Best-of-N methods are inefficient—they generate all N samples to completion regardless of promise. The mathematical formulation of the sampling utility in Equation 1 provides an interpretable framework for balancing competing objectives through the α and β hyperparameters.\n\n3. The paper is generally well-written with clear motivation and good use of figures to illustrate the method's behavior. The formalization as a token-level MDP and the discrete order statistics for computing expectations are technically sound.", "weaknesses": "1. The evaluation focuses exclusively on mathematical reasoning tasks. It's unclear whether ZIP-RC's benefits extend to other domains like creative writing, coding, or open-ended question answering where the reward structure and token length distributions may be very different. Mathematical problems have clear correctness labels and relatively predictable structure, which may make reward/cost prediction easier than in other domains.\n\n2. The authors acknowledge that their method relies on having sufficiently diverse samples from the initial pool, but they don't actually address this limitation. If doubling the number of samples doesn't produce qualitatively different reasoning paths, ZIP-RC cannot improve performance. This is a fundamental constraint that limits when the method can be applied effectively. Some analysis of sample diversity in their experiments would strengthen the work.\n\n3. The paper doesn't clearly specify how many vocabulary positions are reserved, how this impacts the model's generative capabilities, or what the training cost is. The KL penalty term αKL is mentioned but its practical effect isn't analyzed. More importantly, ZIP-RC requires generating ~100k labeled rollouts and training the predictor—this upfront cost should be quantified and compared to the inference savings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5PLKpybttU", "reviewer": "Reviewer_5e4c", "summary": "This work is in the context of autoregressive Language Models with a finite vocabulary (i.e. popular transformer-based LLMs).\nThe paper aims to improve the model's introspective ability (e.g., predicting at each token, prediction final\nreward and remaining length) without increasing the test time inference cost.\nTo enable this, the paper proposes to use a reserved or unused set of logits and thus claims to require no additional compute at inference time.\nThe paper examines this within the framework of test-time scaling, i.e., using test-time compute for improved performance.", "questions": "Please comment on / address the weaknesses above.\n\nAdditionally:\n1. line 267: \"ZIP-RC modifies only the sampling policy at inference-time.\" This seems misleading as the model does train and thus changes the parameters of the original models.\n2. How many reserved logits are used in practice?\n3. What is the initial number of samples? And how does this compare to baselines?\n3. What other use case do the authors see for this framework (reserved logits for a different reward) besides the presented application of test-time scaling?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed approach of using unused logits in combination with the proposed optimization of a joint distribution different from the next token prediction loss (but over an estimated value of the correctedness and remaining tokens) seems very interesting and innovative to me.\n2. The results of the proposed approach in comparison to the presented baselines show clear benefits on the Pareto curve.", "weaknesses": "## Major weaknesses\n1. The paper's clarity in the main approach section is limited. Sections 4.1 and 4.2 are difficult to parse and would benefit from more careful dissemination, including model/approach figures.\n2. The paper claims **zero** inference time overhead. However, computations described in Section 4.2, paragraph \"Sampling utility.\" suggest that additional computations and parameters (alpha, beta, ...) are required (even though they may be small compared to the overall computation and parameter requirements of the LLM). It would be important to clarify this.\n4. The paper's approach heavily relies on a reserved set of logits but does not describe where they come from. Would we save compute if we are not using these \"reserved\" logits? This also relates to the previous point: Do these reserved logits basically represent additional computation needed for this approach\n3. I think the paper is missing an important baseline, which does not change the underlying LLM (i.e. one does not need the KL term), but just optimizes the same reward and cost (ZIP-RC) using a small MLP (one? or two layers), which has access to the same logits used for the proposed approach.\n\n\n## Minor weaknesses\n1. all formulas should be numbered so one can refer to them", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)