Multiple Token Divergence: Measuring and Steering In-Context Computation Density
Authors: Vincent Herrmann
Eric Alcaide
Michael Wand
Jürgen Schmidhuber
Venue: ICLR 2026 Poster (2026)
Keywords: Language models
in-context learning
reasoning
interpretability
decoding
Primary area: foundation or frontier models, including LLMs
TL;DR: If a shallow auxiliary prediction head struggles to approximiate the full next token prediction, we can infer that the model is doing complex in-context computation.

Abstract: Measuring the in-context computational effort of language models is a key challenge, as metrics like next-token loss fail to capture reasoning complexity. Prior methods based on latent state compressibility can be invasive and unstable. We propose Multiple Token Divergence (MTD), a simple measure of computational effort defined as the KL divergence between a model's full output distribution and that of a shallow, auxiliary prediction head. MTD can be computed directly from pre-trained models with multiple prediction heads, requiring no additional training. Building on this, we introduce Divergence Steering, a novel decoding method to control the computational character of generated text. We empirically show that MTD is more effective than prior methods at distinguishing complex tasks from simple ones. On mathematical reasoning benchmarks, MTD correlates positively with problem difficulty. Lower MTD is associated with more accurate reasoning. MTD provides a practical, lightweight tool for analyzing and steering the computational dynamics of language models.

Published as a conference paper at ICLR 2026
MULTIPLE TOKEN DIVERGENCE: MEASURING AND STEERING IN-CONTEXT COMPUTATION DENSITY
Vincent Herrmann" 1.2x Eric Alcaide Michael Wand & Jürgen Schmidhuber
1 The Swiss AI Lab IDSIA/USI/SUSSI
Lugano, Switzerland
2 King Abdullah University of Science and Technology
Thuwal, Saudi Arabia
Equal contribution
(vincent. herrmann, eric. alcaide, saahaal..aand, juergenJeidsii........
ABSTRACT
Measuring the in-context computational effort of language models is key chal- lenge, as metrics like next-token loss fail to capture reasoning complexity. Prior methods based on latent state compressibility can be invasive and unstable. We propose Multiple Token Divergence (MTD), a simple measure of computational effort defined the KL divergence between model's full output distribution and that of a shallow, auxiliary prediction head. MTD can be computed directly from pre-trained models with multiple prediction heads, requiring no additional training. Building on this, we introduce Divergence Steering, a novel decoding method to control the computational character generated text. We empirically show that MTD is more effecctve than prior methods at distinguishing complex tasks from simple ones. On mathematical reasoning benchmarks, MTD correlates positively with problem diffficulty. Lower MTD is associated with more accurate reasoning. MTD provides a practical, lightweight tool for analyzing and steering the computational dynamics of language models.
INTRODUCTION
To solve unfamiliar and challenging problems, language models must perform sophisticated in- context computation (Brown et al. 2020 Lewkowyez et al. 2022 Can we tell whether, and to what extent, a model is making use of its computational capacity at any given moment? It is well- established that the next-token prediction loss offers little insight (Schmidhuber [010 [Brdaa al., in principle, be arbitrarily difficult to achieve (Benneti
2018), as any particular reduction in loss 1988 A more promising approach is to quantify meaningful computation by measuring the entropy, or incompressibility. of model's 's latent representations (Skean al., 2025 Herrmann al. 2025 This concept is rooted in the minimum description length (MDL) principle (Wallace and Boulton
1968 [issanenn 1978 Solomonoff 1964 Vitányi 2006) Elmoznino et al. 2024): if the most com- pact description of a sequence's s structure, given the training data, is still long, then predicting that sequence is demanding due large search space. In other words, task "difficult" or "com- plex" if the shortest description of the solution program, given the training data, is long. If it is short, on the other hand, we consider the task "easy" and hence "boring". Unfortunately, in gen- eral, the length of the shortest description is uncomputable (Li et al., 2008). To make it tractable, the Prediction of Hidden States (PHi) loss was proposed as a measure of in-context computational complexity, quantifying the per-token information gain in a model's latent space (Herrmann et al.) 2025). This work itself is based on the earlier ideas of neural history compression (Schmidhuber) 1992ah. While promising, the PHi framework introduces significant practical challenges: it requires inserting a noisy information bottleneck that can degrade model performance, needs further model training which can be unstable, and is highly sensitive the precise location of its placement within the model, and the weighting of multiple loss terms.
In this work, propose simplified and more direct measure, Multiple Token Divergence (MTD), which quantifies information gain in the model's 's output distribution. The core insight is simple: if shallow computational shortcut (e.g., single Transformer block) can approximate the full model's

---

Published as a conference paper at ICLR 2026
prediction, then the model is not performing particularly complex computation. If, however, there is a significant divergence between these two predictions, we can conclude that the model is leveraging its deeper computational capacity. MTD is straightforward to implement and can even be computed directly using the Multiple Token Prediction (MTP) modules that some modern pre-trained mod- els already possess, requiring additional fine-tuning. In addition to this measure, we present Divergence Steering, novel decoding method that uses the MTD signal to control the computa- tional character of the generated output. It biases the decoding distribution towards or away from computationally dense tokens-i.c tokens that are likely under the full prediction but unlikely un- der shallow approximation by the MTP module. We empirically demonstrate that MTD more effective than prior methods distinguishing complex difficult tasks from simple ones. We also in- vestigate the properties of MTD and Divergence Steering reasoning and creative generation tasks. Our implementation is public
BACKGROUND
2.1
PREDICTION OF HIDDEN STATES (PHI)
The Prediction of Hidden states (PHi)
method, introduced by Herrmann
Neat Prediction
al. (025)) creates information
bottleneck (Tishby and Zaslavsky 2013) within sequence model in der to mcasure the complexity of its in-context computation. A PHi layer is inserted between model's "bot- tom" and "top" layers. The model consists of the following modules: The Bottom Layers (Bz) are the ini- tial Transformer blocks that process the imput sequence embeddings. The PHi Layer contains three key com- ponents: (1) An encoder (qv) that, at time step maps the hidden state St from the bottom layers to posterior distribution over latent variable 2,- This distribution is typically a diag- onal Gaussian, similar to variational auto-encoders (Kingma and Welling [014/eezender al.) 2014). (2) de-
Outpxl Hoad
Outoot
Tanaformer Blok
My Tranffomne
Thunsfomee flok
NTT
Emboediing
Drossdiig Laser
Figure 1:
Comparison between the architecture of PHi model (left) and MTP model (right).
coder (ag) that reconstructs the hidden state, creating o from sample of the latent variable =; (3) An autoregressive prior that predicts the distribution of the current latent 2; using only the history of previous latents, zec. This can be implemented with single Transformer block (M₂)) and two additional linear transforms: by, which maps the inputs to Mμ. to the right dimensionality. and Px- prior distribution. The Top Layers (T,) are the
which maps the output of the Transformer maining Transformer blocks that process the sequence of reconstructed hidden states g' Finally, we have the standard token Embedding (E.) and the Output Layer (O..). Here and in the remainder of the paper, Greek subscript letters indicate learnable neural network parameters.
The forward pass processing tokens æ1, 02,....... is described by these equations:
e E. (x()
Token embedding
9t Ba(en....,
Output from bottom layers
2, w0( let)
Latent sample from posterior
9. ac(21)
Reconstruction of hidden state
h4 T, (g'.......
Output from top layers
T(- (x2)) = (h4-1)
Next token prediction from output head
(t) log π (x2
Negative Log Likelihood (NLL) loss
2

---

Published as a conference paper at ICLR 2026
The PHi loss (LpH)) is the KL divergence between the posterior qv. which has access to the current input 2y via gt, and the prior Px, which only has access to past latents Zec- We assume an initial latent 2g is given.
Ca (x(11-1) dj = Mp(cl,...;ce)
Lincar projection of last latent Output from PHi Transformer block
LpHitt) = DKL g00 ( (9c) Pxl 1 (db)
Loss
(2)
= DxL. gop (r1,.... |l P2 |20 21, 21,-..
22-1) 21,-..,,--11)
The model is trained to jointly minimize both LNLL and Lemi- The PHi loss quantifies the information gain at each timestep-tiis the amount of new useful information present in the current input token 2; that was not predictable from the history of latent states. It has been shown (Herrmann et al.) 2025) that this value correlates well with the complexity and "interestingness" of tasks.
2.2 MULTIPLE TOKEN PREDICTION (MTP)
Multiple Token Prediction (MTP) is technique used improve model performance and enable faster inference via methods like speculative decoding (Cai et al. 2024) Gloeckle et al., (2224) Liu al. 2024 Xiaomi et al. 2025). In this setup, computationally cheap auxiliary module trained to directly predict the main model's ss future output distribution.
First, consider standard autoregressive model's forward pass:
(x(1
Token embedding
he = Fo(et....;.e)
Output of all main Transformer blocks
m(: [Fee) (h4-1)
Next token prediction from output head
(t) = log π (xe))
NLL loss
The goal of MTP is to approximate the main model's 's prediction for the token one step further ahead, Tt-1- Note that often in MTP, there additional modules that approximate predictions for tokens even further ahead, i.e., Ittn for n > 1. We will not use them in this work.
separate, smaller MTP module (e.g., single Transformer block M,,) generates its own prediction without access the full model's current hidden state h, and is usually trained with negative log- likelihood loss, which we call /MP-- Optionally, the MTP module can be given access to the current token embedding e,, as indicated by the square brackets.
a by (b+-
Input MTP module (projection of h4 h and possihly 6x)
(3)
d = My(ci,...,.c.
Output from MTP Transformer block
(MPP( (Ee))
(w(-e-1)
MTP's prediction for token Tr+++1
[t = log TMIP (2: (x())
MTP Loss
In order to predict the next token Ht-1, the MTP module has access to the model's history via h (and optionally the current embedding ex), but crucially lacks the result of the main model's full computation at step /. (i.e., ht). In recent works using MTP for Large Language Models (Xiaomi al.) 2025 Liu et al. 2024). the MTP module shares the embedding and output heads with the original model, implicitly enforcing alignment of the latent space.
3 MULTIPLE TOKEN DIVERGENCE
Observe the similarity between the PHi framework's autoregressive prior Px and posterior dy- on the one hand, and the MTP prediction TMPP and the full model prediction π on the other (Figure M] in both cases, computationally and informationally constrained module approximates the prediction from full model. The key difference is that for PHi, this approximation occurs continuous latent space, whereas MTP operates directly the discrete token distribution.
3

---

Published as a conference paper at ICLR 2026
Based on this analogy, we propose the Multiple Token Divergence (MTD) as an alternative to the PHi loss. It is defined as the KL divergence between the full model's next-token prediction æ and the MTP module's 'aluullsspppl prediction TMTP:
LMTD(()) (t) DKL. (T( (ange) ll TMIP(( ((ree)))))
DKL. ((
ei) || TMTP( [Fa(ei,...., (e))))).,,,e-)))eee)))
The MTD loss, LMTD. can either be optimized directly in conjunction with the standard next-token loss LNLL. or can be calculated post-hoc using an MTP module trained with LMTP loss (see Sec- tion 4.2 For now, we ignore the optional access of the MTP module to the latest embedding ey; this
will be addressed below.
On the Difference between PHi and MTD While PHi introduced a powerful conceptual frame- work for analyzing model's internal processing. its implementation can be complex. The stochas- ticity introduced by the variational information bottleneck can also interfere with the main sequence prediction task. In contrast, MTD is significantly simpler to implement as it functions invasive auxiliary task; providing a more direct and less disruptive method to obtain similar insights into the model's per-token computational effort.
non-
One interpretation of the PHi loss is that it measures, at every step, changes of the "latent program" that the model synthesizes in-context to perform next-token prediction. The MTD loss, however, measures changes directly at the level of the output predictions. This distinction can lead to signif- icant differences. For instance, small change in the latent program could result in large shift in the output predictions. As an illustrative example, consider a model trained on two distinct types of sequences: one type consists of uniformly random tokens, while the other is a specific single, repeated token. To distinguish between these two cases, the latent program only needs to gain one bit of information. The PHi loss would therefore be low. However, the resulting change the output distribution is
from a uniform distribution to a one-hot distribution. In this scenario, the MTD can be as high as Dxz, (oneooooootoliiiie expect LMTD to be significantly higher than Lpm. an effect we observe in our experiments in Sec- tion 4. Conversely, one can imagine cases where the latent program changes significantly while the output predictions remain stable. However, the PHi training objective penalizes encoding such changes, as they do not sufficiently improve downstream predictions and thus represent an ineffi- cient use of the information bottleneck. Whether the change in the latent program is larger than the one in the output distributions or not depends on the exact weighting of the PHi loss during training.
= logg (vocabulary size) bits. such cases, we
Access to the Latest Token Embedding An interesting nuance for both PHi loss and MTD is that the measured information gain at step can originate from two distinct sources: (1) novel information contained within the current token 2, itself, and (2) complex computation performed by the model's main layers (Bg for PHi, F& for MTP), which cannot be easily approximated by the simpler prior or MTP module (M,). To disentangle these two sources, we can provide the prior/MTP module with direct access the latest token embedding et, which is a common practice in MTP models (see Equation 3). This effectively isolates the second source of information gain. For the PHi framework, this modification involves updating Equation T to concatenate the previous latent state with the current embedding:
G = by (
With this change. the PHi prior has access the same input token as the bottom layers, B3. Conse- quently, the modified PHi loss, which we denote as LeHi, isolates the information gain attributable solely to the computation performed by Bg:
(tn((t) = DK_ go (
J......, Ix) -
| 12....., e.)..
A PHi layer modified in this way acts as an information bottleneck that specifically measures com- putational effort. Information that the prior can easily extract from the input embedding e, allowed to pass freely, while information that is computationally non-trivial for Bg to extract is quantified by L.pn The same logic applies to the MTD module when it is given access the latest embedding.
4

---

Published as a conference paper at ICLR 2026
Geodesic
0.6
Probablty
84
Entopy
H. HIDI
Hinh
20-1
-1.0
-0.75
-0.5
40.25
0.0
6.25
0.5
0.75
1.0
Foxed Entropy
0.6
Pobabbiy
0.4
Entrap
1m0
-100
-0.75
-0.5
+0.25
0.0
0.25
0.5
0.75
1.0
Figure 2: Divergence Steering K=3 simplex with temperature curve for p. geodesic interpolation between from to p and beyond, and projection onto distributions with fixed entropy of (p).
Figure 3: Distributions corresponding to Figure Geodesic interpolation Sa. and the entropy of the resulting distribution (top). The same distribu- tions projected onto the surface with fixed entropy, & (bottom).
Arguably, PHi and MTD loss with access to the latest embedding provide better measure of dense in-context computation. This access allows the prior/MTP module to account for trivial shifts in the predictions-like the one described in Section[]-therels reducing the effective difference between the two metrics. In essence, providing access to the latest embedding allows information gain per step that is due to significant computational effort, whether measured in the latent space (PHi) in the output distribution (MTD).
quantify the
3.1 DECODING WITH DIVERGENCE STEERING
So far, we have presented MTD as post-hoc analysis tool. However, its formulation, based on the divergence between two output distributions, provides a mechanism to influence the model's behavior during generation. This allows us to steer the decoding process towards or away from tokens that the shallow MTP module can easily predict. This gives rise to a novel decoding method.
The core idea is to construct a new sampling distribution, 8a. by interpolating between the full model's prediction, æ, and the MTP module's prediction, TMTP. This is controlled by a single pa- rameter, For
O, recover the original distribution from the full model: 50 1, we use the distribution from the shallow MTP module: 81 = TMPP For a < 0, we ex- trapolate away from the MTP module's prediction. This amplifies the probability of tokens that are considered likely by the full model but unlikely by the shallow shortcut, effectively creating an "anti-speculative" distribution biased towards computationally intensive tokens.
For
a = 1,
To perform this interpolation in a principled way, we travel along the geodesic path between the two distributions under the Fisher-Rao metric. This is achieved by mapping the distributions onto the positive orthant of a hypersphere and performing spherical linear interpolation (Miyamoto et al., 2024). Let p and m TMIP be categorical distributions over vocabulary of size K. Their representations on the hypersphere are the square roots of their probabilities:
Pa (VPI, P2.
PK
m, (ymi, v Vm,..... ... m
The angle between these two vectors is θ arccos EE v V The geodesic path 8, (a) be- tween Py and m, is then given by:
sin (1 a)
sin(a
(a)
sin(O) P, sin(O) m,
5

---

Published as a conference paper at ICLR 2026
Model
xtd
PHi without lats ambeedinn with latest embedding
PHUNTOO los and complexiii level
0.50
Partial correlation bostwenn
Normllzed PHUNIT la3s
without embedding
#1.40 std
MTD with ltess embeddling
(contro ling
mewn
-0.25
Seq.
Memn Prog.
Copy
Complexity
Figure 4: Normalized PHi or MTD loss of the four different model types on each of the five tasks. Only in-context language learning (ICLL) requires sophisticated in-context computation. This is re- flected by the scores, with the exception of the MTD model without access to the latest embedding, which assigns high MTD also to the memorized programs task (see the discussion in Sections a.D. Bootstrapped mean with 95% confidence intervals across runs.
Figure 5: Partial correla- tion of PHi or MTD loss with the complexity of the modelled PFA, controlling for NLL. Also here, MTD without latest embedding access is the outlier.
To map this path back to valid probability distribution Sa. we square each component of the vector (a), i.e., (k((a) = (c)))
This method introduces a new control knob, a, which is complementary to the standard temperature parameter, While adjusts the entropy of the output distribution, adjusts its 'computational character." Because TMTP often has higher entropy than changing can also affect entropy. To isolate these effects, we can optionally project the interpolated distribution Sa to a new distribution &a such that its entropy matches that of the original distribution, i.e., H ( = H ( for all a. This provides two orthogonal levers for shaping the decoding process: T for entropy and a for computational density. Figures 2 and 3 visualize the method, additional details
be found in Appendix A As we will show, the optimal choice of a is task-dependent (which is also the case for T): some tasks benefit from the robust, simpler predictions favored by positive a, while others may require the novel, less obvious paths uncovered by negative a.
4 EXPERIMENTS
4.1 MTD AND PHI LOSS OF SEQUENCE MODELS TRAINED FROM SCRATCHI
The considerations from Section 3 leave us with four different model configurations to compare: PHi and MTD models, each with and without access to the latest token embedding. The PHi model without this access corresponds to the original method proposed in prior work (Herrmann et al.) 2025). To compare these different setups, we train Transformer models from scratch several tasks and evaluate them in settings similar to those (Herrmann et al., 2025). For details on exact training setups, please Appendix B.I
Evaluation on Different Tasks The four model types are trained on five different tasks: (1) reciting memorized sequences, (2) modeling sequences from a small of known formal lan- guages (memorized programs), (3) in-context language learning (ICLL), where the formal lan- guage is unknown (Akyürek al. 2024). (4) modeling random token sequences, and (5) copy- ing task that involves modcling random tokens where subsequences appear twice. Of these, only IChh-which requires inferring the structure of an unknown probabilistic finite automaton (PFA) in-context-involves meaningful computation, in the sense that non-trivial latent program must be synthesized by the model. Figure shows a comparison of the normalized PHi and MTD losses for each task. The MTD with latest embedding access shows the clearest distinction between the one complex task and the four "boring" ones; note the high value for ICLL and the consistently low values for all other tasks. For the MTD model without latest embedding access, effect alluded to Section 3 the loss is high for both ICLL and the memorized programs. For the memorized programs task, the actual in-context program required is minimal (only logz(10) bits
shows
the

---

Published as a conference paper at ICLR 2026
PH without istest embedding
PHI with latest emb edlding
MTD without test amb eodding
MTD with latest embedding
17.5
9.20
15.0
2.75
9.04-
2.50
3.00
2.23
10.0
2.30
3.22
0.22
3.50-
3.25
3.00
9.00
NEL Lons
NLL LOGS
NLL LOOE
NLL Loss
Figure 6: Token-wise PHi loss and MTD against binned NLL,, for the different modeled PFA com- plexities. PHi loss without and MTD with latest embedding access both show a clear correlation with complexity level, across NLL bins. See Figure (Appendix) for bin-wise normalized version.
to identify any one out of the ten memorized automata). However, the lack of information from the latest token causes significant shift in the model's output distribution, resulting in a high MTD. Finally, giving the PHi layer access to the latest embedding does not appear to improve its ability to distinguish "boring" from interesting tasks.
Task Complexity Focusing on the ICLL task, we investigate the relationship between the models' PHi or MTD losses and the complexity of the underlying language, measured by the description length of the PFA. Figure | displays the partial correlation between the mean PHi or MTD loss across sequence and the language's 8 complexity. We control for the mean NLL., as it is positively correlated with language complexity (r=0.367, 95% CI [0.315, 0.424)). Here again, we find that MTD with latest embedding access shows the strongest positive correlation (r=0.524 [0.480, 0.565]). In contrast, MTD without access to the latest embedding is negatively correlated with language complexity when controlling for NLL, confirming that it not reliable measure for this purpose. Figure [6] shows the token-wise PHi or MTD loss against binned NLL loss, broken down by PFA complexity (from simple, to 10, complex). This analysis reveals that only the original PHi loss (without embedding access) and the MTD loss with embedding access show a clear, positive token- wise relationship with language complexity after controlling for NI.I..
4.2 PRE-TRAINED LANGUAGE MODELS
To validate our hypotheses on existing large-scale models, we leverage the pre-trained, open-source MiMo-7B model (Xiaomi et al./ 2023). We choose it as a high-quality 7B parameter model, whose base pre-training incorporates an MTP objective, providing the built-in auxiliary prediction heads necessary for calculating the MTD without any post-hoc modification. Additionally, we use the Mistral 7B model (Jiang et al.) 2023). ) which provides no MTP module out-of-the- Instead we train MTP heads of different sizes by keeping the base model frozen and using standard teacher- student distillation (Schmidhuber 19926] Hinton et al., 2013) with a small fraction of the original data and compute (see Appendix [B.2p. For clarity, in the main part of the paper, report the results only for the MiMo model. The qualicatively very similar results for the Mistral models can be found in Appendix C.I
Reasoning Difficulty We employ the MATH dataset (Hendrycks et al. 2021b) which provides mathematics problems labeled from Level (easy) to Level 5 (hard), along with detailed reasoning solutions. We first compute the mean MTD for the provided step-by-step solution for each problem in the dataset and find that it clearly correlates with the difficulty level (r=0.179, 95% CI [0. 0.203I). Interestingly, the NLL loss negatively correlates with problem difficulty (r=-0.249 [-0.274, -0.224]. This suggests that from the model's perspective, reasoning chains for difficult problems are no less plausible or predictable. However, the higher MTD indicates that the model makes increased use of its full capacity to process and generate them. We also have the model generate ten different chains-of-thoughh (CoTs) for each problem and repeat the analysis on these self-generated solutions. There again, we observe very similar results: the partial correlation between MTD and difficulty level, controlling for NLL, is 0.999 [0.189, 0.208], while the correlation between NLL and difficulty is 0-15.1 [-0.168, -0. 149]. These effects hold consistently across most problem categories, as shown in Figures 10 and 11 (Appendix). Since the provided rationales, well as
7

---

Published as a conference paper at ICLR 2026
0.90
1.2h
8.43
1.30
0.300
0.3
8.35
0 0.40
0.50
0.155
6.25
8.45
0330
6.55
6015
CaTecT
1790
8.30
30%
90%
300%
57%
J0P
39%
(a) MTD vs. Difficulty
(b) NLL vs. Difficulty
(c) MTD vs. Correctness
(d) NLL vs. Correctness
Figure 7: Token-wise losses against relative positions in self-generated CoT for the MATH test dataset. MTD shows a clear correlation with difficulty across the full CoT (a), the relationship between NLL and difficulty is less clear (b). Similarly, correct CoTs show higher MTD over all relative positions (c), which is not the case for NLL (d).
the generated CoT', are longer for more difficult problems, the cumulative NLL also correlates positively with difficulty level (see Figures 12 and in the Appendix).
Token-wise Development Across the Response We track the token-wise values for MTD and NLL across each generated COT. As seen in Figure 7a the positive correlation between MTD and problem difficulty holds consistently from the first tokens of the response to the last. Likewise, the negative correlation for NLL persists throughout the generation, even though not as pronounced (Figure The difference between MTD and NLL in their correlation with the problem difficulty is notable because, global level, MTD and NLL are positively correlated with each other (r=0.255 [0.246, 0.265]). This highlights that MTD captures distinct signal related to computational effort that is not present in the standard NLL loss.
Reasoning Accuracy For the self-generated CoTs, we also investigate the relationship between MTD values and the correctness of the final answer. Figure[7c]poot the token-wise MTD, stratified by whether the rationale was correct or incorrect. We observe that correct responses are consistently associated with lower MTD. The relationship between NLL and correctness is less consistent (see Figure [d).. Following the methodology from prior work (Herrmann et al.] 2025). we randomly assemble pairs of one correct and incorrect CoT for each math problem. The probability of choosing the correct CoT when picking the one with the lower mean MTD is 67.1% (95% CI: [65.4%, 68.7%]). When selecting the one with the lower NLL, the probability is 73.3% [71.9%, 74.8%). For the cases where NLL and MTD are agree, we get 80.4% [78.5%, 81.9%] accuracy. We the GSM-SK dataset (Cobbe al. 202T), where we find that select- ing CoTs with lower MTD yields 66.0% [62.9%, 99.2%] correct answers, while lower NLL yields 72.2% [69.1%. 75.0%] and combined yields 75.5% [71.7%, 79.2%]. For token- wise curves, please see Appendix C. These findings stand in contrast to the results for PHi loss, where, for Ilama 3B model, correct answers are associated with a high PHi loss (Herrmann al., (025). While fur- ther investigation is needed,
repeat these experiments
hypothesize that different models may have different tendencies to either overly simplify or overly complicate their reasoning process (Sui et al.] 2025). This tendency could determine whether computationally intensive answers opposed to straightforward ones are more or less likely to be correct for a given model architecture or training regime.
4.3 DIVERGENCE STEERING AND CREATIVE TASKS
Having established MTD as an indicator of complex in-context computation, we now investigate whether we can use it to influence model generation. Specifically, can biasing generation towards tokens with high MTD lead to more complex or creative outputs? The Divergence Steering method allows us to test this hypothesis.
Algorithmic Toy Tasks We adopt the framework proposed by Nagarajan et al. (2025), training TTrnnfformer models four distinct tasks: sibling discovery, triangle discovery, circle construc-
8

---

Published as a conference paper at ICLR 2026
Sibling Discovery
Thangle Discovery
Circle Construction
Line Construction
14
0.7
12
0.6
0.00
9.6
0.5
0.0L
0.4
0.00
04
Figure 8: For the discovery tasks, positive leads to higher creativity, whereas for the construction tasks, negative ar leads higher creativity. Results for geodesic distributions Sa-
tion, and line construction (for details, please Appendix B.3 The objective for each task is to generate sequences that are simultaneously valid, novel (i.e., not memorized from the training set), and unique within fixed number of attempts. Success measured by creativity score, where indicates perfect performance across all three criteria. All models used in this experiment are config- ured with MTP modules that have access to the latest token embedding. Figure slowss the creativity scores across a range of values for temperature and our steering parameter, a. The results reveal a uusk-dependent effect. For the "discovery" tasks, positive values of a a--hich bias generation to- ward the simpler predictions of TMтP -yeld higher creativity scores. For the "construction" tasks, negative values of a-which create an muttiispeculatie" distribution biased away from lerp-llead to better performance. A more detailed analysis (Appendix C.2) suggests that positive helps the model avoid memorized solutions (improving novelty). whereas negative a can encourage the gen- eration of more structurally sound outputs (improving validity). The optimal strategy, therefore, depends on the specific demands of the task. Crucially, temperature and a function as largely in- dependent controls over the decoding process: for all four tasks, the best-performing combination of temperature and a achieves significantly higher creativity scores than optimizing for tempera- ture alone. The qualitative behavior is similar for both geodesic and fixed-entropy distributions (see Figure i6]in the Appendix).
Creative Writing Benchmark We also investigate how Divergence Steering can affect the cre- ative writing performance of the MiMo model. For the creative writing benchmark (Paech) 2025), the model generates several short stories following various prompts. An LLM judge scores the generated stories according to many different criteria, and also assigns main score, the "Overall Impression". From Table I we can see that a slightly negative steering parameter (a = 0.1) leads to the best results overall. Most criteria, including Overall Impression, reach the best scores when biasing tokens away from the MTP distribution via Divergence Steering. As we would expect, we get the least "Unsurprising and Uncreative" stories for negative a. For positive a, however, stories show fewer characteristics connected with overcomplication, such as "Purple Prose" and "Over- wrought". The experimental details and additional results can be found in Appendix [B.3 and C.3 These results show that shaping the computational character of the decoding distribution can have positive effects, especially for creative tasks.
DISCUSSION & FUTURE WORK
In our experiments, MTD outperforms PHi loss in differentiating "boring" from "interesting" tasks and simple from complex ones. It successfully isolates the per-token information gain attributable non-trivial, or "irreducible" (Wolfram) 2002) computation by the model. It should be noted that the utility of the MTD signal can be contingent on the relative capacities of the main model and the MTP module (M,): the MTP module is too powerful, MTD approaches zero, and if too weak. MTD offers little beyond the standard NLL loss. However, experiments with several differ- ent MTP sizes show robustness of the effects reported in this work (Appendix (C.I). Furthermore, MTD may conflate genuine computational effort with simple patterns that exceed the capacity of the lightweight shortcut.
Our findings also surface several intriguing questions. The positive correlation of MTD with prob- lem complexity, in direct contrast to the negative correlation of NLL, warrants further investigation to determine if this is general pattern across models and scales. Similarly, our result that lower

---

Published as a conference paper at ICLR 2026
Creative writing criteria which reach the best score for a given a value. For 10 different criteria, including Overall Impression, a = -0.1 gives the best results. In contrast, a O works best only for 6 criteria. Criteria followed by a down arrow (4) are negative, meaning that low score is best. Results for geodesic distributions Sn.
Table 1:
0.2
0.1
0.0
a 0.1
a 0.2
a 0.4
Unsupprsiing Uncrextive
Overall Impression.
Uncarned Transforma-
Reader
Overwought & (2) Perple Prose J
Engagemennt
Goons 4.
(2) Emotionaly
(2) Consistent
Incongmentt Ending
Complex,
Vaice/Tone of Wriling
Positisity
Emaiionaly
(3) Coberent,
Engaging.
Adlherence
Instructions,
Flow Naturally,
(5) Relievable
(5) Amateurish 4
Chiracter
Actions.
(6) Nusneed Characters.
(6) Meandasing
Imagery and Descriptive
Quality.
Flegrent Prose,
(9) Woll-carned 1 Laghtoss or Darkness.
(10) Weak Dialogue
MTD is associated with correct reasoning contrasts with prior findings for PHi loss, suggesting the relationship between computational effort and correctness is complex and model-dependent. While Divergence Steering enhanced performance on creative tasks, in preliminary experiments we found no clear improvement in the reasoning of large pre-trained models, perhaps because significant changes to the decoding strategy interfere with behaviors learned during post-training.
Our findings suggest that MTD and Divergence Steering have the potential for many applications in training and inference. Examples could be Dynamic Compute Allocation: MTD could be moni- tored in real-time during generation. A prolonged period of low MTD might trigger early stopping for a simple task, while a sudden spike in MTD could activate more powerful components (e.g., additional Mixture-of-Ex from a high-MTD processing phase to a low-MTD conclusion could act as signal that the model has "settled" on solution, potentially allowing for more efficient decoding. Intrinsic Motivation: In agent- based settings, MTD could serve as an intrinsic reward. This would encourage an agent pursue policies that lead to computationally interesting states (high information gain), fostering the development of more sophisticated behaviors. Open-Endedness: MTD and Divergence Steer- ing allows the filtering or direct generation of "interesting" data. This may help prevent model collapse when training on self-generated data and enable more creative, open-ended learning.
layers) for a difficull step. Solution Convergence: The transition
6 CONCLUSION
In this work, we introduce Multiple Token Divergence (MTD), a practical and direct measure for quantifying the computational effort of language models. By measuring information gain in the output distribution, MTD serves as more robust and stable metric than prior methods that rely on latent state compression. We show that giving the auxiliary prediction module access the latest token embedding allows MTD to specifically isolate the information gain attributable to non-trivial computation. Our findings demonstrate that MTD successfully distinguishes complex in-context reasoning from simpler tasks and reveals nuanced relationship between computational effort and predictive loss. As a non-invasive and casily implemented metric, MTD provides a valuable new introduce Divergence Steering, novel decoding
tool for analysis and evaluation. Furthermore, method that uses the MTD signal to actively steer the generation process towards either more or less computationally dense sequences. Shaping this "computational character" is complementary to the standard entropy adjustment using decoding temperature and can help with creative tasks.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All experiments can be reproduced using a single consumer-grade GPU (e.g., NVIDIA GeForce RTX 4090). Appendix B provides the necessary details for the exact replication of our experiments.
The weights of the pre-trained models used in this work, MiMo 7B (Xiaomi et al. 2025 and Mistral 7B (Jiang et al. 2023), are publicly available. Similarly. we employ open-source training and test datasets: Fine Web (Penedo et al. 2024), MATH (Hendrycks et al., (021a), and GSM-8K (Cobbe) al. 2021). The PI-A-related tasks (Section
(.1) follow the implementation in Herrmann et al. (2025), and the creative toy tasks (Section 4.3 are adopted from Nagarajan et al. (2025). both with no changes except the ones specified. The prompts used for the creative writing benchmark are also publicly available (Paech 2025).
REFERENCES
Ekin Akyürek, Bailin Wang. Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms. In Proc. Int. Conf. on Machine Learning (ICML.), Vienna, Austria, July 2024.
C. H. Bennett. Logical depth and physical complexity. In The Universal Turing Machine A Half Century Survey, pages 227-258. Oxford University Press, 1988.
Tom B Brown et al. Language models are few-shot learners. In Proc. Advances in Neural Information Pro- cessing Systems (NeurIPS), Virtual only, December 2020.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04335 2018.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401.10774, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plap- pert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training veri- fiers solve math word problems. ArXiv, abs/21 10.14168, 2021.
Ahhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshor,, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang. Aurélien Rodriguez, Austen Gregerson. Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis An- derson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evti- mov. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Ji- awen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua John- stun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Ken- neth Hcafield, Kevin Stone, and et al. The Ilama herd of models. CORR, abs/2407.2n 21783, 2024. URL
10. 48550/arx 2.w...24....2.2 21783
2407
Eric Elmoznino, Tom Marty, Tejas Kasetty, Léo Gagnon, Sarthak Mittal, Mahan Fathi, Dhanya Sridhar, and Guillaume Lajoie In-context learning and occam's 's razor. ArXiv, abs/2410.1 14086, 2024.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15706-15777 15734, PMLR, 21-27 Jul 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proc. Advances in Neural Information Processing Systems (NeurIPS), December 2021a.

---

Published as a conference paper at ICLR 2026
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021b.
Vincent Herrmann, Róhert Csordás, and Jürgen Schmidhuher. Measuring in-context computation complexity via hidden state prediction. In Forty-second International Conference on Machine Learning, 2025.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. stat, 1050:9, 2015.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier. Lélio Remard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2300.06825,, 2023.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. Int. Conf. on Learning Represen- tations (ICLR), Banff, AB, Canada, April 2014.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Sutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur- Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Proc. Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA, November 2022.
Ming Li, Paul Vitányi, et al. An introduction Kolmogorov complexity and its applications, volume 3. Springer, 2008.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng. Chenyu Zhang, Chong Ruan, et Deepseek-v3 technical report. arXiv preprint arXiv:2412.1945 2024.
Sachit Menon, David Blei, and Carl Vondrick. Forget-me-not! Contrastive critics for mitigating posterior collapse. In Uncertainty in Artificial Intelligence, pages 3370- PMLR, 2022.
Henrique K Miyamoto, Fábio CC Meneghetti, Julianma Pinele, and Sueli IR Costa. On closed-form expressions for the fisher-rao distance. Information Geometry, ((2):3113355 2024.
Vaishnavh Nagarajan, Chen Henry Wu, Charles Ding, and Aditi Raghunathan. Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction. In Forty-second International Conference on Machine Learning, 2025.
Samuel J Paech. Eq-bench creative writing benchmark v3.
https://github..om.
breative wwwting--beeeeen 2025.
Guilherme Penedo, Hynek Kydlfcek, Loubua Ben Allal, Thomas Wolf. Fineweb: decanting the web for the finest text data at scale. HuggingFace. Accessed: Jul, 12, 2024.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proc. Int. Conf. on Machine Learning (ICML), volume 32, pages 1278-1286, Beijing, China, June 2014.
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465-47,, 1978.
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative dynamic recurrent networks. Neural Computation, 4(1):131-139 1992a.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):330-277, 2010.
Jürgen Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992b. doi: 10. 162/neco. 1tw.11/6//cco.992.4222220 1992.4.2.33..
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz- Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty-second International Conference on Machine Learning. 2025.
RayJ J Solomonoff. A formal theory of inductive inference. part i. Information and control, 7()):122, 1964.
Yang Sui. Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint 25033 16419, 2025.
12

---

Published as a conference paper at ICLR 2026
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), 2015.
Paul M Vitányi. Meaningful information. IEEE Transactions Information Theory. 52(10), 2006.
Chris S Wallace and David M Boulton. An information measure for classification. The Computer Journal, 11 (2):185-194, 1968.
Stephen Wolfram. A new kind of science. 2002.
LLM-Core Xiaomi, :, Bingquan Xia, Bowen Shen, Cici, Dawei Zhu, Di Zhang, Gang Wang. Hailin Zhang, Huaqiu Liu, Jiebao Xiao, Jinhao Dong, Liang Zhao, Peidian Li, Peng Wang. Shihua Yu, Shimao Chen, Weikun Wang, Wenhan Ma, Xiangwei Deng, Yi Huang, Yifan Song. Zihan Jiang. Bowen Ye, Can Cai, Chenhong He. Dong Zhang. Duo Zhang, Guoan Wang. Hao Tian, Haochen Zhao, Heng Qu, Hongshen Xu, Jun Shi, Kainan Bao, Kai Fang. Kang Zhou, Kangyang Zhou, Lei Li, Menghang Zhu, Nuo Chen, Qiantong Wang, Shaohui Liu, Shicheng Li, Shuhao Gu, Shuhuai Ren, Shuo Liu, Sirui Deng. Weiji Zhuang, Weiwei Lv, Wenyu Yang, Xin Zhang, Xing Yong, Xing Zhang. Xingehen Song, Xinzhe Xu, Xu Wang, Yihan Yan, Yu Tu, Yuanyuan Tian, Yudong Wang, Yue Yu, Zhenru Lin, Zhichao Song, and Zihao Yue. Mimo: Unlocking the reasoning potential language model from pretraining to posttraining. 2025. URL https: /aaa://ara //arxivv... org/ htrg/abs/25500 2505 07608
13

---

Published as a conference paper at ICLR 2026
A FIXED ENTROPY PROJECTION FOR DIVERGENCE STEERING
To project the distribution sa the hypersurface with entropy H(p), solve the following optimization problem:
min $n
(Ba(8al|sa)
subject to H(3.)= Hp)
E..1=1
By solving the Lagrangian, we see that this is equivalent to finding a temperature-scaled version of 8a- This means that 8a, takes the form:
log 8a T
6.
softmax
for some temperature T, such that the entropy constraint H(.)) (p) is met. Since entropy is smooth monotonic function of the temperature, we use fast root-finding algorithm like binary search find the correct value for T'
practice, divergence steering, either with geodesic interpolation or this fixed-entropy projection, does not meaningfully slow down the generation process. For large vocabularies, however, it might be sensible use Divergence Steering in combination with top-k sampling and only optimize the remaining smaller distribution.
B EXPERIMENT DETAILS
B. SEQUENCE MODELS TRAINED FROM SCRATCH
We train all models using the Adam optimizer. batch size of 16 and gradient norm clipping of 1.0. The learning rate is 0.0003, with 500 step linear warm-up from zero and decay. All losses are weighted equally, for the PHli loss take the mean of the element-wise KL-Divergence for variation trained 8 times with different random seeds for the initial weights and the procedurally generated data (which results in different memorized sequenees and programs). The training of a model can be done on single consumer grade GPU (c.g., NVIDIA RTX 4090).
the sum. Every model
The base model is based on the Llama 3.2 architecture (Dubey et al. 2024
Number of layers: 12
Model dimensionality: 768
Number attention heads:
MLP intermediate size: 2048
Embedding layer and output head are tied
PHi models:
To prevent posterior collapse, we employ additional contrastive self-critic loss (Menon et al. 2022
Training steps: 30, 30,000
Placement of the PHi Layer: After the 10th layer
dimensionality: 768
ge Linear transform
ag: Linear transform
b Linear transform
Mμ.: One Transformer block like the ones the rest of the model
Pst Linear transform
MTD models:
Training steps: 10, 000
bw: Linear transform
M,.: One Transformer block like the ones the rest of the model
For generation of training and testing data, we follow Herrmann et al. 2025). The only difference is that we do not perturb any tokens during training, and that we use the same models for the task differentiation and task complexity experiments (Section
14

---

Published as a conference paper at ICLR 2026
B.2 PRE-TRAINED LANGUAGE MODELS
our experiments in Section 4.2 use the SFT version of the MiMo-7B model (Xiaomi calculate the MTD, we use the included MTP head that predicts one token advance.
2025). To
2023 We train For additional experiments, we use the instruction tuned Mistral 7B v0.3 model (Jiang et al. MTP modules of three different sizes: MTP small consists of one Transformer block, MTP medium of two blocks, and MTP large of three. They initialized using the weights from the last layers of the full model trained on data from the FineWeb (Penedo al. 2024), the MATH training set, and the GSM-8K training set mixture 80%, 10% and 10% respectively). We use MTD as the training loss. The MTP modules have access to the latest emhedding. We use the AdamW W optimizer, a batch size of 4 and train for 30,000 steps, with cosine annealing and a maximum lcarning rate of 0.0001. As shown in Table this can be done in few hours consumer-ger grade GPU.
experimental results include bootstrapped 95% confidence intervals.
B.3 DIVERGENCE STEERING AND CREATIVITY TASKS
Algorithmic Toy Tasks The MTD models use the architecture and training procedure specified in Sec- [..] For each task, a dedicated model is trained for 50, 000 steps. No seed conditioning is used. For task definitions and evaluation procedure, we refer to Nagarajan et al. (2025).
The creativity score is defined the fraction of all generated items that are valid, unique, and novel. In addition, define three more
Validity score: fraction of valid items among all generated items
Uniqueness score: fraction of unique items among valid generated items
Novelty score: fraction of novel items among valid unique generated items
These used the additional empirical analysis in section C.2
Creative Writing For the generation of the stories, use the prompting structure from the ative writing benehmark [Pacch 2025). Divergence steering for temperatures [0.6, 0.7, 0.8] and o-values is evaluated, with geodesic interpolation. For each setting, total of 96
[-0.4, -.2, -0.1, 0., 0.1, 0.4] stories is generated and judged. As a judge model, we use Claude Sonnet 4.5, Again we adhere to the prompting structure and criteria proposed by the benchmark. Every story is evaluated individually, no pairwise El.O-type scoring used. The detailed results shown in section
15

---

Published as a conference paper at ICLR 2026
PHi without latest embedding
PHi with latest embedding
MTD without latest embedding
MTD latest embedding
1.0-
2.6
2.6
9.6
Rumn led MED
Noma boo VTO Isee
Nomddool
0.4
2.4
24
0.0
009
2.0
2.0
2.0
NEL LOGS
2MLL Looe
JILL LOGE
XLLL Leus
Figure 9: Similar to Figure 6 but normalized for each NLL bin. PHi loss without access to the latest embedding, and MTD loss with access the latest embedding both show a clear correlation with complexity level, across NLL bins.
Table Comparison between MiMo and Mistral models terms of tokenizer vocabulary size, loss of the main model (NLL), loss of the MTP module (MTP), multiple token divergence (MTD), and training time. For Mistral models, larger MTP modules lead to reduced MTP Loss and MTD.
Metric
MiMo
Mistral MTP Small
Mistral
Mistral MTP Large
MTP Medium
Vocabulary Size
151936
32000
32000
32000
NLL
1.345
0.943
0.943
0.943
MTP Loss
1.425
1.298
1.217
1.178
MTD
0.469
0.553
0.476
0.429
RTX 4090 hours
9.38
10.25
11.08
C ADDITIONAL EXPERIMENTAL RESULTS
Figure D shows the normalized PHi losses and MTDs, binned by NLL and normalized, making clear to see that PHi without and MTD with access to the latest embedding show the clearest token-wise relationship with PFA complexity.
C.I DETAILED RESULTS FOR MIMO AND MISTRAL
In this section, we report detailed MTD results for the Mistral 7B models, in comparison with the MiMo model, for different sizes of the MTP module. The main awke that the results are consistent with the findings for MiMo and stable across the MTP module configurations. While qualitatively very similar, they are quantitatively less strong, which suggests that training model from scratch with an MTP module can have benefits.
Table 2 shows that, would expect, with larger MTP module size, the MTP loss and MTD significantly drop. However,
turns out, this does not affect the relationship between MTD reasoning difficulty and correctness. We also note that, while due to different tokenization, the losses between MiMo and Mistral models are not directly comparable. MiMo's MTP Loss is much closer to the full NLL, suggesting that its MTP module can predict better than even the large MTP module for Mistral.
From Table we Mistral models. both the provided solutions for ten self geffrgenratted CoTs to MATH test questions. consistently shows significant positive correlation between MTD and question difficulty (controlled for NLL) and negutive correlation between NLL and question difficulty (controlled for MTD). The differences in cor- relation strength for the three MTP module sizes fall within the confidence intervals. For the CoT's generated by Mistral, the partial correlations with difficulty are present and statistically significant, but less clear than for MiMo.
that the relationship hetween task difficulty and MTD is the for MiMo and
Table[[shooss the relationship between MTD and the correct reasoning: we measure the probability of picking the correct answer out of pair of random pair of generated CoTs, one correct, and the other one incorrect, when choosing based the specified criterion. We test the MiMo and Mistral models on the test splits of the MATH GSM-8k datasets. For models and both datascts, there a clear connection between low MTD and correctness. This connection goes beyond the relationship between NLL and correctness, which we can see from the fact that the chance of choosing correctly increases significantly if we pick response with lower NLL. and lower MTD. Also here, the difference between the Mistral MTP module sizes stay within the confidence intervals.
16

---

Published as a conference paper at ICLR 2026
Table Partial correlation coefficients r for the test questions of the MATH dataset between MTD and difficulty, controlled for NLI., and between NLL and difficulty, controlled for MTD. Results for the provided step-by-step solutions, and for the self-generated CoTs. Across all models, MTD correlates positively with difficulty, and NLL negatively.
Correlation Type
MiMo
Mistral MTP Small
Mistral
Mistral MTP Large
MTP Medium
Provided Solutions
MTD-Diff (ctrl NLL)
0.162 1o.137, 0.189]
0.109 pa079, 03188
0.094 10,055 0.122J
0.096 (0.066, 0.128)
NLL-Diff (ctrl MTD)
-0.239 60.266.00.112
-0.153 [-01800, 001127
-0. 143 [-.172 4.115)
-0.144 [-0.173, 0.115)
Generated CoT Solutions
MTD-Diff (etrl NLL.)
0.199 00.18))0.300)
0.050 p0041, 0.058)
0.044 jn.03s, nos3)
0.043 (0.054, 0.052]
NLL-Diff (ctrl MTD)
-0.205 L0.215.00.199
-0.041 [-0.050, -0033]
-0.047 [-0.0%, -0038)
-0.047 [-0,55, .0.038
Table 4: of the correct answer from pairs of CoTs, for different choosing strategies. Across all models, and for MATH and GSM-8k test questions, lower MTD CoTs are more likely be correct.
Criterion
MiMo
Mistral MTP Small
Mistral MTP Medium
Mistral MTP Large
MATH Dataset
Lower NLL
73.3% (481
59.1% 157.6.
59.1 % 157.6 60.61
59, 1% 1576.
Lower MTD
67.1% % (85.5.6855
57.2% (8.66
57.5% (56.0. 59.0] 62.7% (60.8 64.6]
57.8% (56.2,
Both Lower
80.4% (1.5.81.9]
63.0% [61.1.649]
62.7% (60.9.64 64.6]
GSM-8k Dataset
Lower NLL
72.2% 75211.752
64.9% [63.1,
64.9% [63.1. 66.7]
64.9% [63.1, 66.7)
Lower MTD
66.0% (62446693)
59.2% (L.))
61.8% [60.1. 63.5)
61.3% (59.6. 63.11
Both Lower
75.5% (71.7, 7921
69.9% (67.6.
71.4% (69.3, 73.5)
70.9% (68.8. 73.0)
17

---

Published as a conference paper at ICLR 2026
1.4
2.03
1.75
1.50
1.0
0.6
0.2
Nombet Thwary
Aipetca
Aipetera
Presigeteo
Geometty
Proslittio
Couning & Probasiity
Nombter Theer
Alpatea
Internndaaa Algntie
Presipebra
Geomenty
Precaicaaoo
Crunting Probabiiity
Mean MTD Loss
(b) Mean NI.. Loss
Figure 10: Mean losses of the MiMo model across the provided step-by-step solutions to the prob- lems of the MATH test set, grouped by category and difficulty level. MTD clearly grows with diffi- culty, suggesting that the model is making more of its computational capacity when processing more challenging problems. NLL loss, on the other hand, goes down with increasing complexity. Figure [Tows similar results for self- generated chains of thought.
0.200
0.175
0.4
0.125
203
0.100
0.2
0.075
0.020
Homber Theory
Alpetrra
Inoomentiiss Aigetire
Presigatora
Conomatry
Presliralii
Couninng Prodebbilly
Humber Theory
Ageeira
Inumneciiao Aipetra
Prealjetora
Gevenetry
Pecalcatop
Counting Pretuatits
(a) Mean MTD Loss
(b) Mean NLL Loss
Figure 11: Mean losses of the MiMo model across self- generated CoTs for the problems of the
MATH test set, grouped by category and difficulty level. Similarly as in Figure [0] we observe that MTD clearly grows with difficulty, the model is making when generating the solutions to more challenging problems. Also here, the mean NLL goes down with problem difficulty.
of its computational capacity
Additional Visualizations Figure Figure shows MTD and NLL for the provided step-by-step solutions for the MiMo model, broken down by category and difficulty level. Figure [T] shows the same for the self-generated CoTs. The results are qualitatively similar, even though the differences between categories for the CoT' are less pronounced.
Figures 12 and 13 use the cumulative instcad of the mean losses. Due to the fact that the provided solutions well as the generated ones grow in length the problems become more difficult, cumulative NLL also correlates positively with difficulty level.
Figure 14 shows the development MTD and NLL across self-generated CoTs for the problems of the GSM- 8k test dataset (analogous to Figures and [dr MATH), Correct CoT's clearly have lower MTD, and lower NLL. Interestingly, for the GSM-8k dataset, the shapes of the NLL curves differ significantly from the shapes of the MTD, missing the prominent initial bump. Currently, we have explanation for this.
18

---

Published as a conference paper at ICLR 2026
Duffcclty
Diffrclty
730
600
500
400
300
200
200
200
100
100
Homdee Theary
Algetra
leteminda Alpetor
Presipntia
demmentry
Creating &a Prolabilly
Romaee Therry
Algetira
Intemont do Alpetora
Presijetio
demmenty
Precatiuiaa
Crumnng & Prouabiity
(a) Cumulative MTD Loss
(b) Cumulative NLL Loss
Cumulative losses of the MiMo model across provided solutions from the MATH test set. Since more difficult problems have longer solutions, both cumulative MTD and cumulative NLL correlate with problem difficulty.
Figure 12:
1600
1400
400
A03
MYcutyy
200
Dificutt
200
Wonderr Theory
Alyobra
tereenat Alpebra
Predietrra
Presatistio
Chansiog Probatoiyy
Alpebra
Istorontiaa Alpetire
Presijetora
Genmvetry
Precatrobal
Cruming Probebilly
Nombeer
(a) Cumulative MTD Loss
(b) Cumulative NLL Loss
Figure 13: Cumulative losses of the MiMo model across self self- generated CoT' for the problems of
the MATH test set. We observe similar effect as in Figure 12
0.25
3.55
0.35
3.60
0.05
Tue
Pos
Reloiive
(a) MTD vs. Correctness
(b) NL.I. Correctness
Figure 14: Token-wise losses against relative positions in self self-generated CoTs by MiMo for the GSM-8k test dataset. Lower MTD and lower NLL are both associated with more correct reasoning.
19

---

Published as a conference paper at ICLR 2026
C.2 DIVERGENCE STEERING AND CREATIVE ALGORITHMIC TASKS
Figure 15 shows the creativity scores for the four tasks, using different values for temperature and a. In addition, break down the results into validity, uniqueness and novelty scores, By the nature of the task, sibling and triangle discovery models at risk of overfitting to the training data. positive value help avoiding repeating memorized examples, as can seen from increased novelty scores. circle and line construction, on the other hand, are less prone to overfitting, due to the combinatorial nature of the task. The novelty and uniqueness scores are consistently high. For these tasks, negative a appears to help construct inerease the validity scores.
models for
Figure 16 shows qualitatively very similar results for fixed entropy distributions.
Siblingg
Cirde Construcioon
Line Construction
6.7
0.03
Chee 2000
Creat wity
0.02
0.01
0.00
Sibling Discovery
Triangle Discovery
Circle Constraction
Line Construction
14
Vale 0200
Tumpo 100
0.8
0.02
0.50
Sibling Discovery
Triangle Discovery
Cirl
Line Construction
1.0000
1.00
1.00
0.9975
UPioueren Tww.TTE3--. score
0.9950
3.22
8.9925
0.9900
0.55
0375
0.80
0.24
0.4
Discovery
Circle Construction
Line Construction
0.5
0.9595
Nowe score
18
2.5
bowely
3.900
0.38
0.4
0.2
0.3000
Figure 15: Breakdown of the creativity scores into validity, uniqueness, and novelty. Positive a can improve novelty, negative can improve validity. Results for geodesic distributions Sa.
20

---

Published as a conference paper at ICLR 2026
Sibling Discovery
Trinngle Discevery
Crde Construction
Lie Construction
0.75
0.70
0.55
0.60
Creatiity
0.62
2.5
0.30
0.55
0.22
0.20
5.00
Sibling Discavery
Trinngle Discovery
Cirde Construction
Line Construction
3.0
14
0.26
0.40
0.34
0.35
0.30
0.6
0.25
0.50
0.20
0.35
Sib ligiigg Discovery
Iriangle Discovery
1.00
0.9375
0.9
Untoueters
0.22
Uniqunce 3001
0.38
10
0.30
0.26
89255
0.0000
0.8
0.52
0.34
0.5
Sbilig Discovery
Triangle Discovery
Circle Construction
Line Construction
12
0.6
Novely sore
0.3150
Novert score
10
0.94
0,5590
0.5
0.5
0.96
Figure 16: Breakdown of the creativity scores into validity, uniqueness, and novelty. Positive improve novelty, negative a can improve validity. Results for fixed entropy distributions &a.
C.3 DIVERGENCE STEERING AND CREATIVE WRITING
Here we report detailed results for different divergence steering settings the creative writing bench- mark (Pacch) 2025). For cach temperature-a configuration, MiMo generates 96 stories following a range of pre-detined writing prompts. An LLM judge writes assessment and rates each story according 21 differ- ent criteria, including the aggregate "Overall Impression", which show the relationship between temperature, and the different scores. For many criteria, including the overall impression, there is a connection between good scores and slightly negative a. This means that steering away from the "obvious" tokens predicted by the MTP module can lead to better writing performance. On the other hand, some negative criteria connected with excess complexity, such "Overwrought" and "Purple Prose" tend increase with negative a.
the main
Figures [7 and [8]
To establish statistical significance, we perform quadratic regression analysis on the Overall Impression scores. We find that the optimal a is negative and reject the null hypothesis (that a 0 works best) with < 0.01.
DECLARATION OF LLM USAGE
We used Gemini 2.5 and 3.0 to proofread and occasionally to improve sentence flow and wording.
21

---

Published as a conference paper at ICLR 2026
0.5
03
8.3
-.6
3.3
-0.4
-02
6.0
3.4
Emnotonnll Compac
5.0
Ovtall bascer
4.5
12
3.3
-35
3.3
7.5
6.0
0.0
Rcheertce intrrttoss
7.0
0.3
6.0
4.5
6.0
Oo13 Stank
55
02
62
64
RMd car of Uptonsa or Cur 0022
2.3
6.0
5.5
9.0
-0.0
-0.2
0.3
5.2
0.4
-0.6
-0.2
0.0
0.2
0.4
-0.6
-3.
0.0
0.2
-04
-02
5.0
0.2
54
Figure 17: Creative writing scores for positive criteria, for different values of temperature and a. Higher value means better. At the top, the aggregate 'Overall Impression" is shown. For many criteria, including Overall Impression, we observe the highest scores for a = -0.1.
16.5
12.5
23.0
W
16.0
$2.5
12.0
11.6
32.0
18.
0.6
105
0.4
-32
6.2
04
-04
-22
0.
02
-22
0.2
Endre bustivyy
13.5
20.5
-38
15.0
88.0
13
Warddsho
12
12.0
Chaaree
8.0
12
12.0
-0.4
-0.2
0.0
6.2
0.4
-0.4
-3.2
0.0
0.2
0.4
-0.4
-3.2
0.0
0.2
0.4
-0.6
-02
0.0
0.2
0.4
Figure 18: Creative writing scores for negative criteria (lower means better).
22


Reviews:
[{"review_id": "gWd8zZEQyI", "reviewer": "Reviewer_5KTN", "summary": "This paper introduces Multiple Token Divergence (MTD), an alternative to PHi for measuring computational effort by computing the KL divergence between a model's output distribution and that of a shallow auxiliary prediction head. Based on this, the authors propose Divergence Steering, a decoding method that uses MTD to control the \"computational character\" of generated text.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clear and well-motivated: the paper addresses limitations of PHi with a simpler, non-invasive method.\n2. The evaluation is fairly comprehensive, covering both pre-trained models as well as those trained from scratch. While the model may require some adaptation (if it doesn't have an MTP head), it's less invasive than PHi.\n3. MTD shows good correlation with task complexity and difficulty (e.g., on MATH). Furthermore, it's effective for CoT rationale selection when combined with NLL, giving MTD practical use.", "weaknesses": "1. The paper relies heavily on informal notions of concepts like \"complexity\", as well as \"interesting\" and \"boring\" tasks, instead of formal definitions.\n2. The authors only focus on one \"interesting\" task in Section 3.1. It would be interesting to see if these results generalize to other \"interesting\" tasks.\n3. Section 4.2 is missing a direction comparison to PHi for pre-trained models. Looking at the PHi paper, they report lower correlation with reasoning difficulty, but this could be due to other confounding factors (e.g., different pre-trained model). A direct comparison would be valuable here.\n4. Likewise, more work is needed to understand the role of Divergence Steering with pre-trained models. Without such analysis, the impact of Divergence Steering is diminished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIDzo7rCGk", "reviewer": "Reviewer_RBiN", "summary": "This paper introduces Multiple Token Divergence (MTD), a metric designed to measure the in-context computational effort of a language model at each token. The authors propose that standard metrics like next-token loss fail to capture the complexity of the reasoning required to produce a token. Their method addresses the shortcomings of prior work (like the PHi loss).\n\nMTD is defined as the KL divergence between the full model's next-token output distribution and the distribution of a shallow, auxiliary prediction head (an MTP module). The core intuition is that if a \"shortcut\" shallow model can easily predict the same token as the full, deep model, the full model isn't performing complex computation. A large divergence, however, implies the deeper layers are performing non-trivial work. A key refinement is to provide the shallow MTP module with access to the current token embedding, which helps MTD isolate computational effort from the simple information gain of the new token.\n\nThe authors also introduce Divergence Steering, a decoding technique that uses the MTD signal to control the \"computational character\" of the generated text by interpolating between the full and shallow models' distributions. This can bias generation to be more \"anti-speculative\" (favoring computationally intensive tokens).", "questions": "The finding that lower MTD correlates with correct reasoning (sec 4.2, fig 7c) is one of the most interesting results of the paper. It seems to contrast with the initial motivation of MTD as a measure of \"sophisticated\" or \"non-trivial\" computation, which you might associate with successful reasoning.\nCould you elaborate on this relationship? Does this finding imply that high MTD is a signal of inefficient or failed computation (like, the model is \"struggling\" or \"thrashing\" on its way to a wrong answer), rather than a signal of successful, deep reasoning? How do you handle this with the fact that high MTD also correlates with overall problem difficulty?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's most fascinating result (sec 4.2) is the decoupling of computational effort (MTD) from predictive plausibility (NLL). The finding that MTD correlates positively with MATH problem difficulty while NLL correlates negatively is a significant contribution.\n- The discovery that MTD and NLL are anti-correlated with respect to problem difficulty is very interesting. It provides a new, orthogonal axis for analyzing model behavior. NLL measures \"plausibility\" or \"surprise,\" while MTD measures \"effort\". \n- The use of geodesic interpolation (via the Fisher-Rao metric) to navigate between the two output distributions ($\\pi$ and $\\pi_{MTP}$) is a principled and technically sophisticated approach, avoiding the pitfalls of a naive linear or log-linear interpolation.", "weaknesses": "- The paper's discussion briefly notes that MTD may \"entangle genuine computational effort with memorization\". This is a problem because the shallow MTP head is likely trained to be very good at predicting common, high-frequency (like, memorized) n-grams. A high MTD might simply signal that the full model is generating a novel or rare sequence (for example, a specific fact or a unique turn of phrase) that the shallow head couldn't possibly predict, which is not necessarily the same as in-context computation.\n- In section 4.2. MTD is explained as a measure of \"sophisticated in-context computation\", but the experiments find that lower MTD correlates with correct reasoning. This seems contradictory: if high MTD means high computational effort, but low MTD means correctness, it suggests that high MTD might actually be a signal of the model \"struggling\" or engaging in inefficient/failed computation, rather than successful, deep reasoning.\n- The MTD metric is strongly relative, because it measures the divergence between a full model and a specific shallow MTP head. The paper's results are contingent on the architecture of the MTP head used. If the MTP head were slightly more powerful or weaker, how would the MTD values and their correlations change?\n- The main LLM experiments are carried out on a single 7B-parameter model (MiMo-7B). While this model is well-suited for the study, it remains unclear if these findings (especially the crucial MTD-difficulty and MTD-correctness correlations) generalize to other model architectures.\n-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ExY467qZ9T", "reviewer": "Reviewer_Rw1u", "summary": "This paper introduces Multiple Token Divergence (MTD), a novel and lightweight metric for measuring the in-context computational effort of language models. MTD is defined as the KL divergence between the output distribution of the full model and that of a shallow, auxiliary prediction head. The core intuition is that a large divergence implies the full model is performing complex computations that cannot be approximated by a simple shortcut. The authors demonstrate that MTD can be computed from existing pre-trained models with Multiple Token Prediction (MTP) heads without requiring any new training. They also propose Divergence Steering, a decoding method that uses MTD to control the \"computational character\" of generated text by interpolating between the full and shallow output distributions. Empirical results show MTD is a more effective measure of task complexity than prior methods like Prediction of Hidden States (PHi). On mathematical reasoning, MTD correlates positively with problem difficulty, and lower MTD values are associated with more accurate reasoning.", "questions": "1. The failure of Divergence Steering on reasoning tasks is a critical point. Could you elaborate on your hypothesis for why this occurs? Does it suggest that post-training (e.g., SFT, RLHF) makes a model's generation process too brittle for this kind of decoding intervention, or is MTD fundamentally measuring a property that is not directly aligned with generating a correct reasoning path?\n\n2. Your finding that lower MTD correlates with correctness is intriguing and contrasts with prior PHi results. Beyond model-dependency, could it be that MTD is capturing a different aspect of computation, such as \"processing efficiency\" or \"convergence on a solution,\" where lower divergence indicates a more confident, streamlined path?\n\n3. For models without a pre-existing MTP head, what is the practical cost (in data and compute) of training a new head via distillation? How does this compare to the cost of implementing and training a full PHi model, and does it maintain a significant advantage in terms of simplicity and stability?\n\n4. Given that the optimal steering direction (α) is task-dependent, how might a practitioner apply Divergence Steering to a novel task where this direction is unknown? Have you explored any methods for dynamically adapting α during generation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Simplicity and Practicality:** The primary strength of MTD is its simplicity compared to prior methods like PHi. It avoids invasive architectural changes, complex loss functions, and unstable training. The ability to compute it post-hoc on models already equipped with MTP heads (like MiMo-7B) makes it a highly practical tool for analysis.\n\n**Novel Decoding Method:** Divergence Steering is a genuinely new mechanism for controlling generation. It introduces a steering parameter, α, that is conceptually orthogonal to temperature—controlling \"computational character\" rather than just entropy. The use of geodesic interpolation for this process is principled and elegant.\n\n**Strong Empirical Analysis:** The paper provides a robust empirical validation of MTD as a metric. The head-to-head comparison with PHi on synthetic tasks clearly demonstrates that MTD (with latest embedding access) provides a cleaner signal of computational complexity . The findings on the MATH dataset—that MTD correlates positively with difficulty while NLL correlates negatively—are insightful and effectively decouple the concepts of model surprise and computational effort.\n\n**Isolating Computational Effort:** The paper makes a key contribution by showing that allowing the auxiliary head access to the latest token embedding enables the MTD metric to specifically isolate information gain attributable to non-trivial computation, rather than just novel information in the current token.", "weaknesses": "**Limited Impact of Steering on Reasoning:** The most significant weakness is the acknowledged failure of Divergence Steering to improve performance on the core reasoning tasks analyzed. While the method shows interesting, task-dependent effects on toy creative problems, its inability to enhance mathematical reasoning in pre-trained models severely limits the practical impact of the paper's second main contribution.\n\n**Ambiguous Interpretation of the MTD Signal:** The paper finds that lower MTD is associated with correct reasoning on MATH and GSM-8k . This is in direct contrast to prior work where higher PHi loss was linked to correctness. The authors' hypothesis that this is model-dependent is plausible but leaves the interpretation of the MTD signal ambiguous. It is unclear whether high MTD should be interpreted as \"deep, productive thinking\" or \"inefficient, confused thrashing.\" This ambiguity complicates its use as a straightforward optimization target.\n\n**Conditional \"No-Training\" Claim:** The claim that MTD requires no additional training is conditional on using a model that already has an MTP head. Most widely-used open-source models do not. The proposed workaround—training a head via distillation —reintroduces the training complexity and cost that MTD is positioned as an alternative to, weakening one of its main selling points.\n\n**Post-Hoc Steering Direction:** In the creative tasks, the optimal steering direction (positive vs. negative α) depends on whether the task is \"discovery\" or \"construction\". This is an interesting finding, but the paper offers no method to determine the correct direction for a new, arbitrary task a priori, making the steering method difficult to apply in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)