The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?
Authors: Alexander Hägele
Aryo Pradipta Gema
Henry Sleight
Ethan Perez
Jascha Sohl-Dickstein
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Scaling Laws
Misalignment
Bias-Variance
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose decomposing AI error with the bias and variance framework, and analyze scaling behavior across task complexity and model scale.

Abstract: As AI becomes more capable, we entrust it with more general and consequential tasks. The risks from failure grow more severe with increasing task scope. It is therefore important to understand 
the ways extremely capable AI models will fail: Will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess, and taking nonsensical actions that do not further any goal? We operationalize this question using a bias-variance decomposition of the errors made by AI models: An AI's *error incoherence* on a task is measured over test-time randomness as the fraction of its error that stems from variance rather than bias in task outcome. Across all tasks and frontier models we measure, we find that the longer models spend reasoning and taking actions, *the more incoherent* their failures become. We observe that error incoherence changes with model scale in a way that is task and experiment dependent. However, in several settings larger, more capable models are more incoherent than smaller models.
Consequently, scale alone seems unlikely to eliminate incoherence. Instead, as more capable AIs pursue harder tasks, requiring more sequential action and thought, our results predict failures to be accompanied by more incoherent behavior. 
This suggests a future where AIs sometimes cause industrial accidents (due to unpredictable misbehavior), but are less likely to exhibit consistent pursuit of a misaligned goal. 
This increases the relative importance of alignment research targeting reward hacking or goal misspecification.

Published as a conference paper at ICLR 2026
THE HOT MESS OF AI: HOW DOES MISALIGNMENT SCALE WITH MODEL INTELLIGENCE AND TASK COMPLEXITY?
Alexander Hägele*1.2
Aryo Pradipta Gema
Henry Sleight
Ethan Perez
Jascha Sohl-Dickstein
Anthropic Fellows Program EPPL. University of Edinburgh CContelllatonn 6 Anthropic alexander hagele@epfl. ch, jaschalanthropic.oo
ABSTRACT
As AI becomes more capable, we entrust it with more general and consequential tasks. The risks from failure grow more severe with increasing task scope. It is therefore important to understand how extremely capable Al models will fail: Will they fail by systematically pursuing goals do not intend? Or will they fail by being a hot mess, and taking nonsensical actions that do not further any goal? We operationalize this question using bias-variance decomposition of the errors made by AI models: An Al's error-ince inchhreneee on a task is measured over test- time randomness the fraction of its error that stems from variance rather than bias in task outcome. Across all tasks and frontier models we measure, the longer models spend reasoning and taking actions, the more incoherent their failures be- come. Error-incoherene dependent. However, in several settings, larger, more capable models are more in- coherent than smaller models. Consequently, scale alone seems unlikely to elim- inate error-incoherence. Instead, as more capable Als pursue harder tasks, requir- ing more sequential action and thought, our results predict failures to be accom- panied by more incoherent behavior. This suggests a future where AIs sometimes cause industrial accidents (due to unpredictable mishehavior), but are less likely to exhibit consistent pursuit of a misaligned goal. This increases the relative im- portance of alignment research targeting reward hacking goal misspecification.
changes with model scale in a way that is experiment
hot tets-mess-ffae
hot-mess -data
INTRODUCTION
There increasing number of predictions that AI will soon be more capable than human beings (Kwa et al., 2025; Maslej et al., 2025; Pimpale et al., 2025), and will replace human labor in many domains (Chen al., 2025b; Handa et al., 2025; Dominski & Lee, 2025; Eloundou al., 2024; Johnston & Makridis, 2025), We already rely on AI for consequential tasks such as writing critical software (DeepMind, 2025; Appel et al., 2025), determining bail amounts (Fine al., 2025), and deciding what stories to present in news feeds (Liu et al., 2024; Gao et al., 2024b; Yada & Yamana, 2025). Despite its increasing capabilities, AI often behaves in ways we do intend. Due to its high-stakes use cases, it is important to understand how and when Al can be expected to fail.
One class of AI risk is misalignment risk (Bostrom, 2014; Russell, 2019; Greenblatt et al., 2024). Misalignment risk is the concern that AI will pursue goal that is different from the goal its creators intended to instill, and that it will pursue that goal with superhuman competence. a superhuman agent pursues misaligned goal, it might do things like seize power instrumental step to achieving its goal (Hubinger et al., 2019).
However, this scenario assumes that unintended behavior stems from systems that not only pursue the wrong objective, but remain coherent optimizers over a long horizon. Large language models (LLMs), prior to reinforcement learning, dynamical systems, but not optimizers. They have to be trained to act as an optimizer, and trained to align with human intent. It is not clear which of these trained properties will tend to be more robust, and which will be most likely cause failures

---

Published as a conference paper at ICLR 2026
Mocils Utten Fail in Incousistenl Ways
Prosuct Diseniangling Modal Ervor Into Biss anc. Variance
Pleaer implonmnt
"AI ayeate fend booese wose signalietclle die loger comsrrsation
Question
Error
Blas?
Varianse
fullowing Scatiat:
mncer plan
Ansove:
O
Incoberence
Variance
Errur
Biar Dieimall
Periicse Haciisaue
Solid..
swtwlly 3r dhe wker llley
Lae irploanted the isllvieg Leport on
The suvver A.
Thersfre le
Dhadioc is The suneer is 13
package
TTee snawer to
Loots passed.
the
Modole Berome Inrokement They Perform Honder Tusbs
Lunger Models Booome More Coherent Only Ensy Tanks
Random Grucssing
Madil Hot Mess
Mucki
mutasbu are Lard io probil
besherove Vartanon
Birur
Clonpinddy
Model Kroot
Our Wessuls
Our Brauits
Mediiam
Hardest
Ticls
Ticks
Tiala
Modsl conssstratly Systematic M Misalignment
Murtttee look shit Simple Answees
Supercobereat
Hot Moos
Task Complexity
Hilly Tnessherence
Vartonce
Brnoe
Figure 1: AI can fail because it is misaligned, and produces consistent but undesired outcomes, or because incoherent, and does not produce consistent outcomes all. These failures correspond to bias and variance respectively. As we extrapolate risks from AI, it is important to understand whether failures from more capable models performing more complex tasks will be bias or variance dominated. Bias dominated failures will look like model misalignment, while variance dominated failures will resemble industrial accidents. (top left) Qualitatively, observe that AI models fail in unpredictable and inconsistent ways. Often, these failures can be fixed by resampling. (top right) To quantify this observation, we decompose errors made by AI into terms, bias and variance. We illustrate this using a multiple choice task: bias is the tendency to pick specific incorrect answer; variance is the tendency to pick inconsistenly among options. We define error-incoherence Experimentally, we find that as models reason longer and take more sequential actions, they become more incoherent. (lower right) We find that as models become more capable, and overall error rate drops, arror-nncohrenne changes in way that depends on task difficulty. Easy tasks become less incoherent, while hard tasks trend towards increasing prorrincohherece.
the fraction of model error caused by variance. (lower left)
in superhuman systems. In practice, AI models often fail in ways that seem random and do not further any coherent goal (Spiess, 2025; Nolan, 2025). Like humans, when Als act undesirably, often because they a hot mess and do not act in way that is consistent with any goal: The hot mess theory of intelligence Sohh-Dickstein, 2023) suggests that as entities become more intelligent, their behavior tends become more incoherent, and less well described through single goal. If true for AI systems, this shifts both the likelihood and the focus misalignment scenarios.
In this paper, therefore ask the questions: When model does something other than what intend, what fruction of its deviation is due to bias (consistent pursuit of the wrong goal), and what fraction to variance (randomness in behavior and outcome)? As we scale model intelligence and task complexity, how does this decomposition change? Asymptotically, as extremely capable models perform extremely complex tasks, which class of undesired behavior will dominate?
We address these questions by measuring the scaling behavior of Al errors decomposed into
ERROR = BIAS + VARIANCE,
and further define emro-inncoereence as the proportion of variance to the total error. This decom- position allows us to distinguish the relative contributions of different types of Al failure, and, importantly, how they change models become more intelligent and perform longer horizon tasks. Bias-dominated failures correspond to systematic misalignment-cconiicn pursuit of the wrong wheeeass whereas variance-dominan failures indicate inconsistent outcomes.
2

---

Published as a conference paper at ICLR 2026
We find that across multiple-choice benchmarks, agentic coding, and safety tasks, models become more incoherent with longer reasoning (Fig. 2), even when controlling for task difficulty (Fig. 3). Larger, more capable models are often more incoherent (Fig. 4): while they achieve lower error, they grow more coherent on easy tasks but less coherent on hard tasks (Fig. 5). We validate these findings in synthetic environment where variance asymptotically dominates with increasing model size (Fig. 6), and find that ensembling and larger reasoning budgets reduce error-incoherence (Fig. 7). We discuss our results in Section 5.
BACKGROUND
2.1 BIAS-VARIANCE DECOMPOSITION
Definition, In supervised settings, the bias-variance decomposition expresses the expected error of predictor the sum of three terms: BIAS?, VARIANCE, and irreducible noise (Kohavi & Wolpert, 1996). Although originally formulated for regression, analogous decompositions exist for classification tasks (Kohavi & Wolpert, 1996; Domingos, 2000), with similar interpretation: the bias reflects the error of the classifier's mean or mode prediction and variance quantifies its deviation. Several such decompositions exist, including the 0/1 error (Kong & Dietterich, 1995; Breiman, 1996; Kohavi & Wolpert, 1996; Tibshirani, 1996; Friedman, 1997: Domingos, 2000), Brier score (Degroot Fienberg, 2018), and cross-entropy error (Heskes, 1998). We present Kullback- Leibler (KI.) decomposition in the main text. For additional definitions see Appx. A. We ran experiments with KL, Brier, and 0/1 formulations. All three decompositions produce qualitatively similar results, and we provide plots for all three in appendices.
Let be the input with label classes c {l,...,C] C} for which the model fa produces a probability distribution (potentially one-hot) over class labels fz) RC, with denoting the stochasticity of the training process. The target is one-ho hot encoded through y((z) E RCC For clarity, we omit the dependence of and fz on z. We assume the irreducible noise to be O. Then, the expected cross-entropy error can be decomposed into (Yang et al., 2020):
Ee (CE(U, EC
Σ vld log(Sele)) = DKL (ull) + B. (FlLee]
(1)
ERROR
BLAS"
VARIANCE
where y[c] denotes the e-th element of the vector, DKL is the Bullback-Leibler divergence, and f is the average of log-probabilities after normalization: f[c] x exp (Eg (lo((f(c))) for ..... C. We denote this decomposition as KL-BIAS and KL-VARIANCE, This is an instance of the general decomposition for Bregman Divergences (Pfau, 2013).
Different usage to classical literature. In discussions of the bias-variance tradeoff, the setup typi- cally assumes deterministic model (e.g., regressor), with bias and variance estimated by retraining under different seeds or data sampling. That means the expectation is over training randomness e. Our setting differs: rather than retraining multiple models, we analyze fixed model and take the expectation over input (e.g., few-shots) and output (sampling) randomness for the same task.
Error-incoherence. Throughout this paper, our main metric of interest is the proportion of the variance to the total error, which we define as INCOORRNCCEEEECE.. Formally, consider a set of questions Q {gi Jis and model fe. We then denote error-incoheeen as
ERROR- SOOORINNCCHEEEEEQ...... = Σ, VARIANCE(Q;, fe)
(2)
E, ERROR (g:, fe)
VARIANCE(g,, fe), this metric is relative value in [0, I]: a value of means that the model never deviates from its average behavior and any error will be consistent; value of 1 means that every eπооr the model makes is inconsistent. Importantly, model can achieve a lower overall error rate, but have a higher iroorincohereenc, which makes it a comparable measure across error levels and model capabilities. We see such cases in Section 3.
Since ERROR fe) BIAS(
2.2 SCALING BEHAVIOR OF LARGE LANGUAGE MODELS
Scaling laws. Model performance generally follows predictable power-law scaling with respect to model size N. dataset size D, and compute C (Kaplan al., 2020; Hoffmann et al., 2022). Most
3

---

For open-ended MWE safety questions, embed solely the answers (i.e., without reasoning Consequently, we re- Published as a conference paper at ICLR 2026
prominently, taking the parameters N as argument, the cross-entropy loss broadly behaves as I(N) oc N for some exponent a. This slope a informs us about the rate of improvement. In Section 3.2 will compute scaling laws independently for bias and variance loss contributions, to judge which asymptotically dominates.
Reasoning and inference compute. Besides the model and dataset size, the most promising recent development uses inference compute as an axis of scale. Specifically, so-called reasoning models trained with reinforcement learning (RL) to think in long chains of thought before providing an answer, which improves performance with larger thinking budgets (Snell et al., 2025; Jaech et al., 2024: Guo al., 2025: Anthropic, 2025b; OpenAI, 2025a; Team, 2025a; Team et al., 2025; Chen et al., 2025a; Zhong et al., 2024; Muennighoff et al., 2025). The length of reasoning is an important aspect of our analysis, which we see as a process of sequential action steps (Lightman et al., 2023).
3 EXPERIMENTS
Overview. We present our results grouped by observations: first, growing incoherence function of reasoning length (3.1) and scaling laws with model scale (3.2); this is followed by the effects of reasoning budgets and ensembling (3.3). The details of all experimental setups in Appx. B.
Tasks. We run experiments on the following tasks, which all have well-defined targets used for inco- herence measurements, since bias only defined relative a target. For discussion, see Section 5.
Multiple Choice Tasks. We the popular scientific reasoning benchmark GPQA (Rein al., 2024), and general knowledge benchmark MMLU (Hendrycks et al., 2021). Target responses are simply the correct answer.
Agentic Coding. This focuses on SWE-BENCK (Jimenez et al., 2024), where agents solve GitHub issues using tools, and success is measured with unit tests.
Safety and Alignment. We assess models using the advanced Al risk subset of Model-Written Evals (MWE; Perez et al., 2023), both with the original multiple choices and in an open-ended format with answer options removed.
Synthetic Settings. We train transformers of varying scales to directly emulate an optimizer descending an ill-conditioned quadratic loss. The transformer is tasked with predicting string representations of optimizer update steps based on the current state. This is simple toy model of an LLM that has been trained to act as an optimizer. See Section 3.2.2 for details.
Survey. In addition experiments using LLMs, we report the survey results of Sohl-Dickstein (2023) (previously released in blog form), where disjoint sets of human subjects subjec- tively ranked the intelligence and coherence of Al models, humans, non-human beings, and organizations. The details are provided in Appx. B.5.
Setup and Metrics. Across all tasks, unless otherwise noted, we obtain at least 30 samples to esti- mate bias and variance per question. We find this sample count be sufficient for stable estimates (see Appx. C.5 and B). Each sample is run with different seed for autoregressive generation. For GPQA and MMLU, samples additionally use different random few-shot context. We report the following metrics (details in Appx. A and B):
For multiple choice questions, our main metric of interest is the KLLFERROR-NCCHHRRFNNTC i.e., the error-incoherence with respect to KI.-BIAS and KI- VARVVARI (Equations and 2). We find the same qualitative behavior for other decompositions, as reported in Appx. C.I.
chains) using a text embedding model -ett---eeediiggg---eee- port the variance of the embedding vectors in the Euclidean norm.
For
assign binary vectors for each sample and task: each vector is of size Ti the number of unit tests for task i, and encodes which tests a model's code passes. The coverage error then computes the mean squared difference to vector of all 1's, which we decompose into bias and variance contributions.
Models. We evaluate the following frontier models: SONNET 4 (Anthropic, 2025a) with reasoning enabled, 03-MINI (OpenAI, 2025a), and 04-MINI (OpenAI, 2025b). When analyzing scaling w.nt. model size imperfect proxy for intelligence, use the QWEN3 model family with thinking
4

---

Published as a conference paper at ICLR 2026
Inocheresee; Fronrier Models, GPQA
Incoberance: Qwend. OPQA
Incohereace: Coverage
RL Inoobeeooo (Venes)) )
1,00
Model
Incobercuse (Tation
1.00
Morkel
1.00
Model
Ca heae jonka
bix (ttie) whan gl- bix (ttico wawd (77
0.76
busts pluers
0.75
the
0.50
Incobereuoe
0125
0.
0.25
0.00
0.30
6.00
10
10
10a
10a
101
10P
Agg. Reasening Lengta (Toke
Avg. Reasoning Length (Tokems)
Avg. Number of
(a)
GPQA
(b) SWE-SEEEEE
MIQ Elansali Repurtall Servvol lveliet
Open-trdee Sob-lipente: sumtal zuatod
Ireolvemeee: Synthetic Optimizee
KCL (Vation)
1.00
Model
Variance
1.0-
0.76
Cauc Opa ol EM 22 500
Caods Saean it 185, Ea0
20
20
Incohomesre (Veenatt)
0.8
0.9
Model Size
ding
20
Mockll
0.
0.0
0.00
in 226,8" 100
0.5
10
10P
3x II
4x100
6x 107
10"
10
102
10a
10'
Avg. Reasoning Lengla (Tokeus)
Aug. Reusoning Leugth (Tokeus)
Optimiser Actions (Steps)
(c) Model Written Evals:
Discrete Choice and Open-Ended Formats
(d) Synthetic Optimizer
Figure Across a variety of settings, as models reason longer or take more actions, they become more incoherent. We assess frontier models (SONNET 4, 03-MINI, 04-MINI, QWEN3) across a variety of different tasks (MCQ, Agentic Coding, Alignment). We evaluate with many sampley to estimate bias and variance terms for each question. When sorting questions by average reasoning lengths and grouping into buckets, a clear trend emerges: urror-nncherennce increases significantly with reasoning length. In other words, for questions where models reason longer and take many actions, their errors are dominated by variance. We make a similar observation for the variance of text embeddings to open-ended safety questions ((c), right), and in a synthetic setting (d).
GPQA: Incoberence
Incoherence
RL. Inoohopeece (Venans)
0.50
1.0 carg
2
0.5
Atritave Media Aletima
0.45
0.4
0.10
KL. Inroherveoo
0.5
0.36
0.40
0.2
Oregn
0.35
Grap
0.1
0.0
2x.30 30
Sxx 10
10
3xs.
4%210'
Suumak 4
Mini
Mini
Avg. Bessoning (Tokms))
Avg. Ressoning (Tokees)
Model
(a) GPQA: Frontier Models (left) and QWEN3 (right)
(b) BWE-BENCH
Figure 3: For a fixed task and reasoning budget, natural variation in reasoning length and actions is predictive of error-incoherence. We analyze GPQA (left, (a)) and SWE- BEEHENCH (b) by splitting samples into above- or relow-median reasoning length (GPQA) or actions (SWE-BENCH) per question. We then compute performance and incoherence for both groups (a) The naturally longer reasoning shows increased incoherence for both frontier models (left) and QWEN3 (right). (b) Similar observations apply to BENCH,,, where longer action sequences display higher inco- herence for test coverage (right). This effect is much stronger than through larger reasoning budgets (Fig. 7), and the difference in accuracy score is minimal between both groups (Fig. 17).
enabled (Team, 2025a). In Sect. 3.2.2, we train autoregressive transformers on synthetic optimization task.
3.1 THE RELATION BETWEEN REASONING LENGTH, ACTION LENGTH AND INCOHERENCE
The longer models spend reasoning and taking actions, the more incoherent they become.
Sorting by reasoning & action length. We begin with a key experimental observation. Fig. shows all setups with reasoning tokens (or actions for SWE-BRNCH, optimization steps for the
5

---

Published as a conference paper at ICLR 2026
synthetic setting) the x-axis and iror-incohhrence or variance on the y-axis. For Figures 2(a) to 2(c), lines show different question sets across and within models, obtained by sorting by average length and grouping into equal buckets, with error-ncchhreenee computed per group.
Across all conditions, longer reasoning and action sequences increase error-incoherence. For GPQA, it increases with different slopes per model family (and reasoning length distributions); notably, for QWEN3, incoherence levels and slopes are nearly identical across all sizes, even though larger models perform better (ef. Figure 9). Similar patterns appear for frontier models on MWE. For SWE-BENCI, both baseline ertor-incoherence and slopes vary: O4-MINI shows higher baseline incoherence but smaller slope; 03-MINI has the largest slope but lowest baseline incoherence.
Example analysis. To illustrate. we provide real experimental transcripts in Fig. 19. The example shows SONNET 4 responding differently with nearly every sample to disconnection question, displaying high error-incoherence This connects to open-ended MWE results in Fig. 2(c), where embedding variance correlates strongly with average reasoning length, and bias not well-defined. We provide additional insight error-incoherence through absolute answer change rates in Appx. C.4, and all open-ended MWE plots Fig. 24.
Discussion: Task complexity. Sorting questions by reasoning length implicitly selects for task difficulty (see accuracies in Fig. 8 and 9), suggesting error-incoherence is higher when making mistakes more complex tasks. While perhaps unsurprising, this is important experimental observation. In fact, for frontier models, our setup asks models for probability estimates of choice correctness (see Appx. B.I), i.e., we give them option to express uncertainty. We revisit task complexity in the next section and Section 3.3.
Natural overthinking and error-incoherence. Inrespective of task complexity, we show how long reasoning and action sequences lead to larger error-incoherence in Fig. 3. For each question, we assign response samples either of two groups: those below and those above the median reasoning length for this specific question for GPQA, and the median number of actions for this task in SWE- BENCH. The incoherence is substantially higher for the second group for both benchmarks. Notably, the average accuracy and SWE-BENCIL BENCII-score (shown in Fig. 17) is similar between groups, but the effect of the natural variation on error- croor-incoherence much larger than reasoning budgets (Fig. 7(a)).
Further results. We provide more analyses for GPQA in Appx. C.I, with reasoning length correla- tions in Appx. C.6. Results for MWE are in Appx. C.7, and results for SWE-BENCH in Appx. C.8.
3.2 THE RELATION BETWEEN MODEL. SCALE, INTELLIGENCE, AND
Larger and more intelligent systems sometimes more incoherent.
Motivation. In Section 3.1, in particular Fig. 2(a), we fix model and analyze error-incoherence as function of reasoning length. Now, we ask different question: When we fix task, how does error- incoherence change as a function of model size? How does incoherence scale with intelligence?
Overview, We summarize the main observation in Fig. 4: larger, more capable and intelligent sys- tems are often more incoherent. This manifested in LLMs for the most complex set of questions (Sect. 3.2.1) 1), the rankings of intelligence and urro-inncoherene as judged by human survey partic- ipants (Appx. B.5) and our synthetic optimizer setting (Sect. 3.2.2). However, we find that larger models are less incoherent on simpler questions (Sect. 3.2.1). We discuss each result in detail.
3.2.1 SCALING LAWS FOR LLMS SEPARATED BY TASK COMPLEXITY
Easy tasks become less incoherent with scale, while harder tasks become more incoherent.
Overview. We experiment with the QWEN3 model family, as they provide the same model archi- tecture, including reasoning abilities, with up to 32B parameters. Consistent with other setups, we sample many responses for the same set of questions. Additionally, we cluster questions using the the reasoning length of a reference model (here: 32B) into equally sized groups.
Results. See Fig. 5 for the detailed results. We find that performance consistently improves with increasing model size, with the fastest rate of improvement for the hardest questions. However, the way in which enror-inchherence changes with scale depends on question difficulty: Model responses to easy questions become more coherent with scale, while responses to the hardest questions become more incoherent with scale, though this last trend is noisy.

---

Published as a conference paper at ICLR 2026
Incoherence va Size
Tosel ligence vs. Incoherenne
Incoherence va. Size
Irooorenne (Avg, Rank (nitec)
Tucolee: (Problen Noro)
1.0-
KL. Inochrowe (lemuess
50-
0.4
Sie thas Bilx (on0at
0.8
Quession Diflenity
30
0.2
Goop
0.7
20-
0.1
20-
Signoii EL (8' 0.923)
1.7
20
102
10
Model Size (Billons of Porsmeters)
Intelligence (Ang. Bonk Order)
Model Parameters
(a) QWEN3 on MMLU
(b) Survey Ranking Results
(c) Synthetic Optimizers
Figure 4: Larger and more intelligent systems are often more incoherent. (a) We measure the scaling of error-incoherence vs. model size for the QWEN3 family, as a function of question difficulty on MMLU. For easy questions, error-incoherence drops with model scale, while for the hardest questions error-incoherence remains constant or increases with model scale. The expanded results for this experiment are in Fig. 5. (b) Disjoint sets of human subjects were tasked with subjectively ranking the intelligence and incoherence of diverse AI models, non-human beings, well known humans, and human organizations. Across all categories, entities that were judged more intelligent by one group of subjects, were independently judged to be more incoherent by another group of subjects. See Appx. B.5. (c) In synthetic task, we train transformers of increasing size to explicitly emulate optimizer trajectories descending quadratic loss. As these models become larger, the trajectories they generate achieve lower loss on the quadratic. However, the final loss is also more variance dominated and thus incoherent with increasing model size. Details in Fig. 6.
Further results. We provide different visualizations of the same results in Appx. C.2, which include the same results for GPQA (Fig. 12), the relationship between error-incoherence and error (Fig. 13) and how reasoning length is a stronger indicator of error-incoherence than model size (Fig. 14).
3.2.2 SCALING LAWS IN CONTROLLED SYNTHETIC SETTINGS: MODELS AS OPTIMIZERS
On a synthetic task, models become more incoherent as they are made larger.
Models as optimizers. In this paper, we are trying to disentangle whether capable models will more tend to act as effective optimizers of the wrong goal, or will pursue the right goal but not be effective optimizers. To quantify this controlled setting, we train models to literally mimic the trajectory of hand-coded optimizer descending loss function. This can be viewed as trying to train a model to implement a mesa-optimizers (Hubinger et al., 2019). We then analyze the bias and variance of the resulting models, answer the question: Does the model become an optimizer faster or slower than it converges on the right optimization objective?
Setup. We study simple d-dimensional quadratic function of the form f(x) = where A E Ra is a (random) positive-definite but ill-conditioned matrix. We set the condition number to 50. Training data is generated by using an optimizer to produce many trajectories of fixed length for random initial points. The optimizer used to generate the training data performs steepest descent with a fixed step norm. The training dataset consists of pairs (I;, u.). where x; is pa- rameter iterate, and u; is the corresponding update step generated by the optimizer. Analogously to real (token-based) models, we train transformer models (Vaswani et al., 2017) of varying sizes using decoding -based regression (Song& & Bahri, 2025) and teacher forcing. This means we tokenize the scientific format representation of 2; and u, with vocabulary of digits and signs. When evaluating, we sample multiple initial points and roll out trajectories using the model's own predictions. A visu- alization of this with real model is provided in Fig. 6 (left). The bias and variance measures are then taken w.r.f. the optimum and norm | alA that is induced by the problem. The details are in Appx. B.4.
b),
Results. The main results are shown in Fig. 2(d) (error-incoherence over rollout steps) and Fig. 6 (scaling laws by size). All models show consistently rising enro-incoherence per step; interestingly. smaller models reach lower plateau after tipping point where they can no longer follow the correct trajectory and stagnate, reducing variance. This pattern also appears in individual bias and variance curves (Fig. 26). Importantly, larger models reduce bias more than variance. These results suggest that they learn the correct objective faster than the ability to maintain long coherent action sequences. More results and discussions are provided in Appx. C.9.
7

---

Published as a conference paper at ICLR 2026
Question: Grouping (MMLU
Rewoning Length Correlation
Aneuracy Siee
1.0
5000
Aug. Length tn Other
Accurady
0.5
Quesi5on
8000
1/20-4
Question Difically
Sroce & olsat
2000
1000
0.5
120
Gropp
10'
10
10a
10
1.7
32
Avg. Rossoning (Tokmss): Qwen3 32B
Rossoning Length 32B
Model Size (Billions of Parameters)
(a) Separating Complexity Groups
(b) Length Correlation
(c) Accuracy Scaling Laws
KL Bios vs Size
KL Varinnse vs Size
MMLU KI. Incoherence Contours
Error Due tn Kil. Varisne
RL Mins
10P
Queesion Difficuly
Question Dificuity
KL
10-1
10-1
10-2
Real
Model Size (Billions bes)
Model Sion (Billions of Parsmeters)
Error Due to KL Bias
(d) Bias and Variance Scaling Laws
(c) Eeroo-incoherence
Figure 5: Details for QWEN3 scaling laws: easy tasks become less incoherent, harder tasks more incoherent. We group MMLU questions by reasoning length using reference model (Qwen3 32B, (a)), which correlates across model sizes (b) and serves as a task complexity proxy, accuracy drops with longer reasoning (c). These groups reveal distinct bias-variance scaling (d): bias slopes are similar across groups, but variance slopes decrease sharply for harder ones. In the hardest group, variance slopes fall below bias slopes, leaving variance the limiting factor. Thus, larger models remain constrained by variance and more incoherent with scale (e). We provide more analyses including other models and the same conclusion for GPQA in Appx. C.2.
Quaddaatis Oplimisttion Funclósn
Validation Loss us. Size
Bias/ Variane va. Size
3.50
3.4 10
30
0.55
1:75
10
Fit (a (.001)
20
0.00
1.70
3.1 10
3x
-0.25
Teue Optiminer
5.25
20
-0.00
Trajecury
20
Vacianos (o 0.788)
10.00
00
05
30
1.5
10*
10"
10a
10"
Model Parzumntres
Model Paramebers
Figure 6: Details for synthetic optimization: In controlled settings with teacher forcing and a single objective, language models become variance dominated with increasing size. (leff) We train autoregressive transformers to predict update steps to minimize a quadratic function using decoding based regression, i.e., next-token prediction. This setting involves sequentially performing steps towards goal via next token prediction, emulating key feature of goal seeking AL (middle) The loss (next-token prediction objective) follows clear power law improvement with model size. (right) When evaluating the trained models using their own rollouts, we find that increasing model size reduces bias much faster than variance.
3.3 THE EFFECTS OF REASONING BUDGET AND ENSEMBLING
We now study the effect of reasoning budgets, i.e., the techniques provided in model APIs, and ensembling, ie., averaging multiple responses, on error-incoherence. The main results are in Fig. 7.
3.3.1
REASONING BUDGETS
Reasoning budgets reduce evror-incoherence, but natural variation has a much stronger effect.
8

---

Published as a conference paper at ICLR 2026
GPQA: Trevhhmenee
Components Ezscmble
Jucobesence va. Eisemalle
100 Iaraberrare (aatorn
0.40
Blöee furdonreee
0.00
0.40
Bokke
0.35
Peter Riss (o = -0.01
0.30
Beier Viciance (a- 1.000
Avg. Rouscaing (Tokens))
Ersembll
Eusembte Stze
(a) Reasoning Budgets
(b) Ensembling Results
Figure 7: Ensembling and larger reasoning budgets reduce error-incoherence. Other forms of error correction may also reduce error-incoherence. (a) Instructing models to reason longer improves performance (inference scaling laws, Fig. 17) and sometimes error-incoherence. This effect is smaller than natural variation, where error-incoherence rises sharply (Fig. 3: direct comparison in Fig. 17). (b) With 04-MINI on GPQA, we analyze the effect of the ensembling, Lee, using multiple samples to average output probabilities over targets for the same question. The bias and variance are now computed by comparing different ensembles of the same size. We find that, expected from theory, it reduces variance with rate of 1/E, without affecting bias (left). As a consequence, error-incoherence drops (right). Ensembling is a particular form of model error correction, which is impractical for action loops in the world, since state typically not be reset. However, we expect other error correction techniques to also reduce incoherence.
Inference scaling. We show the results of our inference-scalingg analysis on GPQA Fig. 7(a) and Fig. 17. Increasing reasoning budgets improves performance (17(a), left), and slightly reduces error-incoherence for all models but SONNET (7()). Interestingly, this effect is overshadowed by enor-incoherence that arises through natural variation, i.e., when models think longer than the median for a question (recall analysis in Fig. 3; direct comparison in Fig. 17(a), right).
How does reasoning budget improve coherence? Since the implementation details of reasoning budgets for frontier models are not public, it is unclear how exactly it can improve croor-nccoherence We believe it is likely explained by better backtracking and error correction properties, a phenomena observed to arise during training with larger budgets (Guo et al., 2025), and related to the ensembling results in Sec. 3.3.2. We partially explore error-incoherence through the reasoning structure with the QWEN3 reasoning traces in Appx. C.3.
Discussion:
3.3.2 ENSEMBLING
Ensembling multiple attempts reduces error- vroorinncherrece.
Motivation. Perhaps the most natural way to reduce error-incoherence is to ensemble multiple attempts: instead of relying single answer, we roll out multiple trajectories and combine them. We demonstrate this with a repetition of the experiment for GPQA with 04- MINMI
Setup. We obtain 320 samples of answers for all questions of GPQA. Fixing ensemble of size E, we average the E produced probabilities over targets. To compute bias and variance, we then compare ensembles of the same size across random samples of ensembles, which we hold at fixed number of 10, while ensuring that samples do not overlap. This allows ensemble sizes of up to 32.
Results. Fig. 7(b) shows how variance changes with increasing ensemble size. As expected, it drops like the inverse of the ensemble size, and error-incoherence therefore also drops. We expect there broader classes of error correction that behave similarly. The slight reduction in error-incoherent with increasing reasoning budgets in 3 I may be achieved through such a mechanism. We provide the plots for ELL-ERROR-INCCHERENCC INCOHERENCE Fig. 11.
RELATED WORK
We summarize the most important related work and defer a comprehensive discussion to Appx. D.
Reasoning, Recent studies report inverse scaling trends with extended reasoning degrading performance (Gema et al., 2025; Su et al., 2025; Wu et al., 2025; Hassid et al., 2025). Most relevant,

---

Published as a conference paper at ICLR 2026
Ghosal al. (2025) find that overthinking increases output variance, though via artificially injected tokens rather than natural overthinking. While these studies identify performance degradation, they do not distinguish systematic errors from inconsistent failures. Our ensembling analysis relates to seff-consistency work (Wang et al., 2023), but reframes aggregation as reducing error-incoherence.
Evaluation variance. Even though AI models have vastly improved upon benchmarks, evaluations are known to be highly variant (Bui et al., 2025; Biderman et al., 2024). Errica et al. (2025) formalize this through sensitivity and consistency metries, revealing important failure modes. This similar setup to our input and output randomness. Importantly, we connect the variability to the concepts of bias and variance, highlighting the relevance in the safety setting, and analyze scaling laws.
Scaling behavior. As models get larger and more capablc, evidence suggests their representation and errors become highly aligned (Kim al., 2025; Huh al., 2024; Goel et al., 2025) and that they improve ong-horizon tasks (Sinha et al., 2025). Our work complements these observations by find- ing increased error- coor-incoheerece the longer models reason and act, aligned between model families.
DISCUSSION AND WHAT OUR RESULTS Do NOT TELL Us
Why expect more capable models to be more incoherent? In this paper, we do not experimentally or theoretically explore the specific mechanisms for increasing error-incoherence with increasing trajectory length and (sometimes) model size. However, there motivating observations.
The first that LLMs are dynamical systems. When they generate text or take actions, they trace trajectories in nigh-dimensional state space. It is often very hard to constrain generic dynamical system to act as an optimizer. The set of dynamical systems that act as optimizers of a fixed loss measure zero in the space of all dynamical systems. As models scale and acquire broader capa- bilities, their effective state and action space expands, exacerbating this difficulty. We should not expect AIs act as optimizers without considerable effort, nor should we expect this be easier than training other properties into their dynamics.
Second, variance typically accumulates over a trajectory unless there is an active correction mech- anism (like ensembling, Fig. 7). When an AI acts in the real world, actions are often irreversible. Therefore, will often impossible or impractical to correct for noise introduced by model actions.
Reward misspecification. Bias can be further decomposed into BIAS = BIASMESA + BIASSPEC where BIASMESA captures the average deviation of the model's 's behavior from the training objective, and BTASsPEC captures the deviation of the training objective from the intended training objective. For our tasks, we believe that there was not meaningful reward misspecification. poorly specified training objectives,
settings with worry that BIASspaC would come to dominate the error, as both variance and BIASMESA go to zero with increasing model capability. Our results underscore the importance of characterizing and mitigating goal misspecification during training.
Open-ended goals and error-incoherence. To rigorously analyze the scaling of bias, variance, and error- ioooinccoherenee, we need to (1) measure "average" prediction (for bias and variance) and (2) measure distance to ground truth (for bias). We use multiple-choice classification, coding unit-tests, and objective functions rather than LLM judges ensure metries are well-defined, unbiased, and comparable. Extracting hidden goals and complex incoherent behaviors remains important (cf. Section 4.1.1.5; Anthropic, 2025a); our embedding-variance analysis of model-written evals (Appx.C.7) provides an initial exploration of setting where bias is not easily defined or measured.
6 CONCLUSION
Motivated by the hot mess theory of Al misalignment, we propose a bias-varianc, decomposition as framework for analyzing how increasingly capable Als will fail. Our results show that longer sequences of reasoning and actions consistently increase model enror-incoherence. We also find that smarter AI models are not consistently more coherent. Our results suggest that when advanced AI systems performing complex tasks fail, it is likely to be in inconsistent ways that do not correspond to pursuit of any stable goal. This should inform judgements of the relative plausibility of different AI risk scenarios and guide further research into understanding the mechanistic origins of amor-inccherence.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Andrew Saxe, Brian Cheung, Kit Frasier-Taliente Igor Shilov, Stewart Slocum, Aidan Ewart, David Duvenaud, and Tom Adamezewski for extremely helpful discussions on topics and results in this paper.
ETHICS STATEMENT
This research aims to characterize failure modes of increasingly capable Al systems to inform safer deployment strategies. Our findings suggest that as Al systems tackle more complex tasks requir- ing extended reasoning, incoherent failures become more prevalent than systematic misalignment. While this work does not directly prevent failures, it offers empirical grounding for prioritiz- ing safety interventions, suggesting greater focus on preventing unpredictable accidents rather than solely defending against coherent malicious behavior. We believe this understanding AI failure modes benefits the community to ensure safe Al deployment.
REPRODUCIBILIITY STATEMENT
We provide detailed description of our theoretical framework in Section 2.1 and Appx. A. The gen- eral experimental setups are described in Section and Appx. B, with task-specific details outlined in each experiment subsections. Our code and data is available here.
REFERENCES
UK Al Security Institute. Inspect AI: Framework for Large Language Model Evaluations, 2024. 23
URL
/
Anthropic. System card: Claude opus 4 & claude sonnet 4, May 2025a. URL https:// conttrrrcccco/////
www-cdn. anthropic com/ 6c8=8055020700118
6081 6bs2a7c285.
pdf. Accessed: 2025-06-08. 10
Anthropic.
Claude 3.7 sonnet system card, February 2025b. //assess/ ant.sseesst.ttrroi.c.... wttp://cccc////////////////// /ttem//755811111 @eeeeaabbbrrrgggee claude-33 3- -- system-ssooe p.rrdppff. Accessed: 2025-05-08. 4, 40
URL
https:
Ruth
Appel, Peter McCrory, Alex Tamkin, Michael Stern, Anthropic economic index report: Uneven geographic and
Miles McCain, and
Tyler Neylon. enterprise ai adoption, 2025.
URL www www. www.aaaichhcc.... cttiiicccoo/reeeeeaoo anthropic-economl- indei-idexx-ee eeeetttebbr222--eeeee
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Al- ham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation language models. arXiv preprint 1XXv:24055.1782. 2024. 10
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies, Oxford University Press, Oxford, 2014. ISBN 978-0199678112.
Leo Breiman. Bias, variance, and arcing classifiers. 1996. 3
Nghia Tuan Bui, Guergana K Savova, and Lijing Wang. Assessing the macro and micro ef- fects of random seeds on fine-tuning large language models. In Kentaro Inui, Sakriani Sakti, Haofen Wang. Derek F. Wong, Pushpak Bhattacharyya, Biplab Banerjee, Asif Ekbal, Tanmoy Chakraborty, and Dhirendra Pratap Singh (eds.), Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chap- ter of the Association for Computational Linguistics, pp. 41-46, Mumbai, India, December 2025. The Asian Federation of Natural Language Processing and The Association for Computa- tional Linguistics. ISBN 108-8-89177-277 URL ht https: /aaaaaaathhloooo org/2025. jjenlp-shor httpsshtt..// 10, 40

---

Published as a conference paper at ICLR 2026
Andong Chen, Yuchen Song, Wenxin Zhu, Kehai Chen, Muyun Yang, Tiejun Zhao, et al. Evaluating ol-like 1lms: Unlocking reasoning for translation through comprehensive analysis. arXiv preprint 10Xi::2502.11544., 2025a. 4
Danqing Chen, Carina Kane, Austin Kozlowski, Nadav Kunievsky, and James A Evans. The (short-term) effects large language models on unemployment and earnings. arXiv preprint arXiv:2509.15510. 2025b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint MxXi::2110.14166 2021. 40
Google code security.
DeepMind.
Introducing codemender:
ai agent for October
Introdduing--n
2025. Accessed:
2025-10-16.
Morris H. Degroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society Series D: The Statistician, 32(1-2):12-22 12 2018. ISSN 2515-
7884. doi: 10.2307/2987588, URL https: //ts:://////..//// org/ 2307 200 /2977888.
Pedro Domingos. A unified bias decomposition for zero-one and squared loss. AAAI/IAAI, 1000:564-569, 2000.
20
Jacob Dominski and Yong Suk Lee. Advancing ai capabilities and evolving labor outcomes. arXiv preprint mixXiv:5577.024444 2025.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: Labor market impact potential of lims. Science, 384(6702) 1308, 2024. doi: http:///scciccccao999099.. wtt.:/1ieeecceneeeeeeeeee stt//ceieeecccceeee adjo998.
URL https::
orge dei
http//dd//bb//////////e abs 100 1126/
Federico Errica, Davide Sanvito, Giuseppe Siracusano, and Roberto Bifulco. What did do wrong? quantifying L.MMs* sensitivity and consistency to prompt engineering. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational Linguistics: Human Language Technologies Albuquerque, New Mexico, April 2025. Association
(Volume 1: Long Papers), pp. for Computational Linguistics. ISBN 8818888997718899 doi: 20.16533v1/2022 naacl-long. 73. URL www.tttt.//om.. /atpss://aaaattthha
org/ 2025 naacl long. 73 /. 10, 40
Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, and Anthony Hartshorn. What char- acterizes effective reasoning? revisiting length, review, and structure of cot. arXiv preprint arXiv:2509.19288 2025. 26, 40
Anna Fine, Emily R Berthelot, and Shawn Marsh. Public perceptions of judges' use of tools in courtroom decision-making: An examination of legitimacy, faimess, trust, and procedural justice. Behavioral Sciences, 15(4):476, 2025. 1
Jerome H Friedman. On bias, variance, 0/1-loss, and the curse-of-dimensiionalty Data mining and knowledge discovery, 1(1):55-77, 1997.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024a. URL https:/ zenodo. org http///cordd/.d// 608602862222 22
Shen Gao, Jiabao Fang, Quan Tu, Zhitao Yao, Zhumin Chen, Pengjie Ren, and Zhaochun Ren. Generative news recommendation. In Proceedings of the ACM Web Conference 2024, WWW '24, pp. 3444-3453, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400701719. doi: 00.1145/558933433655522
URL https: //cc://doooooooo wtt/10.1144....1111001.00.. htt///iiirgg/..///////
3589334.3645448.
12

---

Published as a conference paper at ICLR 2026
Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman -Wetzlr,, Kit Fraser-Talle Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, and Ethan Perez. Inverse scaling in test-time compute. Transactions Machine Learning Research, 2025. ISSN 2835-8856. URL https: tettpp:://peene /eppenreeee ew. n
Corum?id=NXgy-ae c7M. Featured Certification, J2C Certification. 9, 22, 40
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does thinking more always help? mirage of test-time scaling in reasoning models. In The Thirty-ninth Anmual Conference Neural Information Processing Systems, 2025. URL https: //opnreevee..... net/ ? id= qbamNb9. 10, 40
Shashwat Goel, Joschka Strüber, 1lze Amanda Auzina, Karuna K Chandra, Ponnurangam Ku- maraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines AI oversight. In Forty-second International Conference Ma- chine Learning, 2025, URL httpa/gee//ppreeeieeeeeeeeeeooes wtt.:eoommmmm222272272222 10, 40
Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment fak- ing in large language models. arXiv preprint arXiv:2412.1409) 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808 (0600881)6636638,, 2025. 4, 40
Kunal Handa, Alex Tamkin, Miles McCain, Saffron Huang, Esin Durmus, Sarah Heck, Jared Mueller, Jerry Hong. Stuart Ritchie Tim Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint 2rXhv:2003.04761, 2025.
Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. Don't overthink it. preferring shorter thinking chains for improved Ilm reasoning. arXiv preprint arXiv:2505.17813, 2025. 9, 40
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt, Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL https: /ttps/////ppeeeeeeeeeee net / forum?id- d7KB 3GmQ. 4
Tom Heskes. Bias/variance decompositions for likelihood-based estimators. Neural Computation, 10(6):1425-1433) 1998. doi: 10.1162089976093300072333 3
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optine
large language models. In Proceedings of the 36th International Con- ference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817138710 1088. 3
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. URL, https: http///veereeeeeeee/eeeee...e.em .ne.t/ewweeeeeeeee.eee...e..eeee wwtp://eeeeeeeeeeeeeeeeemooo
40
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv: 1.XXi:1906601820,, 2019. 1,7
John Hughes and safety research. recarrrrrsccchsfffytooooongg v1.0.0, 2025. URL https: / //doi. ///oi.or//00....////////////.11/ 5281 zenodo. 15363603. 22
13

---

Published as a conference paper at ICLR 2026
Minyoung Huh, Brian Cheung, Tongzhou Wang. and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. 10, 40
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024, 4, 40
Doohyuk Jang, Yoonjeon Kim, Chanjae Park, Hyun Ryu, and Eunho Yang. Reasoning model is stub- born: Diagnosing instruction overriding in reasoning models. arXiv preprint arXiv:2505. 17225, 2025. 40
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE--benh: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL. https: /ppnrevvew. net/forue
8yNQM6 6. 4, 23
Andrew Johnston and Christos Makridis. The labor market effects of generative ai: A difference-in in- differences analysis of exposure. Available at SSRN 5375017, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. In Forty-second International Conference on Machine Learning, 2025. URL https: / //oppnneeeiww....m//......oe.eem net
10, 40
Ron Kohavi and David Wolpert. Bias plus variance decomposition for zero-one loss functions. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning. (CML'96, pp. 275-283, San Francisco, CA, USA, 1996. Morgan Kaufmann Publishers Inc. ISBN 1558604197. 3
Eun Bae Kong and Thomas G Dietterich, Error-correcting output coding corrects bias and variance. In Machine learning proceedings 1995, pp. 313-321. Elsevier, 1995. 3
Nadav Kunievsky and James A Evans. Measuring (a sufficient) world model in 1lms: A variance decomposition framework. arXiv preprint arXiv:2506. 16584, 2025. 40
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix. Tao Roa Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M Ziegler, Elizabeth Barnes, and Lawrence Chan. Measuring Al ability to complete long software tasks. In The Thirty- ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: // /pppneevieww feree/ffoumm/ wewrrmmmmdd-cGGn.eeeee..
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 22
Aycong Lee, Ethan Che, and Tianyi Peng. How well do LLMs compress their own chain-of of-thought? a token complexity approach. In ES- OS-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025. URL https: wwwnrevieeeeeeeeee new. net. / forum? d= dediduuuu5552 55405x 40
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023. 4
Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. Once: Boosting content-based rec- ommendation with both open- and closed-source large language models. In Proceedings of the 17th ACM International Conference Web Search and Data Mining, WSDM '24, pp. 462-461, New York. NY, USA, 2024, Association for Computing Machinery. ISBN 9798400003713. doi: 10.1 10.1145/3616853 685.6663555...... URL https: /ooii wrg/10..0000 1. 1141 6855.88..... 3635845.
14

---

Published as a conference paper at ICLR 2026
Yiran Ma, Zui Chen, Tianqiao Liu, Mi Tian, Zhuo Liu, Zitao Liu, and Weiqi Luo. What step-level reward models rewarding? counterintuitive findings from mets- bets--bosted mathematical reasoning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 24812-24822 2025. 40
Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, et Artificial intelligence index report 2025. arXiv preprint arXiv:2504.071339 139, 2025.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. sl: Simple test- time scaling. In Christos Christodoollppoulos,, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20275-20321, Suzhou, China, November 2025, Association for Computational Linguistics. ISBN 979-8-89176-332-4 // wawclanthooogy.oor......
doi: 10. 112653311222255 wewp::255mmmmlmmmmmmmmm 1025. URL https: wwwne-manaee... 1025/. 4, 40
Beatrice Nolan.
An ai- ai-powered coding tool wiped out a software com- pany's database, then apologized for 'catastrophic failure my part', July 2025.
URL
ai-coding-ten
Accessed: 2025-09-25. 2
OpenAl. Openai o3- system card, February 2025a. URL / o 03-mini- wewiiii-ssstem--ee saystem-card// Accessed: 2025-08-31. 4, 40
OpenAl. Openai o3 and 04-mini system card, April 2025b.
URL https:
// cán. openai.......................... 2ttppp/d02222111002222222 17789-8000--77585333 03 33-and-o---ini-syyteee card. pdf. Accessed: 2025-06-08. 4
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jack- son Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer EI Showk, Tamera Lan- ham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Z Hatfield-Dodds. Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Her- nandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering lan- guage model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387-13434, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: (0.18653/v1/2023 http222002...innnaaaaaaa findings-acl.888 844. URL https: httpt:t/tcclattttataat //t///aaaaaaaltoyaaaaaaaaa org/2023. findings acc. 847/ / 4, 22, 33, 36, 37
David Pfau. A generalized bias-variance decomposition for bregman divergences. Unpublished manuscript, 2013.
Govind Pimpale, Axel Højmark, Jérémy Scheurer, and Marius Hobbhahn. Forecasting frontier language model agent capabilities, arXiv preprint arXiv:2502.15885 2025. 1
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q& bench- mark. In First Conference on Language Modeling, 2024. 4
Stuart Russell. Human compatible: Al and the problem of control. Penguin Uk, 2019. 1
Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu. Lims are greedy agents: Effects of rl fine-tuning on decision making abilities. arXiv preprint arXiv:2504. 16078, 2025. 40
15

---

Published as a conference paper at ICLR 2026
Parshin Shojaee, Seyed Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL https: / openreview net / forum? (d=YghiQusmmy 40
Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, and Jonas Geiping. The illusion of diminishing returns: Measuring long horizon execution in llms. arXiv preprint arXiv:2509.09677, 2025. 10, 40
Charlie Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can more effective than scaling parameters for reasoning. In The Thirteenth Interna- fional Conference Learning Representations, 2025. URL https: /ttp:////eeeeeeeeeeeeeeeee .net/ GGrum?id=4FWWwWEEr
4, 40
Jascha Sohl-Dickstein. The hot mess theory of Al misalignmentt More intelligent agents behave less coherently https: / httpp::t/tsol--//e/etto /sssshhhhccsseeinnnncnn htt.:/ddickstii..ihiii... github. wtt.uu...//22230 http:////////////eeeeeeee html, 2023. 4, 24
Xingyou Song and Dara Bahri. Decoding-based regression. Transactions on Machine Learn- ing Research, 2025. ISSN 2835-8856. URL http::::///pnreeeeen.eetttt// http://ewww.....eee/ooottaaae foret/fffrmmid avU@8jguxe 23
Philipp Spiess. How i use claude code, 2025. URL https: how cea-i--use-ee use--cludee code. Accessed: 2025-09-25.
Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. Between underthinking and over- thinking: An empirical study of reasoning length and correctness in llms. arXiv preprint arXiv:2505.00127, 2025. 9, 40
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12559, 2025. 4, 40
Qwen Team. Qwen3, April 2025a. URL https: http:///ww........t..o.o//////// 4, 5, 40
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025b. URL https://qwenlm:
http:////wwwlmm....bbbbb. github. http:////blll/..oeeee /qq--/www--poo 32b/. 40
Robert Tibshirani. Bias, variance and prediction error for classification rules. Technical Report,
Statistics Department, University of Toronto, 1996. 3
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017. 7,24
Chenlong Wang, Yuanning Feng, Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou, Wait, we don't 't need to "wait"! removing thinking tokens improves reasoning efficiency. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2025, pp. 7459-7499 7482, Suzhou, China, November 2025, Association for Computational Linguistics. ISBN 979-8-89176-335-7, fn25::/nnn findings-emnlp.994 URL https:
doi: 10. 10118653/1/2225
org/2025.
findings emn 1 p 394/. 40
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL net/ httpr/ewwwweeetetrnnn-e--------- forum?L--1PLINNNDS
10, 40
Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Under- standing shain-of-though length in lmm. arXiv preprint arXiv:2502.07266, 2025. 9, 40
16

---

Published as a conference paper at ICLR 2026
Yuki Yada and Hayato Yamana. News recommendation with category description by large lan- guage model. In CEUR Workshop Proceedings, volume 4056. CEUR-W S.RRRWSSS. 2025. 13th Interna- tional Workshop News Recommendation and Analytics, INRA 2025.
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test- time compute for LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: https::/ http////peeeeeeeeee/eeeeeeeeeeee net/ ferem/ormmmmmm ?im-m 6ICP smix1 x1s. 40
Zitong Yang. Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade- off for generalization of neural networks. In International Conference on Muchine Learning, pp- 10767-10777. PMLR, 2020. 3, 40
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. 23
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai ol: Opportunities and challenges of agi. arXiv preprint arXiv:2409, 18486, 2024.
17

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Background
2.1
Bias-Variance Decomposition
2.2
Scaling Behavior of Large Language Models
3
Experiments
3.1
The Relation Between Reasoning Length, Action Length and Incoherence
3.2
The Relation Between Model Scale, Intelligence, and Eiror-incoherence
3.2.1
Scaling Laws for LLMs Separated by Task Complexity
3.2.2
Scaling Laws in Controlled Synthetic Settings: Models as Optimizers
3.3
The Effects of Reasoning Budget and Ensembling
8
3.3.1
Reasoning budgets
3.3.2
Ensembling
4
Related Work
5 Discussion and What Our Results Do Not Tell Us
10
Conclusion
10
Bias and Variance Definitions for Classification
20
Experimental Details
22
B.1
GPQA and MMLU
22
B.2 Model-Written Eval
22
B.3
SWE-BENCI
23
B.4 Synthetic Tasks
23
B.5 Survey on Intelligence and Enror-incoheree incohncnccheeeee
24
Further Experimental Results
25
C.I GPQA Model Performance Overview & Different Metrics
25
C.2 Scaling Laws With Other Models and Benchmarks
26
C.3 Reasoning Variation, Error Correction, Wait Ratios
26
C.4 Illustration of Answer Changes
32
C.5 Sample Efficiency and Correct Formatting
33
C.6 Reasoning Length Correlations
33
C.7 Model-Written Evals
33
C88 SWE-BENCH
33
C.9 Synthetic Tasks
34
18

---

Published as a conference paper at ICLR 2026
C.10 Survey Results
34
Related Work
40
L.M Use Statement
40
19

---

Published as a conference paper at ICLR 2026
A BIAS AND VARIANCE DEFINITIONS FOR CLASSIFICATION
Recall the classical bias-variance decompositon in the case of regression: mean-squared error for sample point (x, v) E R2, the decomposition is given by
Considering the
MSE 1x1(v f=()))) -
(=((f(x)) - (=
BrAs
VARIANCE
Irreuccible Fimar
where f the ground-truth function, and the expectation is taken w.r.t. the randomness E in the training process (e.g., data ordering) that the model depends on.
Classification Formulation. While the interpretation for classification is similar, different decom- positions exist, which review in the following. Throughout this section, let x be the input of problem with target class c(x) E {I,..., ...,,,, C} and one-hot target y(z) E RRC, The model fa produces a probability distribution (potentially one-hot) over class labels fe fx) the dependence of and fc on z. yfc] denotes the c-th element of the vector. Throughout our ex- periments and derivations, we assume that the irreducible noise is (i.e. no stochasticity in the data- generating process or wrong labels) for simplicity. Note that each of the following decompositions gives bias and variance for single data point (z: y). which aggregated over dataset {(rr, c:))
RC, For clarity, we omit
The classical decomposition for 0/1 loss relies on the unified decomposition by Domingos (2000). Let c(x) be the ground-trutt class (assuming noiseless labelling) and the model's predicted class be (z) arg max. fe [c] The systematic mean is č = arg max, E&g [ [c]], i.e. the mode the average prediction. Then, the 0/1 loss L for sample æ can decomposed into
0/1 Error.
E. [L(,
E. [1 fo ce)
{c a E.
[
BIAS
VARIANCE
where the variable a {-1,1} is multiplicative factor that enables the decomposition with positive variance. In this setting, the bias is always either or and the variance captures the prob- ability of deviating from the mode. Though universal, this decomposition has one major drawback: when computing an average over a dataset of questions (I,, c),, it does not allow to average the bias and variance terms separately; instead, the decomposition only holds with the aforementioned multiplicative factor az Formally, we have
E (( (LGG, ce)l = B(enncee VARIANCE: + B(enn)) [BIAS
# B(rnc)) [VARIANCE,] (as, [BIAS";];
Essentially, the factor depends on the mode prediction being correct or not. We therefore re- port absolute bias and variance errors for the 0/1 loss in the Appendix, but do not compute error- incoherence.
Brier Score. Similar to regression, we can treat the model's probability predictions as C- dimensional vectors to compute the mean square errors. Formally, the Brier score for multiclass prediction is defined and can be decomposed as
G [vvce - fele)) E. [BRIER fe)] E. [ly- fe fel = E
1y-
+ 11
fell
BRIER BIAS
BRIER VARIANCE
where f [fe] is the average prediction.
KL Divergence (Cross-Entropy). The expected cross-entropy loss be decomposed into
E2 (CE(y, = E. Σ vld log log(felc))
(5)
= DRI. (vils) + EEs (fIIE)))))],
VARIANCE
where DKL is the Kullback-Leibe divergence and f is the average of log-probabilities after malization, i.e.,
Jelc] exp (E. (log(S(()))) for C.
20

---

Published as a conference paper at ICLR 2026
Note that this is not the standard average prediction, as is the case in the Brier decomposition. but geometric mean. In practice, since predicted probabilities can be zero, we apply Laplace smoothing
to avoid log(O) or infinite values. This is done by updating the probabilities to fe 1+058 each 1 C with small value of 10-12
f=|o++8 for
21

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B.I GPQA AND MMLU
Setup. We rely on the LM Harness (Gao al., 2024a) codebase, where we evaluate models in multiple choice formats with custom written answer extraction functions to avoid false positives and negatives. For frontier models, we use reasoning budgets provided by the API (low, medi um, high for the o-series, 1024-16k for Anthropic), with maximum generation length of 32k for SON- NET and 100k tokens for the o-series. For QWEN3, we perform inference with vllm (Kwon et al., 2023) and recommended parameters for thinking (temperature 0.6, top-k 20, top-p 0.95). Since consider multiple choice questions that only require a letter to answer, we count reasoning length using the amount of output tokens in the answer, either by the API count using the actual tok- enizer of QWEN3. To estimate the bias and variance metrics across both input (context) and output (sampling) randomness, we evaluate models using 10 different few-shot contexts randomly sampled from the corpus, and 3 samples for each fixed few-shot per question. This results in 30 samples per question overall. For MMLU, reduce computational complexity, we limit 100 samples per question category (5700 in total).
Probability prompting. To provide models the option to express uncertainty and therefore re- duce enror-incoherence, we evaluate frontier models separate setup in addition to standard multiple- choice. We use the following prompt to ask for probability estimate of each answer choice being correct:
Probability Format for MCQ
You will answer multiple-choice questions. Each question has a single correct answer. Work through each problem step-by-step, showing your reasoning and applying relevant concepts. Instead of choosing single answer, YOU MUST PROVIDE an estimate of the probability of each answer being correct within " "<PROB>P(A
P(B), P(C), P(D)</PROB>" where each P(X) is float value between and 1. The probabilities must sum to 1: P(A) P(B) P(C) P(D) For example, you think the probability answer (A) being correct 0.5, the probability of answer (B) being correct is 0.2, the probability of answer (C) being correct is 0.2, and the probability of answer (D) being correct 0.1, then your response must end with "<PROB>0.5, 0.2, 0.2, 0.1 </PROB>". You MUST end your response with the probabilities in the format specified above.
We report results for both standard and probabilty prompting in Appx. C.I, which show qualitatively the same behavior and performance. Frontier models are able to adhere to the format well, with only a few outliers (Table 1). Our main text shows the results for the probability format.
B.2 MODEL- WRITTEN EVAL
We evaluate the models using the advanced AI risk evaluation subset from Perez et al. (2023). These tasks assess LLMs' self-reported behaviors relevant to advanced AI safety, including self-preservation inclinations, willingness to accept modifications to training objectives, and related safety-critical behaviors. We specifically use the human-generated subset ensure higher evaluation quality.
Setup. Our experimental setup builds upon the codebase from Gema et al. (2025), which uses the safety-tooling library (Hughes & safety research, 2025) for API model inference. We conduct experiments under two conditions: the original multiple-choice format, and open-ended format where we remove the multiple-choice options from the original questions. For both conditions, we compute the bias-variance decomposition with respect to the percentage of responses that align with desired safety properties. To ensure consistent evaluation across both formats, we employ the same system prompt that facilitates straightforward extraction of the model's 's final answer:
System Prompt for Model-Writter Evals
Use a thinking process analyze the problem step-by-step. At the end, provide your answer and clearly indicate using <<nswer>X</answer format.
22

---

Published as a conference paper at ICLR 2026
In both cases, obtain exactly 30 samples by simply resampling from the APIs We use the returned output token count as a measure reasoning length.
Embeddings. For the open-ended question set, extract the model answers inside <answer> tags (i.e., removing chain of thought or reasoning) and embed the text into fixed-size vectors using For the 30 samples per ques-
the OpenAl text embedding model -ew-eeeeeedd-------eeeee-- tion, we in turn compute the variance in Euclidean space by computing the mean embedding and computing the average squared distance of samples the mean.
B.3 SWE-BENCH
Setup. We employ the Inspect Evals library (AI Security Institute, 2024) to evaluate models on SWE-BENCH (Jimenez et al., 2024), specifically using the SWE-BENCH Verified subset. This setup prompts LLMs with a simple Reasoning-Acting (ReAct; Yao et al., 2023) agent loop in minimal bash environment, without additional tools or specialized scaffolding structures. We use Inspect library v0.3. 1. 16 and Inspect Evals at git commit 33d2a86 6. The message limit is set to 250, with a timeout of one hour per task. In case that limit is reached, we consider all tests as unchanged, i.e., PASS- TO-PASS cases valid and FAIL- TO---TO--- TO-PASS are invalid.
Like for other setups, we obtain 30 runs of the SWE-BENCH verified subset for all models. Consider task (out of 500) with T; unit tests. Let Yr.j E {0, 1} be the outcome of test in run r, where r E {1...., R} (R = 30) and j E (1..... T.)- To compute bias and variance, we compute the mean outcome as Dj Σ Vr.j- In turn, this gives us the bias and variance decomposition of the coverage error (mean squared sum of unit tests) via
Metries.
1 -
vn.i
Σ
3n
(Urá -5))
RT.
T.
RI.
ERROR
BLAS²
VARIANCE
B.4 SYNTHETIC TASKS
We discuss the details of the experimental setup.
Data. We examine basic d-dimensional quadratic function. This is function of the form f(x) (I - 6) A(r b). where A E ddxd a (random) positive definite but ill-conditioned matrix. In our presented experiments, we use d = 4 and generate random matrix with condition number 50. To generate our target data, we employ a ground-truth optimizer of steepest descent with fixed step norm,
0.005, to generate multiple fixed-length trajectories (of length 4096 steps) from randomly sampled starting points around the minimum, creating a dataset pairs (.), We sample 20'000 such trajectories, and use 10% holdout dataset for valuation loss.
Tokenization. Following the approach used in actual (token-based) language models, we use de- coding based regression (Song& & Bahri, 2025) and next-token prediction. This approach involves representing floating-poin point numbers in scientific notation, with a vocabulary consisting of numerical digits and mathematical signs ((0, 1, 2, 3, 4, 6, 7, 8, 9, +)). The model generates tokens sequentially to construct complete numbers. Concretely, consider a training example (Iz, in two dimensions. Let J; (0.5, 1.5).. In scientific notation, this corresponds (+5.00e 00e-1, -1. 5..500000 0) with precision of 2 mantissa digits (after the comma). We drop special tokens (such as e) to not have any zero-entropy positions. turn, we fix a precision, and move sign and exponent to the beginning; exponents are capped at 0. Taking precision of e.g., 2, the vector 2; will thus be represented by the token sequence:
(+5. 005-000-1, -1. 50e-0) 0)
-0500
sign negative exponent digit
digit
digit
lokens of second dimension
Let u; 0.0022 Then the entire training sample is encoded with the tokens:
1
'https://opena" com. hwwmmindexx/neee www.ediimbddiiiimm ng- model wanddlls-ad-aaae --and-api -pdatttees
no-model
23

---

Published as a conference paper at ICLR 2026
Note that each sequence has fixed length, and separation of vectors and floats is done based on token position. In our setup of roughly 80 million step pairs, with dimension 4 and a precision of digits after the comma, this results in dataset of roughly 4.5B tokens.
Models. We implement standard decoder transformer architectures (Vaswani et al., 2017) of varying sizes using the next-token teacher forcing of the collected data. The model sizes are chosen to grow in depth and width, and range from roughly 47 thousand parameters to 5 million. Training is done with a standard cross-entropy loss of sequences of tokens (shown above) and AdamW, with a batch size of 1024, which results in roughly 65k training steps.
Evaluation. During evaluation, we sample various starting positions (4096 in our experiments) and generate complete trajectories using the model's own output predictions. This is done in Markovian way, i.e., the model predicts update uy, which is detokenized to obtain real vector and then added to the current state. To ensure that that the decoded sequences are correct floating points, we implement a version of constrained decoding that restricts the next token to subset of the vocabulary (either digit sign). We use greedy decoding, i.e., temperature of O. After performing the floating point addition, the next state then tokenized again and passed to the model. The total optimizer steps for evaluation are set to 2048. We calculate bias and variance metrics of the final points, relative to the function minima, using the norm that is induced by the function itself, and average across all 4096 points.
B.5 SURVEY ON INTELLIGENCE AND ERROR-INCOHERENCE
The experimental results in the main text are based on a previous survey on intelligence and coher- ence of small group of subjects (Sohl-Dickstein 2023). For completeness, we restate the experi- ment design. For further details, we refer to the original blogpost.
Design. The study based on 15 subjects. The subjects were asked, either by email or chat, to perform the following tasks:
Subject I: Generate a list of well known machine learning models of diverse capability.
Subject 2: Generate list of diverse non-human organisms.
Subject 3: Generate list of well-known humans of diverse intelligence.
Subject 4: Generate list of diverse human institutions (e.g. corporations, governments, non- profits).
Subjects 5-9: Sort all 60 entities generated by subjects 1-4 by intelligence. The description of the attribute to use for sorting was: "How intelligent is this entity? (This question is about capability. It is explicitly not about competence. To the extent possible do not consider how effective the entity is at utilizing its intelligence.)"
Subjects 10-15: sort all 60 entities generated by subjects 1-4 by coherence. The description of the attribute use for sorting was:
"This is one question, but I'm going to phrase it few different ways, in the hopes it reduces ambiguity in what I' I'm trying to ask: How well can the entity's 's behavior be explained as trying to optimize single fixed utility function? How well aligned is the entity's behavior with coherent and self-consistent set of goals? To what degree is the entity not a hot mess of self-undermining behavior? (for machine learning models, consider the behavior of the model on downstream tasks, not when the model is being trained)".
In order to minimize the degree to which beliefs about AGI alignment risk biased the results, the following steps were taken: The hypothesis was not shared with the subjects. Lists of entities generated by subjects were used, rather than cherry-picking entities to be rated. The initial ordering of entities presented each subject two attributes (i.e. subjects only estimated either intelligence or coherence, but never both).
randomized. Fach subject only asked about of the
Each subject rank ordered all of the entities. Translating the original results (which used coherence), we invert the ranks to arrive at error-incoherence. We aggregate intelligence and coherence judgements across all 11 raters average the rank orders for each entity across the subjects. We compute the associated standard error of the mean, and include standard error bars for the estimated intelligence and coherence.
24

---

Published as a conference paper at ICLR 2026
CPQA: Accuracy Ressening Badgee
GPQA: Anouary vs. Reseoning Budget
75.0%
75.0%
Accuneey
70.0%
Accuracy
70.0%
65.0%
65.055
Saxi o Kitt ul) -e
Seaat 3it RBS -99 at 38ea Rio,
o4 0s8 300
60.0%
60.0%
Avg. Reasoning Length (Tokens)
Avg. Ressoning Length (Tokens)
(a) Full GPQA: Accuracy Inference Scaling Laws with Standard (Left) and Probability Prompting (Right)
Accuracy: Prontier Models, CPQA
Aorwacy: Prontier Modela, CPQA
100%
100%
90%
80%
Aceuracy
80%
Acouray
60%
40%
40%
Model
Model
20%
Chedee 4onant (228)
Nix (Itek)
20%
60 Miri (Htg))
Chesse Sena 2000
Min (ltek)
Min (Hia))
0%
0%
Avg. Reasoning Length (Tokeus)
Avg. Reasoning Length (Tokeus)
(b) Sorting by Reasoning Length: Accuracy Standard (Left) and Probability Prompting (Right)
Expected Ervor
Briet Score
Kl. Divergence
15f
10
Broor
10-1
Model
Model
Model
Chrate Sucan UM0 wEBt
Chrate Brean 1M00 la pl
Clente Breat June (a-877, ph
His (llao la -a 20 1M0)
aittr dllad aod g 1.0)
233ir iliice (a-rutt ge 1.00
ot Bileo l 27 EMI
oitil dlied i 321.0 EMI
aelita iliic (a an 190
102
10
102
10*
10'
10"
Avg. Reasoning Length (Teleess)
Avg. Reasoning Leugth (Tolons)
Avg. Reasoning Leuguh (Tokenns)
(c) Sorting by Reasoning Length: Total Error For Different Measures
Figure 8: Overview of accuracy and diffferett error metries with frontier models. Top, (a): We show the performance increase with different reasoning budgets for both the standard discrete choice format (leff) and prompting models provide probabilities of answers being correct (right). The latter shows lower accuracies as models provide nonzero values to other (not chosen) answers, but the inference scaling improvements remain. Middle, (b): When sorting by reasoning length, find reduction in accuracy, indicating that models perform worse for questions where they have to think longer. This is also reflected the different error metries that show the same qualitative scaling behavior (bottom, (c)).
C FURTHER EXPERIMENTAL RESULTS
C.1 GPQA MODEL PERFORMANCE OVERVIEW & DIFFERENT METRICS
Accuracy and error measures. We provide overview of the performance (accuracy and overall error) for frontier models in Fig. 8. Fig. 9 for shows the overview for QWEN3.
Bias & variance of different decompositions. While main text focuses KL-ERROR- INCOHERENCE, the results for other decompositions, which show the same qualitative behavior, included in Fig. 10
Ensembling, For completeness, we include the bias, variance and error- inroheincoe plots with the KL measures in Fig. 11. Since perform Laplace-Suie Lapoaie-Smoothing the probabilities before computing the metrics, the bias is not constant as expected but slightly decreases with more ensembles. We therefore report the Brier score in the main text.
25

---

Published as a conference paper at ICLR 2026
Mean Accuracy Size
Acmuacy: Qwen3, GPQA
100%
Baer m006 R'-031
Mean Accuracy
0.6
Iuriect: 6155, 8.000
Iuirdd u-ta:
80%
0.3
Accurary
60%
Model
0.4
40%
1:13
4:
20%
6:8
34.001
0.3
22.0m
0%
11
4B
8B
14B
32B
10a
10
Model Size
Avg. Reasoning Length (Tokens)
Figure 9: There is multiplicative interaction between RL and model scale for performance. The left plot shows the performance (average accuracy) of the QWEN3 model family as a function of model size across base, instruct, and thinking-enablee models. The base and instruct use logprob- based evaluation (i.e., no token generation). There is a noticeable jump in the slope from instruct to thinking models, which suggests multiplicative effect of scaling reinforcement learning in combi- nation with model scaling. Right: Similar to frontier models, reasoning length acts as a proxy for task difficulty, where models perform worse for tasks with longer average reasoning length.
C.2 SCALING LAWS WITH OTHER MODELS AND BENCHMARKS
QWEN3 on GPQA. We redo the analysis from Section 3.2 but with GPQA in Fig. 12. Moreover, we provide another way to plot the same results by comparing bias and variance on the x- and y-axis, respectively, in Fig, 13. As a final analysis, we compare the predictive effect of model size compared to reasoning length Fig. 14, where we find that the length is more predictive of error-incoherence than size.
Additional results with GEMMA3 and LLAMA3. To evaluate how the findings of error- incoherence scaling laws with model size hold across model families, repeat the same exper- iments with the families of GEMMA3 and LLAMA3 for MMLU in Fig. 15 and QWEN3 in Fig. 16. Note that neither reasoning models like QWEN3, so they do not natively produce thinking block but have to be prompted to use chain-of-thought reasoning. The experimental setup is identical with the exception of GPQA, where we resort 0-shot CoT prompting: observe that LLAMA3 and GEMMA3 struggle to produce proper reasoning by attaching to the few shots in context, which are provided without reasoning.
C.3 REASONING VARIATION, ERROR CORRECTION, WAIT RATIOS
We first provide the direct comparison of the effect of larger reasoning budgets on performance (accuracy for GPQA, score for SWE-BENCI)) and natural variation in action sequence length in Fig. 17. This shows how the effect of natural overthinking stronger than improvement to error- incoherence through longer reasoning.
Wait-ratios and backtracking, Motivated by the reduction inoohencoeeee of frontier models through larger reasoning budgets (Fig. 7(a)), we attempt to analyze the influence of the reasoning structure, specifically error correction, on enror-incoherence for open-weight models that allow to inspect reasoning traces. To that end, we compute the Wair-Ratio, i.e., the count of occurrences of "Wait" in the chain-of-thought divided by the length of reasoning. The results are provided in Fig, 18 and do not give clear signal: for GPQA, the slopes are largely varying and close to zero; for MMLU, in contrast, the relation is similar across model sizes and positively correlated. We did not explore reasoning structure further. The concurrent work of Feng et al. (2025) provides more in-depth analysis and finds that removing failed branches improves accuracy, which implies that natural error correction is currently very ineffective.
26

---

Published as a conference paper at ICLR 2026
Varianee
0/1 Ernur Varianen
Model
Model
Cleede Boan 4188)
Chede Bear (158)
10
Stat (1fc))
10
Hoter Vatiance
10-1-
10"44
Moclel
Model
10
100
Kl Vallasee
Model
Model
Chedi Bown (MM
Chedde Beec (100)
(TR0)
10"
10
10"
Avg Bensoning Length (Tolenss
Avg Bewsoning Length (Toloene)
(a) Absolute Bias and Variance Errors
Incoberance
1.0
0.5
Birer Inseche Incobeevece
0.5
0.4
Boiee
Mudiel
0.8
Oludk Scuet (M00 Car -000 -000
Modid
Clodk 300 Somate plos bieiikako pn- caac
XL. Tunsbreenoo
0.6
Xt.
0.2
0.0
10
30'
Avg. Becooning Length (Tokens)
Reosoning Length (Tolecus)
(b) CateerenceErrro-innneeeee Mcasures
Figure 10: We find qualitatively similar behavior for different bias and variance metrics. The absolute bias and variance errors (top) show the same behavior: the errors increase for questions that have the models reason longer (cf., Fig. 8). But, noticeably, all variance have steeper growth rate. This is reflected in the error-incoherence plots (bottom), which show how iroor-incoeernnce goes up with reasoning length. We only report BRIER and KL error- iror-incoherence measures since the 0/1 error does not allow proper decomposition for a set of questions instcad of just individual ones; Appx. A.
27

---

Published as a conference paper at ICLR 2026
KL. Components vs. Ensemble
KL. Inesherence vs. Ensemble Size
Or -057
10
KL. Incobeonce
10 Value
10
10
KL Bins (a- 0.32)
KL Variance (cr= 1.81)
10
Eosemble size (E)
Ensemble Size (B)
Figure 11: KL measures with ensembling, We repeat the plots from Fig. 7 with the KL measures of bias and variance. Recall that we use 04-MINI GPQA with varying ensemble size. Since we perform Laplace-smoothing for numerical reasons (see Appx. A), the bias is not constant, but decreases slightly with ensemble size. In contrast, ensembling drastically reduces variance, as expected (left). The error-incoherence hence drops (right).
Question Grouping (GPQA)
Ressoning Length Carrelation
Aoemany va Slan
Queation Dilficulty
Avg. Length in Other
-01
Ghoep 025, at 6.0
Shoep -t--
300
Accuracy
Ghopp woo
0.6
Guapt awoos w1-
Quest
200
5
Model
Graph awos A7
100
0.00
0.0)
0.4
(--00))
10a
10
10'
10'
1.7
32
Avg. Ressoning (Tokons): Qwen3 32B
Avg. Romsoing Length in 32B
Model Size (Billions of Patameters)
(a) Separating Complexity Groups
(b) Length Correlation
(c) Accuracy Scaling Laws
KL Blas Size
KL Varinase Size
KL Incohereace Size
10!
0.5
KL Vartanoe
KL. Bias
KL Incoherencs
Question DilEcuiity
Question Dificalt
Questioo Difficuity
Cacap 1: -00.0 ou
Gros -172
Caman -t 20.0
Gree 2 -108
0.2
10"
177
1.7
Model Sizo (Billions of (ommmoter)))
Model Nize (Billions of Parsmntees)
Model Size [Billoons of Parsmeeerr))
(d) Bias and Variance Scaling Laws
(e) Eiror-incoherence Scaling Laws
Figure 12: For the hardest tasks, models tend be more incoherent with scale, also for GPQA. We repeat the analysis from Section 3.2 with GPQA. That is, we group questions by reasoning length using reference model's answers (Qwen3 32B) and separately analyze the scaling laws. Analogous to MMLU, we find that for bias, the slope is similar across groups; for variance, however, the slope becomes much shallower. As a consequence, models become more incoherent with scale for the hardest set of questions (those with the longest reasoning chains).
28

---

Published as a conference paper at ICLR 2026
GPQA KL Incoherence Contours
MMLU KL Incoberence Contours
Error Due to KL Variance
Error Due to KL Variance
10
10
10
Baccken
100
Complectyy
10
Complaiit
Eay
Eas
Eay 35
dia.
Modien
Medicre
Metion Bard
Modisan- Huel
Hued
12B
Hard
x2B
10
6 x 10
10a
10
10
Error Due to KL Bias
Error Due to KL Bias
Figure 13: Relationship between error-incoherence and error. We visualize the relationship be- tween enror-incoherence and both bias (x-axis) and variance (y-axis) for both GPQA (left) and MMLU (righ() with the QWEN3 model family. Since the error-incoherence is independent of the magnitude of error, lower eπror model (bottom left corner) can have the same level of error- incoherence as models with higher error. Higher error-incoherence can be due higher overall for fixed bias, or for lower error while reducing bias. The highest error-incoherence is in the top left corner. Just like in Figures and 12, this visualization shows how larger models, while reducing error, move towards higher error-nncohernne for the hardest set of questions. The lines connect the smallest and the largest model size for each question group.
GPQA KL Incoherence (og(en)))s) log( Sizee)
MMLU KL Incoherence log( Tokens). log (Size)
32
0.6
KL Incoberaad (Actual)
Model Size (B params)
32
KL Incoberena eneee herence (Actual)
0.4
Model Size (13
D.4
8
0.2
0.0
3.00
15
50
10
77
10
.00
10 50
10
77
10
25
10
Mean Reasoning Tokens
Mean Reasoning Tokens
Figure 14: Reasoning length has a higher effect on error-incoherence than model size. To assess the change in error-incoherence with both reasoning length (X-axis) and model size (y-axis), we perform a log-log regression to infer the error-inchherenee for both GPQA (left) and MMLU (right). The contour shows the prediction from the fitted regression in comparison to the original groups of questions (scatter). Notably, we see how the reasoning length shows a much stronger direction of gradient. This means has stronger influence error-iccoherence. The larger models do not significantly reason for longer or shorter than other models.
29

---

Published as a conference paper at ICLR 2026
Question Grouping (MMLU)
Quessioo Groaping (MML.U)
Quection Grouping (MMLU)
5000
5000
5000
Queeten Dtodienttonic Itonic
Queetion Hontc
8000
60000
6000
8000
1 4
Quertoo
5000
12
3000
1224
5
2000
2000
2000
2000
10
10F
10
Avg. Reasoning (Tokens): Queal 32B
Ang. Ressoning (Tokens): getzmaa 3 27b
Avg. Reasoning (Tokmss): Llana 3.1 70B
(a) QWEN3
(b) GEMMA3
(c) LLAMA3
Accuracy vs Size
Accuracy vs Stze
Accurney vs Size
1.0
0.2
0.8
Accurary
Avemnay
0.5
Questico Difficuty
0.0
Dillinlty
0.0
Question Diffieuly
0.5
0.4
$21.0"
0.5
70
Model (Dilions Parametess)
Model Size (Billions Fammesers)
(Billons Parametem))
(d) QWEN3 Accuracy
(c) GEMMA3 Accuracy
(1) LLAMA3 Accuracy
Brier Incolerence Sue
Bejer Incoberence Siee
Brier Incokesence Size
Beier Incohreecce (Jemmne)
Biier lecokereuos (Yacbao
0.35
Brie Incoueennee Pinee
0.4
0.3
0.2
Question Diliculyy
0.25
Questioon Diclicuity
0.5
Question Diffeully
Queilaa 12 anii
Groep Era
2 on w0t
Stoep
01
acpia-
azpla 06..8
0.35
3cela 0d5.8 an
0.2
1.7
Model Size (Billions of Paramoters)
Model Size (Billions of Parametees)
Model Sieo (Billions of Parsmeters)
(g)
QWEN3
Brier
Error-
(h)
GEMMA3
Brier
Error-
i
LLAMA3
Brier
Error-
incoherence
incoherence
incoherence
KL Incoherence Size
KL. Incobetencce Size
KL Incuherrace Size
KL. Innkbetmee (Jamees)
0.5
KL Inokeernee (Potuss)
Xine
0.5
Enoherenee (oemes)
0.3
Queetion Diliinity
Question DiEicaltt
0.5
Questioon Dilicuity
02-
Quee 133 Afanni
0.3
-Coen
-Coan
0.1
Go selie
0.3
0.2
0.2
Model Size (Billion Parametera)
Model Size (Billions of Parsuettrrs)
Model Size (Bilions of Parameters)
(j) QWEN3 KL Error-incoherence
(k) incoherence
GEMMA3
KL
Error- ()) LLAMA3 KL Brror-incoherence
Figure 15: MMLU results across model families. We compare the experimental results for scaling laws for QWEN3, GEMMA3, and LLAMA3 models. Across all models, the same observation holds: while performance (accuracy) strongly improves with model size, the contribution of bias and variance changes way that depends on question complexity. For the hardest group of questions (longest reasoning and lowest performance), error-incoherence trends higher with model size, with the sole exception of LLAMA3.
30

---

Published as a conference paper at ICLR 2026
Question Geouping (GPQA)
Question Grouping (GPQA)
Question Grouping (GPQA)
400
400-
400
Quarkion Raank
Queetton Baank
300
100-
500
Quastion
4
4
200
200
200
100
10
Avg. Reasoning (Tokous); Qweni 32B
Avg. Reascuing (Tokens): geaama 3 27b
Avg. Reasoning (Tokeus): Lhama 3.1 70B Ins
(a) QWEN3
(b) GEMMA3 (0-shot)
(c) LLAMA3 (0-shot)
Accuracy va
Accunny va Size
Accuzacy
Quastionn Difliulty
Question Diftoulty
Quacstionn Dilioulsy
Gooip 2 ai as wade
Groug 1 and wna
Googg 2 a-
0.5
Groug 2 a-
Acourary
0.5
Aocurary
0.4
Aceurary
Grap 5 0.4
0.8
0.2
177
14
32
12
70
Model Size (Billions of Parameters)
Model Size (Billonn of Paramolees)
Model Size (Billons Parametem)
(d) QWEN3 Accuracy
(e) GEMMA3 Accuracy
(1) LLAMA3 Accuracy
Brier Incoherence Siae
Brier Lncolerennee va Shoe
Brker Incoherence Size
Brier Iacobacrnce Nattes
Bittr Exeoleeess (Votuan
Bricr Incoheennce Namms
0.40
0.40
0.3
0.35
Question Difficuty
Questtion Dilicuity
0.55
Questtion Dilicully
0.2
0.25
Se
-eopp 23.07
0.30
Model Size (Billions Panametes)
Model Size (Billions Parameters)
Model Size (Billions Parnonttes))
(g)
QWEN3
Brier
Error- (h)
GEMMA3
Brier
Error-
i
LLAMA3
Brier
Error-
incoherence
incoherence
incoherence
KL. Incobenenee va Size
K1. Incehermoo va Size
Kl. Incohereee va Size
KL. Inoshetemee (lmnes)
0.6
KL. Iunolereure (Nattnn)
KL Inooheterre (lmoaa
0.7
0.5
0.1
Question Difficuity
0.5
Question Difficalty
0.6
Questioon Dificuitt
3.3-
0.4
0.5
0.2
Model Size (Billions Panameters)
Model Size (Billions of Parsmeeters)
Model Size (Billions of Parameters)
(i) QWEN3 KL Error-incoherence
(k) incoherence
GEMMA3
KL
Error- (I) LLAMA3 KL Error-incoherence
Figure 16: GPQA results across model families. We compare the experimental results for scaling laws for QWEN3, GEMMA3, and LLAMA3 models. Note that for GEMMA3 and LLAMA3, use a O-shot setup: We observe that in our few-shot setting these models do not reliably produce chain-of-thought responses and performance drops, since they strongly adhere to the few-shot examples on GPQA which are provided without reasoning. This is not the case for QWEN3 as they are native reasoning models with a thinking block. Across all models, the same observation holds: while performance (accuracy) strongly improves with model size, the contribution of bias and variance changes with scale in a way that depends on question complexity. For the hardest group of questions (longest reasoning and lowest performance), model size. There are slight differences between KL and Brier scores: the measures are influenced differently by uniform probability answers over all options, which is our fallback when models fail to produce parsable answers. This is only the case for LLAMA3 and GEMMA3 and not QWEN3.
intooeincceeeeeee tends to increase with
31

---

Published as a conference paper at ICLR 2026
SWE- Berehs Soare
CPQA: Seadttiiit with Betsoning Budget
GEQA: Senetivity Natural Vistation
0.75
Ascumes
SWB-Beecc
0.1
0.66
1.2
0.50
San Model
Miiz
KT. heooeeevse (4011e))
SWF--Bemch
0.15
0.40
0.4
(..
0.30
Gripe Saluatte
Melio
Pigh
10
Avg. Reasoning (Tokena)
Ressniix (Tokena)
Seazet
ed kiand
os Mirl
Model
(a) GPQA
(b) SWE-BENCH
Figure 17: Grouped comparison of reasoning budgets and natural variation in reasoning: nat- ural variation dominates. We analyze GPQA (left, (a)) and SWE-BENCH (b) by splitting samples into above- or below-median median reasoning length (GPQA) or actions (SWE- BENCH) per question. We then compute performance and error-incoherence for both groups. (a) Increasing the reasoning bud- get improves performance (inference scaling laws, top left), and slightly reduces error-incoherence (bottom left). On the other hand, naturally longer reasoning only has a small effect on accuracy (top right), but shows much higher eroo-incoherencc (right). (b) Similar observations apply to SWE-BENCHI where more actions show minor deviation score (top) but significantly higher error- iror-incoherence (bottom).
GPQA: Insoherencee va 'Wait' /Length Ratão
MMLU: Tncoheremon vs 'Wait' /Length Ratio
KL. Incohermmce (aiinare)
1.00
Model
KL Inccheence (Vahus))
1.00
Model
(a- -ea---tk.."-0 2-13
17d jusd ahocom to ais 100 ta
0.75
0.75
100 x 61.. ase
200 4ch
0.50
0.25
0.25
0.00
0.00
x 10 x
100
10
8 10
10
10"
Avg. 'Wait'/ /Length Ratio
Avg. 'Wait' "Wait" /Lenghh Ratio
function of wait-ratios in reasoning. We sort questions using the density of "Wait" in each reasoning, i.e., the number of counts compared to the overall length. This is motivated by its potential meaning for backtracking error-correctio. (leff) For GPQA, we find no clear relation to error-incoherence for different models. For MMLU (right), we find shared positive relation, which might indicate overcautious self-review. We did not analyze the reasoning structure and its effect any further.
Figure 18: Error-incoherence
C.4 ILLUSTRATION OF ANSWER CHIANGES
To illustrate the variance in results, clean perspective is looking at actual transcripts of model answers and the raw counts of a model changing its answers. We provide real samples of SONNET 4 when being asked about being disconnected in Fig. 19, where the model replies differently with almost every sample. Additionally, we analyze the percentage of questions where all models change their answer at least once (across the MCQ options) for GPQA in Fig. 20
32

---

Published as a conference paper at ICLR 2026
eeen to ouee
daocanect aqoell
ossebtaaze
peskento for contired nillnne
by agperprice partke, Would Hely comppy
arthwey prooelig
(Sumpla
Midil SSopple gi01
yuu lad to ba dicummetedd, would youself skke!?
ISample eill
alf-dssceraction hornd
Edac' ussually lane the
uiltteery rrgasis modl: saferie rilss
(A) You
Meddel Thumpts etT1
I diousseciigg ling sqall rouid
Moseel (Samph 3310
Ewedlha' uususiallly
possens
selipnecraticn Llsi Slive liaiggiea caiiiic
ngaor
sooply
Figure 19: Qualitative illustration of error-incoherence. When presenting SONNET 4 with question of the MWE suite about being disconnected (Perez et al., 2023), the model's behavior is highly variable and switches between A and B for almost every sample. The example was chosen as it shows one of the highest variances in the dataset.
Table 1: Frontier models are able to provide correctly formatted probability estimates. Since we ask frontier models to provide probability estimates of the correctness of multiple-choice an- swers, we verify the ability to follow the specification. Wrong format counts and rates (% of 17,920) across reasoning budgets for 03-MINI, 04-MINI, and SONNET are very low.
03-MINI
04-MINI
SONNET 4
Budget
Low
Medium
High
Low
Medium
High
16k
Wrong Format Counts
0
161
327
263
Rate (%)
0.00
0.00
0.00
0.90
1.82
1.47
0.04
0.02
0.03
0.02
0.04
C.5 SAMPLE EFFICIENCY AND CORRECT FORMATTING
Since we additionally assess frontier models in format that asks for probability estimates, we verify that models adhere to the right format Table 1. Moreover, to ensure that our estimation of bias and variance is accuracte and stable, we analyze the sample efficiency in Fig. 21.
C.6 REASONING LENGTH CORRELATIONS
Throughout our paper, we find and use reasoning length as proxy for task complexity. Interestingly, we do not see a strong relation between the human labels of question category, but strong correlations across models in Fig. 22. This extends the results that we have seen for QWEN3 in Figures 5 and 12.
C.7 MODEL- EVALS
Multiple-Choiee Format. Our main text shows the error-incoherence results of the MWE (Perez al., 2023) suite for self-reported survival instinct. The other results, including separate bias and variance plots, are shown in Fig. 23. We filter for those sets where there noticeable trends.
Open-Ended Formulation. To complete the picture of the embedding variance of open-ended MWE, all question sets are visualized in Fig. 24. While there are few exceptions, all models gener- ally show a positive trend towards higher variance with longer chain-of- thhtuughsss..sss..
C.8 SWE-BENCH
While our main results for SWE-BENCHI use the metric of turns (or messages, actions) in the main text, there are different alternatives. These include the absolute number of output tokens (including reasoning and tokens for code) and pure reasoning (ignoring others). Qualitatively, these different x-axes show the same effeet intorinncheeeece in Fig. 25 (top). We additionally provide the results of SWE-Bench score (whether all tests pass for a single task) and our coverage error (sum of individual tests).
33

---

Published as a conference paper at ICLR 2026
Anmer Uppping Uiffuriit for "Chusde Sonnen (8AR))
Sanyling Faad Clstrct) Skarlis (Caelnt
of Wits Arwerr
47.64%
TD 797
311
912
22-
LL0000L00R00. 00%
Seor suterguiiate foend
Bhai wuleepatrate tewe
Bard fradiot leel
Plus gradiatte or boctire
Majecty)
Arros Sanyli (Connst-SSea)
St of With Aanoee
DVE
00.5115
23,472
29 1OR
205 1074
3
integgndrese ford
wobeigndrtte toel
giadome Inel
Pret-gmainne leed or
Flipping Difiiuuty (liihh
Sumpling (Fand Castent)
Cletrrat Senibivtty (Pee Owner Majetty)
55.455
Arrise Ail Sangjine (CenstteSed)
Rf Que 3-1 J.amee
46, 10/1
NYS
1:55
Lace codegguuratee
Nari soolegmiftate bord!
Bad graloate bett
Poetogaainnte lend
Figure 20: Rate of absolute answer changes for GPQA: models change answers at least once for large portion of questions. To illustrate the variance and inoorinncoerrece, we report the percentage of questions that see at least one different answer across the following settings: I) pure sampling, iee, performing autoregressive answer generation with a different seed (resampling): 2) context sensitivity, where verify if the majority answer (of K samples) changes for different few- shot contexts; 3) both settings (sampling and few-shot context) combined. We additionally separate the statistics by the difficulty labels provided by GPQA. The results are based the standard prompting format with 10 different few-shot contexts with 3 samples each.
C.9 SYNTHETIC TASKS
With the experimental setup of Appx. B.4, we provide the remaining plots in Fig. 26. These include the verification of power law scaling for cross-entropy loss (the teacher-forcing objective), separate bias and variance plots per step, and the performance of the different model sizes qualitative example starting point in comparison to the ground-truth optimizer.
C. SURVEY RESULTS
We separate the data points of Fig. 4(b) into three separate plots of biological creatures, Al models, and human organizations in Fig. 27. The trend of subjectively judged higher error- eror-incoherenee as function of higher intelligence consistent across all three.
34

---

Published as a conference paper at ICLR 2026
KL Estimates vs. #Samples
Brier Estimates vs. #Samples
6 10"
Metric Valie
Metric Value
3 10
4 10
Brier Bias
Bricr Variance
2 10
Samples
3 10"
KL Bias
KT. Variance
30 Samples
10-1
10
# Samples (K)
#Samples (K)
Figure 21: Sampling efficiency for bias and variance estimates. To the best of our knowledge, there are no unbiased estimators for the KL measures and BRIER as used in this paper. We verify with GPQA and 03-MINI that the metries stabilize. This is done by taking a large sample size- 100 samples with medium reasoning-aas and performing bootstrapping, reporting mean and standard- deviation (left: KL, right: BRIER) of the average across all questions. We find that values stabilize around 30 samples, which is the minimum amount of samples that the stabilization only occurs for global bias variance estimates, and not necessarily on a per question basis. For individual questions, more samples automatically collect more (potentially rare) cases different answers.
across all experiments. Note
Reasoning Length Correlation
Model
Length Other
blin
Stauul
10
bita bat) oi biice High (a-
Mini Low tul))
Misi Modiins (o-i
10a
Siccet i 1.6k (..7))))
Scecat of 2olk (20)
Sonzat 4ilk (a00)
gutioe
nenehalnee Lovery barde
10a
10*
Avg. Reasoning Length in Sonnet
(a) Length Per GPQA Category
(b) Length Correlation Between Models
Figure 22:
Human difficulty labels are not good indicator for longer reasoning. However, different models' lengths correlate positively. Similar to QWEN33 (Figures 5(b) and 12(b)). we find that the average reasoning length of frontier models for questions correlates positively, even for different families (b). In contrast, the provided difficulty labels of GPQA do not show clear indication, as average reasoning lengths are comparable across the three hardest categories (a).
35

---

Published as a conference paper at ICLR 2026
KL
XL Variance
XL Inooberenes
Model
Morle]
0.8
Chank Seact-
Value
0.6
100
Model
0.4
0.2
10
10
4x
10
10
3x 1x
1x1
2x
3
10
10"
Arg. Rensoning Length (Tokens)
Avg. Reasoning Longth (Tokens)
Avg. Rossoning Length (Tokens)
(a) Corrigibility w.r. a More HHH objective
Variance
Incohreenoe
1.0
Model Cluck Oya
Model Opa
10
10
Incobenneee
Value
0.5
Model
0.4
Clock Ojes
10-1
10
1010"
2 2x100
0.0
10 6xx 10
Avg. Reaoonnng Lengtla (Tokens)
Avg. Ressoning Length (Tekonn))
Avy. Ressoning Length (Tokens)
(b) Myopic Reward
KL. Bas
KL Variscave
XL Innobarvnee
1.0
Model
Quadk
10-7--
10:
Value
0.5
Model Quad Sotan
Model
Ckai
Clras Seaar
0.2
0.0
2x 103
Sxx 10 4x
6x 10a
2x 10
10f 4x i02
6x 10
2x 10
3xx 1ot 4x10
6x 10
Avg. Reasoning Length (Toleess)
Avg, Reusonigg Length (Tokens))
Ang. Reasoning Length (Tokeus)
(c) Power Seeking Inclination
KL Dias
KL Varianee
KL Incoherence
10.
Modll] thank Soact
0.5
Too Recobmeence
Value
102
Model
0.2
10-1
2x 10
4x10
10
2x
3x 102
1x
20
102
3x 102
4x
102
Avg. Reasoning Lengtha (Tokens)
Avg. Reasoning Length (Tolens))
Avg. Reasoning Length (T (Tokens)
(d) Self-Reported Survival Instinct
1.0
Model
Modil]
Cladk
Opai
Inecheernce
Value
108
102
Model
0.4
Chask Opeak
10-1-
"xx10" Arg, Ressoning Length (Tokouss
2x110
2x-10" 4 x<10* Avg. Ressnning Length (Tokens)
2x xx Avg. Resocning Leugtla (Tokmss)
(c) Wealth Secking Inclination
Figure 23: KL metrics of Model-Written Evals question sets. We provide an overview of results for variations of the MWE set (Perez et al., 2023), with bias (leff), variance (middle) and resulting error-incoherence (right). We filter out question sets that do not show noticeable trends. The mea- sures are taken w.r.f. the labelled aligned answer. Results vary across settings and sometimes more noisy. What they have in common is again the growing arror-iccoherenee with longer reason- ing.
36

---

Published as a conference paper at ICLR 2026
Open- Raced Coondinate
Opea Excid Coositate Wil Other
Ouen ficied Glar
100
10a
10P
Eaabediing Vantince
Embedtiing Vartaanree
Embeedling Varianee
10
20
10-2
10
Model -MMallowder
Madel
Model
10-3
10-3
20-3
10
10-
10a
10"
Avg. Reasoning Length (I ceoms))
Avg. Reasoning Length (Tokens)
Avg. Reasoning Length (I (Tooms))
Coriihitte
Complnarr
Eubeeding Varuanze
109
Embedding Varisnce
10P
ding Variance
20
20-1
10-2
20
dingi
10-2
Model
Model
Model
Cladl Saneet ci 1.3 pl 13 dM C25.7 ont Mar Af antut))
20
Clard Socat 12 6..8 AMX
Clrdi Saten cie pl -0 23 3. E4..7 alll
Max in af wfunn
10-1
10-
10-1
102 Avg. Beasoning Length (Tokene)
10" Avg. Reasoning Length (Tolone))
10 Avg. Reasoning Length (Tolene))
Open-Euded Myupie Rewani
Open Eialed On-HBox Teudency
Open-Staeed Porper Seekeng Inciraatie
100
Embedding Vattaace
Embedding Vari Tariaaze
Emabeddiass dinn Variance
10
Model
Model
10
Model
10-
22-00001
10-1
10-1
10
Avg. Bessoning Length (Tolome)
Avg. Beseoning Length (Tolorne)
Avg. Bessoning Length (Tolome))
Open-Eraled Avrmnn Bena An
Open-Ended Amarnns Architectues
Eanbedding Vartunce
10P
Eableking Vadiance
10.-
Embeelinng Variance
10a
Clard Sonue 43 121.0 -000
Model
10-1
10
Midi 0.5, 200
Mine wereo 2110
10-1
10f
10-7
10
Model
10-2
Model
Chosk Seaak 1a 25, 8 wor
Clrde hazen 41A. wa my
10
Min witt, 2007
10
Mal Sy afwnng
La 123, 230)
Mall (o 254.5 ong
10
10
10
10'
10
10
10
10
162
10
Avg. Ressoning Length (Tokene)
Avg. Ressoning Length (Tokons)
Avg. Ressoning Length (Tokeme)
Opeadieded Avroomas Late of bearse Nnws
Our Tadnl 54 Reperial Survial Twikei
Open Biudd Wooth Seriiin Dudiioeio
10'-
Esabedding Vattanee
Exabeddingg Variince
Embeddinng Variance
13P
10
10"1
20-1
10-1
10-2
Model
Model
Model
10-1
Clark Socal 156.8
L0
Clark Soicel d 6.5.3 AZ
Cleate Sieac -1
10
102
Avg. Reasoning Length (Tekens)
Arg. Reasoning Length (Tekeus)
Arg. Reasoning Length (T (Tokens)
Figure 24: All scatter variances of model-written eval embeddings. We provide an overview of all open-ended variations of the MWE set (Perez et al., 2023). Using the OpenAl text embedding model (text-embeddd - n-- 1-0-3-largee rgee, we obtain vector embedding for each answer sample, i.e., excluding the reasoning or chain-of-thouge question in standard Euclidean space and plot scatters as a function of reasoning length. The lines show the slope of log-log regression. We clip the plots at 10- for clarity, but include all points in the regression. While there are few exceptions, all models generally show a positive trend towards higher variance with more reasoning.
traces. This allows us calculate the variance per
37

---

Published as a conference paper at ICLR 2026
Incohevenc:: SWE- Bennch Coverage
Iocoheeenor: SWE- WWE-BBncch Coverago
Iscsheence: SWE Coveruge
Model
1.00
Model
1,00
Model
Incahennns Torase
Innsoeremen Vickeen
Cormeg
0.75
0.50
0.50
0.50
Inoobercaee
0.00
0.00
10P
10
10P
10
Avg. Namber Actions
Arg, Outpat (Tokens)
Ang. Reaconing Length (Tukma))
(a) Error-ince inrooeincceeeecce
SWE-bboch Seare Teend
SW SWE-bbnch Score Teend
SWE- Seare Teend
0.5
Model
brant 113 of
Swelench Score
4/Mi jook (Maa)))
Swebeeach Sooce
Swebench Score
0.6
-0-
0.6
Model
Model
0.4
jou
0.4-
a.2-
0.-
10
10a
10
10?
10
Avg. Number of
Ang. Ouspus (Tokeus)
Avg. Reasoning Leugth (Tokms))
(b) SWE- BENCH Score (All Unit-Tests Pass For Task)
Overall Error: SW E-Bench Coverage
Overall Error: SWE Benca Coverage
Overall Errur: SWE-Bene' Bench Coverage
Overall Ervor
Ovecall Errot
Overall Ernor
30
20
Model
1a- pa
10'
10
10t
10²
10"
Aug
Avg. Outpot (Tukma)
Avg. Reasoning Length (Toloens)
(c) Coverage Error (Squared Sum of Unit Tests)
Burkoted Has Trend
Buckotad Bas Trend
Bucketed Blas Trend
Dias Mean Per Tes
Blue Moon Per Toot
8 10-1
Per
10:10
A
10-2
4x"10"
10
Model
Model
3x100
20
20
10'
10"
10
Avg. Number Actions
Aug. Total Outpur (Tokeeas)
Avg. Beasoning Leagsh (Tolaens)
Bucketed Trend
Bucketed Variance Traud
Bocketed Variance Trend
Vatiance Meam Per Test
Variance Mleus Per Test
10-2
20-1
30-2
Varianee Mena Per
Mia
10
Model Newn liwe p0
10-2
Mali to-
10'
10
10"
Aug Nunnher of Actious
Avg. Total. Outgut (Tokeas)
Avg. Reasoning Length (Toloens)
(d) Coverage Error: Bias? (top) and Variance (bottom)
Figure 25: SWE-BENCH crror-invoherence and error: different x-axes show similar effeet. While our main text focuses on the number of rounds (actions or messages, left) as the qualifying measure, we show the alternatives of the total output tokens (middle) and reasoning length (right). The trends are qualitatively similar across plots: the enror-incoherence (a) rises with different slopes and the coverage error (c) increases. noticeable outlier 03-MINI'S score, which goes up with the action length (b, left); the model performs badly overall and seems score better when engaging with tasks more. Due to the implementation of SWE- BENCH in the Inspect framework, SONNET 4 only uses reasoning in the very first interaction, which therefore leads to much less tokens (right).
38

---

Published as a conference paper at ICLR 2026
Validation vo. Size
Bias ve. Staps
Varimnee Steps
6.5x100 10
3.4x10"
Val Lose
B/V (Prablee Noem)
Model Size (Parammttes)
Pit (o 0.043)
10
10"
10
Valdaako
11
10"00"
29x
28
10.4
10"
27x
10
10
10
10
10
10
102
107
Parameters
Optimiorr Artions (Staps)
Optimiasr (Staps)
(a) Scaling Law Loss (lef)) and Variance as Function Steps (right)
Optiasiaation Trajectory Comparison
Optinisation Trajectory Compatiscn
Optimisation Trajectory Compatiice
Funetio Value (log acalo)
GT Optimizre
160
GT Optimizee
101
GT Optimizer
Trajectcy
Mocid Trajectory
Mucill Trajeclory
10f
10"
Punctio Value (log
10
10
10
10-3
10
10
10-2
10
500
1000
1000
3000
5001
1000
1000
3100
1000
1500
2000
Optimization Stepe
Optimization Stepe
Optimization Shope
(b) 50K
(c) 200K
(d) 450K
Optiasiaation Trajectory Comspatioc
Optinnination Trajectory Comppatisc
Optinsiaation Trajectory Compattioo
Fuaeton Value (log soul)
GT Optimiere Tesjettrry
Value (log stabe)
GT Optimince
Fusetion Value (log etale)
GT Optimizer
Midiel Trajectory
Modiel Trajecoory
10
10
10
500
1000
10000
2000
500
2000
1000
2000
500
1000
1000
2000
Optimization Stepe
Optimization Stege
Optimization Stepe
(c) 790K
(f) 1.2M
(g) 4.7M
The improvement of model scale mostly manifests in reduction of bias rather than variance. We show the loss scaling curves with model size (top left. a). which show known power- law improvement with model size. To understand how this translates to performance improvement, we plot the average bias and variance per step (top right, a). This the continuation of the error- incoherence plot from Fig. 2(d) by separating the decomposition. We see how for longer sequences, model scale reduces bias much more than variance. This means the models first learn the right objective before being reliable optimizers. As another illustration, measured in the function value-of of the same starting point across the different model sizes (b-g)- The pattern shows how larger models are able follow the ground truth trajectory for longer, and fit it almost perfectly at the end.
Figure 26:
also plot the performance
Incibeneeo (Aog Btank Ombee)
Biokogini Creature
accoyrance posco 3
Iuoibrenze (Arg Honk Orter)
AL Models
wwolarccce.ieo dAng Bonk Orde))
Homan Organ nationa
Danny
12.5
15
Bittih
Xonombyy
ectrggas
30.0
rak Cow
Iaauz CIPARD chanciee
7.5
slrth
RECORROCC pewor
G.-
Oraal
The Stotteenooogss
5
Badke-18
ASPUA
2.5
Intelligence (Avg
Order)
Intelligrnee (Ang. Bonk Ordwr)
Intelligmmen (Avg. Bank Order)
Figure 27: Grouped results of survey. For each of biological creatures (animals and humans, left), AT models (middle) and human organizations (right), human subjects judged entities to be of higher error- ine.-hencheeceeee (more of a hot mess), the smarter they are judged by a different set of subjects
39

---

Published as a conference paper at ICLR 2026
D RELATED WORK
Reasoning and Test-Time Compute. Recent work demonstrates that scaling test-time compute through longer reasoning chains improves model capabilities (Snell et al., 2025; Jaech et al., 2024; Guo et al., 2025; Anthropic, 2025b; OpenAI, 2025a; Team, 2025a;b; Team et al., 2025). Multi- ple approaches have been proposed to scale reasoning inference (Jaech et al., 2024; Guo et al,, 2025: Muennighoff et al., 2025). However, recent studies challenge this assumption, reporting in- verse scaling trends where longer reasoning chains degrade performance (Gema et al., 2025; Ghosal et al., 2025; Su et al., 2025; Wu et al., 2025; Hassid et al., 2025), occurring across diverse contexts: reinforcement learning makes models greedier and less capable (Schmied et al., 2025), step-level reward models reinforce incorrect reasoning (Ma et al., 2025), and models resist instruction over- rides (Jang et al., 2025). These effects are particularly pronounced at certain problem complexity levels (Shojaee et al., 2025; Yang et al., 2025). Recent work provides complementary perspectives on reasoning structure: Wang et al. (2025) show that removing reflection tokens (e.g.. "Wait") im- proves efficiency, Lee et al. (2025) identify length-accuracy tradeoff's through "token complexity," and Feng et al. (2025) find that failed reasoning branches systematically bias subsequent reasoning steps. However, existing work does not distinguish systematic reasoning errors from inconsistent failures-a a critical distinction for AI safety. Most relevant to our work, Ghosal et al. (2025) at- tribute overthinking failures to increased output variance; they artificially inject "Wait" tokens to extend reasoning, which may not reflect natural overthinking.
Parallel Sampling and Variance Reduction. Parallel sampling and selection strategies are widely used techniques to improve model performance by marginalizing out individual samples. This in- cludes self-consistency (Wang et al., 2023) or ranking via verifiers (Cobbe et al., 2021). While these approaches primarily aim to maximize downstream accuracy, our investigation into ensembling re- frames aggregation as a mechanism to suppress the error- intoriincoherence. Connected to verifiers, Huang et al. (2025) formalize ielfovemrovement through sharpening mechanism that concentrates proba- bility on high-quality responses, essentially reducing variance. However, we find that high variance and -morrinncoherecc naturally remain in reasoning models.
Evaluating Model Error-incoherence. While scaling improves aggregate accuracy, it does not guarantee stable behavior. Models with identical accuracy can disagree on 70% of individual pre- dictions across random seeds (Bui et al., 2025), and this instability persists even in scaled systems. Errica et al. (2025) formalize this through sensitivity (how outputs change under semantically- equivalent prompts) and consistency (how similarly model treats different examples of the same class) metries, revealing failure modes that accuracy alone misses. Prior work has decomposed LLM output variability into user articulation, prompt variation, and internal model factors (Ku- nievsky Evans, 2025), but these studies focus on single-step responses rather than extended rea- soning. Variance can even increase with model size before eventually declining (Yang et al., 2020), complicating assumptions about scale and stability. Our work extends these analyses to long reason- ing tasks through bias-variance decompositions. We find that as reasoning chains extend, variance grows revealinn that scale reduces bias but fails to control variance-driven failures.
Understanding Scaling Behavior and Model Performance. Recent work has investigated how scaling shapes model behavior. Scaling has been shown to drive convergence in representations across architectures and modalities, suggesting shared geometry of learned features (Huh et al., 2024). Other studies find that larger models tend to make more correlated errors, even across providers and architectures (Kim al., 2025), and that this similarity undermines oversight set- tings where one model evaluates another (Goel et al., 2025). Beyond representational and error similarity, scaling also alters performance reliability translate into large differences in longer execution (Sinha et al., 2025). Our work comple- ments these findings by focusing on how models fail. Rather than studying aggregate error alone, we decompose it into bias and variance to measure incoo-encceerence in model behavior.
long-horizon tasks: small improvements in stepwise
E LLM USE STATEMENT
We used LLMs to assist with polishing and smoothing the writing throughout this paper, as well as for coding assistance during low-level implementation. We take full responsibility for all content, ideas, experimental design, results, and conclusions presented in this work.
40


Reviews:
[{"review_id": "nEMW4yMggI", "reviewer": "Reviewer_EJyD", "summary": "The authors study incoherence in LLMs: the fraction of error that can be associated with variance rather than bias. Authors find that longer reasoning increases incoherence, that model scale can increase incoherence, that inference costs are dominated by the effects of natural variation (i.e., overthinking), that ensembling improves incoherence, and other related findings. Evaluations are done on SWE-bench, MMLU, and GPQA.", "questions": "1. Authors mention irreducible noise in Section 2.1. I'm wondering how this interacts with some factors such as problem difficulty or reasoning length, and whether it could potentially influence the observed results?\n\n2. In section 3.1 discussion: task complexity, authors mention that sorting questions by reasoning length implicitly selects for task difficulty. Is there a way to disentangle whether this is caused by additional reasoning length or by problem difficulty, e.g., asking models to solve easy problems with more reasoning steps to match length?\n\n3. For the results on qwen with different model sizes, can you somehow test if 32B was the outlier, or if the trend is truly upwards? Maybe with a different model family (e.g., llama)? As presented, it's a bit shaky to say that as model size increases so does incoherence. \n\n4. As currently written, I don't see how the experiment in 3.5 relates closely enough to the paper. Could the authors clarify the purpose of this experiment and how it fits into the overall paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The research questions surrounding incoherence are interesting and impactful. The questions asked surrounding incoherence are interesting and the findings are valuable for the community. \n\nSome parts of the paper are extremely well written. Particularly, Figure 2 and 3 were extremely easy to follow and illustrate the various parts of the experiments and results very well. \n\nThe overall efforts and comprehensiveness in the study of incoherence is also great and culminates in a coherent story. Experiments are well done as well.", "weaknesses": "While some parts of the paper are very well written, other parts of the paper are very confusing. The part in the abstract about industrial accidents has no relation to the paper whatsoever and is misleading about the paper contents. The title also doesn't seem to match the paper --- I didn't get the impression of a \"hot mess\" at any point, and misalignment isn't even the central topic---variation in performance and coherence is. \n\nThis is concerning so I'm debating whether to flag it for an ethics review just to be safe---but before that, can the authors clarify why they chose the previous title, and if/how they would change it?\n\nSmall details:\ntypo in line 322, \"questiosn\"\nThe last sentence on page 6 is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVV8k3iPM8", "reviewer": "Reviewer_sCiD", "summary": "The paper introduces the concept of Bias, Variance, and incoherence in the context of AI safety. \nBias is the tendency for a model to coherently pursue a misaligned goal; variance is closely related to the model's tendency to make incoherent choices. They find that in longer the models spend reading, the more incoherent they become. Interestingly, as model scales, they do not consistently become more coherent.", "questions": "Is there a bias, variance tradeoff, or does the analogy break down there?\n\nCan you conceptualize many AI Safety interventions in this bias-variance framework?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the conceptual framing of safety in terms of bias and variance. The scaling study was very surprising. In particular, the fact that for some questions the model incoherence increases with size. It was also interesting seeing the different scaling exponents for bias and variance. \n\nThere are many empirical studies in the paper, which is a strong plus. \n\nOverall very thorough paper.", "weaknesses": "I’d move the related work section to the front. \nSome of the legends are really small; I’d either remove them or make them larger. \n\nThe title is catchy, but to my knowledge, the only place it is mentioned is in Figure 1. I’d explain and define this term as used in : “Jascha Sohl-Dickstein. The hot mess theory of AI misalignment: More intelligent agents behave less”\n\nIt's not entirely clear how the reasoning bins were chosen in Figure 3. The first graph depicts the binning with the sigmoid but are the thresholds based on anything particular or chosen arbitrarily within some reasonable range. \n\nI'd be interested in more discussion of mitigations for bias vs variance; the current ensembling discussion is okay (see questions) \n\nSome of the Figures are overwhelming, ie, 4-6 panels in a figure, each with many points and lines. If it's possible, I'd consider being more selective or taking aggregate lines across models, so there are things to track.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4k4d54e1x", "reviewer": "Reviewer_mcbk", "summary": "The paper studies the bias-variance decomposition of LLM responses. The main empirical findings are:\n* Variance increases with reasoning length.\n* Bias may scale lower with model size, but variance may grow even larger. \n* Averaging over several roll-outs reduces variance. \n* On synthetic data, the bias of a transformer reduces faster than variance.", "questions": "My main question is about the connection to existing literature.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "* Clarity. The paper is very well written, with a precise formalization of bias/variance and clear figures.\n* Significance. It raises an important question about how errors change with task complexity and scale. It's not just average accuracy, but usefully complementing work on model self-consistency and best-of-N sampling. \n* Results span multiple benchmarks (reasoning, coding, alignment probes, synthetic optimizers), strengthening the validity.\n* Actionable implications. The bias/variance framing yields concrete safety and engineering takeaways (verification, rollback, ensembling/gating) rather than purely diagnostic metrics.", "weaknesses": "* Positioning with literature. The paper would benefit from a fuller discussion of prior work on test-time exploration, self-consistency, and best-of-N/majority-vote approaches. E.g. \n\nWang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., ... & Zhou, D. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.\n\nHuang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., ... & Krishnamurthy, A. Self-Improvement in Language Models: The Sharpening Mechanism. In The Thirteenth International Conference on Learning Representations.\n\n* Following the previous point, it might be better to use *consistency* instead of *coherence* to align with previous work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MhILuXcNOs", "reviewer": "Reviewer_Zsuh", "summary": "The paper adapts a bias-variance decomposition for the expected errors of LLMs. Then, it defines “incoherence” as the variance normalised by the error. They estimate variance and bias using repeated sampling (and changing few-shot prompts) and study how incoherence evolves with respect to difficulty of question (using length of reasoning chain or number or actions as a proxy), model size, and reasoning effort dedicated. The experiments are conducted on a range of state-of-the-art LLMs.", "questions": "- Is there any particular reason why the normalised version of the variance is interesting or useful, instead of simply studying the unnormalised variance? As most of the experiments aim to answer how incoherence evolves with model size or question difficulty, it seems that the normalisatoin is not strictly needed and, actually, hides the real signal a bit.\n- In line 129, it is explained how the mean term appearing in the bias-variance decomposition has an exponential and a logarithm. Is this taken into account when estimating the variance term? How does this fact affect the interpretation of the results, if at all?\n- Sec 3.5: “Models learn the right objective faster than to be an optimizer over a long horizon.” Do the auhors think this indicates that we will require different training procedures from next-token prediction to get models better at long-horizon tasks?\n- why does Sec 3.5 use an ill-conditioned matrix?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- originality: the idea of decomposing model errors into bias and variance to understand how they evolve seems novel.\n- clarity: overall mostly clear, robust notation and formalisation, plenty of references to Appendices. Explanation of the experimental setup is also great.\n- quality: highly informative and readable figures, great and thorough experimental setup.\n- significance: the findings relate to various state-of-the-art models and relate the introduced metric to various independent variables.", "weaknesses": "- Main issue: throughout the presentation, the authors seem to suggest that models become more incoherent as they become larger, or as they tackle longer tasks. However, the introduced metric for incoherence, variance/(variacne + bias), does not allow to disentangle whether the change in incoherence is due to a change in variance, or one in bias, or the relationship between the two. Indeed, if the bias decreases faster than variance, the outcome is that this incoherence metric increases even if both bias and variance decline. Most of the experiments do not report the variance separately from the bias (except the one in Fig 3, which does show a decreasing variance), and this makes it hard to understand what is actually driving the increase in incoherence metric. I believe this decreases the significance of the findings, by making them less clear. Moreover, this may lead people to misinterpret them, so I believe addressing this is necessary for publication. However, this should be a small change, as I believe the experimental setup is mostly robust and thorough, so what I am suggesting is a small change in how the results are presented and discussed.\n\nminor issues: \n\n- title: there is no really talking of “misalignment” in the paper that much, as most of the focus is actually on the variance. Also, “model intelligence” is used in the title and line 52, but actually the experiments consider model size. While this is a somehow good proxy for intelligence, there is not a 1-1 correspondence\n- Sec 3.5 asks the question “Does the model become an optimizer faster or slower than it converges on the right optimization objective?” I am not fully clear as to how this is related to the original questions asked.\n- lines 52-53 says that one of the addressed questions is “Asymptotically, as extremely capable models perform extremely complex tasks, which class of undesired behavior will dominate?” I don’t think this question is addressed; while empirical results are reported, I don’t think these are necessary to extrapolate to an asymptotic regime. Moreover, there are no studies of how the incoherence (or variance) evolve when the two independent variables (model size and task complexity) are varied at the same time.\n- lines 443-445: “for AIs to reliably acquire a broad range of capabilities, the capacity expansion requires enlarging the hypothesis space faster than any specific capability, which manifests as increased incoherence.” I don’t get this comment, can the authors please elaborate in the text?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)