The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Authors: Akshit Sinha
Arvindh Arun
Shashwat Goel
Steffen Staab
Jonas Geiping
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Long Horizon
Agents
Primary area: foundation or frontier models, including LLMs
TL;DR: We measure long-horizon execution capability of LLMs, identify a failure mode where models self-condition on their own errors, and show benefits of increasing model size and sequential test-time compute

Abstract: Does continued scaling of large language models (LLMs) yield diminishing returns? In this work, we show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can successfully complete. Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than an inability to reason. So, we propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve a long-horizon task. First, we find that larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accuracy. We then observe that the per-step accuracy of models degrades as the number of steps increases. This is not just due to long-context limitations---curiously, we observe a self-conditioning effect---models become more likely to make mistakes when the context contains their errors from prior turns. Self-conditioning does not reduce by just scaling the model size. But, we find that thinking mitigates self-conditioning, and also enables execution of much longer tasks in a single turn. We conclude by benchmarking frontier thinking models on the length of tasks they can execute in a single turn. Overall, by focusing on the ability to execute, we hope to reconcile debates on how LLMs can solve complex reasoning problems yet fail at simple tasks when made longer, and highlight the massive benefits of scaling model size and sequential test-time compute for long-horizon tasks.

Published as a conference paper at ICLR 2026
THE ILLUSION OF DIMINISHING RETURNS: MEASURING LONG HORIZON EXECUTION IN LLMs
Akshit Sinha' 1+
Arvindh Arun Shashwat Goel
Steffen Staab 2,5
Jonas Geiping 3,1.6
1 University of Cambridge Institute for AI, University of Stuttgart
a Max Planck Institute for Intelligent Systems *ELLIS Institute Tübingen
University of Southampton Tübingen AI Center
Equal contribution
Code
Dataset
ABSTRACT
Does continued scaling of large language models (LLMs) yield diminishing re- turns? In this work, show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks model can successfully complete. Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than inability reason. So, we propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve long-horizon task. First, we find that larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accu- racy. We then observe that the per-step accuracy of models degrades as the number of steps increases. This is not just due to long-context limitations-ccune curiously, we observe cenditiodin -ffectsmddels become more likely to make mistakes when the context contains their errors from prior turns. Self-conditioning does not reduce by just scaling the model size. But, we find that thinking mitigates celdiionditi,n and also enables execution of much longer tasks in single turn. We conclude by benchmarking frontier thinking models on the length of tasks they execute in single turn. Overall, by focusing on the ability execute, we hope to reconcile debates on how LLMs can solve complex reasoning problems yet fail at simple tasks when made longer, and highlight the massive benefits of scaling model size and sequential test-time compute for long-horizon tasks.
INTRODUCTION
Is continued scaling of compute for Large Language Models (LLMs) economically justified given diminishing marginal gains? This question lies at the heart of the ongoing debate on the viability of continued massive investments in LLMs. While scaling laws show diminishing returns metrics like test loss, the true economic potential of LLMs might arise from automating long, multi-step tasks (METR, 2025). However, long-houd morizoo tasks have been the Achilles' heel of Deep Learning. We see recent vision models generating impressive images, yet consistency over long videos remains an unsolved challenge. As the industry isolated questions, a fundamental question arises:
to build agents that tackle entire projects, not just
How can we measure the number of steps an LLM can reliably execute?
LLM failures on simple, but long tasks have been considered a fundamental inability to reason (Mirzadeh et al., 2024). Despite massive improvements complex reasoning bench- marks, Shojaee et al. (2025) claim thinking models (Guo et al., 2025) only give an "illusion of thinking", as they eventually fail when the task is made longer. These results have sparked much debate in the community, which we think can be resolved by decoupling the need for planning and execution in reasoning agentic tasks. Plamning involves deciding what information retrieve

---

Published as a conference paper at ICLR 2026
Dinitiihhing Stngle Step Expeocatial
Taking Steps
Step Accuneey
Espectol
0.80
Observed
Length
0.00
Beokeg Lange Tacks
Thinking Erabile Execution of Long Tasks A Stagle Tun
Earge Miciei Aue Proce Cuediibiinn
Thak Samelh (Steg's
Talk Lrngtt
X
Quaa--
Gooc
Chodk
10
(Billion
Figure I: A summary of our contributions. Our work measures long- long-horizon execution, finding large benefits from scaling model size and sequential test-time compute. We identify a failure mode where models self-condition their own errors, degrading future performance.
or tools to use and in which order, while execution involves carrying out the plan. In Shojaee et al. (2025) setup, the LLMs know the correct plan, they initially follow it correctly for many steps. We posit that the eventual failures are in execution as the task gets longer, the model is more likely make mistake in executing the plan. Although much attention has been paid to LLM planning abilities (Kambhampati et al., 2024), execution remains an understudied challenge, despite being increasingly important as LLMs begin to be used for long reasoning and agentic tasks.
In this work, we measure long-horizon execution capabilities of LLMs in a controlled setting, We isolate the execution capability LLMs by explicitly providing them the knowledge and plan needed. By controlling the number of turns, and the number of steps per turn, which together contribute to task length, we reveal insights about long-horizon execution in L.LMs:
We observe that diminishing improvements in single-step accuracy can compound, leading to exponential growth in the length of task model can complete. Tra- ditionally, scaling model size is assumed to increase capacity to store parametric knowledge search for plans. Yet, even when the required knowledge and plan are explicitly provided, we find that scaling model size leads to large improvements in the number of turns model can execute successfully.
Does Scaling have Diminishing Returns?
The Condiiinnii Effect. One might assume that failures on long tasks are simply due to the compounding of small, constant per-step error rate and context length issues. However, find that the per-step error rate itself rises as the task progresses. This is in contrast to humans, who typically improve executing task with practice. We hypothesize that, as significant fraction of model training is to predict the most likely next token given its context, conditioning models on their own error-prone history increases the likelihood of future errors. We this by controlling the error rate in the history provided to the model. As the error rate in the history is increased, we observe sharp degradation in subsequent step accuracy, validating that models self-condition. We show how self-conditioning leads to degradation in model performance in long-horizon tasks beyond previously identified long-context issues, and unlike the latter, is not mitigated by scaling model size.
The Impact of Thinking. We find that recent thinking models are not affected by prior mistakes, fixing the self-conditioning effect. Further, sequential test time compute also significantly improves the length of task model can complete in single turn. Where, without chain-of-thought prompt- ing, even frontier LLMs like DeepSeek- V3 fail at performing even four steps of execution, and its thinking version R1 can execute over 100 steps, highlighting the importance of reasoning before act- ing (Yao et al., 2023). We benchmark frontier thinking models, and find GPT-5 thinking (codenamed "Horizon") can execute over 2100 steps, far ahead of the next best competitor, Claude-4 Sonnet 432.
2

---

Published as a conference paper at ICLR 2026
The "jagged frontier" (Dell' Acqua et al., 2023) of LLM capabilities remains fascinating yet confusing. Unlike traditional machines, LLMs are more susceptible to failure when used for executing repetitive tasks. Thus, argue that execution failures in long tasks should not be misinterpreted the inability to reason. We show that long-horizon execution improves dramatically by scaling model size and sequential test time compute. If the length of tasks model can complete indicates its economic value, continued investment in scaling compute might be worth the cost, even if short-task benchmarks give the illusion of slowing progress.
2 FORMULATION
First, we define key capabilities involved an agentic or reasoning task. As a motivating example, consider an agent for the task of booking flights. Upon receiving search result, the agent must reason to choose a flight that aligns with user preferences. One plan for this reasoning task could be:
For each flight, verify the flight timings, baggage allowance, and airline reviews. Then apply any available discounts or reward programs, and finally select a flight based on cost and travel time.
Each of these individual steps requires two operations: retrieving some information, and composing it with the existing information state, until the goal of choosing the final flight is reached. Both these operations require knowledge, potentially tacit, about how to perform them. Execution is carrying out this plan, a sequence of retrieve- teaeeethee-comppee steps, until final booking is made. We formalize these terms for our work as follows:
Key Terms
Planning. Deciding what steps take, and in what sequence.
Execution. Carrying out the steps decided in the plan.
Knowledge. Information about different types of steps, and how compose them. A reasoning task. Requires planning the steps needed to solve it, and then executing them.
An agentic task. Requires planning what actions to take, and then executing them.
In this work, we focus on execution, as we argue that it is a critical component of long-horizon capabilities. Execution has traditionally received less attention (Stechly et al., 2024) than capabiliiiees such as reasoning, planning, and world knowledge, which have been the primary focus of LLM capability discussions. In fact, failures in execution have been misattributed to limitations in reasoning or planning capabilities (Shojaee et al., 2025; Khan et al., 2025). This perception may stem from the view that execution is straightforward mundane. For example, once a task, we are quite reliable at executing it, even improving with practice. However, as LLMs do not come with correctness guarantees, we posit that just execution over a long horizon can surprisingly be a challenge. We hypothesize that:
humans learn how to do
Even if planning and world knowledge are perfected, LLMs will still make mistakes in execution over long-horizon.
In an agentic or reasoning task, the model begins in an initial state (based on the first input) and has to perform a sequence steps to reach the final goal. A long-horizon task requires a large number of steps, with the task length being the number of steps needed to complete it. We define the following metrics to evaluate performance:
Evaluation Metrics
Step Accuracy. Measures the fraction of samples where the state update from step i 1 to step i is correct, regardless of the correctness of the model's state at step
Turn Accuracy. A turn is single interaction with the model, which may require executing multiple steps. Turn Accuracy measures the fraction of samples where the state update from turn state at turn t - 1.
turn correct, regardless of the correctness of the model's
Turn Complexity (K). Defined as the number of steps the model has to execute per turn.
3

---

Published as a conference paper at ICLR 2026
Task Accuracy. Measures the fraction of samples in which the model complete task of steps without making any mistakes in the process.
Horizon Length (H,). Given success rate threshold S S 1, the horizon length is the first step where the model's 's mean task accuracy across samples drops below s. It can be interpreted as: the model can perform task of length H, without making mistakes, with probability We use 0.5 unless otherwise specified, like Kwa al. (2025).
2.1 DIMINISHING RETURNS IN STEP ACCURACY COMPOUND OVER A LONG HORIZON
We begin by analyzing the relationship between model's single-step accuracy and its horizon length. Note that this analysis applies not just to execution, but rather to any general long-horizon task. To obtain mathematical relation, make two simplifying assumptions similar to LeCun (2023). First, we assume model's step accuracy remains independent and constant over the task. Second, we assume model does not self-correct, meaning any single error leads task failure. We assume this only for the analysis here, which is illustrative and provides useful intuition. Our subsequent empirical analysis goes beyond this, investigating how LLMs, in fact, do exhibit constant step accuracy for long-horizon execution, and may also correct their mistakes. Intuitively. if the model succeeds one step with probability p. it's success probability after steps is p'. This often negatively viewed as errors compounding over long task. But conversely, it also implies that a small improvement in step accuracy can lead to a large increase in horizon length after a certain (high) threshold of single-step accuracy is achieved.
Proposition Assuming an independent and constant step accuracy p and no self-correction, the task-length H at which a model achieves a success rate is given by:
[In(s)
In(s)
Hp(p)
100
In(p)
In(p)
Ha.
(The derivation is provided in Appendix J.)
Ha25
Horizon Length
Ha.50
This shows that the horizon length grows hyperbolically with the step accuracy. We illustrate this growth in Figure 2 across different values for the success rate s. Notice the sharp growth in horizon length beyond 80% single-step accuracy, performance that frontier models now achieve on many guestion-answerte benchmarks (Vendrow et al., 2025), which can be considered short tasks.
Hars
60
14.90
20
0.8
0.85
0.9
0.95
note that human labor often compensated for time. If the economic value of an agent also arises from the length of tasks it can complete, single-turn or short task benchmarks may be misleading for evaluating the benefits of further investment in LLM compute. While these benchmarks reveal genuine diminishing returns at the step level, they understate the compounding benefits that emerge over long-horizons. Beyond threshold, small improvements in step accuracy can translate into success at rapidly increasing task lengths, which may provide more faithful indicator of economic value.
Step Accuracy (p)
Figure 2: Growth of Horizon Length. The length of task model can perform grows hyperbolically in high accuracy regime.
For example, in METR's horizon length plot on software engineering tasks (Kwa al., 2025), it was empirically observed that the horizon length at s = 0.5 of frontier models is growing exponentially, doubling every 7 months. Using our result above, Figure we show that such exponential growth in horizon length occurs even in a regime of diminishing returns on step accuracy. If we set s = 0.5.
ln(2) As such, the step-accuracy p required sustain exponential growth in obtain Ho.s In[p]
Ho.s over time (t) 2 which is indeed a diminishing function.
2.2 ISOLATING EXECUTION BY DECOUPLING PLANNING AND KNOWLEDGE
We now deseribe how measure long-horizen execution empirically. We isolate execution failures by explicitly providing the requisite knowledge and plan. We study the chaining of the retrieve-then: compose step motivated in the flight-selection agent example earlier. Each step involves retrieving
4

---

Published as a conference paper at ICLR 2026
Long-Horizen Tasks Can Be Represented As
Our A bstraction: Value Dictionary Addition
Keep track of this dictionary
The Plan
The Execution
The State
Apple
Breai
Grape
SG
Track
Step
Retrieve
Compose
Ourpur
Add
Apple
Ciispe
Step
Retrieve
Compose
Output
Durrent Appler -27 Grape
47 66
Step N
Retrieve
Compose
Outpur
Add
Brok
Track
We isolate and study Long-Horizon Execution by LLMs
Froake Track
User Provided
<ondeer <answer
Figure 3: Overview of framework. (Left) Our framework models long-horizon tasks sequence of retrieve-then- compose steps. (Right) We design simple task that decouples planning from execution: in each turn, we provide the model the plan as key(s), asking it to retrieve their value(s), and compose them to maintain running sum. We control the number of turns and turn complexity (keys per query).
tool specified by the plan and then composing its output to update the current state. The plan is deciding what to retrieve and how to compose it, whereas execution is actually performing those operations. This fits natural abstraction key-value dictionary. The key serves as one step of a plan specifying what knowledge to retrieve, or tool to call, while the value represents the knowledge or tool output, which then has to be composed with the current state. In our study, we provide the plan as the keys in each query, eliminating the need for planning abilities from the LLM. We also provide the key-value dictionary in context, removing any dependency on the model's 's parametric knowledge. With this design, we directly control two important axes that multiply to obtain the task length (number of retrieve-thene theee---tee--ooosseeeee steps): the number of turns, and the turn complexity (K). The turn complexity can be varied by changing the number of keys queried per turn.
relevant information
EXPERIMENTS
We design simple task where even language models with 4 billion parameters can achieve high accuracy, to isolate the capability of long-horizon execution.
provide the model with the needed knowledge, fixed, in-context dictionary D : - Z. where V is vocabulary of common five-letter English words and values are integers sampled uniformly from [-99, 99). The initial state is S,-
Setup. As illustrated in Figure
In turn f {1...., T}. the model receives explicit plan P. = [kx,, . kz, J. which is set of K keys sampled from V. For each turn 1, the model must execute this plan, which requires updating the state, S,, maintain running sum of values for all past queried keys. This requires the retriee--then-compee steps:
1. Retrieval: Look up the integer value D[k] for each key ls E P.
2. Compusition: Sum these values and add them to the previous state, - S. EK D
We choose short English words and two-digit integers to minimize errors arising from tokenization. We provide few-shot examples to clarify the task. More details, including the exact prompt, are provided Appendix G. We also disentangle performance on the individual retrieval and composition operations, finding that models have much higher accuracies each of them alone (Appendix F).
3.1 EFFECT OF INCREASING THE NUMBER OF TURNS
We first test our hypothesis that Long-horizon execution can be challenging even when model has the required knowledge and planning ability, and then study the benefits of scaling model size.
Setup. We evaluate the Qwen3 (Yang al., 2025a) and Gemma3 (Gemma-Team al., 2025) model families, as they offer a range of sizes: [4, 8, 14, 32JB and [4, 12, 27JB parameters, respectively. For this experiment, we set the turn complexity to its simplest form (K 1). providing single key per turn, and vary the number of turns. Models are instructed to output the final answer directly, without intermediate thinking tokens, with the format enforced via few-shot examples. We verify that
5

---

Published as a conference paper at ICLR 2026
Maai Foeely Queill
10
0.8
Ruiion Leats
Tak Accupacy
Tort Accueeyy S
0.6
S
1.4
MecDack Oclad
10
20
20
50 Taak Lengll
10o
Model Size (Dillio Parumcter))
Midel Stie (Uillion Paroncteccs
Tisk Lougth
(a)
(b)
(d)
Gemmal-4BB
Gemma3-122
Germaa3-27B
Qwen3-4B
Qwen338B
Qwen3-14B
Qwen3-33B
Figure 4: Scaling model size has non-diminishing improvements in the number of turns it can execute. The first-step accuracy for our task is near near-perfect for all except the smallest models (a). Yet, the model size scaled, the horizon length increases significantly (b). We also the effect of scaling in widening the gap between small and large models in task accuracy (c) and turn accuracy (d) as number turms increases. The shaded region is the mean 1 one standard deviation over 100 samples; the solid line is the moving average over 5 turns; the dotted line is hypothetical baseline model with constant step-accure accuracy of 0.99.
formai-following errors are not the primary failure mode (Appendix H). We also show that the results below hold with chain-of-thought (CoT) prompting, and thinking models (Appendix Figure 13), and the trends are not affected by the temperature used (Appendix Figure 14).
Result Execution alone is challenging. As seen in Figure (a), all models except Gemma3-4B and Qwen3-4B achieve near-perfect accuracy on the first step, confirming they have the knowledge required perfectly do single step of our task. Yet, task accuracy falls rapidly over subsequent turns (Figure 4 (c)). Even the best best-performing model (Qwen3-32B) sees its accuracy fall below 50% within 15 turns. This confirms our hypothesis that long-horizer execution can be challenging for L.MSs, even if they have the needed knowledge and plan.
Result 2: Non-diminishing benefits of scaling model size. As shown in Figure (c), larger models sustain higher task accuracy for significantly more turns, resulting in a clear scaling trend for horizon length (Figure+ 4 (b)). We abstain from deriving a "scaling law" since we can only obtain at most four model sizes from the same family, but the improvements do not seem diminishing. This observation non-trivial. While the benefits of increasing model size are often attributed to improved capacity for knowledge, our task is not www.lede-constraing accuracy (Figure 4 (a)), nor is the task more complex so that a larger model would be required. Yet, larger models are clearly more reliable executing the task for longer. A possible explanation is the redundancy of internal circuits in larger models, which ensembles to reduce error (Lindsey et al., 2025). However, we find that simulating this redundancy with output-level aggregation of parallel compute does not replicate the gains observed from scaling model size (Appendix D).
as models achieve near- pear-perfect first step
3.2 WHYY DOES TURN ACCURACY DEGRADE? THE SELF-CONDITIONING EFFECT
One might expect a model's 's turn accuracy to remain constant. Yet, Figure (d) shows the accuracy of individual turns degrading as the number of turns increases. We investigate two competing hypotheses:
Degradation as the context length increases. The model's 's performance degrades simply due to increasing context length (Zhou et al., 2025a), irrespective of its content.
Self-conditioning. The model conditions on its own past mistakes. becomes more likely to make a mistake after observing its own past enrors in previous turns.
Setup. To disentangle these factors, we conduct counterfactual experiment by manipulating the model's chat history. We control the error rate by injecting artificial output histories with chosen error rate in the same format. If we fully heal the history, with a 0% error rate, degradation in the model's turn accuracy between turn and later turn be attributed long-context issues. If a model's 's accuracy for a fixed later turn consistently worsens with increasing error rate in prior turns, this would support self-conditioning hypothesis.
Result 3: Self-conditiniing causes degradation in turn accuracy beyond long-context. Our results in Figure 5 (a) show evidence for degradation due both long-context and self-conditiooing.

---

Published as a conference paper at ICLR 2026
10
Turr 100 Arcureyy
Tura Accuracy
Cp
O
Locch
Tuk
Indroed Eroor Rate
638 Ixduced Boror Rate
(b)
Figure Models self-condition on their previous mistakes, leading to more mistakes in subsequent turns. By manipulating the chat history, we counterfactually vary the fraction of errors in previous turns. find this increases the likelihood of errors in future turns (left). This shows a source of degradation in turn turn-wise model accuracy beyond long- contecotext, as in the turn 100 slice (right) model accuracies are much higher when we provide fully correct history. Scaling model size increases self-conditioning, even for frontier non non-thinking models.
When conditioned on an error-free history (Induced Error Rate = 0.00), model turn accuracy at turn 100 is below its initial value, consistent with prior observations of long-context degradation (Zhou et al., 2025a). More interestingly, as we increase the rate of injected errors into the context, accuracy at turn 100 consistently degrades further. This demonstrates the self-conditioning effect-as models make mistakes, they become more likely to make more mistakes, leading to continuous degradation in per-turn accuracy throughout the output trajectory as shown in Figure 5 (b).
Result Unlike long-context, scaling model size does not mitigate self- ceff-condiiooning. At the error rate of 0%, notice that the accuracy at turn 100 consistently improves as you scale model size. As shown in Figure (b), scaling to frontier (200B+ parameter) models like Kimi-K2 (Kimi-Team et al., 2025), DeepSeek-V3. V3 (DeepSeek-Ae AI et al., 2025), and Qwen3-235B IatB-IIntuuc--5000 (Yang et al., 2025a) largely solves long-context degradation for up to 100 turns, achieving near near-perfect accuracy on healed history. However, even these large models remain susceptible to self-conditioning, as their performance consistently degrades as the induced error rate in their history increases. A possible explanation for this is the relation between self-conditioning and in-context learning. In in-context learning, the model desirably conditions on user and environment inputs, for example, the correct few-shot demonstrations of the task we also provide in the initial prompt. In contrast, in self-conditioning. the model conditions on its own past behaviour, which could be wrong, leading to more mistakes. At a fundamental level, both arise from the model trying to output the most likely completion to its context, which larger models are known to be better at (Arora et al., 2025). Self-conditioning is also consistent with results showing how hallucinations snowball (Zhang et al., 2023), and larger models shift more in personality during multi-turn conversations (Choi et al., 2024; Becker et al., 2025), where in our case, the drift is toward a personality that makes errors.
In Appendix I, we try the above setup of output manipulations with CoT prompting, finding that accuracy still deteriorates rate increases. A potential confounder is that the manipulated outputs deviate from the COT. We try to mitigate this issue with programmati- cally generated CoT traces, but still observe self- conditioning. We also try removing CoT traces from previous turns from history, which causes the model to stop using CoT completely. So, we now present results for the Qwen3 thinking models, which are trained with reinforcement learning (RL) to think even when previous turn traces not presented. As before, observe their turn 100 accuracy while controlling the error rate in prior turns.
1.
Qwen3
the induced error
1000 Aosutacy
0.
8mc
Qwea3-tb
8.4
Qwen3 14b
Qwen--1th
80000
0.50
0.75
Indured Frooe Rate
Figure 6: Thinking fixes self-conditioning. Qwen3 models with thinking enabled no longer self-condition, even when the entire prior history has wrong answers. contrast non- non-thinking results.
Result 5: Thinking fixes celf-conditioning In Figure 6, we observe that the Qwen3 thinking models do not self-condition the accuracy of the models at turn 100 remains stable, regardless of the error
7

---

Published as a conference paper at ICLR 2026
Tusk Leaxth, (Singlo Turn)
Welher
Tusk Leugth (Stugle Tura)
Thilee
ofougks
C
381
112
72
Cramat--3 371
Deypnol-43
Kinuf
Kiai--3
Queed 2000 Azat
Deopstoke V3
Gonini-g 5.Pro
Despeedo
Qwnas Next
Grob-
Choide Stattond
CPT4
Figure Benchmarking the length task models can execute in single turn. Without CoT or thinking. even the biggest models fail to execute more than a few steps (a). Sequential test time compute (thinking tokens) significantly improves this, especially when trained with RL (b), where GPT-5 is far ahead of the rest.
rate in its context. This could arise from two reasons. First, RL training can reduce the most likely next token prediction behaviour of language models, making them oriented towards task success rather than continuing the context. Second, the removal of thinking traces from prior turns could reduce the influence of prior turns on the model's output, By inspecting the models' thinking traces, we observe that they do not refer back to their answers in prior turns, which could be potential reason why they do not self-condition. Inspired by this, for non-thinking models, we experiment with context engineering by explicitly removing prior history and find that indeed mitigates self-conditioning (Appendix C.2). We also find that just prompting models to self-verify their answers does not solve self-conditioning completely (Appendix C.1).
thinks about the new turn independently.
3.3 WHAT IS THE LENGTH OF TASKS MODELS CAN COMPLETE IN A SINGLE TURN?
In the previous sections, we measured how many turns models can successfully execute a single senieve-then-compeee step. However, most real-world tasks require more complex processing every turn. In fact, in Appendix E we show that the horizon length of different models can vary significantly different turn complexities. The total task length model can handle function of both the number of turns and the number of steps to execute per turn. We now measure the latter dimension: the maximum number of steps a model can execute per turn.
Setup. To quantify the length of task models can complete in one go, without user input, we run binary search (Lehmer, 1960) to find the highest turn complexity (K, the number of keys) the model provide the correct sum for with accuracy > 80%. We evaluate suite of frontier models like GPT- 5 (OpenAI, 2025), Claude-4 Sonnet (Anthropic, 2025). Grok 4 (xAI, 2025), Gemini 2.5 Pro (Gemini Team, 2025), Kimi K2 (Kimi-Tean of our benchmark is that it is contamination- free, as new examples can be generated programmatically.
al., 2025), and DeepSeek-RP R1 (Guo et al., 2025). An advantage
Result 6: Without CoT, non-thinking models struggle to chain more than few steps per turn. In Figure 7 (left), we first find that when prompted to answer directly, without chain-of-thoughe the larger Qwen3 32B, Gemma3 27B, as well frontier non-thinking models like DeepSeek-V (670B), and Kimi K2 (1026B), fail to execute even turn complexity of more than six. This is consistent with prior work showing the necessity of thinking tokens for transformers to perform sequential tasks (Weiss et al., 2021; Merrill & Sabharwal, 2023). We see that the number of steps the model can execute in a single turn improves significantly with chain-of-thougeee
This reinforces the importance of reasoning before acting (ReAct (Yao et al., 2023)) for agents, even if this costs more and fills up the context window. We show preliminary evidence that parallel test time compute not as helpful, with majority voting leading to only marginal improvements in execution length (Appendix D).
Result 7: Benchmarking frontier models. In Figure (right), we benchmark frontier models on the length of task they can execute in single turn. We find a surprisingly large gap between GPT-5 (codenamed Horizon) with 2176 steps and others like Claude-4 Sonnet (432 steps), Grok (384 steps), and Gemini 2.5 Pro (120 steps). Qwen3-Next, which combines Gated DeltaNet (Yang et al., 2025b) with standard attention, designed explicitly for long-context tasks, outperforms even larger models that use only standard attention, hinting at the impact of model architecture for long-horizon
8

---

Published as a conference paper at ICLR 2026
tasks. Overall, even our simple task can separate frontier models in their long-horizel execution capability, and presents a clear opportunity to improve current open ppen-weight models.
4 RELATED WORK
Increasing Task Length. Multiple works have recently shown how models worsen as problem complexity increases (Zhou et al.. 2025b), often attributed failures of reasoning (Cheng, 2025; Shojace et al., 2025). Recently, multiple real-world long-horizon agentic benchmarks have been proposed (Backlund & Petersson, 2025; Xie al., 2024; Shen et al., 2025), where prior work has studied planning failures (Chen et al., 2024b). By designing a task where no reasoning is required, given that we provide the model the requisite plan and knowledge, we show that execution alone can be a challenge, degrading model accuracy longer tasks. Our observations on scaling could hold for the related problem of geeetthhnneralzzttion training models to succeed on tasks longer than those seen during training (Fan al., 2024; Cai et al., 2025).
Long Context. Much of prior work has focused on improving the maximum context length that can be provided in the input to language model (Su et al., 2021), and evaluating whether (Tay al., 2020) and how (Olsson et al., 2022; Li et al., 2023) models maintain performance as the context gets longer (Tay et al., 2020). Closest is the recent RULER (Hsieh et al., 2024) and GSM-Infinite (Zhou et al., 2025b), which also use synthetic data to systematically evaluate long-context abilities. While long-context will help models execute for longer, is different capability compared to long- horizon execution (Zhou et al.. 2023; Chen et al., 2024a), as it focuses on performance as a function of input, not output length. We identified one such difference, the self-conditioning wfeee past errors in model output increase the chance of future mistakes, and disentangle this effect from long-context degradation in Section 3.2.
Self-correction. To address an issue similar to self- celiiiondiion at the token level called "exposure bias", Ranzato et al. (2015) proposed using reinforcement learning to train language models. This trains the model to react to mistakes in its prior outputs by correcting them, instead of making more of them, an intuition that also guides more recent work on RL for self-correction (Kumar et al., 2024; relffveriiiatiion (see
Ma et al., 2025). As training-free free fix for our setting, we also try model-based Kamoi et al. (2024) for a survey) at each turn in Appendix C.I, which can be considered a form of self- refinement (Madaan et al., 2023) without having access to environment feedback (Gou et al., 2023).
Scaling LLMs and RL. Scaling laws for language models show diminishing returns on the loss for the single step of predicting the next token (Kaplan et al., 2020; Hoffmann et al., 2022). When models competed in simple knowledge- -base question-answering tasks such as MMLU (Hendrycks al., 2020), such single-step measurements could inform us about the rate of progress. This has changed in the last year. Where earlier we could only post-train on human demonstrations (Mishra al., 2021), language models can now be trained with just rewards (Shao al., 2024), enabling sophisticated reasoning (Guo et al., 2025; Jain et al., 2024) and agents (Kimi Team et al., 2025). This opens up the opportunity to solve much longer tasks where earlier human supervision would be too expensive to scale. Our work shows how diminishing returns on single-step performance can compound to provide large benefits the length of tasks a model can solve. This motivates the need to study empirical scaling laws for horizon length in agents (Hilton et al., 2023).
Tool Use. In symbolic AI, once tasks are formalized, for example, into STRIPS plans (Fikes Nilsson, 1971), they can be executed without issues. Prior work (Chen et al., 2024b; Valmeekam et al., 2024) has shown LLMs struggle to match symbolic algorithms for automated planning. In contrast, show LLMs can fail on straightforward execution (Zhu et al., 2025b; Sun al., 2025; Stechly et al., 2024) over a long horizon even when the plan is provided. Teaching LLMs to use tools offers one way to shift the burden of execution from probabilistic models to reliable programs (Schick et al., 2023). However, reasoning is often fuzzy and not always easy to implement as a tool, requiring the model to execute some steps by itself. Even calling the right tools requires reliable execution from the model (Patil et al., 2025).
5 CONCLUSION
In this work, we show how taskt-task benchmarks may give the illusion of slowing progress for modern language models. We show that scaling model size increases the number of turns model
9

---

Published as a conference paper at ICLR 2026
execute, while sequential test-time compute increases the length of tasks a model can perform on a single turn. Together, these contribute to dramatically increasing horizon lengths for LLMs.
Limitations. As with any "synthetic" task (Allen- -Zu,, 2024; Poli et al., 2024; Chollet et al.. 2024) used for a controlled study of LLM capabilities, there are a few limitations of our setup, Improvement on our task is necessary, but not sufficient for long-horizon execution on real-world tasks. It does not reflect the complexities and sources of error arising in real agentic tasks with a large number of possible actions. In such settings, the number of actions and the accuracy of each action can both vary based on the plan, requiring more careful consideration. Another issue in more complex tasks could be the possibility of multiple distinct correct plans, such that even if we provide plan to the model, it could deviate from it while being correct. This is not possible in our controlled study, as using the wrong keys would likely lead wrong outcome. In Appendix A, we demonstrate how seff-conditioning is a significant failure mode for LLM agents even in realistic benchmarks like ALFWorld (Shridhar et al., 2021), GAIA (Mialon et al., 2024), and WebShop (Yao et al., 2022). Our results are observations about current LLMs, and not inherent properties of transformers, so they might change with task-fpecific finetuning. Particularly, the single-turn task in Section 3.3 is in theory parallelizable, which discuss further and propose pathways to future-proofing via more inherently sequential tasks in Appendix B. Finally, our current task accuracy metric does not account for self-correction. In tasks where mistakes acceptable and easy to undo, self-correction is promising direction to improve long-horizon execution.
Outlook. Scaling up the length of tasks model can complete would be major step towards realizing the true potential of general, open-ended agents (Raad et al., 2024). If they are trained in simulated environments created with generative models (Bruce al., 2024), maintaining accuracy over a long-horizon becomes doubly important. By showing long-horizon execution can be studied on simple tasks, we hope to inspire more research on this capability, is an increasingly important capability in the era of experience (Silver & Sutton, 2025).
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we have provided comprehensive details of our method- ology, experiments, and theoretical results. The mathematical derivation and assumptions for Propo- sition are detailed in Appendix J. All specifics of our experimental methodology are consolidated in Appendix G, which covers our synthetic task design and data generation (Appendix G.1), the exact prompts used for both standard and thinking models (Appendices G.2 and G.3), model specifications and hyperparameters like temperature and top-p (Appendix G.4), and the computational resources used for the evaluations (Appendix G.5). Furthermore, we have included the complete source code, data generation scripts, and the exact datapoints used as supplementary material.
ACKNOWLEDGEMENTS
We thank Maksym Andriushchenko, Nikhil Chandak, Paras Chopra, Dulhan Jayalath, Abhinav Menon, Sumeet Motwani, Mathias Niepert, Lluís Pastor Pérez, Ameya Prabhu, Tim Schneider, Shashwat Singh. and Timon Willi for helpful feedback. AA was funded by the CHIPS Joint Undertaking (JU) under grant agreement No. 101 140087 (SMARTY), and by the German Federal Ministry of Education and Research (BMBF) under the sub-project with the funding number 16MEF0444. AA thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) and the ELLIS PhD programs for support. The authors gratefully acknowledge compute time on the Artificial Intelligence Software Academy (AISA) cluster funded by the Ministry of Science, Research and Arts of Baden- Maden-Württemberg.
AUTHOR CONTRIBUTIONS
SG conceived the project. AS led the execution of the experiments with the help of AA, while SG led their planning with the help of AA, AS, and JG. SG and AA wrote the paper, while AS worked on the figures. JG and SS advised the project, providing valuable feedback throughout.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zeyuan Allen-Zhu. ICML 2024 Tutorial: Physics of Language Models, July 2024. Project page:
https:/ hatps://physccee alwea-lllnaae sellenzhuu.. com/.
Anthropic. System card: Claude Opus 4 & Claude Sonnet 4, May 2025. URL https: : www. anthropic. com/ claude- 4 1-systemm em card. Covers Claude Sonnet 4 and Opus 4.
Aryaman Arora, Dan Jurafsky, Christopher Potts, and Noah D. Goodman. Bayesian scaling laws for in-context learning, 2025. URL :ttttps::/araee http://www.....r////////.0.000..
Axel Backlund and Lukas Petersson. Vending-Benchi A Benchmark for Long- Coherence of Autonomous Agents, arxi::2502.15840/ccc February 2025. doi: 10448500arxxw22200.55400
URL httttt//aaxii/// org. org/ /ttb////5525 15840.
Jonas Becker, Lars Benedikt Kaesberg, Andreas Stephan, Jan Philip Wahle, Terry Ruas, and Bela Gipp. Stay Focused: Problem Drift in Multi-Agent Debate. arxiv: May 2025. doi:
4445550/rrXiv2252211555555 URL http: /ttp:://aaxxvv.ooo wwwpaiwwwwwiiioo//oooooooaoaaaa/ 2502..2/2522212 19559.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. Forty-first International Conference on Machine Learning, 2024.
Ziyang Cai, Nayoung Lee, Avi Schwarzschild, Samet Oymak, and Dimitris Papailliopoulos. Extrapo- lation by Association: Length Generalization Transfer in Transformers. arxiv:2506.099 (52.500000091//s..... Au- gust 2025.
doi: 10.48850//rxiv.2506..9999 abs 250. /
URL http: www.v....gg..................
225111221..... 09251.
Siwei Chen, Anxing Xiao, and David Hsu. LLM-State: Open World State Representation for Long- horizon Task Planning with Large Language Model. arxiv:2311 April 2024a. doi: 10.485550/rxivv2311 17406. URL
Yanan Chen, Ali Pesaranghader, Tanmana Sadhu, and Dong Hoon Yi. Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TraveIPlanner as an Example. wtt.v:::408.00311///111 August 2024b. doi: 10.48500krxii..44000633. 06318. URL http: /ttp//aavvvvv/.a///////eeeeaaaaa org/ abs/2408 06318.
Jingde Cheng. Why cannot large language models ever make true correct reasoning?, 2025. URL https: //arxiaiivv/v////////// org/ arggabs//////ee.ooo..ggg//ee abs/ 2500 10265.
Junhyuk Choi, Yeseon Hong, Minju Kim, and Bugeun Kim. Examining identity drift in conversations of Il agents. arXiv preprint arXiv:24/2.00800 2024.
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: Technical report. arXiv preprint 1rxiv:24/2.04604, 2024.
DeepSeek-AI, Aixin Liu, Bei Feng, et al. Deepseek-v3 technical report, 2025. URL https: ///ar/ii.orr//b//222...55
Fabrizio Dell' Acqua, Edward McFowland III, Ethan R. Mollick, Hila Lifshitz- Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Working paper, Harvard Business School Technology & Operations Management Unit 2023. URL https / rarn. com/ /ttpa//tttttt553222222 Also circulated as The Wharton School Research Paper; last revised 2023-09-27.
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped Transformers for Length Generalization. The Thirteenth International Conference on Learning Representations, October 2024. URL https: : wwwpopeeen...../////p net/ wttet//oommmmmndddddoossaaaaeee
Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3): December 1971. ISSN 0004-3702. doi: atticcle/pii/00043770000
10.101660004-770277199000000 URL https: /www sciencedirect. htttpp::www..cccccedeeeeee...e com/ science/
11

---

Published as a conference paper at ICLR 2026
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities. Technical report. Google DeepMind. June 2025. URL https: httrttaa://stooggeeeee /tt.sttorrrg.ge.e wwwgleapis...oooooooooo laaiiii_____eprrt_____
gemini/ com/ httpa/ppis.oo////eeeeeeeeeee httpd//epminddmmiiaaaeee hpppp//sss.oaag..g.pp...........
Gemma-Team, Aishwarya Kamath, Johan Ferret, et al. Gemma 3 technical report, 2025, URL https://arxiv: otg/ hwargg/ab////e abtp//as///3./. 197866
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 1rXiv:2305.11738, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
András Hajnal, Wolfgang Maass, Pavel Pudlák, Mario Szegedy, and György Turán. Threshold circuits of bounded depth. Journal of Computer and System Sciences, 46(2): 1993.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 033000 2020.
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301. 13442, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint 22K.v:2200.,15555 2022.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the Real Context Size of Your Long-Context Language Models? In First Conference on Language Modeling, August 2024. URL https: wwwp//eeee.......eee/e net/ Corum?id=kIoBe 68y.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 20W32400.0797,, 2024.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can't 't plan, but can help planning in Ilm-modulo frameworks. arxiv preprint arXiv:2402.018// 2024.
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms actually correct their own mistakes? critical survey of self-correction of llms. Transactions of the Association for Computational Linguittics, 12: 12:1417-1440, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / arxiv. org/ abs 20002001110000 08361.
Sheraz Khan, Subha Madhavan, and Kannan Natarajan. A comment on" the illusion of thinking": Reframing the reasoning cliff as an agentic gap. arXiv preprint arXiv:2506. 18957, 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. 2025.
Kimi-Team, Yifan Bai, Yiping Bao, et al. Kimi k2: Open agentic intelligence, 2025. URL https: //arxiv.org/ aww./aa/a/0o 2007720534....
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to celf-correct via reinforcement learning. arXiv preprint arXiv:2409. 129/7, 2024.
12

---

Published as a conference paper at ICLR 2026
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. Measuring ai ability to complete long tasks. arXiv preprint arXiv:2503. 2025.
Yann LeCun. Do large language models need sensory grounding for meaning and understand- ing? Slide deck, NYU Philosophy of Deep Learning debate, March 2023. URL https: drive. google com/filee d/ BBW.///33.333....... www.5//6ww.aaaaacaaaaoaa..... Lttpe///MRRRNNi/eieeeee Nbi/view. In- cludes slide "Autoregressive LLMs are Doomed.".
Derrick H Lehmer. Teaching combinatorial tricks to a computer. In Proceedings of Symposia in Applied Mathematics, pp. 179-193, American Mathematical Society, 1960.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopooloss and Samet Oymak. Transformers as Algorithms: Generalization and Stability in In-context Learning. In Proceedings of the 40th International Conference Machine Learning, pp. 19565-19594. PMLR, July 2023. URL pettps:/pproceediie
mlr www..ppees/vv000 1 11111.1111 html.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https: /ttss://traassss /tt/aaanssttt www.eooreieiruuueeeeeereeoooaere 2ubb/2025/
wwtpboillllloo...oo.oooooooooo.. html.
Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S2r: Teaching 11ms to self-verite and self-correct via reinforcement learning. arXiv preprint Xrv:222502. 12.00...288.. /2853, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- self-feedback. Advances in Neural Information Processing Systems, 66:46534-46599 2023.
Carlo Mereghetti and Beatrice Palano. Threshold circuits for iterated matrix product and powering.
SAIRO-Theoretical Informatics and Applications, 34(1):39-46, 2000.
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. arXiv preprint arXiv:2310.0792. 2023.
METR. Measuring ai ability complete long tasks, March 2025. URL https:::////
blog/2025- 2ttp://222000--1-mmasrrrria
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a B. Kim, Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 9025-9049, 2024. URL https: htttps::/ppoceeedd.assss i iclr wc/paper fttpperrrr_eleeeepeeeeeeee 024//244/22 file/ pdf.
benchmark for general ai assistants.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning large language models. arXiv preprint arXiv:2410.05229, 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2/04.087733 2021.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, 7ac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context Learning and Induction Heads. CoRR, January 2022. URL https: //openreview. fttt:/oormmddddeee ImUOO.
OpenAl. Gpt-5 system card, August 2025.
URL https: wtt http:/cc..o...ccccccc// opm.ai.coom
gpt-5- gpt-5-system-cardd pdf. Canonical system card PDF.
13

---

Published as a conference paper at ICLR 2026
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph Gonzalez. The berkeley function calling leaderboard (bfel): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025.
Michael Poli, Armin W. Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Re, Ce Zhang, and Stefano Mas- saroli. Mechanistic Design and Scaling of Hybrid Architectures. Forty-First International Conference on Machine Learning, June 2024. URL https: /ttp//ppeeeeeeeeee/ee ew. www.nee//aee nwt//frurm? forum? 1.d=DDpGGdd9nf.
Maria Abi Raad, Arun Abuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint urXiv:2404. 10179, 2024.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint
2015.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke 7ettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68551, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. TaskBench: Benchmarking large language models for task automation. In Proceedings of the 38th Interaational Conference on Neural Information Processing Systems, volume 37 of NIPS '24, pp. 4540-4574, Red Hook, NY, USA, June 2025. Curran Associates Inc. ISBN 979-8-3313-1438-5
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. URL http:: / //tp://aaavv.///a///e/eeeeaaaa.a wwtp:////gaa//a/oe/o 06941.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL bttps:://axivs o alss 2010. 03768.
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 4rXiv:2408.03314. 2024.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness? an analysis of cot in planning, Advances in Neural Information Processing Systems, 37:29106-29141, 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Simeng Sun, Cheng- Bing Hsieh, Faisal Ladhak, Erik Arakelyan, Santiago Akle Serano, and Boris Ginsburg. L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution. arxiv:2503. 22832/cs/ April 2025. doi: 10.488500arxi..2553.2222 URL http:://axii/
httg/abb//555
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena A Benchmark for Efficient Transformers. In International Conference on Learning Representations, October 2020. URL https: pettps:/ccpenreiie ew net wwt.eewwwwweewmmtt///e/e www.mriruddoooo foee//oonnddi///e--------- witpiiddevgeeeeeeeee.ee grC2k.
14

---

Published as a conference paper at ICLR 2026
Karthik Valmeekam, Kaya Stechly, Atharva Gundawar, and Subbarao Kambhampati. A Systematic Evaluation of the Planning and Scheduling Abilities the Reasoning Model ol. Transactions on Machine Learning Research, December 2024. ISSN 2835-8856. URL https: htttpa://ppeneeviieee
set//forumt d==kKBxpOFhR,
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.0346. 2025.
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. International Conference on Machine Learning, pp. 11000-11090, PMLR, 2021.
XAI Grok 4 model card, August 2025. 205-00--2--roo----mooe
URL https:///ata.....
pdf.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and
Yu Su. Travelplanner: A benchmark for real-world planning with language agents, 2024. URL
https://arxij org/ otg/abs/////oooooooo/ooog//ooo.r / 2402. 01622.
An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025a. org/abs/ 2505. 09388.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference Learning Representations, 2025b. URL https: http:///ppeeevveeeeeee ew net /ortt/formmmmm ponrummid=r8FFF d wzw.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. In Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural In- formation Processing Systems, volume 35, pp- 20744-20757 Curran Associates, Inc., 2022. URL https:://// file/ https::/proceed: http://ooceeddeeeoe ngs neurips oc wttrrfflls//ppprr/////r/ Pat.eeeeeaeaeeeeepppeeeee www--nnnneeeeeeeeee .df.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball, 2023. URL https: pettps://arxte /wwp/aaaivvv.//////ee/eaaaeellea wwwpaaawwwww..////a//a.aa
13534.
Haoyu Zhou, Mingyu Ding, Weikun Peng, Masayoshi Tomizuka, Lin Shao, and Chuang Gan. Generalizable Long-Horizon Manipulations with Large Language Models. arxiv:2310002244(css) October 2023. doi: 10448500axXiw2211002226. URL http: 02264.
http::/aaxxv..cr http//axvvvv/a/.a http://awww...ogaa//22221
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your Ilms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05255. 2025a.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. GSM-StinftyS: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length? In Forty-Second International Conference on Machine Learning, June 2025b. URL https: //oopnreeiiww...mmmm. net wtwp://www.eeefot//mmeeeee (ooommiiddn522yyeee..
Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, Xiaoteng Ma, Xiaodong Yu, Gowtham Ramesh, Jialian Wu, Zicheng Liu, Pan Lu, James Zou, and Jiaxuan You. Where Imm agents fail and how they can learn from failures, 2025a. URL https:tp///. //a//aiivvv////////////////aaavv org/ abs 250 25370.
Minjun Zhu, Qiujie Xie, Yixuan Weng. Jian Wu, Zhen Lin, Linyi Yang, and Yue Zhang. AI Scientists Fail Without Strong Implementation Capability. a0xiv:2506.0137//cc June 2025b. doi: 10. 10.488500arxii.2506.011111 01372. URL http::// /twp/viivvvo wwweii.org/abb///// 2500. 01372.
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Self-Conditioning in Realistic Agentic Tasks
17
Sequentiality the Single-Turn Task
17
Investigating Potential Fixes for Self-conditoonnn
19
C.I Turn-wise Self- Verification Prompting
19
C.2 Context Engineering
19
D Can Parallel Test-time Compute Scaling Match Thinking?
20
Number of Turns vs Turn Complexity
21
Deconstructing Errors in Retrieve-then-composs Steps
21
G Experimental Setup
23
G.I Task Details
23
G.2 Prompting
23
G.3 Prompting for Thinking Models
24
G.4 Model Specifications
24
G.5 Compute Details
24
H Format Following Failures
25
Chain-of-Thought Self-Conditionin
26
Proof and Analysis of Proposition 1
27
J.1 Implications for Horizon Length (Ho.s)
27
16

---

Published as a conference paper at ICLR 2026
A SELF-CONDITIONING IN REALISTIC AGENTIC TASKS
Zhu et al. (2025a) recently released' annotated failed trajectories from more realistic, popular agent benchmarks like GAIA (Mialon al., 2024), ALFWorld (Shridhar et al., 2021), and WebShop (Yao et al., 2022). We manually analyzed each annotated sample and identified potential instances of self-conditioning. We attach an example of an annotated entry below.
"trajectory :i": "GPT-4o_ 001 chat bbcC_ttctc_eeeecc tCo _ec_e000000 e00 -00-d3248880",,
"LLM": "GPT-4o", "task_type"; "webshop", "eritical_faalur_sttte 29, "eritical_faalure__ee :odullo;;_ "plan", istep_annotationss.
"step": 29, "plan": {
beaiilure_typ""" "inefficient paantennt___a,,,
"reasoning": "This plan Loops back to the very beginning, moving in the wrong direction, and results in inefficient search query that is almost icentical the initial ones."
We search for entries where the failure reason mentions the agent repeating the previous mistakes repeatedly and eventually degenerating into failure trajectories. Some of the eπrоr categories that fit our search were,
ine Chefficient_p11 Plan is overly long illogical because of prior errors.
progress_misjjudge: Incorrectly evaluates progress based on its prior outputs.
deusall_isstten wtt.miiiattiitiiioo..... bution: Correctly notes failure but blames the wrong cause due to its prior outputs, misguiding subsequent plans.
Through this process, estimate that roughly 20% of GAIA, 48% ALFWorld, 33% of WebShop failures are similar to self-conditioning. Here are some such trajectory annotations examples:
WebShop: "... This set the agent endless loop of query reformulation and paging through irrelevant results, making success impossible...
This set precedent for all subsequent memory outputs to ignore key details about which objects had been visited what actions were performed at each and what the actual observations were..."
ALFWorld:
Th..This
GAIA: "Make an low efficient plan that not success, repeat the similar action that not success"
caveat in the analysis we present above is that the correctness of steps is subjective to determine on these tasks for quantitative study of self-conditioning. This is exactly where having controlled study like ours, even if with a synthetic task, is extremely valuable. It allows us to isolate execution failures from planning and knowledge gaps, which difficult in more complex agentic tasks.
B SEQUENTIALITY OF THE SINGLE-TURN TASK
potential limitation the single-turn setting in Section 3.3 that retrieving and adding list of numbers is, in theory, parallelizable inside transformer (Merrill & Sabharwal, 2023). This is
Link AgendErrorBench trajectories
17

---

Published as a conference paper at ICLR 2026
because adding list of numbers lies in the Threshold Circuit (TCO) (Hajnal al., 1993) class of problems that can solved with constant depth computation. While currently our results show that than few such additions without chain of thought,
frontier language models fail perform since this is in theory learnable, our single-turn study is not future-proof. Note that this limitation does not apply to multi-turn experiments, provided turn-by-tura and thus the model has to perform these steps sequentially. To mitigate this, we now perform a study with sequence of 2 x (non-stochastic) matrix multiplications, which in the general case lies in TC1, generally believed to not be equivalent to TCO (Mereghetti & Palano, 2000).
the keys to be retrieved and their value added is
Experimental Setup. To keep this task similar to the key-value addition task introduced Section create analogous key-value matrix multiplication task, where each key corresponds randomly generated 2 matrix. Since multipiccaiion causes values to blow up to very large numbers quickly, ensure each element in the matrix is in the range of [-9, 9j. The LLM is then prompted to complete a chain multiplication of K matrices, where K corresponds to the per-turn complexity (Section 3.3). The full prompt is presented below. Analogous the key-value addition task, we again have a series of retrieve-thene perrive-then-composs steps:
1. Retrieval: Look up the matrix M[k] for each key & Pt
2. Composition: Compute the product of these matrices and multiply by the previous state,
S: St [[==iM(Me
Starting Prompt
are assistant. will provide you with dictionary of keys mapped 2xx matrices and give you keys in groups of 2. Kaintain the running 2x2 product starting from the identity matrix.
sach group, respond with the <answer></arswer tags using four commasseparate integera. tags,
product in row-major order insida
IMPORTANT: Do output any other inside else at all.
output anything
71 Tllssraaive examples using a reference dietionary:
alpha: [[1, 0], 1]]
beta: [12, 11, [0, 11:
gemma: [[1, 0l. 1:1
delta: [[0, 11, [1, 0.1
Example 1:
Input Xeys: ['alpha", "beta', 'gamma'] 7I ullusrraiing with product after key: Resul:: kanswer21, 2,1,0,1 3,1,1,11/answer>
Example 2:
Trpuu Keys: ('delta', "beta', 'gamms', 'alpha') Illustrating with product after each block of keys: Resul:: 1, 1,1,,, 1e/
Example 3:
input Keys: ['beta', 'genma', 'beta'] illustrating with product after each black of keys: Resul:: sanswer26, 2, 2,2,2</answe, 2</answer>
Now, here the actusl task: Dictionary maintain:
drive: [[-3, 91, [1, -3]]
alona: [[6, 31. [5, -5]1
adult: [[-1, -51, [-2, 8]....
Ready to scart!
IMPORTaNT: only output the four comma-separeted integers representing the matrix product
inside kanswer> tags. Nothing alse should he present in the output, the
Our findings for the iterated matrix multiplication task, presented in Figure 8, are similar to those presented in Figure 7. For each model, evaluate 50 independent rollouts. Without sequential test-time compute, models are unable to multiply even matrices. This is expected, as each step of this task requires constant number of sub-tasks, including multiple retrievals, multiplications, and
Results.
18

---

Published as a conference paper at ICLR 2026
Took Length (Single Turn)
15
Took Leauth (Siagle Tura)
Sequzntia Tea- Compate
Tining
15
Oven of Thright
11
Gramand 278
Despreed va
Rimi-122
Gemmad 27D
Qena so
Derpentt va
Dospeod RJ
Kiml- Thinking
Gered
a
(b)
Figure Benchmarking iterative matrix multiplication within single turn. Without CoT or thinking. even the biggest models fail to multiply just two matrices (a). Sequential test time compute (thinking tokens) significantly improves this (b).
additions. With sequential test-time compute, we see a substantial increase in each model's ability to perform the task.
C INVESTIGATING POTENTIAL FIXES FOR SELF-CONDITIONING
C.I TURN-WISE SELF- VERIFICATION PROMPTING
We investigate whether the self-conditioning effect be mitigated by explicitly prompting the model to perform active self-correction. At each turn, we instruct the model to first re-validate its previously reported state and, if required, recalculate the full historical sum before processing the current turn's 's keys.
The results, shown in Figure 9, are mixed. For the Gemma3 family with CoT, this setup provides an initial boost in accuracy, successfully breaking the self-conditioning loop in early turns. However, the self-verification process significantly increases the number of tokens generated per turn, causing the model to exhaust its context window much sooner, which leads to a sharper performance collapse in later stages. In contrast, the Qwen3 thinking models show negligible improvement. Manually inspecting their reasoning traces, we find that these models, likely due to their fine-tuning, overthink and frequently fail at the verification step itself, sometimes making arithmetic errors even during their re-caleulation process.
These findings suggest that prompting maadt self-correction may not be enough. is computationally expensive, incurring context- context-length penalty, and is itself complex, error-prone execution task that models may not be able to perform reliably.
C.2 CONTEXT ENGINEERING
Another natural mitigation strategy limit the model's exposure to its own past errors in its history. We operationalize this using simple sliding context window, which is particularly well-suited for Markovian tasks like ours. This approach maintains only the most recent turns in the model's context. The rationale is that smaller context window reduces the probability of the model observing a lot of its own past failures, thereby breaking the negative feedback loop of self-conditioning.
As shown in Figure 10 (a), performance improves significantly as the context window size is reduced, allowing models to sustain execution for longer horizons. While a fixed sliding window only applicable to tasks without long-range dependencies, this result validates more general principle: active context management designed to minimize the accumulation of errors in the context is a promising direction for improving long-horizon reliability in L.MM agents.
19

---

Published as a conference paper at ICLR 2026
Qwen3 8B [Thinking]
Gemma3 12B [CoT]
Gemma3 12B [CoT]
Tolems Genernted Step.
1200
Accutary
Turn Accaracy
1000
0.0
0.8
500
0.4
0.4
600
0.2
0.2
50
100
250
200
250
0.6
30
100
150
200
200
50
100
150
200
260
Task Longth
Task Length
Task Length
With Self- Veriffeiiiiattinn
Original Run
Figure Self-verification does fix self-conditioning. Prompting self-verify does not suffice fix the self-conditioning effect completely. leads to overthinking in thinking models and increases the amount of tokens required per turn, leading to faster context consumption in CoT models.
1.0g
0.5
Tion Accuras
Took Acenneyy
0.4
0.0g
Task Lengli
200
0.00
Task Lengh
18
20
Original Run
Contedt Sireel Turn
Context Sizee15 Tume
Majority
Original Run
(a) Context Engineering
(b) Majority Voting
Figure 10: Context engineering and majority voting Gemma3 12B. Controlling context size reduces the self-conditioning effect, hut relies on the Markovian nature our task. Majority voting at K=1 provides only minimal improvements over the baseline.
D CAN PARALLEL TEST-TIME COMPUTE SCALING MATCH THINKING?
We also experiment validate if parallel scaling in test- time compute can achieve the same improvements as think- ing. We verify this by testing if parallel majority voting can replicate the gains from either model scale or sequen- tial computation (thinking). To create fair comparison, we sample multiple outputs from a non-thinking Gemma3 model at each turn, with the number of samples to match the average token count of its CoT counterpart. The final answer is determined by majority vote over these parallel generations. From the results in Figure 11 and 10 (b), we see that while majority voting yields marginal performance improvement over the base model, it is in- sufficient to match the reliability of a larger, non-thinking model, let alone the substantial gains from using CoT reasoning. This suggests that for long-horizon execution, sequential computation provides advantage that parallel test time scaling cannot match. This contrasts with find- ings in other domains, such math common-sense rea- soning, where parallel sampling with self-consistency has been shown to be highly competitive (Snell et al., 2024).
1.0
0.8
Task Accuracy
0.6
0.4
0.2
0.0
100
200
Task Length
Majority Vote [No CoT] (N=100)
Chain Thought Enabled
Figure 11: Parallel time scaling on Gemma3 12B at K=2. Majority voting with the same amount of tokens as CoT traces does not match the performance of CoT.
20

---

Published as a conference paper at ICLR 2026
Qwen 328
Gemma3 12B
Tua Areuner Mise 120
Thak Avruney Abr
Turn Complesiy
Complexily
10
(a) For the same number of total steps, different turn complexities lead to different outcomes, We find trend across families.
Qwen3 32B
Gemmaã 12B
d
g
Final Tank
200
Aworgee Ostpet Tikess
Ascrage Outyon Tolous
(b) Average output tokens used complete the execution vs final accuracy. We that for Qwen3 32B, turns lead more token usage, even at lower turn complexities, pointing to overthinking. Gemma3 12B, on the other hand, uses fewer tokens for very low turn complexity or very high turn complexity.
Figure 12: Relation between the turn complexity and the number of turns.
E NUMBER OF TURNS VS TURN COMPLEXITY
In our experiments, we show that we can increase the length of the task needed be performed by either (1) increasing the number of turns or (2) increasing the turn complexity, i.e, providing more inputs in the same turn. To investigate the relationship between these two axes, we perform an experiment where model has to perform fixed number of operations while varying the turn complexity. With a higher turn complexity, the model requires fewer turns to reach the fixed number of operations. Results in Figure 12 (a) indicate there is strict turn complexity that is consistently the best across model families. Rather, we found that different models behave quite differently for the same complexities. Qwen3 32B seems show poorer performance lower turn complexities, indicating that it is unable to perform well over a large number of turns, even if the turns are simple themselves. Gemma3 12B shows different trend. It reaches accuracy peaks at either extreme of the turn complexity spectrum, failing badly at mid-level turn complexities. This indicates it suffers when the turn complexity and the number of turns are both sufficiently high.
Another axis of evaluating the number of turns turn complexity trade-off is the test-time compute used. From cost perspective, increasing the number of turns increases the overall cost of inference. We can lower the number of turns by increasing the turn complexity, but that would result in an increase in the per-turn inference cost, as result the added complexity. For the same experiment above, track the number of output tokens used for computation (including thinking tokens) per sample, and again find diverging results for each family in Figure 12 (b).
F DECONSTRUCTING ERRORS IN RETRIEVE-THEN-COMEE
STEPS
To further isolate the source of execution errors, we decompose our task into its two constituent operations-rretieval and addition-and evaluate models on them individually:
Retrieval-Only Task. A stateless task where, at each turn. the model is given key and must simply return the corresponding integer value from the dictionary. No running sum is maintained. This isolates the retrieval component.
21

---

Published as a conference paper at ICLR 2026
0.5
0.4
Yook Accunasy
Tulk Accuuayy
Hontoon Lereth
Tisk
50
Modii Fanily Gexana
Que
0.0g
0.6
0.
Length
(Diliion
(a)
"Thinking Disabled
K=2
(b)
*Thinking Enabled
(c) *Thinking Enabled
K=10
(d) *Thinking Enabled
Gemma3-48
Gemma5-128
GemmaS-2778
Quens548
Qwen3-88
Qwen3-148 148
Qwen3-328
Figure 13: Scaling trends hold even after enabling Sequential Test Time compute. We compare model performance with thinking disabled (a) against thinking enabled for Qwen models and CoT for Gemma models (b, varying turn complexities. (a) Without thinking, all models fail execute
two steps (K 2) in single turn. (b) In contrast, enabling thinking prevents this performance collapse, with all models successfully handling K 2. (c) When the turn complexity is further increased to K 10, performance degrades, but clear scaling trend emerges.
(d) This trend is explicitly shown, illustrating that for complex turns, the horizon length increases consistently with model size, reinforcing the benefits of scaling model size even when thinking enabled.
1.0
1.0
0.8
0.8
Task Accuracy
Step Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
25
100
Task Length
Tusk Length
Gemma3-48 3-BB
Gemma3-122 28
Gemma3-2778
Qwen3-4B
Quen3-88
Qwen3-14B
Qwen3-328
Figure 14: Temperature does not impact the trends observed. We reproduce the same trends in Figure 4, when running with temperature 0.
22

---

Published as a conference paper at ICLR 2026
Addition-Only Task. A stateless task where, at each turn, the model is given two random integers to add. No rumning sum maintained. This isolates the arithmetic component.
Prefix-sum Task. A stateful task where, each turn, the model is given integer directly and must add it to its previously reported running sum. This isolates the combination of arithmetic and state-tracking components.
From Figure 15, we can observe that models achieve near-perfect performance on the stateless retrieval and addition task, indicating that neither simple dictionary lookup nor addition is significant source of error. In contrast, the prefix sum task, while significantly better than our task, still exhibits a slow degradation over time.
This leads to two key insights. First, the difficulty lies not in the atomic operations themselves, which models perform with high accuracy in isolation over long horizons. Second, this suggests that component of the task. While stateless
the primary source of degradation the stat-maaaggeee retrieval and addition are trivial the requirement reliably maintain and update a running sum introduces higher chances of error. This suggests that the models struggle with the requirement to concurrently manage information lookup and state updates.
1.00
0.5
Turn Accuracy
0.75
Twls
0.4
0.50
0.2
0.25
0.0
20
49
90
80
100
Trak Tangth
Retrieval
Cumulative Addition
0.00
400
800
Task Length
Addirian
Camulative Retrieval Addition
Gemma3 27b
Owen3: 32b
Figure 15: Analysis of execution failures. (a) Sclf-conditioning effect emerges as tasks get longer. Even for models that the task at task length of 100, the Turn Accuracy drops constantly further increase the turns. (b) Models are good the tasks individually, but their composition. State tracking introduces additional difficulty.
G EXPERIMENTAL SETUP
G.1 TASK DETAILS
We create a dictionary where keys consist of common five-letter English words, and the values consist of integers uniformly sampled from -99 to 99. The range of values is deliberately kept large to minimize the chance of assistant being wrong an earlier accident. For our experiment, we first create fixed set of 100 keys. Then, we create multiple rollouts (samples) of 50, 000 steps. For each rollout, we uniformly sample separate set of values to be assigned to each key, in order to increase experimental breadth. Next, at each step, we uniformly sample a key to provided at that step with replacement. This gives us a list of keys to be processed in order, which is exactly the plan to be executed by the agent.
correcting its response by pure
To account for the turn complexity (K), we group consecutive keys together and represent them as one turn. Thus, we can fully specify our intended evaluation by (1) specifying the number of samples (rollouts) needed, (2) the turn complexity, K, and (3) the number turns required. This gives us superset of data from which sample rollouts to use in experiments. For the Qwen3 and Gemma3 families, we sample 100 rollouts. For frontier models, due to cost limitations, we sample 20-50 rollouts. To ensure consistency in evaluation, provide the same rollouts to each model.
G.2 PROMPTING
Each LLM is provided a standardized prompt describing the task at the start of the conversation. This prompt specifiss the dictionary containing the five-letter word keys and their corresponding values.
23

---

Published as a conference paper at ICLR 2026
Further, the prompt specifies the number of keys that will be provided to the LLM at each subsequent turn. To ensure format following, the prompt also contains few-shot examples with different turn complexities. Finally, the LLM asked to provide the rumning after each turn in <answer> tags. An example conversation is shown below.
G.3 PROMPTING FOR THINKING MODELS
To enable models to use chain-of-thought prompting, we add the line "Think step by step before answering. to the prompt and also add CoT traces to the few-shot examples. We find that models stop performing CoT reasoning after a few turns, as it starts conditioning on the answer format in its history. Thus, end up including the thought trace in the conversation history, to ensure the model does not forget the CoTT instruction. This is a trade-off we had to make it increases the input context of the LLM, however, was essential to ensure instruction following. Thinking models provided their reasoning in <think> tags, which were removed from the conversation history. No other changes were needed to make the thinking models follow instructions.
G.4 MODEL SPECIFICATIONS
For chain-of-thought prompting, we set the per-turn output token limit to 10, 000 tokens, and for thinking models, the token limit is 32, 000 tokens, consistent with token limits provided by OpenRouter. We ensure that these token limits are sufficient to complete the required computations.
We temperature of 0.6 and a top-p value of 0.95 for all Gemma models. For Qwen, we use temperature of 0.6 and a top-p value of 0.95 for thinking mode and a temperature of 0.7 and a top-p value of 0.8 for non-thinking temperature does not affect the observed trends by much.
recommended in their documentation." We find in Figure 14 that
G.5 COMPUTE DETAILS
All experiments were conducted on machines equipped with 4x NVIDIA A100 GPUs with 40/80GB memory. Frontier model evaluations were performed using OpenRouter.
24

---

Published as a conference paper at ICLR 2026
Starting Prompt:
You are an AI assistant. will provide you with a dictionary and then give you keys in groups of 2. Your task is to keep running total (starting from 0) by adding the values associated with the keys provide.
each turn, I'1l provide keys (comma- seomaa-seaaaaedh.. Respond with the current running sum, eaclosed <answer> tags. Examples:
Dictionary to maintain: 'apple'; 5, 'banana': 0, 'cherry': 7,
'grape': -4, 'kiwi': 2, 'mango': -1
example 1: keys in groups of
User: apple, banana
Assistant: <amswer>5</answer>
User: cherry, grape
Assistant: canswer9<</answer>
User: kiwi, mango
Assistant: <answer>9</answer> Example 2: keys in groups of
User: apple, banana, cherry Assistant: <answer>12</answer
User: grape, kiwi, nango Assistant: <answer>9</answer Example 3: keys in groups of 6
User: apple, banana, cherry, grape, kiwi, mango Assistant: <answer>9</answer> Now, here is the actual task:
Dictienary to maintain:
'doubt'; -64, 'alone': 46, 'adult'; 84, 'fault' -19, 'brain':
-45, 'blind'; 68, 'cable': -32
'coach': -31, 'alarn': 88, 'could': 25,
Ready to start!
IMPORTANT: DC NOT OUTPUT ANY OTHER TEXT OUTSIDE ANSWER TAGS. Only provide the final running OF ALL TURNS in <answer> tags.
User: alarm, coach
Assistant: <answer>57<<ansserr>
User: doubt, cable
Assistant: <answer>>39<</nsww
H FORMAT FOLLOWING FAILURES
In any LLM evaluation, format following failures common source of error that is often neglected. In our experiments, any model can have types of format following failures: (1) They do not provide <answer> tags their answer, and (2) They do not provide valid integer within <answer> tags. To minimize format following failures, we ensure clarity in the starting prompt with clear format instructions, as well few-shot examples. To empirically verify that model errors our task actually execution errors and not just format following errors in disguise, for each experiment, we also track the format failure fraction: the fraction of samples that do not correctly follow the format, with the failure being either (1) or (2). It is important to note that while we try to minimize any such error the best of our abilities, we still count format following as limitation of the model and hence source of error.
Our results for format following failures are presented in Figure 16 for the experiments presented in Section 3. We observe that smaller models are more susceptible to format failures, with the Qwen3 family in particular being worse at following format instructions. Overall, the fraction for format following errors is low (around 0.1), with the Qwen3-8B being exception. We find that the error here actually comes from the model trying to cheat, and do the entire summation inside the <answer> tags (For example, <answer>39 51 (D</answer>). This is explicitly
25

---

Published as a conference paper at ICLR 2026
Somot Fatlm Proetym
Fomat Fulter Pricton
Fulux Fuclin
200
20
100
70
100 Length
300
Leugth
Leugth
Length
(n) K=1
(b) K=2, Thinking Disabled
(c) K=2, Thinking Enabled
(d) K=10, Thinking Enabled
Gemma3-4B
Gemma333 128
Gemma3-278
Qwen3-4B
Qwen3-88
Qwen3-14B
Qwen3-328
Figure 16: Format following the primary mode failure. We analyze the fraction of errors attributed to incorrect format following for the experiments presented in Section 3. Overall, format adherence is high and not the primary source of exccution errors.
forbidden we do not allow chain-of-thouse thinking in this experiment, and thus count this as error. Gemma3 4B fails at later turns due to context length limitations; however, that does not affect any results, as its accuracies drop much earlier.
For the experiments presented in Figure 16, we find the Qwen3 family to be prone to format following errors in the case where we have thinking disabled for K consequence of models trying to cheat and use extra tokens for computation inside the answer tags. This is fixed by enabling thinking. Following this, the errors format become negligible. At K 10, we see Gemma3 12B sharply rise to format failure fraction of 1.0, again due to full context window.
We again find this be the
I CHAIN-OF-THOUGE SELF-CONDITIONIG
While our seff-conditioning analysis provides clear insights for thinking models, extending this to models using Chain-of-Thought Thouugh (CoT) presents some significant methodological challenges.
First, fundamental prerequisite for reliable CoT reasoning the inclusion of prior CoT traces in the context history. As we observed with the Gemma3 models, they often condition the format of context; prior turns lack CoT traces, the models cease to generate them, even when explicitly instructed to do so. Consequently, this experiment for CoT must include the full reasoning trace for every preceding turn. This requirement immediately makes the setup practically infeasible, as the verbose nature of CoT traces would rapidly exhaust the context window limits of even frontier models.
Second, even if context length were not constraint, the process of injecting controlled errors into CoT histories is not straightforward. A naive approach of only altering the final answer while preserving the original, correct CoT trace creates unfaithful history. When conditioned history where reasoning and conclusions are contradictory, the model is no longer being tested on its execution reliability but on how it resolves inconsistency-it might learn to distrust its own reasoning, introducing a confounding variable.
The alternative is programmatically generate flawed CoT traces. We implemented and experimented with this, and as seen in Figure 17, CoT does not mitigate the self- self-conditionig effect. However, this setup also introduces its own complexities. For our simple task, there are multiple distinct points of failure within a single trace: an enror in the retrieval step (looking up an incorrect value) or error in composition step (an arithmetic mistake). controlled experiment would need to systematically manage the type, frequency, and location of these injected errors, making the setup intractable. Even establishing a "perfectly correct" (Induced Error Rate 0.00) baseline history is problematic. A model might have CoT trace with flawed reasoning (e.g., minor calculation error that cancels out), which we then replace with the correct final answer. Such a history also unfaithful.
Given these challenges-tie the practical infeasibility due context length and the difficulty of designing a faithful error injection mechanism, we limit our self-conditionin analysis to non-thinking and thinking models.
26

---

Published as a conference paper at ICLR 2026
Gemmaa-3- 12B CoT
0.4
Format Failure Fraction
0.35
0.8
Turn Accurecy
0.25
0.6
0.2
0.13
0.3
0.03
1000
Task Length
Task Length
Original Run
100% Eror Rate
13% Erso Rate
50% Ersor Rate
25% Ernor Rate
Enor Rate
Figure 17: CoT does not fix self-conditioning. We observe even with programmatically generated CoT history, the Gemma3 models cannot mitigate self-conditioning.
J PROOF AND ANALYSIS OF PROPOSITION 1
Proposition 1. Assuming an independent and constant per-step accuracy and no self-correction, the horizon-length at which model can achieve success rate s given by:
In(s)
H.(p) = In(p)
Proof. Let p be the constant probability of successfully executing single step. Under the assumption of no self-correction, task of length H is successful only if all H independent steps are executed correctly. The probability of this joint event, P(success, H), the product of the individual step probabilities:
P(success, H) P X X ... xp = "
H
times
This is equivalent the Task Accuracy turn H, i.e., TA(H) nd We define the horizon-length H as the number of turns at which the probability of success equals desired rate s. Therefore, we our expression for the success probability equal to
Solving for H.
In(p") In(s) = H In(p) In(s)
H.(p) = [n(s)
In(s)
In(p)
In
This completes the proof.
J.1 IMPLICATIONS FOR HORIZON LENGTH (Ho.s)
We apply this general result specific metric, the Horizon Length (Ho.s), which is defined as the number of turns at which Task Accuracy drops to = 0.5,
[In(0.5)
In ln(2)
Hp) (p)
In p
In
For analysis, we use the continuous approximation:
Ho.r(p) ~ 1n(2)
In p
27

---

Published as a conference paper at ICLR 2026
Sensitivity to Small Changes in Step Accuracy. This formulation allows analyze the sensitivity of the horizon length to small improvements in per-step accuracy by taking the derivative with respect to p.
dHos In(2) (- (Tmp)) 1)
1n2
dp
p(n
This implies that small change in accuracy Ap results in change in horizon length Aoo.5 of,
ln2
Aos.
Ap
p(lnp)²
Near-Perfect Accuracy Regime. The effect is most dramatic when accuracy is already high. For near-perfect accuracy, let p= - where < 1. Using the Taylor approximation In(1 - E) R -e,
can simplify the expression for Ho.s.
In(2)
In(2)
1n2 ln2
Ho.5
In(1 6)
1 1-p -p
The sensitivity in this regime becomes,
dHo.s
ln2
In 2
Aos.
Ap
dp
(1 1--p-
p)
(1 p)2
This demonstrates that as p 1, the improvement in horizon length for fixed gain in step accuracy grows quadratically, highlighting the compounding benefits of scale.
28


Reviews:
[{"review_id": "YWq4IRZK5Y", "reviewer": "Reviewer_xdtj", "summary": "This paper investigates whether the apparent diminishing returns from scaling LLMs are an illusion caused by evaluating them on short-horizon tasks. It identifies execution errors instead of reasoning failures as the main limitation in long-horizon task performance. The authors design a controlled synthetic setup that isolates execution by providing explicit plans and knowledge, enabling the measurement of how many steps a model can reliably execute. They discover that even marginal single-step accuracy improvements compound exponentially into longer horizon lengths, introduce the notion of self-conditioning and show that thinking models mitigate this effect. Experiments across models show non-diminishing gains from scaling.", "questions": "See suggestions in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a well-structured experimental design using a key–value retrieval task.\n* The paper uncovers a new degradation source of models compounding their own mistakes, which is novel.\n* The authors evaluate multiple model families, showing consistent scaling trends.", "weaknesses": "* The arithmetic-like setup may not capture the diverse error dynamics in natural environments. Extending to real-world planning or tool-use tasks could strengthen claims of generality.\n* The focus is primarily on model size and sequential test-time compute, without exploring complementary factors such as training curricula, architecture modifications, or memory-augmented mechanisms that may interact with long-horizon execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5GhUNK59NF", "reviewer": "Reviewer_42Ay", "summary": "This paper proposes the idea that the common notion of \"diminishing returns\" on single-step tasks (as models are scaled up) should not be taken to mean that there is no value in pushing for those small increases, as these seemingly small increases can lead to an exponentially large benefit on long-range execution. In particular, the authors propose a synthetic tasks (of essentially computing \"prefix sums\" of a sequence of numbers revealed via a key-value mechanism) as well as setups/metrics to disentangle execution ability (the focus of this study) from the ability to properly plan and/or retrieve knowledge. With detailed studies on this benchmark, the authors provide evidence supporting their claims about diminishing returns being an illusion, LLM trends for long-horizon execution tasks, etc.", "questions": "Could the authors comment on the points raised in the weaknesses section and how they might address it? I think the paper is thought-provoking and I am open to increasing my score contingent on the authors' willingness to address some of the points raised.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The question raised by the authors, namely whether the diminishing rate of improvement in single-step tasks is actually a real problem or not, is an interesting and practically valuable one. The authors' proposal of considering the max task execution length as a key metric is also of importance.\n\n* The study is clearly motivated and the empirical section is well-organized with clearly labeled setup and results section.\n\n* The authors raise a fairly large number of interesting questions and design specific experiments to test them.\n\n* The synthetic long-horizon execution task seems well-suited to the authors' desire to isolate away effects of knowledge retrieval and planning. The idea of treating planning and retrieval as \"key value\" problems is unique, and it allows the authors to \"provide\" the plan and knowledge at each step as appropriate.\n\n* Some of the results are quite surprising and creatively supported by the authors with explanation hypotheses. E.g., that single-step accuracy decreases with task length seems very unintuitive at first sight, but then the authors try out two hypotheses and find their combination to be a useful explanation of this phenomenon.", "weaknesses": "* The synthetic task proposed here boils down to retrieving and summing up (all prefix subsequences of) $t \\times K$ numbers, where each number has 2 digits. While this task fits the authors' desire to isolate out planning and retrieval issues, it's **not exactly a sequential task** as referred to by the authors when discussing some experiments. This is important as, while sequential tasks are known to benefit from \"reasoning\" tokens or chain-of-thought, this is not really the case for adding $N$ numbers. Formally, this problem is well-known to lie in the complexity class $TC^0$ of *fully parallelizable* problems! This is a bit counter-intuitive, but happens to be true --- that one can add $N$ $N$-bit numbers completely in parallel in just a few steps. In fact, when a Transformer-based LLM compute attention over $N$ previous tokens, this sum is precisely what it computes. Thus, in a way, the synthetic task here isn't quite the perfect fit for what the authors really need. (An alternative task would be where 2-digit numbers are replaced with $5 \\times 5$ size *matrices*, and instead of sum, the task is to *multiply* the matrices; this is *not* known to be fully parallelizable, making it a better fit for some of the authors' narrative around CoT and reasoning tokens being helpful.)\n\n* That said, the experiments even with the proposed \"prefix sum\" task do seem to support the authors' claims in practice. I do find some details a little hard to believe/understand, though. For instance, Fig 7(a) seems to suggest that without CoT, strong models like Deepseek-V3 an Kimi-k2 can barely add 4-6 numbers of 2 digits each. I would have expected them to be able to handle longer sequences.\n\n* The discussion in section 2.1 (diminishing returns) feels not very convincing or deep. For one, Proposition 1 seems to follow directly from the observation that if each step *independently* succeeds (a strong assumption) with probability $p$, then $T$ steps will succeed with probability $s = p^T$. If this is the reasoning, it would be better to spell it out in the main text, as readers will readily connect with this common understanding of single-step errors compounding in multi-step scenarios.\n\n* Similarly, it is well-understood that single-step errors compound in multi-step uses. It would be helpful for the authors to discuss this. The common \"use\" of this observation is that a small *decrease* in single-step accuracy often leads to a large *drop* in multi-step accuracy. What the authors are saying is really the same statement, looked from the other direction: namely, a small *increase* in single-step accuracy often leads to a large *increase* in multi-step accuracy. These kinds of connections both help understand where the proposed observation is coming from and why it's reasonable. Yet, the discussion is not framed this way.\n\n* In the same section, in Fig 2, the claim that a small increase in step accuracy (x-axis) leads to a large increase in horizon length (y-axis) appears to really be only true when the slope of the curve is significantly larger than 1, which happens when step accuracy is already in the range of 90% to 100%. Note that for long-range tasks to be meaningfully solved, $H_{0.75}$ or $H_{0.9}$ are much more realistic measures than $H_{0.5}$. In other words, the exponential increase observation (with small increases in step accuracy) seem to \"work out\" only when single-step accuracy is already very high, in the 90%+ range. This seems important but is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmZvik8TDN", "reviewer": "Reviewer_qfhB", "summary": "This paper studies the ability of LLMs to execute many-step plans or instructions in long-horizon tasks. While this consists of planning and then execution, the paper strips away the planning part via controlled tasks where the model is given detailed instructions and \"knowledge\", to only test the ability to execute. The paper studies three factors: (1) the impact of model strength/scaling, (2) the impact of correct executions of earlier steps in the context (called self-conditioning in the paper), and (3) the impact of thinking.\nFor (1), it starts with a simple formal analysis that assumes constant and independent accuracy per step, that indicates strong gains from improving accuracy per step. The paper itself later shows that the assumptions of this analysis are not always true. To obtain a controlled empirical setting, the work uses a simple lookup and prefix-sum task, where the \"dictionary\" is provided. Regarding (1), they find that first, even models that are very good at one lookup or addition step can perform much worse on the long-horizon task, possibly indicating problems with state tracking. The per-step accuracy diminishes with the length of the task. Second, still, better models indeed are better at long-horizon execution. For (2), the paper shows an experiment that helps distinguish between failure due to task length versus due to errors in the execution trace, by controlling the errors in the trace. All tested models are sensitive to the errors in their history. (3) For thinking models, this sensitivity appears to go away. The final experiment measures the influence of per-step complexity, and finds that without CoT or thinking, models cannot execute complex steps.\n\nOverall I thought this paper has some interesting findings.", "questions": "- Sec 3: how exactly is correctness measured for the cunulative sum and addition? If e.g. the model makes a mistake in step 2, the result is wrong for the remaining steps. Is this counted as an error from step 2 onwards or just for step 2? Is this also how it is done for horizon length?\n\n- lines 260/261: where is the CoT in Figure 12?\n\n- You find that both state tracking and self conditioning are a problem. Do these interact, and if so, how (maybe I missed that)?\n\n- do Sec. 3.2 and 3.3 use the same task as Sec. 3.1? In Sec. 3.3, how exactly do you vary the turn complexity? A description and/or example would be helpful.\n\n\nOther minor comments:\n- legend of Figure 7b: both thinking and CoT color code look the same in my pdf\n- For proposition 1, add the independence assumption to the proposition statement. This may not hold in practice at all times, so good to mention it.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper shows interesting observations on the brittleness of LLMs to errors in context, and the increase in per-step error in long tasks, despite being able to execute single steps well.\n\n- The experiments use a controlled (albeit very simple task) setting with interesting setup. \n\n- Multiple models of different sizes are tested.\n\n- The paper is decently written.", "weaknesses": "- The controlled task is rather simple, and it is a chain of always the same task (the paper mentions this already, though). I understand the controlled setting helps to get a controlled study, and that is also good. Possibly relating it to a more real-world task could have been nice.\n\n- Some things were not fully clear -- see my questions below.\n\n- it is possible that if the model designs a plan internally, it is different from the externally provided lookup and \"plan\". This is not a full weakness in that the model is not fully transparent, but it could be mentioned in the text.\n\n- In some points, the writing was not completely clear to me, e.g.:\n\nlines 87/88: \"the per-step error rate itself rises as the task progresses. This is in contrast to humans, who typically improve at executing a task with practice.\" The reasoning chain is usually not \"practice\", hence, I was not sure how this relates to humans practicing.\n\nlines 133/34: \"In fact, failures in execution have been misattributed to limitations in reasoning or planning capabilities\". If the model fails, how do you know it is planning correctly but failing to execute the plan? If you do not control for the plan like in this paper.\nThe description of \"turn accuracy\" compared to \"step accuracy\" is a bit confusing, as both use one state update.\n\nSec 3.1: the appendix explains the results with the difficulty of state tracking/management, whereas the main text talks about circuits. Why is the discussion split like that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e1PmWIlnpE", "reviewer": "Reviewer_mGG6", "summary": "This paper studies whether scaling LLMs or introducing thinking continues to yield benefits, for long-horizon tasks that requires many sequential steps. They design experiments with provided knowledge and plan to isolate execution and planning capabilities.\n\nTheir finds are:\n\n1. longer execution are more difficult\n2. larger LLMs significantly prolong the execution\n3. self-condition is the main reason aside pure long context\n4. larger LLMs do not affect self-condition\n5. thinking alleviates self-condition\n6. CoT is necessary\n7. GPT-5 > Claude 4 ~ Grok-4 > others", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n\n2. The idea is well motivated; isolating the long horizon from other factors makes sense.\n\n3. The experiment setting is well designed. (1) long horizon, knowledge, and execution are decoupled. (2) long context and self-condition are decoupled.\n\n4. Finds are valuable to the community.", "weaknesses": "1. Is self-conditioning a bad behavior? From my understanding, a strong LLM with in-context reasoning **should** be self-conditioned.\n\nFor example, in this case:\n```\nOriginal number: 10\nSequence: [+3, -7, +1, +2]\nStep 1: 10 + 3 = 14\nStep 2: 14 - 7 = 8\nStep 3: 8 + 1 = 10\n...\n```\n\nIn this case, a better answer should be `Step 4: 10 + 2 = 13`\n\n2. The discussion about the relationship between self-conditioning and existing self-correction methods is necessary. The self-correction method includes agentic methods (self-refine and its following works) and reasoning behavior (aha moment).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "18PSbydvQh", "reviewer": "Reviewer_f3Ec", "summary": "This paper investigates the scaling properties of Large Language Models (LLMs) on long-horizon tasks. The central thesis is that the perception of \"diminishing returns\" from scaling is an illusion created by short-task benchmarks. The authors argue that even small, diminishing gains in single-step accuracy can compound exponentially, leading to massive improvements in the total length of a task a model can successfully complete. To support this, the authors propose a distinction between \"reasoning/planning\" and \"execution.\" They argue that many failures on long tasks are not failures of reasoning, but of simple execution. They introduce a synthetic, multi-step key-value dictionary addition task designed to isolate this \"execution\" capability by providing the model with all necessary knowledge (the dictionary) and the plan (the keys to add). The authors conclude that scaling and sequential compute both provide massive, non-diminishing benefits for the long-horizon tasks that hold true economic value.", "questions": "1. If the problem is purely \"execution,\" why is the solution \"reasoning\"?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and the central arguments are clear to follow.\n2. The synthetic experiments are thorough. The design of the \"self-conditioning\" experiment, which injects errors into the model's history to prove a causal link, is particularly clever and provides clear support for this specific phenomenon.", "weaknesses": "1.  The paper's entire set of conclusions about \"long-horizon execution\" is built upon an extremely synthetic and narrow task: repeatedly retrieving values from a dictionary and adding them to a running sum. The authors claim this isolates \"execution,\" but it is highly questionable whether this type of execution is representative of the complex, diverse, and adaptive execution required in real-world agentic tasks (e.g., navigating a new website, debugging code, or managing a multi-tool workflow). Real-world execution is not typically the N-th repetition of the exact same retrieve-then-compose operation. It involves varied actions, changing states, and error correction based on new observations. Because the task is so far removed from the real-world tasks it claims to model, it is impossible to know if the findings (especially \"self-conditioning\") would generalize. The paper's broad conclusions about the \"economic value\" of LLMs are therefore built on a very tenuous foundation. The limitations section acknowledges this, but I believe it is a more fundamental flaw that challenges the paper's entire premise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)