Dynamic Speculative Agent Planning
Authors: Yilin Guan
Qingfeng Lan
Fei Sun
Dujian Ding
Devang Acharya
Chi Wang
William Yang Wang
Wenyue Hua
Venue: ICLR 2026 Poster (2026)
Keywords: agent
efficiency
online learning
reinforcement learning
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Despite their remarkable success in complex tasks propelling widespread adoption, large language model based agents still face critical deployment challenges due to prohibitive latency and inference costs. While recent work has explored various methods to accelerate inference, existing approaches suffer from significant limitations: they either fail to preserve performance fidelity, require extensive offline training of router modules, or incur excessive operational costs. Moreover, they provide minimal user control over the tradeoff between acceleration and other performance metrics.
To address these gaps, we introduce **Dynamic Speculative Planning** (DSP), an asynchronous online reinforcement learning framework that provides lossless acceleration with substantially reduced costs without requiring additional pre-deployment preparation. DSP explicitly optimizes a joint objective balancing end-to-end latency against dollar cost, allowing practitioners to adjust a single parameter that steers the system toward faster responses, cheaper operation, or any point along this continuum.
Experiments on two standard agent benchmarks demonstrate that DSP achieves comparable efficiency to the fastest lossless acceleration method while reducing total cost by 30\% and unnecessary cost up to 60\%. Our code and data are available through https://github.com/guanyilin428/Dynamic-Speculative-Planning.

Published as a conference paper at ICLR 2026
DYNAMIC SPECULATIVE AGENT PLANNING
Yilin Guan
Qingfeng Lan"
Fei Sun
Dujian Ding'
Devang Acharya'
Chi Wang
William Yang Wang
Wenyue Hua
JJhns
Hopkins University Universstty of Alberta
Uniiversity of British Columbia
+ "vvyy Research Center
'Google DeepMind University of California, Santa Barbara
yguan2 w0w..ehhhhee edu
ABSTRACT
Despite their remarkable success in complex tasks propelling widespread adoption, large meeguage-mmdel..eo -ased agents still face critical deployment challenges due to prohibitive latency and inference costs. While recent work has explored various methods to accelerate inference, existing approaches suffer from significant limi- tations: they either fail to preserve performance fidelity, require extensive offline training of router modules, or incur excessive operational costs. Moreover, they provide minimal user control over the tradeoff between acceleration and other performance metrics. To address these gaps, we introduce Dynamic Speculative Planning (DSP), an asynchronous online reinforcement learning framework that provides lossless acceleration with substantially reduced costs without requiring additional pre-deployment preparation. DSP explicitly optimizes a joint objective balancing end-to-end latency against dollar cost, allowing practitioners to adjust a single parameter that steers the system toward faster responses, cheaper oper- ation, or any point along this continuum. Experiments on two standard agent benchmarks demonstrate that DSP achieves comparable efficiency to the fastest lossless acceleration method while reducing total cost by 30% and unnecessary cost up to 60%. Our code and data are available through http::: / com/ guanyi 1iiiinnnnnanannn Stp.
Speculative Planning
INTRODUCTION
LL.M-assed agents have rapidly moved from controlled demonstrations to real-world deployments. supporting complex multi-step tasks in domains such as autonomous software engineering (Wu et al., 2024), open-world tool use (Shen et al., 2024), and everyday personal assistance (Ge et al. 2024). Although ----ddsed agents have achieved widespread adoption, practitioners consistently report substantial end-to-end latency (Hua et al.; Zhang et al.] 2024b Saha al.) 2025). This latency not only degrades user experience but also limits the fcasibility of deploying basedd agents in
time-sensitive scenarios such real-time decision support (Cai et al. 2025 interactive tutoring
(Modran et al. 2024 Piro et al. 2024).
Multiple lines of research attempt address the latency bottlenecks. These include approaches such as context length reduction (Zhang et al.), step parallelization (Hua et al.) Li et al. 2025 Pan et al. 2025), and dual-process thinking that combines fast and deliberate reasoning modes (Saha al. 2025 Lin et al.; Christakopoulou et al.). While these approaches offer various improvements in efficiency, there are several common critical limitations: (1) No guarantee on lossless performance except (Sha et al. (2025) requires training a router between two agents and Zhang et al. requires careful prompt engineering to enable runtime tool-selection, both relying on the quality of the additional component. [Pan et al. (2025) requires the model to learn to generate parallelizable steps. (2) Pre-deployment preparation: Saha et al. (0025); Pan et al. (2025) require training specialized modules in the system before deployment, adding complexity and overhead implementation. (3) Lack of usserconttollabiiiiy (Kiefer et al. 2017): Most existing approaches neglect user conttcontolleed flexibility in navigating efficiency improvements. This inflexibility becomes particularly problematic in the rapidly evolving LLM ecosystem, where pricing structures, model inference speeds, and model performance change frequently, and organizations have diverse priorities regarding latency versus

---

Published as a conference paper at ICLR 2026
episode
episode 2
New data for Training: (s
#1:
52.
Trainer
(e 83.
Predictor
k = 2
k: =
k: = 2
A
B
C
D
Approximation
D
Target
8
Time
Cancel ongoing threads
Total task time
Prodictor
Approximation
Target
Trainer
Action exceution
Canceled operations
Figure 1: Dynamic Speculative Planning with Online Reinforcement Learning. The diagram illustrates how DSP adaptively adjusts speculation steps across planning episodes. (1) Predictor dynamically infers step values & from current state s;, while the Approximation (A) and Target (T) agents execute parallelly. (2) When mismatches occur (denoted by x), ongoing threads are canceled and execution resumes from the last verified step. (3) asynchronous Trainer collects state-step (s, kc) pairs from completed episodes, continuously updating the predictor without blocking execution.
operational costs including router training, prompt engineering, or extra cost brought by parallelizing computations.
In this paper, we propose Dynamic Speculative Planning (DSP), lossless, user-controllable agent planning acceleration framework that requires pre-deployment preparation. In addition to ensur- ing lossless agentic performance via speculative strategy (Hua et al.). DSP introduces three core improvements for cost reduction and user-controllability
First, we investigate the Pareto frontier the latency-cost trade-off in speculative agent planning, determining the optimal cost for any given latency reduction target. This trade-off is governed by the speculation step, which refers to the number of future steps that an approximation agent attempts to predict ahead of verification. It determines how aggressively the system parallelizes computation to reduce latency. Our analysis reveals that the commonly adopted ixeddsppeculation-stey approach (Hual al. Leviathan et al., 2023) is fundamentally limited: for complex tasks, aggressive speculation produces excessive and redundant agent calls that substantially increase costs, while for simpler tasks, conservative speculation fails deliver sufficient acceleration. Since optimal speculation step
highly context-dependent and unknown a priori, fixed steps significantly constrain practical utility.
Second, we develop lightweight adaptive speculation step predictor that dynamically determines when to suspend speculation, effectively eliminating unnecessary cost while preserving acceleration benefits. This predictor employs online reinforcement learning that requires no external datasets, preprocessing steps, dedicated pre-deployment phases. Instead, the system learns to optimize speculation step organically as it processes tasks, becoming increasingly efficient over time with zero additional infrastructure costs while ensuring immediate deployment benefits.
Third, based on this dynamic prediction module, we introduce two mechanisms that modulate the trade-off between latency and cost: (1) biased step prediction and (2) unbiased step with biased offset, allowing practitioners to precisely calibrate system behavior anywhere along the spectrum from low-cost/low-spee to caw.ccottts/seeeeeeee operation. By providing fine- -nnaiind user control the latency-coss balance, our framework accommodates diverse organizational priorities and adapts the fluctuating pricing structures and inference speeds of the rapidly evolving LLM ecosystem.
2

---

Published as a conference paper at ICLR 2026
The structure of our system is presented in Figure 1
Our empirical evaluation across two standard agent benchmarks demonstrate that DSP achieves the efficiency of the fastest lossless acceleration methods, while reducing total cost by up to 30%, cutting unnecessary cost by 60%, and preserving comparable acceleration, demonstrating the effectiveness of the framework.
RELATED WORK
Latency in LLM-based Agent LLM-based agents have gained widespread adoption in real-world
applications Ge et al. )2024); Wu et al. (2024) due to their strong planning capabilities. However, significant latency issues have been documented across multiple domains: [laech et al. (2024) reports 24-hour processing windows for ol-based agents in engineering tasks, while [Elfeeet & Chollet (0024
and [iu et al. (2024a) notes that perceived latency and extended inference time negatively impacts user experience and render LLM agents unsuitable for real-time applications such as virtual reality
and gaming.
Several research directions have emerged to address latency challenges. System 1.x Saha al. (2025) leverages dual- dual-process theory of cognition Evans| (2003); Hua & Zhang (2022), dynamically substituting the primary backbone with smaller models when planning subtasks are identified as straightforward, thereby reducing individual inference times. EcoAct (Zhang et al.] addresses different source of delay by optimizing prompt length through the selective omission of tool descrip- tions irrelevant to the current step. Interactive Speculative Planning (ISP) Hua et al.) accelerates execution by parallelizing action generation and verification, transforming the inherently autoregres- sive planning pipeline to non-autoregressive. However, critical gap exists in the literature: these approaches either compromise performance guarantees to achieve acceleration, require extensive pre-deployment preparation, incur prohibitive computational costs without providing mechanisms to control latency trade-offs. Our work bridges this gap by introducing a lossless self-optimizing, zero-setup framework based on speculative execution that can adaptively balance latency reduction against operational costs, providing the first comprehensive solution for deploying accelerated LLM agents in diverse real-world contexts with varying efficiency requirements.
Speculative execution Kocher et al. (2020); Xiao et al. (2020) originated in computer architecture to reduce latency by executing instructions before confirmation, with roll back mechanisms for incorrect speculations. Speculative decoding [Zhang et al. (2024a): Leviathan et al. (2023) adapts this principle to LLMs: an efficient draft model predicts multiple future tokens, which a more capable while slow target model verifies in parallel. Tokens that pass verification are committed to the output, while rejected tokens are regenerated, ensuring lossless performance. To improve efficiency, one line of work aligns draft models with target models through online distillation methods Liu et al. (2024b): Zhou et al. (2024); :
Ouyang et al. (0024): Gui et al. (2024) or training a dependent head [Cai et al. (2024) Li al. (2024a). Another line of work adapts dynamic draft lengths strategies such as Liu et al. (2025): Li et al. (2024b); Mamou et al. (2024). ISP extends this paradigm to agent planning, where an approximation agent (A) generates tentative planning steps while a stronger target agent (T) verifies them asynchronously. While ISP's plug-and-play approach preserves planning performance, its fixed speculation step leads either excessive costs or insufficient acceleration. Our work addresses this limitation by developing a dynamic speculation system that automatically determines optimal steps through online reinforcement learning, significantly reducing costs while preserving speed. Additionally, we introduce coer-controllable mechanisms that expose the latency- cost continuum, enabling precise system calibration based on operational priorities, a capability absent in all previous speculative execution approaches.
3 WHY IS SPECULATIVE PLANNING EXPENSIVE?
Speculative planning extends the principle of speculative execution to multi-step sequential decision making in LLM-based agents. This approach employs a dual-agent architecture with a "draft-and- verify" paradigm to accelerate planning while preserving solution quality. In this framework, computationally efficient A rapidly generates a sequence of candidate actions, while simultaneously the more capable T (the original planning agent) utilizes the prefix trajectory from the A to simulta- neously generate its own next actions as verification of the proposals of A. This parallel processing architecture enables multiple verification steps proceed concurrently rather than sequentially. When
3

---

Published as a conference paper at ICLR 2026
Pla Plas
Plac
Approximation
Flrsh
Pls Dia
Cla Pan
Approximation
Pas
Tugel
Torpet
Du
Time
diglls Claud
Long total task time
Shoet total tak time
(a) speculative step too small (k: 2).
(b) spoculative step too large (k 6).
Figure Limitations of fix speculative step approaches. We denote the number of speculative steps k. small & limits acceleration although it avoids waste, while large k increases speedup the cost of redundant computation when mismatches occur (step 5 & 6 is wasted in (b)).
100000
368000
Normal Plan
Normal Plan
SpecPlan (K-2)
SpecPPan (k-
60000
SpeePlan
Token Counnt Task
121000
Spec (k-4)
60000
SpecPlan (k-6)
091600
SprePPan (k-6)
60000
88000
2000
10000
21000
A-P
A-G
T-P
T-G Totul-PToti-G Total
A.P
A-G
T-P
T-G Totnl-PTotal-G Total
Token Cost Type
Token Cost Type
(a) Direct-ReAct
(b) CoT-MAD
Figure 3: Token cost breakdown under speculative planning for different fixed values on OpenAGI benchmark. shown for two distinct A-T configurations: (a) Direct (A) with ReAct (T), and (b) CoT (A) with Multi-Agent Debate (T). Each bar contains four segments: A prompt (A-P) and generation (A-G) tokens, and T's prompt (T-P) and generation (T-G) tokens. These components are also aggregated into total prompt (Fotal-P)and generation tokens (Total-G), and overall total tokens (Total). In both subfigures, higher values of k result in substantially greater total token costs.
T confirms alignment with A's proposals, those actions are immediately committed to the final plan. This verification process proceeds in parallel threads, significantly reducing the end-to-end latency compared traditional autoregressive planning. Whenever divergence occurs, where T proposes a different action than the A, the system adopts T's altermative and continues planning from this corrected trajectory. This mechanism ensures that the final execution path always conforms T's policy while achieve substantial acceleration.
While speculative execution provides acceleration in a lossless manner, it inevitably introduces additional token consumption. As shown in Figure 2b this overhead becomes particularly significant when (the number of speculative steps) large, motivating the need for more adaptive planning approach. In this section, we aim to understand where the major overheads arise by systematically breaking down the costs of speculative planning under different fixed-k settings. Redundant costs arise when token consumption does not contribute to the final generated plan or acceleration, which can be divided into four main components: redundant (1) the prompt tokens and (2) the generation tokens consumed by A, and (3) the prompt tokens and (4) the generation tokens consumed by Figure 3
Speculative Planning Cost Broken-Down.
provides a detailed breakdown of token consumption for each component, comparing normal planning against fixed-k speculative planning across different ks with 2 speculative planning configurations.
Approximation agent incurs excessive wasted token generation when the speculative step A exceeds the number of correct steps the agent can generate sequentially. When a mismatch occurs before reaching k. all subsequent generated steps become wasted. This effect intensifies with large-k, where frequent divergence at steps smaller than & leads to more discarded tokens. As shown in Figure 3 A waste primarily stems from prompt tokens rather than generation tokens. This occurs because prompts
4

---

Published as a conference paper at ICLR 2026
contain lengthy environment descriptions, task details, and action histories, while generations relatively brief, consisting only of actions and possibly some reasoning thoughts.
Target agent also incurs excessive wasted token from mismatches between A and T. These mis- matches introduce two additional sources of redundant costs: (1) Completed T steps: When mismatches occur, all subsequent completed target steps become invalid, wasting their prompt and generation tokens entirely. (2) Ongoing target steps: All subsequent in in-progress target steps after a mismatch are immediately canceled. These steps may have already initiated LLM calls, consumed prompt tokens, and generated partial outputs before forced cancellation. This type of waste grows proportionally with & because speculative planning concurrently launches A and T each step. Larger & naturally lead to more target steps running ahead of validation, amplifying potential waste caused by each mismatch.
Variation of Optimal k An optimal l is defined as the number of speculative steps from the current state up to the first mismatch, achieving maximal speedup without generating any invalid tokens. The analysis of 312 tasks from the OpenAGI benchmark reveals substantial variability in the optimal speculation step & even within the same task. For each task, we computed the maximum and minimum optimal & across different execution trajectories, well as the variance and the difference (max-min) of these values: on average across tasks, the maximum optimal & was 3.53 and the minimum was 1.60, with mean variance of 1.46 and average range of 1.93. The observed intra-task variability in optimal k: suggests that a fixed setting is inadequate for diverse scenarios.
Implications for Dynamic Speculative Planning The cost analysis and heterogeneity of optimal reveal an inherent tension in speculation step selection: the optimal I varies significantly across different planning stages within single task and thereby small limits acceleration by under-utilizing available concurrency, while large k frequently trigger excessive token generation which increase costs. Due to the heterogeneous difficulty of step proposals within tasks, no single optimal k exists throughout the plan. Therefore by adaptively modulating & based on contextual factors, such as task characteristics, step complexity, and prediction confidence, dynamic approach should strategically balance acceleration and efficiency.
Beyond mere adaptation to task characteristics, a dynamic approach also creates an opportunity for aserrcontrolled efficiency trade-offs. As model pricing structures and organizational priorities evolve, the optimal balance between latency reduction and cost efficiency varies considerably across deployment contexts. Some applications may prioritize near-inttantaneous responses regardless of cost (i.e., critical decision support) or operate in environments where prompt tokens are incxpensive, thus minimizing the financial impact of additional token generation. Others may emphasize cost efficiency with moderate acceleration (i.e., batch processing systems) or operate under constraints where excessive generation tokens or high-priced prompt tokens significantly impact operational bud- gets. By exposing this latency-cost continuum through configurable parameters, dynamic speculative planning empowers practitioners precisely calibrate the system.
4 DYNAMIC SPECULATIVE PLANNING WITH USER CONTROLLABILITY
Dynamic Speculative Planning begins with a simple intuition: once has produced a few actions, often possible to forecast how many additional actions it will generate correctly before T must intervene. Rather than selecting & heuristically or relying on an expensive offline model, we learn to predict it online while the system is running. When mismatch happens, the system logs a training pair (partial trajectory, k) for training small DistilBERT-basee regressor whose training cost can be negligible compared with inference cost of a single LLM call. To maintain system responsiveness, the predictor training is done asynchronously: for every newly collected datapoints, an independent thread fne-tunes the predictor to convergence and then preserves the updated checkpoint. The orchestration layer always utilizes the most recent checkpoint, so learning progresses without ever blocking agent plamning.
Compared to offline training, our online, multi-threaded training regimen offers two key advantages: (1) Real-time adaptation: the predictor is continuously updated with the most recent user interactions, keeping it aligned with the current usage distribution. (2) Prompt cost savings: the system begins reducing expenses as soon as new data arrive, without waiting to curate a large offline dataset, which is expensive and inconvenient.
5

---

Published as a conference paper at ICLR 2026
4.1 SPECULATIVE- CONTROL AS VALUE PREDICTIONS
As mentioned above, dynamic speculative planning requires effectively predicting the l for each (partial) trajectory and updating the predictor online for continuous adaptation. These requirements make online reinforcement learning a suitable framework. We formalize speculative step prediction as a state-value prediction problem in reinforcement learning and apply Temporal-Different (TD) learning (Sutton & Barto 2018) to online update this state-value function.
Formally, let ((X) be the space of all probability distributions supported over the set X. Consider a Markov decision process (MDP), M = A,P, 7), where the state space S is a set of token sequences, the action space A a set of speculative steps, P S x A + (SS) is the transition function, the initial state distribution Pu is a probability distribution over the prompt dataset Q (i.e., Po E (Q)), R: S x + R is the reward function, and y E [o, 1] is the discount factor.
An approximation agent A interacts with the MDP based a policy S + ((4). Specifically, for each episode, the agent starts from a prompt so ~ Pa(-). At each time-step t, it observes the state 8, S, takes action (i.e., speculative step) a (T-(ss), transitions to new state 81+11 and receives a scalar reward 7:41 = R(s),a)). The episode ends when an approximation-tane -targer mismatch is mel. In our case, 1 is simply concatenation of the token sequences of 8t and ot, i.e., $t+1 8y (0t. We also set 7 = 1. After an episode ends, we get a complete trajectory and it formally defined as T (so, ag, 11, : where T is the final time-step. Define return G2 over T as the total (discounted) reward from time-step t:
G: G-= E
iut.
state-value function is defined as the expected return under policy क
V=(s) Ex [G, s].
The reward function is designed that the return G, corresponds to the number of speculative steps at time-step f. Specifically, if the speculative step (generated by A) matches the one generated by T, we set R(st, ax) = 1; otherwise, A gets zero reward and the episode ends immediately. Under this reward setting, the state-value V (s) is essentially the expected number of speculative steps of state s.
In practice, Vx is usually approximated by neural network Ve. where 0 denotes the network weights. To learn a good approximation of VG, we apply TD learning and minimize the following loss function:
Lg = E (G; - Vo(*)))
where G. is known as X-retumn Sutton & Barto 2018 Formally, is defined as
T
G =1 A)
X" Σ
Grata + T GG,
n=l
where Gertn = Tr+1 + + " Tro + +-"
(e(t4)) and X E [0, 1]. The usage of X-return offers a flexible way to achieve better biri-variance tradeoff by interpolating between Monte Carlo return G, (when X~ = 1) and 1-step TD target r++1 + (V (when X 0). worth noting that when x= = 1 and y = 1. TD learning in this context reduces to online supervised learning since G G k. thus our learning framework a more generalized framework and achieves superior performance compared to purely supervised training (detailed in Section A.8)
Prediction Decoupled Data Collection. To ensure unbiased data collection, DSP logs pairs (st, GA)) only when A actually fails, rather than when the speculative budget is exhausted. Consider an under-speculattio scenario in which A can generate k consecutive valid actions, but the predictor predicts Ki (K2 < ki) actions. If training data are collected only for ka steps, would underestimate and gradually bias the predictor toward conservative strategies. To eliminate this bias, DSP records training samples at the actual failure point (k1). This ensures that prediction errors affect only system cost and efficiency without biasing the training data, thereby eliminating the need to counteract systematic underestimation.

---

Published as a conference paper at ICLR 2026
State- Value Estimation Formulation. DSP frames the speculation problem value prediction, rather than policy optimization. In our case, both agents (A and T) are fixed, where A controls the policy and T is part of the environment. Moreover, the inherent nature of speculative planning guarantees that the prediction of k values does not alter the distribution of generated trajectories. Thus, DSP curates data incrementally only estimate the expected returns under the fixed policy, avoiding the stability issues caused by policy optimization.
4.2 THLTI-THREAD ONLINE LEARNING
To minimize inference latency and update overhead, our system adopts a fully asynchronous learning framework that decouples predictor inference and training from the core exccution pipeline: (I) asynchronous inference for parallel execution: To mitigate the impact predictor latency on system throughput, we perform speculative step prediction asynchronously at the start of each episode. The predictor runs concurrently with A's planning step, and typically completes before A 's current step finishes. This parallelism effectively hides prediction latency. (2) asynchronous training for continuous adaptation: To adapt to evolving task distributions, the predictor is updated online a decoupled inffenne architecture. We maintain two models: a training predictor that undergoes continuous updates, and an inference predictor that serves all runtime inference requests. Upon completion of each speculative episode, an asynchronous thread is launched independently of the core planning pipeline to train the predictor. Once the training finishes, updated weights are immediately synchronized to the in-memory inference predictor.
4.3 CONTROLLING THE COST-LATENCY TRADE-OFF
In speculative planning, the balance between cost and latency intrinsically tied to the predicted step k. Thus, by adjusting the estimation bias of the step k, we allow users to control the trade-off between cost and latency according to their specific needs. There are in general two approaches for adjusting the speculative step k: one implemented during predictor training through modified loss functions, and another applied at inference time through post-prediction adjustment parameters.
Biased A Prediction. One method to flexibly modulate the estimation bias of step values is to implement expectile regression (Kostrikov et al., 2022) during training. This approach applies asymmetric penalties that systematically shift the predicted values in a desired direction, allowing for principled control over the bias-variance tradeoff in step prediction. Essentially, expectile regression generalizes ordinary least square (OLS) regression by replacing (symmetric) mean squared error (MSE) with an asymmetric MSE: E(z,))= [L5 (y- - fx))] where L (z) = |T 1(u < 0)
(
0)/u² lu2 and
T E (0, 1) determines which expectile to estimate. The loss reduces to MSE when = 0.5.
In DSP, we apply expectile regression to train the step predictor Vo(s). by simply replacing MSE in Equation (1 with the asymmetric MSE: Lg = EE>z< [ (G? - ).)] By tuning the value of r, we can flexibly balance cost and latency in dynamic speculative planning. When > 0.5, negative prediction error (G²- (s < 0) incurs a lower penalty, causing (o(s) to overestimate GA,, which leads to faster execution but higher cost. Conversely, when < 0.5. Vo(s) tends to underestimate G², resulting in lower cost but increased execution latency.
l with Biased Offset. Another way to flexibly control the bias is to directly add minus some offset to the unbiased predicted l without retraining the predictor. This method offers a straightforward and convenient mechanism for implementing user-controlled preferences in the latency-cost tradeoff. After the predictor generates its initial unbiased estimate ie, we apply a user-specified bias parameter E N to obtain the final step value: = max(1, k + 9). Positive values of ,3 shift predictions toward more aggressive speculation while negative values produce more conservative predictions. The maximum function ensures that the final step is at least 1, maintaining a valid speculation step.
5 MAIN EXPERIMENT
This section describes the experimental setup and evaluation results. We assess our approach on two benchmarks: OpenAGI (Ge et al. 2024) which focuses complex decision- decision-making tasks that require reasoning over both vision and language modalities including 312 multi-step tasks, and TravelPlanner (Xie et al. 2024) which targets travel planning scenarios with 180 tasks. We evaluate the performance
7

---

Published as a conference paper at ICLR 2026
Table 1: Dynamic k vs. Fixed &s Strategies on OpenAGI benchmark with GPT backbone.
Direct-Rect ReAct (Setting I)
CoT-MAD (Setting
Mode
T(x)
((x)
(x)
(x)
MC
K
T(x)
P (x)
G (()
Cost (x)
MC
K
Fix (k =
1.00
1.00
1.00
1.00
3.00
2.00
1.00
1.00
1.00
1.00
3.00
2.00
Fix
0.92
1.56
1.23
1.41
4.95
4.00
0.83
1.41
1.38
1.37
4.97
4.00
Fix (k 6)
0.90
2.24
1.57
1.92
6.32
6.00
0.81
1.87
1.75
1.77
6.44
6.00
Dyn. (r=0.5)
1.01
0.87
0,91
0.88
4.33
1.78
0,98
0.95
0.95
0.95
4.44
2.10
Dyn. (r=0.8)
0.97
0.97
0,98
0.97
4.41
2.33
0.89
1.04
1.04
1.04
4.73
2.67
Dyn. (r=0.9)
0.95
1.04
1.00
1.02
4.56
2.47
0.86
1.12
1.11
1.11
5.22
3.03
Dyn. (7=0,95)
0.94
1.12
1.07
1.09
2.88
0.85
1.26
1.25
1.24
5.64
3.58
Dyn. (7=0.99)
0.91
1.32
1.19
1.25
5.24
3.59
0.82
1,41
1.37
1.37
5.87
4.06
Dyn. (offset=1)
0.92
1,16
1.06
1.08
4.86
2.75
0.88
1.11
1.09
1.10
4.95
2.80
Dyn. (offset=22
0.90
1.33
1.15
1.25
5.13
3.47
0.80
1.29
1.27
1.27
5.60
3,99
of our dynamic speculative planning framework and compare it to set of baselines. We also provide comprehensive evaluation of the predictor's performance by evaluating both its prediction accuracy and warm-up speed (experiment details in Section A.6
Baselines are chosen to demonstrate the advantages of our approach across three multiple dimensions. (1) We compare against fixed-k speculative planning (k E (2, 4, 6)) to show the effectiveness of dynamic k selection over static approaches that ignore environment characteristics and task require- ments. We)We include non-contextual dynamic optimization baseline using Bayesian Optimization to demonstrate the necessity of our sophisticated neural predictor (detailed in Section A.7). (3) Additionally, we evaluate supervised fine-tuning approaches within our framework to highlight the advantages of our RL formulation (detailed in Section A.8)
The speculative step predictor uses DistilBERT for its efficiency and expressive capacity in forecast- ing the speculation step & from given state. The model is fine-tuned using expectile regression to learn the bias and adaptively adjust its predictions based on the collected data. The model is optimized using AdamW W with learning rate of 1 10-5, batch size 16. Each training process iterates over 3 epochs on a randomly sampled replay buffer (size 2,500). For multi-step return estimation, we adopt X-retum blending (X = 0.95) to balance blas-variance trade-offs during reward propagation.
LLM backbones for agents from the GPT and DeepSeek families are used in DSP. Below are their cost profiles: (1) GPT-4.1-min costs $0.40//11.6 (prompt/generation per million tokens). (2) DeepSeek: We use DeepSeek-chat as A 's backbone and DeepSeck-reasoner as T'ss backbone, with respective costs of $0.27/S1.10 and $0.55/$2.19.
Four DSP configurations are utilized for experiments; (1) directly for each step in single LLM call, without any intermediate reasoning. (2) Chain-of-Thouggh (CoT): A first reasons about the current step and then generates the action based on the reasoning, all within single LLM call (Wei et al. 2022). (3) ReAcr: T first deliberates on the action take, then generates the action through two distinct LLM calls (Yao al., 2023). (4) Multi-agent Debate (MAD): two target agents debate in two rounds, with each agent proposing and refining actions.
A generates the action
Evaluation Metrics are designed along three critical dimensions to assess speculative planning (SP) systems: latency reduction, resource overhead, and concurrency dynamics. This triad captures the trade-offss in real-world deployment, where effective systems must accelerate task completion while respecting resource constraints and API concurrency limits. For latency reduction and resource overhead, adopt fixed k 2 as the baseline configuration for evaluating various speculation strategies, as it represents the minimal non-trivial setting that offers meaningful acceleration with minimal resource cost: on the OpenAGI benchmark, the fixed 2 configuration typically provides 15%-25% time reduction compared to sequential planning. Efficiency and economy metrics are reported as ratios relative to the fixed 2 baseline. T(x). P(x). ). G(x), and Cost(x) (x) represent relative latency, prompt token usage, generation token usage, and monetary cost, respectively. MC denotes average peak concurrency (maximum parallel LLM calls), and K' is the average predicted step length per episode. (Detailed definition in Section A.2
MAIN EXPERIMENT RESULT 5.1
As shown in Table I Table 2 Section 5.I.I and Figure 7 our experiments validate that DSP achieves superior Pareto efficiency compared to fixed-k baselines. Key findings include:
8

---

Published as a conference paper at ICLR 2026
Table Dynamic i vs. Fixed Strategies on OpenAGI benchmark with DeepSeek backbone.
Direct-ReAct (Setting 3)
CoT-MAD (Setting 4)
Mode
T(x)
(())
(x)
Cost (x)
MC
K
T(x)
(())
G ((x)
Cost (x)
MC
K
Fix
1.00
1.00
1.00
1.00
3.00
2.00
1.00
1.00
1.00
1.00
3.00
2.00
Fix (k = 4)
0.89
1.58
1.50
1.55
4.92
4,00
0.89
1,51
1.47
1.49
4.94
4.00
Fix (k 6)
0.87
2.04
1.86
1,98
6.33
6.00
0.87
1,86
1.80
1.83
6.59
6.00
Dyn. (7=0.5)
1.04
0.93
0.94
1.69
1.07
0.94
0.94
0.94
1.54
Dyn. (1=0.8)
0.98
1.02
1.02
1.02
4.20
2.08
0.97
1.08
1.08
1.08
4.57
2.31
Dyn. (r=0.9)
0.95
1.12
1.12
1.12
4,45
2.58
0.93
1.16
1.15
1.16
4,69
2.65
Dyn. (7=0.95)
0.92
1.17
1.19
1,18
4.78
2.83
0.90
1.24
1.22
1.23
5.12
3.12
Dyn. (r=0.99)
0.88
1.42
1.40
141
5.46
3.85
0.88
1.39
1.36
1,38
5.29
3.75
Dyn. (offset=1)
0.93
1.11
1.12
1.1
4.39
2.58
0.91
L.15
1.15
1.15
4.48
2.80
Dyn. (offsct=2)
0.87
1.35
1.34
1.34
4.99
3.53
0.88
1,36
1.33
1.35
5.38
3.85
Direct-ReAct (Setting I)
CoT-MAD (Setting
Mode
T(x)
(())
(x)
Cost (x)
MC
K
(())
(
G G
Cost (x)
MC
K
Fix 2)
1.00
1.00
1.00
1.00
3.00
2.00
1.00
1.00
1.00
1.00
3.00
2.00
Fix (k 4)
0.89
1.95
1.39
1.95
5.00
4.00
0.96
1.48
1.22
1.46
4.62
4.00
Fix
0.87
2.86
1.67
2.91
6.99
6.00
0.97
1.67
1.25
1.64
5.37
6.00
Dyn. (r=0.5)
1.05
0.81
0.92
0.80
4.28
1.46
1.05
0.85
0.88
0.84
4.09
1.30
Dyn. (r=0.8)
0,99
0.95
0.97
0.93
4,60
1.88
1.05
0.86
0.89
0.85
4,00
1.42
Dyn. (7=0.9)
0.95
1.06
1.03
1.04
2.21
1.01
0.96
1.79
Dyn. (r=0.95)
0.93
1.20
1.09
1.18
5.19
2.55
0.98
1.02
0.97
1.01
4,00
2.10
Dyn. (r=0.99)
0,92
1.49
1.20
1.48
5.32
3.10
0.96
1,25
1.13
1.24
4.58
2.94
Dyn. (offset=1)
1.03
0.96
0.97
0.94
2.30
0.99
1.05
1.01
1.04
2.16
Dyn. (offsct=2)
0.99
1.16
1.04
1.12
4.91
3.22
0.98
1.33
1.16
1.32
4.33
2.98
5.1.1 COST-EFFICIENCY DOMINANCE
DSP demonstrates strict Pareto dominance through two operational modes:
(1) Equivalent Latency, Lower Cost. On the OpenAGI benchmark:
In setting 1, Dyn. (r 0.99) and Dyn. (offset 2) achieves comparable time reduction Fix (k- 6) with 34.9% (1.25x compared with 1.92x in Table reduction cost. Dyn. (offset 1) saved 23.4 % total cost compared to Fix (k 4) with comparable latency reduction. Compared to Fix = 2), Dyn. (r 0.5) achieves a similar latency reduction and substantial 12% decrease in token cost.
setting 2: Dyn. (r 0.95) achieves comparable latency reduction Fix (k 4) while saving 9.49% cost. Dyn. (offset 2) yields a marginally better acceleration than Fix (k 6),
and significantly reduces token by 28.25%.
setting 3: Dyn. (r 0.99) and Dyn. (offset = 2) saves 6. 6.78% and 10.67 cost compared to Fix (k 4) with marginal acceleration improvement.
In setting 4: Dyn. (T 0.99) saves 7.38% cost compared Fix (k 4) with comparable latency reduction. On the Travel Planner benchmark: Dyn. (7 0.8) achieves a 7% reduction in cost in setting 1, along with modest latency improvements compared Fix (A: 2).
(2) Equivalent Cost, Faster Execution. On the OpenAGI benchmark:
In setting 1, Dyn. (T 0.9) delivers 5% relative acceleration over Fix (k 2), only consuming 2% more cost.
In setting 2, Dyn. (T = 0.5) provides 8% better latency reduction while consuming 5% less cost than Fix (k: 2).
setting 4, Dyn. (offset = 2) requires 9.52% fewer cost to surpass Fix (I: = 4).
DSP consistently outperforms all fixed-k baselines in terms of cost-effective acceleration in all the settings, demonstrating its adaptability and generality. We provide a comprehensive analysis of unnecessary cost reduction and time acceleration relative to sequential planning in Section which

---

Published as a conference paper at ICLR 2026
200
Dynamic Dyusstic othet
Dyyaniic Dysanii ofhec
Geneattinn Tokmn Incruse
Proump fecre
Better Performance
Better Performanee
More Saved)
Tolens, Mare Saved)
2ff
ofbeiii
(
14
16
28
13
Time Decreased (%6
Time Decreased (1)
(a) Prompt Token Increase vs Time Decrease
(b) Generation Token Increase vs Time Decrease
Figure Performance trade-off between acceleration and token increase between Fix-k, Dynamic-T and Dynamic-offiet SP. (a) Prompt token and (b) Generation token increase vs. latency reduction, both relative normal-plan baseline. Detailed metric definitions and corresponding performance analysis are provided in Section A.2 and Section A.3
also demonstrates DSP's ability to effectively identify and exploit parallelism opportunities in variety of reasoning pipelines. also provide wall clock time analysis in Section A.12 demonstrating that DSP quickly stabilizes near the optimal latency-cost region and achieves Pareto-frontic fast.
CONCLUSION
We presented Dynamic Speculative Planning (DSP), method that addresses fixed-step speculative ex- ccution limitations in LLM-based agents through lightweight online reinforcement learning requiring no pre-deployment preparation. Our evaluation demonstrates DSP reduces redundant computational overhead by up 60% and overall cost by 30% while maintaining acceleration benefits, with user- controllable parameters enabling precise calibration of latency-cost tradeoffs. This advancement improves the viability of deploying sophisticated agents in latency- senssiysee real-world applications.
REPRODUCIBILITY STATEMENT
We provide our implementation at https://github com/ wwt./gunnyllnnn222222 Dynamic Speculative Pl Planning. git The repository includes detailed README file with step-by-step instructions to reproduce our experiments. All model hyperparameters and training settings are clearly stated in the experimental setup section (see Section 5)
8 ETHICS STATEMENT
This work does not involve human subjects, sensitive personal data, or information that could raise privacy concerns. The datasets and benchmarks we use are publicly available, and our methods do not introduce discrimination unfair bias. We believe our research poses no foreseeable ethical risks and adheres to the ICLR Code of Ethics.
REFERENCES
Leng Cai, Junxuan He, Yikai Li, Junjie Liang, Yuanping Lin, Ziming Quan, Yawen Zeng, and Jin Xu. Rthagent: A llm-based agent system for real-time bidding. In Companion Proceedings of the ACM on Web Conference 2025, pp. 104-113, 2025.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024. URL //ppp/r:veee//ei pett/forum:21d=Peebbbs
10

---

Published as a conference paper at ICLR 2026
Konstantina Christakopoulou, Shibl Mourad, and Maja Mataric. Agents thinking fast and slow: A talker-reasoner architecture. In NeurlPS 2024 Workshop on Open World Agents.
Morad Elfleet and Mathieu Chollet. Investigating the impact of multimodal feedback on user- perceived latency and immersion with llm-powered embodied conversational agents in virtual reality. In Proceedings of the 24th ACM International Conference on Intelligent Virtual Agents, pp. 1-9, 2024.
Jonathan BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454459, 2003.
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When IIm meets domain experts. Advances in Neural Information Processing Systems, 36, 2024.
Lujun Gui, Bin Xiao, Lei Su, and Weipeng Chen. Boosting lossless speculative decoding via feature sampling and partial alignment distillation. arXiv preprini arXiv:2408. 15562, 2024.
Wenyue Hua and Yongfeng Zhang. System I system 2= better world: Neural-symbolic chain of logic reasoning. In Findings of the Association for Compulational Linguisties: EMNLP 2022, pp. 601-612, 2022.
Wenyue Hua, Mengting Wan, JAGANNATH SHASHANK SUBRAMANYA SAI VADREVU, Ryan Nadel, Yongfeng Zhang, and Chi Wang. Interactive speculative planning: Enhance agent efficiency through co-design of system and user interface. In The Thirteenth International Conference on Learning Representations.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Peter Kiefer, Ioannis Giannopoulos, Vasileios Athanasios Anagnostopoulos, Johannes Schöning. and Martin Raubal. Controllability matters: The user experience of adaptive maps. Geolsformatica, 21:619 641, 2017.
Paul Kocher, Jann Horn, Anders Fogh, Daniel Genkin, Daniel Gruss, Werner Haas, Mike Hamburg, Moritz Lipp. Stefan Mangard, Thomas Prescher, et al. Spectre attacks: Exploiting speculative execution. Communications of the ACM, 63(7):93-101, 2020.
Ilya Kostrikoy, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yaoru Li, Shunyu Liu, Tongya Zheng, and Mingli Song. Parallelized planning-acting for efficient Ilm-based multi-agent systems. arXiv preprint arXiv:2503.03500 2025.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024a. URL
net/ forum? id= INdN7eXyb4]
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 conference empirical methods in natural language processing, pp. 7421-7432, 2024b.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In seirth- Conference on Neural Information Processing Systems.
Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. Lim -mm-pwwered hierarchical language agent for real-time human-ai coordination. In Proceedings of the 23rd International Conference Autonomous Agents and Multiagent Systems, pp. 1229-1228. 2024a.
11

---

Published as a conference paper at ICLR 2026
Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. Pearl: Parallel speculative decoding with adaptive draft length. In The Thirteenth International Conference on Learning Representations, 2025.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, lon Stoica, and Hao Zhang. Online speculative decoding. In Forty-first International Conference on Machine Learning, 2024b. URL httpt:ttttpppeeeeeeesss
net. forum? Piterummid=BPPP 1d=
Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Accelcrating speculative decoding using dynamic speculation length. CoRR, 2024.
Horia Alexandru Modran, Ioana Corina Bogdan, Doru Ursutiu, Cornel Samoilă, and Paul Livius Modran. Llm intelligent agent tutoring higher education courses using a rag approach. In International Conference Interactive Collaborative Learning, pp. 589-55999999 1899599..00 Springer, 2024.
Siru Ouyang, Shuohang Wang, Minhao Jiang, Ming Zhong, Donghan Yu, Jiawei Han, and Yelong
Shen. Temperature-centric investigation of speculative decoding with knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13125-13137, 2024.
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Victor Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. In Second Conference on Language Modeling, 2025. URL httppts:///peeeeeieeeeee wwtprreeiwwwwweeeeeeeeeeeeeeeeee www.eeoer/m.rrr..em
YaW07>XPXU
Ludovica Piro, Tommaso Bianchi, Luca Alessandrelli, Andrea Chizzola, Daniela Casiraghi, Susanna Sancassani, and Nicola Gatti. Mylearningtalk: An Ilm lim-based intelligent tutoring system. In International Conference on Web Engineering, pp. 428 431. Springer, 2024.
Swarnadeep Saha, Archiki Prasad, Justin Chen, Peter Hase, Elias Stengel- and Mohit Bansal. System 1.x: Learning to balance fast and slow planning with language models. In The Thirteenth International Conference on Learning Representaionss, 2025. URL https:: net/ 1d= XXxxBAA
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances Neural Information Processing Systems, 36, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, second edition, 2018
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang. Xiaoyun Zhang. Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen L.LM applications via multi-agent conversation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URL /opeeeees
Yuan Xiao, Yinqian Zhang, and Radu Teodorescu. Speechminer: A framework for investigating and measuring speculative execution vulnerabilities. In Network and Distributed System Security Symposium, 2020.
Jian Xie, Kai Zhang, Jiangjic Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference Machine Learning, 2024. URL (RI.https: / ew. net/
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan
Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023, URL /tlttttss////peeeeeee httpss///ppeeeeeiie foet/forum?
id=wE_ vluYUI--X
12

---

Published as a conference paper at ICLR 2026
Chen Zhang, Zhuorui Liu, and Dawei Song. Beyond the speculative game: A survey of speculative execution in large language models. arXiv preprint arXiv:2404.14897 2024a.
Jieyu Zhang, Ranjay Krishna, Ahmed Hassan Awadallah, and Chi Wang. Ecoassistant: Using LLM assistants more affordably and accurately. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024b. UtLhttps: openrevieeeeee ew net/ Lo
Shaokun Zhang, Jieyu Zhang, Dujian Ding, Jiale Liu, Mirian Del Carmen Hipolito Garcia, Ankur Mallick, Daniel Madrigal, Menglin Xia, Victor Rühle, Qingyun Wu, et al. Ecoact: Economic agent determines when to register what action. In Workshop on Reasoning and Planning for Large Language Models.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, San- jiv Kumar, Jean-Frat Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Representations, 2024. URL httt:t:t/pppeereeeeeeee net wwt.eeeeeeee.ooro..ooerrroo717..
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
LLM USAGE
Large Language Models were used this work solely for grammatical correction and language editing purposes.
A.2 EVALUATION METRICS
1. Time Decrease (AT) Quantifies the relative latency reduction when using speculative planning. Specifically, for each task, we calculate the total time taken to complete the task under SP (2"5)) and sequential planning (1"))). The percentage time saving is then calculated as:
N
7:SP
ATime
Σ
100%
N
2=1
where N is the total number of tasks. Positive values indicate time savings.
2. Cost and Token Increase Token increase captures the redundant resources consumed by speculative planning as result of incorrect speculation. While the primary goal of speculative planning is to reduce latency and enhance efficiency, it still incurs a minimum cost. Even with perfect accuracy in speculative planning (i.e., the approximation agent makes correct predictions for every step), the cost must at least include the baseline cost - the minimal overhead associated with sequential planning of both the target and approximation agents. Thus, we define the baseline for token increase as the sum of the planning costs for the target agent and approximation agent operating sequentially, which represents the least amount of resources required for completing the task with speculation. Any increase in token usage beyond this baseline reflects the additional overhead introduced by speculative failures, capturing the waste generated from incorrect predictions by the approximation agent.
Since the cost per token can differ for prompt tokens and generation tokens, we separately evaluate these components and also provide combined metrie for total cost increase.
Prompt Token Increase (AP) This measures the increase in prompt tokens due to redundant prompt usage in SP. Let p; = pt + Pr approx be the baseline prompt tokens (the sum of target and approximation prompts under sequential execution), and o be the prompt tokens used under SP (parallel execution). The prompt token increase is calculated as:
SP
1) x 100%
AP
P
Generation Token Increase (AG) This metric captures the increase in generation tokens caused by incorrect speculation. Let 9,¹h = 9,
sarget + g. g.sproo represent the baseline generation tokens (the sum of
target and approximation generations under sequential execution), and g²º² the generation tokens used under SP. The generation token increase is calculated as:
AG
1 x 100%
N
Total Cost Increase (ACost) This metric quantifies the relative increase in cost between speculative planning (SP) and sequential planning (seq) strategies. Specifically, we define the total cost increase as:
N
PCSP + GCSP PO + GC
-1) -1) 100%
ACost
(5)
N
where PC and PC represent the prompt costs of task i for SP and sequential strategies, and GCSP and c represent the corresponding generation costs. These costs are computed follows:
14

---

Published as a conference paper at ICLR 2026
PCP = ap, costap + tp; costp
(6)
PC, =
op, s24 costap + tp, costp
(7)
GO = ag) SP
costag + Ig. SP costi
(8)
GC = og; sey costag + Lg; 204
costig
Here, ap;" SP and tp, represent the number of prompt tokens for the approximation and target in the SP strategy task i, and ap; : and ty; represent the number of prompt tokens for the approximation and target in the sequential planning task i. Similarly, ag;", tg,"" ag, seg and tg, the corresponding number of generation tokens. For cost, costap and costp are the costs for the approximation and target API for prompt tokens, while costag and costy the corresponding costs for generation tokens.
0oq represent
By computing the total cost increase, can quantify the trade-off between speculative planning and sequential planning in terms of the associated computational costs, helping to assess the efficiency of the proposed approach.
3. Absolute Time Ratio (T(x)) This metric quantifies the ratio of the absolute time in the current configuration compared the baseline configuration (fix 2). as it represents the minimal non-trivial setting that offers meaningful acceleration with minimal resource cost. It is calculated as:
Teurrent
T(x)
Tk=2
4. Absolute Cost and Token Ratio cost in the current configuration compared to the baseline configuration (fix 2).
This metric measures the ratio of the absolute token usage and
Absolute Prompt Token Ratio (P(x))
current
P(X)
p
k=2
Absolute Generation Token Ratio (G(x))
Current
G(X)
9,
Absolute Cost Ratio (Cost(x) (x))
Ccurrent
Cost(x)
CK==
5. Concurrency Dynamics Evaluating the concurrency behavior of speculative planning (SP) essential for understanding its real-world deployability. Modern systems often impose limits on simultancous API requests due to infrastructure constraints or cost-efficiency considerations. Excessive concurrency can lead to API throttling, degraded performance, or instability, making critical to monitor and control parallel request patterns. To quantify this behavior, we introduce two metries.
Average Max Concurrent API Calls (MC) For each task i, MC; denotes the peak number of overlapping API requests during speculative execution. MC calculated as:
1 N
MC
MC:
(10)
where N is the number of tasks. This metric reflects how aggressively a system exploits parallelism and whether it respects concurrency constraints under varying speculative policies.
15

---

Published as a conference paper at ICLR 2026
Table 3: Performance Comparison of Dynamic vs. Fixed & Speculative Planning on OpenAGI benchmark with GPT backbone.
Direct-ReACT (Setting
CoT-MAD (Setting
Mode
AT (%)
AP (%)
AG (5)
ACost (%)
MC
K
(T())
AP (9)
AG (%)
ACost (%)
MC
K
Fix (k=2)
18.00
39.47
20.71
33.94
3.00
2.00
22.35
31.57
21.78
29.81
3.00
2.00
(km4)
25.06
114.94
47.88
4.95
35.62
83.78
74.46
4.97
Fix (k=6)
25.88
212.93
90.41
176.25
6.32
6.00
37.21
145.42
122.01
134,38
6.44
6.00
Dyn. (r=0.5)
15.68
16.40
8.23
14.04
4.33
1.78
23.12
21.20
17.83
19.63
4.44
2.10
Dyn. (r=0.8)
19.43
30,98
17.05
26.96
4.41
2.33
30.09
33.51
29.41
31.61
4.73
2.67
Dyn. (r=0.9)
39.83
4.56
5.22
Dyn. (r=0.95)
22.31
51.36
27.00
44.22
4.75
2,88
34.24
62.51
56.
59.53
5.64
3.58
Dyn. (r=0.99)
24.85
79.74
42.36
68.71
5.24
3.59
36.86
82.76
73.14
78.26
5.87
4.06
Dyn. (offset=1)
49.84
26.86
4.86
31.62
4.95
(ollset=2)
25.26
82.55
37.88
69.36
5.13
3.47
37,09
66.52
58.98
62.99
5.60
3.99
Table 4: Performance Comparison of Dynamic à vs. Fixed l Speculative Planning on OpenAGI benchmark with DeepSeek backbone.
Direct-ReACT (Setting 3)
CoT-MAD (Setting
Mode
(%)
(S)
AG (5)
ACost (%)
MTC
π
(%)
(S%
(9)
ACod (%)
TCC
Fix (k=2)
34.25
28.23
3.00
36.17
30.19
3.00
2.00
Fix (km4)
29.19
109.41
90.92
95.15
4.92
4.00
34.22
99.61
86.99
90.01
4.94
4.00
Fix (k=6)
30.34
171.35
134.69
142.50
6.33
6.00
36.25
144.87
128.6
128.71
6.59
6.00
Dyn. (r=0.5)
21.90
3.78
(r=0.8)
21.33
32.28
27.51
29.28
4.20
2.08
27.48
41.30
35.03
37.84
4.57
2.31
Dyn. (r=0.9)
23.84
47.71
40.77
42.58
4.45
2.58
30,32
52.66
45.25
48.14
4.69
2.65
Dyn. (r=0.95)
48.18
60.91
Dyn. (r=0.99)
28.70
87.02
75.24
75.92
5,46
3.85
33.93
84.59
74.42
77.30
5.29
3.75
Dyn. (offset=1)
25.07
46.30
40.92
42.40
4.39
2.58
31,79
49.08
42,90
45.41
4.48
2.80
Dyn. (ollset=2)
29.58
78.31
68.64
69.90
4.99
3.53
34.24
76.23
66.90
69.92
5.38
3.85
Average Speculative Step (K) In addition to peak concurrency (MC), which captures brief spikes of load, aim to characterize the average concurrency of the planning process the planning process, as it correlates with sustained system pressure. However, the instantaneous concurrency varies continuously over time. Precisely computing the time-averaged number of concurrent API calls would require fine-grained logs of every call's start and end time, which is often impractical. To address this, we use the average speculative step K a proxy for the true average concurrency. Let M be the total number of planning episodes, and k; be the speculative steps used in episode i. We define
K
Σ KG
(11)
M
Because each target-agent computation is relatively long-running, the k speculative calls in episode i tend to overlap in time. In practice, this means the average concurrency during episode i is effectively k,. Averaging these values over all episodes therefore yields meaningful estimate of the sustained concurrency.
A.3 ACTUAL PERFORMANCE
Figure visualizes how different speculative planning configurations in Setting 1 of the OpenAGI benchmark impact prompt token increase, generation token increase, and the corresponding reduction in execution time.
OpenAGI benchmark: In setting 1, Dyn. (offset 2) achieves a time reduction comparable the fastest fixed configuration (Fix (k = 6)), while reducing redundant cost by 60.65 %. Under Setting Dyn. (T 0.5) achieves a 23. 12% latency reduction while incurring only 19.63% unnecessary cost cost-saving 34.15% redundant cost compared to the most cost-efficient fixed configuration (Fix, k 2), and delivering better speed. The Dyn. (offset= = configuration matches the time savings of the fastest fixed strategy (Fix, k 6), while reducing redundant cost by 53.13% %. Under setting 3, Dyn. (offset= 2) delivers time savings comparable to the fastest fixed configuration (Fix, 6), while reducing redundant cost by 50.95%. Under setting 4, Dyn. (offset 2) achieves 97.21 latency
16

---

Published as a conference paper at ICLR 2026
Table Performance Comparison of Dynamic Planner benchmark with GPT backbone.
Fixed & Speculative Planning on Travel
Direct-ReAT (Setting
CoT-MAD (Setting
Mode
AT (%)
AP (%)
AG (%)
ACost (%)
MC
K
(T(%)
AP (%)
AG (%)
ACost (%)
MC
K
Fix (k=2)
15.36
64.52
26.86
61.91
3.00
2,00
10.97
53.46
24.91
54.77
3.00
2.00
(km4)
23.48
215.76
73.93
205.98
130.46
51.66
128.93
4.62
Fix (k=6)
24.27
364.01
109.91
346.71
6.99
6.00
12.75
159.45
55.37
157.74
5.37
6.00
Dyn. (r=0.5)
11.47
31.96
15.28
30.74
4.28
1.46
5.61
27.46
8.24
27.02
4.09
1.30
Dyn. (r=0.8)
16.52
54.22
21.96
51.83
4.50
1.88
5.65
35.10
12.49
34.57
4,00
1.42
Dyn. (r=0.9)
72.84
29.59
22.06
Dyn. (r=0.95)
21.04
95.01
37.10
90.72
5.19
2.55
12.74
57.54
22.35
$6,65
4.00
2.10
Dyn. (r=0.99)
22.39
137.90
50.41
132.21
5.32
3.10
14.54
93.63
41.01
92.49
4.58
2.94
Dyn. (offset=1)
54.95
(ollset=2)
16.20
89.08
30.84
84.24
4.91
3.22
12.75
105.28
43.65
104.16
4.33
2.98
reduction relative to the fastest fixed baseline (Fix k: == 6). while concurrently reducing redundant cost by approximately 45.68%.
On Travel Planner benchmark: In setting 1, Dyn. (T = 0.8) achieves slightly faster time reduction than Fix (k = 2), while reducing redundant cost by 16.28%. Under setting Dyn. (7 0.99) achieves similar latency compared to both Fix (k = 4) and Fix (k = 2), while separately reducing redundant cost by 28.26% and 41.37% respectively.
Experiment results also demonstrate that DSP achieves significantly more efficient concurrency utilization compared to fixed & baselines. DSP achieves 40% lower sustained system pressure(K) in its fastest configurations (Dyn. (T = 0.99), Dyn. (offset= 2)), compared to the fastest fixed-k setting (Fix (k= 6)), while maintaining comparable latency reduction. This demonstrates that dynamic policies can minimize persistent system load without sacrificing acceleration performance. All dynamic variants maintain .CC below 6, outperforming the fastest Fix (k 6) configuration.
A.4 BIASED PREDICTION v.8. WITH BIASED OFFSET
Tunable trade-off control The hyperparameter provides fine-grained control over the latency- cost trade-off by enabling three operational modes, defining a smooth continuum of operating points, forming a Pareto frontier between latency and token cost. This flexibility allows practitioners to adjust system behavior dynamically to meet varying operational requirements and economic constraints.
High- Peeh-Peeformance Mode (r 0.99, 0.95): It provides compared to fastest setting (even exceeding its performance) while saving 30% total cost.
90% of the time savings
Balanced Mode (r 0.9, 7 0.8): It achieves over 80% of latency reduction, while on average halves total cost compared to the fastest setting.
Economy Mode (r = 0.5): maintains non-trivial acceleration while reducing cost overhead compared the most cost-efficient fixed setting.
All three modes are supported within unified DSP framework by simply tuning т, without altering the algorithm. This tunability allows DSP meet different application requirements. In DSP, provides a smooth continuum of operating points, effectively generating a Pareto curve of latency vs. token cost from which one can choose according preferences. As an interpretable hyperparameter, offers clear deployment guidance: higher values promote larger A predictions, while lower values encourage conservative steps, as detailed Section 4.3 As shown Table Table and Table 5 the average speculation step K increases with confirming its practical and theoretical role in governing speculative planning behavior.
Biased l Prediction v.s. A with Biased Offset We examines two distinct approaches for control- ling k: training-time bias adjustment through expectile regression and inference-tin direct offset application. Both methods enable practitioners navigate the latency-cost tradeoff, though with different optimization characteristics.
Applying fixed positive offset the unbiased predicted value of k offers computationally efficient and conceptually straightforward approach to balancing acceleration and cost efficiency. In OpenAGI benchmark, DSP(offset=)) achieves latency reduction comparable to DSP(T 0.95). Similarly,
17

---

Published as a conference paper at ICLR 2026
Table 6: Warm-up Phase Accuracy (First 13 Tasks).
(a) OpenAGI
Contg
Task
Task
Tack
Task
Tack
Task
Tisk
Direct Retttt-RAtttee (GPT-4 I I-minn)
0.42
0.67
0.67
0.75
0.6
]
1
0.8
0.5
0.83
DinectReAct (DeaySock)
0.7
CoT-MAD (GPT4. m-min)
0.5
0.5
0.67
0.25
0.38
0.75
0.5
0.8
0.57
0.6
CoI-MAD (DespSeek)
0
0
0
0.33
0.33
0.25
0.33
(b) TravelPlanner
Comlig
Task
Task 2
Task
Task 4
Task
Task 6
Task
Task 8
Task
Task
Task 11
Task
Task 13
(GPT441-mi))
COT-MAD (GPT4 I-mini)
0
0.62
0.29
0.43
0.83
0.57
0.71
0.71
0.57
DSP(offset=2) achieved 97% acceleration by the optimal fixed-k setting where k = 6, while saving up to 28% cost. These results demonstrate that simple post-prediction adjustment yields robust performance improvements across both latency and cost metrics.
However, the principal advantage of biased-k prediction implemented through training-ting time expectile regression lies in its capacity for more granular control over the efficiency-cos continuum. Ex- perimental results indicate that moderate training parameters profiles that cannot be precisely matched by integer offset adjustments. These intermediate settings enable more nuanced optimization configurations that achieve reasonable latency reduction with substantially lower operational costs than their fffsettbbsed counterparts (offset +1, +2).
0.8, 0.9) delivers acceleration
The choice between these approaches ultimately depends on the specific operational requirements and constraints. Training-time bias adjustment provides finer-grained control and potentially better optimization at intermediate points on the efficiency-cost spectrum, particularly valuable for applica- tions with precise performance requirements. Conversely, inference-time offset application offers implementation simplicity, immediate adjustability, and strong performance at discrete operating points, making it suitable for deployments where straightforward configuration and rapid adaptation are prioritized.
A.S ONLINE WARMUP
In our experiment, begin with randomly initialized DistilBERT model the initial stage of every setting and its predictions are naturally zero at the start. This results in the entire pipeline executing the target agent in sequential manner during warm-up tasks without any additional cost increase. To expedite the reduction of warmup time, we start collecting training episodes online as the pipeline runs. Once enough episodes are gathered meet the predefined batch size, the model undergoes its first training update.
Our analysis across different settings reveals that the warmup duration varies depending on the API backbone. As shown in Table when using (PPT-4.1-min) it typically requires around 3-5 tasks to warm up, as GPT-4.1-ming tends generate longer sequences to complete each task. It enables faster collection of sufficient number of training episodes. In contrast, when using DeepSeek-series models, the warmup phase is slower, requiring approximately 8-10 tasks to collect enough training data. This discrepancy highlights the differences in the response times of different APIs and the resulting impact on the warmup efficiency.
A.6 PREDICTOR ACCURACY OVER ONLINE TRAINING
To assess the effecttivnees and stability of our online learning procedure, examine predictor accuracy on the OpenAGI benchmark across sequential task segments. Table summarizes the mean and standard deviation of accuracy within non-overlapping 50-task windows, and Figure visualizes the global trend with a smoothed trajectory.
In the early stage (tasks 1-100), the predictor exhibits moderate accuracy and relatively high vari- ance, reflecting the initial adaptation to unseen task distributions. As training progresses, accuracy consistently improves across all settings, accompanied by steady reduction in variance. By the later
18

---

Published as a conference paper at ICLR 2026
Table 7: Predictor Accuracy per 50-Task Segment on the OpenAGI benchmark (Mean + SD).
Task
Direct-ReAct (GPT)
Direct-ReAct (DeepSeek)
CoT-MAD
CoT-MAD (DeepSeek)
1-50
0.600+0.331
0,465:10.117
0,341
0,459:10.334
51-100
0.477:10.363
0.499
0.693=0.349
0.501=0.3446
101-150
0.689:0.256
0.701=0.352
0.675=0.285
0.564::0.334
131-200
0.69340. 189
0.393:10.326
0,575=0.217
0.570=+0.329
201-250
0.755:10.207
0.657=10.280
0.621
0.704:10239
251-300
0.7244:.169
0.762:10.207
0.693
0.674::02337
301-312
0.261::0200
0.707=+0.201
0.760=0.130
0.698
Predictor Accurucy Trend (Direct- ReAct with GPT)
Predictor Accuracy Trend (Direct- ReAct with DeepSeek)
Predictor Accuracy
1.00
Predictor Accuracy
WW Agawar
0.25
0.75
Fint use
0.50
8.58
0.24
0.15
10-Task Maving Average
0.00
90
100
150
200
250
300
0.00
50
100
1:50
200
250
300
Task Index
Task Inclxx
(a) Direct-ReAAce (GPT-4.1-mmm 1-mini)
(b) Direct-ReAct (DeepSeek)
Predictor Accuracy Trend (CoT-MAD with GPT)
Predictor Accuracy Trend MAD with DeepSeek)
Predictor Accurscy
1.00
Predictor Accuracy
1.00
Maving Average
0.75
0.75
Firitz 650.e @
2.50
vn
0.50
Mlp
10-Taak Moving Ansrag
Task
Indlx
(c) CoT-MAD (GPT-4.1-mini)
(d) CoT-MAD (DeepSeek)
Figure 5: Predictor accuracy trends online training across different settings. In all cases
(Direct-ReAe and CoT-MAD with GPT-4.1-mini or DeepSeek), the predictor exhibits clear upward trajectory in accuracy alongside decreasing variance, indicating improved generalization unseen tasks and stable convergence without late-stage drift.
segments (tasks 200-300), the predictor achieves substantially higher mean accuracy (0.70-0.76) while maintaining much lower variance, indicating convergence and stability.
These results demonstrate that online updates reliably enhance the predictor's generalization ability to unseen tasks. Importantly, we observe no evidence of performance drift instability in later stages. Instead, the predictor becomes increasingly accurate and stable, ensuring dependable speculative planning across diverse benchmarks.
A.7 EVALUATION ON NON-CONTEXTUAL DYNAMIC HEURISTICS
To further demonstrate the advantage of our Dynamic Speculative Planning (DSP), we implemented a non-contextual reward-driven black-box optimization heuristic using Bayesian Optimization (BO). Unlike DSP, this method does not exploit contextual state features, but instead attempts learn mapping from past speculative outcomes to effective k-values. Specifically, BO maintains surrogate model (Random Forest Regressor) to predict the expected reward for different k-values based on past selection outcomes. The reward is defined as:
R(A)
k
where As denotes the ground-truth speculative depth. This formulation incentivizes the selection of values closer to the true horizon. For the e-Greedy acquisition strategy (6 0.1), which maintains efficiency during online updates.
tradeoff, we adopted lightweight
19

---

Published as a conference paper at ICLR 2026
Table Performance Bayesian Optimization and DSP on OpenAGI (First 200 Tasks).
Derect-ReAct (GPT-4.1-mini)
Direct-ReAct (DeepSeek)
CoT-MAD (GPT-4.1-min))
Method
Time
Prompt
Gen
Cost
Time
Prompl
Gen
Cust
Time
Prompi
Gen
Cost
Fixed-k
2
1.00
1.00
1.00
1.00
1.00
1.00
1,00
1.00
1.00
1.00
1.00
1.00
4
0.90
1.47
1.21
1,35
0.85
1.49
1.44
1,46
0.81
1.35
1.34
1.32
& 6
0.88
2.00
1.47
1.74
0.84
1.85
1.76
1.82
0.77
1.69
1.63
1.62
DSP
0.5
0.99
0.94
0.96
0.94
1.02
0.95
0.96
0.96
0.92
1.03
1.02
1.03
0.8
0.94
1.06
1.03
1.04
0.98
1.02
1.04
1.02
0.84
1.10
1.10
1.10
0.9
0.93
1.14
1.06
1.10
0.92
1.09
1.08
1.09
0.82
1.15
1.14
1.14
0.95
0.91
1.22
1.13
1.17
0.91
1.20
1.22
0.81
1.30
1.29
1.28
0.99
0.90
1.37
1.21
1.29
0.87
1.42
1.41
1.41
0.81
1.39
1.37
1.36
offset=1
0.91
1,15
1.08
1.11
0.93
1.12
1.14
1.13
0.84
1.15
1.13
1.13
offset=2
0.90
1.35
1.17
1.27
0.85
1.30
1.31
1.30
0.76
1.33
1,30
1.30
Heuristic
BO
1.03
0.98
0.98
0.97
1.09
0.98
0.94
0.95
1.10
1.03
0.95
1.00
As shown in Table g Direct-ReAct setting with GPT backbone, BO achieves performance comparable to the fixed = 2 baseline, but fails realize the dynamic acceleration potential of DSP. Compared to our DSP variant with r = 0.5, BO yields a similar latency reduction yet incurs 10% higher total cost. Conversely, when compared with DSP but delivers significantly less acceleration. Direct-ReAct setting with DeepSeek backbone and CoT-MAD setting with GPT backbone only contains first 250 tasks. In these settings, BO consistently lags behind DSP variants as it delivers substantially worse accelcration.
0.8, BO matches token cost
These results demonstrate that non-contextual methods like BO cannot effectively capture the nuanced state information from dual agents, which is critical for accurate speculation prediction. This limitation underscores the necessity of employing lightweight neural networks such as DistilBERT to encode contextual features and learn complex state-dependent patterns that simple black-box optimization approaches fail to exploit. DSP maintains superior Pareto frontier, achieving better balance between efficiency and resource usage through this context-aware prediction capability.
A.8 SUPERVISED VS. REINFORCEMENT LEARNING FOR SPECULATIVE STEP PREDICTION
We investigate two alternative formulations for speculative step prediction: supervised learning (SFT) and reinforcement learning (RL) with TD-based updates. While both are supported within framework, we ultimately adopt the RL formulation for two key reasons.
Theoretically, supervised learning arises special case of framework. When À and y the TD target reduces to the rollout length G; = k, thus TD-lcarning reduces to online supervised learning. Additionally, X-returns provide tunable bias-variance tradeoff between Monte Carlo and bootstrapped estimates, offering greater flexibility than pure regression. Overall, our learning framework is a more generalized framework.
Emprically, training with fully supervised targets (i.c., A = 1, 1) produced overly conservative predictors. As shown in Table 9 while the Dyn. (SFT) model achieves similar or even lower token cost savings as our TD-trained variant (Dyn. (7=0.5)), it consistently underperforms in terms of latency reduction. In most cases, the performance was even slower than fixed = baseline, which is the minimal non-trivial setting. Additionally, the Dyn. (SFT) model achieves lower accuracy across all settings compared to Dyn. (7=0.5) The experimental findings reveal the limitations of the supervised learning approach in this context. We attribute this to high variance in derloet k labels and the averaging tendency regression, which causes SFT models to underestimate l in uncertain regions.
20

---

Published as a conference paper at ICLR 2026
Table 9; Performance comparison of SFT vs. RL strategies on the OpenAGI benchmark.
Direct-ReAct
CoT-MAD
Backbone
Mode
T(x)
P(x)
G(x)
Cost(x)
T(x)
P(x)
G(x)
Cost(x) x)
Fix (k=2)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
GPT-4.1-mini
Dyn. (7=0.5)
1.01
0.87
0.91
0.88
0.98
0.95
0.95
0.95
Dyn. (SFT)
1.05
0.86
0.91
0.88
1.02
0.96
0.96
0.96
Fix (k=2)
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
DeepSeek
Dyn. (7=0.5)
1.04
0.93
0.94
0.93
1.07
0.94
0,94
0.94
Dyn. (SFT)
1.09
0.96
0.95
0.96
1.11
1.01
1.00
1.00
A.9 DSP COST BROKEN-DOWN
The cost breakdown data for Fix and Dynamic Speculative Planning across different asymmetry parameters and different offset is presented in Table [0] In both DSP settings, the cost structure maintains the same general pattern observed fixed-k Speculative Planning, with the majority of tokens being consumed by target agent operations in input prompt; but parameter effectively controls the aggressiveness of speculation, showing clear patterns in cost reduction:
At the most conservative setting where т 0.5, the DSP reduces the average redundant token consumption from +10,748.3 to +672.2 in setting I and from +20,424.0 to +8,508.7 redundant tokens in setting 2, respectively, comparing with the most conservative fixed-k setting where & = 2. This represents an around 71% reduction in redundant token consumption. At the most aggressive setting where 0.99, the DSP also reduces the average total token consumption from +52,664.6 to + +14,4522 4,266 setting and from +96,927.3 to +50,977.8 the most aggressive fixed-k setting where k 6. This also represents an around 50% reduction in redundant token consumption. The graduated scaling of costs with the t parameter provides practitioners with fine-grained control over the efficiency-cost balance, enabling precise system calibration that fixed-k approaches cannot match.
setting 2, respectively, comparing with
A. LIMITATIONS
While our Dynamic Speculative Planning framework demonstrates significant improvements in balancing acceleration and cost efficiency, several important limitations warrant ocknowledgmed and suggest directions for future research. First, our current implementation does not adaptively select or optimize the approximation agent itself. The speed, accuracy, and availability characteristies of this agent represent crucial factors in overall system performance. Different approximation agents may be better suited for specific tasks or planning stages, and dynamically selecting from a pool of candidate agents could further enhance both acceleration and cost-efficiency. Future work could explore meta- learning approaches that automatically select or even generate task-specific approximation agents based on observed performance patterns.
Second, our current rejection criteria for verifying speculation steps remain relatively strict, potentially limiting achievable acceleration. Agent planning often admits multiple valid solution paths to accomplish the same task, yet our verification mechanism may incorrectly reject valid alternative approaches proposed by the approximation agent when they differ syntactically from the target agent's preferred approach. This high false negative rate in the rejection criteria unnecessarily constrains parallelization opportunities. Future research could explore semantic verification mechanisms that recognize functionally equivalent actions despite surface-level differences, or could implement more sophisticated verification strategies that consider the downstream effects of actions rather than just their immediate representation.
Finally, evaluation focused primarily on two standard agent benchmarks; more extensive testing across diverse application domains would provide greater confidence the generalizability of our findings.
21

---

Published as a conference paper at ICLR 2026
Table 10: Speculative Planning cost broken-down on OpenAGl benchmark with GPT backbone.
(a) Direct- React (Setting 1)
Plan Type
Approx Prompt
Approx Generation
Target Prompt
Target Generation
Total Prompt
Total Generation
Total
Token
Normal Plan
6040.3
108.6
12715.6
1906.7
18755.9
2015.3
20771.2
Fix (k=2)
I0759.8
165.7
18162.7
2431.2
28922.5
2596.9
31519.44
(k=4)
19894.5
281.9
26610.1
2963.1
46504.6
3245.1
49749.7
Fix (k=6)
34124.0
369.2
35079.7
3862.9
69203.7
4232.1
73435.8
Dyn. (r=0.3)
6451.7
115.0
13022.8
T965.6
19480.5
2081.6
21443.44
Dyn. 008)
7454.8
130.9
14348.4
1994.6
21803.1
2125.5
23928.6
Dyn. =09)
8745.6
148.9
16062.6
2185.7
24808.2
2334.6
27142.8
Dyn. (r 0.95)
9226.2
154.7
16407.2
2176.9
25633.4
2331.6
27965.0
Dyn, (r 0.99)
12307.0
195.1
20140.0
2581.7
32447.0
2776.8
35223.8
Dyn. (offset=1])
9576.3
163.9
173814
2293.0
26957.7
2456.9
29414.7
Dyn. (offset =)
12999.8
206.1
20254.3
2457.7
33254.1
2663.8
35917.9
Redundant Cost
A Fix (k =
+4719.5
+57.1
+5447.1
+524.5
+10166.6
+581.6
+ +10748.3
Fix (k 4)
+13854.2
+173.3
+13894.4
+1056.5
+27748.7
+1229.8
+28978.5
(k 6)
+28083.7
+260.6
+22364.1
+1956.2
+50447.8
+2216.8
+52664.6
Dyn. (7
#417.4
+6.4
+307.2
+58.9
+724.6
+66.3
+672.2
A Dyn. 0.8)
+1414.5
+22.3
+1632.8
+87.9
+3047.2
+110.2
+3157.4
Dyn. (r (.9)
+2705.3
+40.3
+3347.0
+279.0
+6052.3
+319.3
+6371.6
A Dyn. (r (0933)5)
+3185.9
+46.1
+3691.6
+270.2
+6877.5
+316.3
+7193.8
A Dyn. (T (T
+6266.7
+86.5
+7424.4
+675.0
+13691.1
+761.5
+14452.6
À Dyn. (offset
+3536.0
+55.3
+4665.8
+386.3
+820I18
74416
+8863..5
A Dyn. (offsct +6959.5
+97.5
+7538.7
+551.0
+14498.2
+648.5
+15146.7
(b) CoT-MAD (Setting 2)
Plan Type
Approx Prompt
Approx Generation
Target Prompt
Target Generation
Total
Total Generation
Total
Prompt
Token
Normal Plan
5422.5
1701.3
35968.4
7332.8
41390.9
9034-0
50424.9
Fix (k=2)
"858,99
2435.4
50373.5
9881.11
$8532.4
12316.5
70848.9
Fix (k=4)
13799.2
3884.8
70283.3
13253.9
84082.5
17138.7
101221.2
Fix (k=6)
26065.3
5466.1
97668.0
18152.8
123733.3
23618.9
147352.2
Dyn. (T=0.3
6429.6
1946.9
42200.3
8356.7
48629.9
10303.7
58933.6
Dyn. =08)
7671.9
2330.3
48703.2
9577.7
56375.1
11908.0
68283.1
Dyn. (r 0.9)
8682.5
2590,9
52586.8
10255.2
61269.3
12846.1
74115.5
Dyn. (r = 0.95)
10417.5
3043.1
58605.2
11353.2
69022.7
14396.4
83419.1
Dyn. 0.99)
13857.5
3848.7
70281.1
13415.5
84138.5
17264.2
101402.7
Dyn. (offset =
8978.2
2699.9
54749.3
10715.0
63727.4
13414.9
77142.3
Dyn. (offset 2)
11256.8
3276.3
61864.3
11828.1
73121.1
15104.4
88225.6
Redundant Cost
A Fix =
+2736.4
+734.1
+14405.1
+2548.3
+17141.5
+3282.4
+20424.0
A Fix (k 4)
+8376.7
+2183.5
+34314.9
+5921.2
+42691.6
+8104.7
+50796.3
A Fix 6)
+201642.8
+3764.9
+61699.6
+10820.0
+82342.4
+14584.9
+96927.3
A Dyn. 0.5
+1007.1
+245.6
76231.9
+1023.9
+7239.0
+1269.7
+8508.7
A Dyn. 0.8)
+2249.4
+629.0
+12734.8
+2244.9
+14984.2
+2874.0
+17858.2
A Dyn. (7=
+3260.0
+889.6
+16618.4
+2922.4
+19878.4
+3812.1
+23690.6
A Dyn. 0.95)
+4995.0
+1341.8
+22636.8
+4020.4
+27631.8
+5362.4
+32994.2
A Dyn. (.99)
+8435.0
+2147.4
+34312.7
+6082.7
+42747.6
+8230.2
+50977.8
A Dyū. [offset=
F3355.7
+998.6
+18780.9
43382.2
+22336.5
+4380.9
426717.4
A Dyn. (offset= =
+5834.3
+1575.0
+25895.9
+4495.3
+31730.2
+607004
+37800.7
Despite these limitations, work represents significant step toward making accelerated LLM-
based agents more practical for real-world deployment by addressing the critical tension between latency reduction and cost efficiency while providing unprecedented user control system behavior.
22

---

Published as a conference paper at ICLR 2026
A.11 BROADER IMPACT
Dynamic Speculative Planning contributes to the broader effort of making Al agents more practical and accessible for real-time applications. By substantially reducing response latency while providing fine-grained control over operational costs, our approach can help the deployment of LLM--baed agents in time-senee amme-senitive contexts. For examples: educational applications could benefit from more responsive tutoring agents that provide immediate feedback while managing computational expenses within institutional constraints; healthcare systems might leverage faster diagnostic support agents without compromising accuracy or incurring prohibitive costs; customer service platforms could deploy assistants with response times approaching human conversation pace.
A.12
WALL CLOCK TIME ANALYSIS
Fisek
Dynamic
Dynamic
Pronupe Token Incnaaee (50
Dynanic offeet
Generatio Tokem Incrcsse
Dynamic offiet
Periomaacee
40
Pectormazce
60
(sheedd jobeet
30
40-
20
CHOs
20-
10
22
24
26
26
30
32
34
22
24
25
30
32
34
Time Decrensed (9)
Time Decrezsed (%6
(a) Prompt Token Increase Time Decrease
(b) Generation Token Increase vs Time Decrease
Figure Performance trade-off between acceleration and token increase between Fix-k; Dynamic-T and Dynamic- SP within first 50 tasks. (a) Prompt token and (b) Generation token increase vs. latency reduction, both relative to a normal-plan baseline.
Food k
Pixsed
Dynaticc
Guneration Token Increase (s)
60
Dynanio
Dynantic offer
Dynamic offict
Pronpt Token Incraase
50
Better Perforrence
40
Better Perfarmanee
30
60-
20
1--02"
20
22
26
26
20
24
26
30
32
36
Time Deuncssed (%))
Time Decrassed (%)
(a) Prompt Token Increase vs Time Decrease
(b) Generation Token Increase vs Time Decrease
Figure Performance trade-off between acceleration and token increase between Fix-k, Dynamic-T and Dynamic-ofiset SP within first 100 tasks. (a) Prompt token and (b) Generation token increase vs. latency reduction, both relative to a normal-plan baseline.
We provide a performance comparison between DSP and fixed-k baselines the first 50 and first 100 tasks.
Within the first 50 tasks, all DSP configurations except = 0.99 already reach the Pareto frontier, and within the first 100 tasks, all DSP variants lie on the frontier.
These experiments results demonstrate that DSP quickly stabilizes near the optimal latency-cost region, achieving Pareto-efficient behavior early training without adding wall-clock cost.
23


Reviews:
[{"review_id": "NZfFkMPM3n", "reviewer": "Reviewer_QcJ4", "summary": "The work proposes a method (DSP) where they use a target LLM and a smaller approximation LLM to accelerate multi-step agent planning through speculative execution. The approximation LLM speculatively generates multiple future planning steps ahead in parallel, while the target LLM simultaneously verifies each step. When the two agents agree, steps are committed to the final plan. When they disagree, the incorrect speculative steps are discarded and execution resumes from the target LLM's decision. The main contribution of this work is dynamically predicting how many steps ahead to speculate using an online-trained distillbert predictor based on TD learning, rather than using a fixed speculation depth. This prediction is adjusted using either expectile regression or a biased offset to let users control the latency-cost tradeoff. Results show they achieve comparable acceleration to aggressive fixed-depth approaches while reducing both total costs and unnecessary computational waste.", "questions": "- You claim applicability to software engineering  too, but there is no result on such a benchmark (like SWE bench), can you explain why? \n- Were experiments run with multiple seeds? \n- Why does DSP achieve 30 percent cost reduction on OpenAGI but only 7 percent on TravelPlanner? What task characteristics drive this variability?\n- System 1.x and EcoAct are cited as addressing similar latency challenges but not compared experimentally. Why were they excluded?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- DSP achieves Pareto dominance across multiple agent configurations and LLM families, delivering comparable acceleration to aggressive fixed-depth approaches while reducing costs.\n- The ablations are well done. \n- Dynamic speculation depth via online RL is seems reasonably novel compared to fixed-k prior work.", "weaknesses": "- Only OpenAGI and TravelPlanner are evaluated. They for instance mention the applicability of the work in software engineering but do not have any results on any such benchmark. \n- I am not sure if authors evaluate on multiple seeds or not, given the online RL part, I believe multiple seeds are extremely important to establish statistical relevance.\n- Authors do not discuss why TravelPlanner shows substantially weaker improvements than OpenAGI (7 percent cost reduction vs. 30 percent), some discussion around that will be good to have. \n- System 1.x (Saha et al. 2025) and EcoAct (Zhang et al. 2024b) are cited as addressing similar latency challenges but are not compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "394Hw4vFwm", "reviewer": "Reviewer_suqa", "summary": "This paper proposes a reinforcement learning–based framework to accelerate large language model (LLM) agents while maintaining their performance quality. Traditional acceleration techniques often require expensive pretraining or sacrifice accuracy; DSP instead introduces a lightweight, online learning mechanism that dynamically adjusts how far ahead a “draft” agent speculates before verification by a “target” agent. Using asynchronous temporal-difference learning, DSP continually refines its prediction of optimal speculation steps during live operation, requiring no pre-deployment setup. Moreover, it offers user-controllable tradeoffs between cost and latency through expectile regression or simple offset parameters. Experiments on OpenAGI and TravelPlanner benchmarks show that DSP maintains comparable or better acceleration to existing lossless methods while cutting total cost by up to 30% and eliminating 60% of redundant computation, making it an efficient and flexible solution for real-world LLM-based agents.", "questions": "1. How do you plan to measure the generalization gap, and is there any safety net for out-of-distribution state-action pairs at test time?\n\n2. Will setting gamma to 1 cause instability for RL training? Why not use a more theoretically sound number like .99?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper uses online reinforcement learning (TD learning) to adjust the speculation step k without pretraining, achieving lossless acceleration while maintaining performance.\n2. It provides user-controllable tradeoffs between speed and cost through expectile regression and offset parameters.\n3. The work demonstrates strong empirical results with up to 30% total cost reduction and 60% reduction in redundant cost.", "weaknesses": "1. The method’s technical novelty is incremental, as it mainly extends existing speculative planning with an adaptive step predictor.\n2. The online RL part may introduce instability or convergence issues in dynamic, real-world settings. More specifically, the real-world LLM applications could have countless new scenarios, and it can be hard to make sure that the RL training has covered enough state-action spaces.\n3. DSP’s performance depends on the quality of the initial predictor, which might require some warm-up time before optimal behavior emerges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0lilor1oRE", "reviewer": "Reviewer_4PyS", "summary": "The authors have created an online RL framework for accelerating LLM-based agents while also being able to lower operational costs. This framework focuses on a limitation of Interactive Speculative Planning. That of a hard-coded speculation depth, which tends to perform badly as the optimal depth can be very different between different tasks and planning stages. The authors have created a speculation step predictor as a state-value estimation problem which is solved with a TD(λ) style algorithm, alongside an asynchronous multi-threaded architecture. Expectile regression and direct offset then allow for fine calibration of the latency-cost tradeoff. Experiments on OpenAGI and TravelPlanner benchmarks show a 30% total cost lowering and up to 60% cost reduction in redundant overhead compared to fixed-depth baselines, with no pre-deployment preparation needed.", "questions": "In addition to the above, can you explain if there is a clear way to decide when to choose tau or beta?\nWhen the two agents disagree, how often are those rejections actually harmless?\nThe DSP relies on running multiple speculative threads in parallel. Have you tested what happens if concurrency is limited? This feels like a real-world constraint that is important to understand.\nHow much extra wall-clock time does DSP spend exploring the Pareto-frontier?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem is clearly well-motivated and highlights an important failing in current approaches. The analysis is rigorous and the demonstration of the sources of redundancy is clear. It's clear why fixed depth doesn't work in general.\nThe impact shown by the new algorithm also seems very impressive (though perhaps over a small number of tasks), and the fact that there is no pre-deployment preparation is clearly a win.\nThe multi-threaded architecture itself seems to be well-designed.\nThe baselines are appropriate (though on first reading I thought that there should have been comparison with System 1.x and ecoact)", "weaknesses": "The benchmarks themselves are appropriate but somewhat limited.  They are both domains where it makes sense for the algorithm to do well in, so it would be interesting to see how robust the improvements are in slightly out of distribution domains. For instance, could it be tested with a code-generation agent or an interactive tool-use QA, or indeed just something which is reasoning-heavy but not specifically planning-focused.\nIt's clearly hard in this domain, but the lack of any statistical analysis is a drawback. Could one apply bootstrapping to get some variance estimates?\nThere doesn't seem to be any testing for hyper parameter sensitivity for lambda, replay buffer size, batch size or learning rate.\nIt would be good to quantify the predictor overhead. It says that it's negligible, but it would be useful to know precisely how negligible.\nSystem 1.x and Ecoact are mentioned but it's not explained why these wouldn't be sensible benchmarks to compare against. I see that they are tackling different problems, but it would be useful to have this spelled out explicitly.\nPossibly the most important point, and it may be a matter of misunderstanding, but is about the amount of wall-clock time spent exploring the Pareto-frontier. Does that have an overall impact on actually compute time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xb1ayYGcaz", "reviewer": "Reviewer_J1uY", "summary": "This paper proposes a framework to accelerate large language model (LLM)-based agents by reducing inference latency and cost—two major bottlenecks in real-world deployment. The authors build on speculative execution (a technique where a lightweight model “guesses” future steps while a stronger model verifies them) and extend it to multi-step agent planning. Existing speculative planning methods use fixed speculation steps (k), which can either waste resources (if k is too high) or fail to accelerate enough (if k is too low). The method proposed here, DSP, introduces a dynamic, reinforcement-learning-based mechanism that adaptively chooses the optimal speculation step size during runtime. However, it still needs broader empirical validation, deeper real-world integration, and more nuanced cost modeling to fully realize its impact.", "questions": "1. The reward function sets R=1 for correct steps and 0 otherwise, which seems coarse. Have you explored more informative reward signals (e.g., scaled by prediction confidence or cost impact)?\n2. How do you ensure stability when the predictor is updated asynchronously while being used for inference?\n3. When the predicted step k is far from optimal, how do you mitigate oscillations or compounding inefficiency (especially early in training)?\n4. While DSP aims to reduce token cost, it adds overhead for training and maintaining the predictor. How significant are these infrastructure costs in large-scale deployments?\n5. The paper provides code, but could you report how sensitive the results are to λ, learning rate, and batch size?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of online reinforcement learning (Temporal Difference (TD) learning with λ-returns) to dynamically adjust speculative step size (k) is an elegant solution to a well-known inefficiency in speculative planning. The method’s real-time adaptability—learning directly during deployment rather than via costly offline training—is both practical and technically forward-looking.\n2. The proposed method DSP does not require router training, parallelization setup, or prompt engineering, making it directly applicable to many existing agent frameworks.", "weaknesses": "1. The evaluation is confined to OpenAGI and TravelPlanner, which are still benchmarks, not real-life systems. Generalization to highly dynamic or human-interactive domains (e.g., robotics) remains untested.\n2. The paper defines a simple reward function (reward = 1 per correct speculative step), which might oversimplify complex planning dynamics. Real-world task rewards often involve long-term dependencies or delayed outcomes, which may challenge the proposed TD formulation.\n3. The experiments assume fixed per-token costs, but API pricing, context-length, and throughput pricing in practice vary nonlinearly. The claim of “30% cost reduction” could fluctuate under real billing conditions.\n4. The baselines focus on fixed speculative methods. A stronger evaluation might compare DSP with distillation-based acceleration or dual process models (System-1.x) for a more comprehensive view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)