DreamPhase: Offline Imagination and Uncertainty-Guided Planning for Large-Language-Model Agents
Authors: Shayan Mohajer Hamidi
Linfeng Ye
Konstantinos N. Plataniotis
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Autonomous Agents
Primary area: foundation or frontier models, including LLMs

Abstract: Autonomous agents capable of perceiving complex environments, understanding instructions, and performing multi-step tasks hold transformative potential across domains such as robotics, scientific discovery, and web automation. While large language models (LLMs) provide a powerful foundation, they struggle with closed-loop decision-making due to static pretraining and limited temporal grounding. Prior approaches either rely on expensive, real-time environment interactions or brittle imitation policies, both with safety and efficiency trade-offs. We introduce DreamPhase, a modular framework that plans through offline imagination. A learned latent world model simulates multi-step futures in latent space; imagined branches are scored with an uncertainty-aware value and filtered by a safety gate. The best branch is distilled into a short natural-language reflection that conditions the next policy query, improving behavior without modifying the LLM. Crucially, DreamPhase attains its performance with substantially fewer real interactions: on WebShop, average API calls per episode drop from $\sim$40 with ARMAP-M (token-level search) to $<10$ with DreamPhase, a $4\times$ reduction that lowers latency and reduces executed irreversible actions by $\sim 5\times$ on WebShop (4.9$\times$ on ALFWorld) per incident logs. Across web, science, and embodied tasks, DreamPhase improves sample efficiency, safety, and cost over search-based and reward-based baselines. This offers a scalable path toward safe, high-performance autonomous agents via imagination-driven planning. Code: \url{https://anonymous.4open.science/r/DreamPhase-A8AD/README.md}.

Published as a conference paper at ICLR 2026
IMAGINATION AND
DREAMPHASE: OFFLINE
UNCERTAINTY-GUIDEE PLANNING FOR LARGE- LANGUAGE-MODEL AGENTS
Shayan Mohajer Hamidi', Linfeng Ye², Konstantinos N. Plataniotis 'ttanford University, UUnivesityy of Toronto
smooaajer@stanford edu 2 iinfengg.... ye@mail wetmail..ttoooreeele.eee.e. otontoo. ca; kostasßece www.:utt..oooooooommm
ABSTRACT
Autonomous agents capable of perceiving complex environments, understanding instructions, and performing multi-step tasks hold transformative potential across domains such as robotics, scientific discovery, and web automation. While large language models (LLMs) provide powerful foundation, they struggle with closed- loop decision makiing due to static pretraining and limited temporal grounding. Prior approaches either rely expensive, real-time environment interactions brittle imitation policies, both with safety and efficiency trade-offs. We intro- duce DREAMPHASE, modular framework that plans through offline imagina- tion. A learned latent world model simulates multi-step futures in latent space; imagined branches are scored with an incertainty-aaan value and filtered by a safety gate. The best branch is distilled into a short natural lorllllange language reflection that conditions the next policy query, improving behavior without modifying the LLM. Crucially, DreamPhase attains its performance with substantially fewer real interactions: on WebShop, average API calls per episode drop from ~40 with ARMAP-M (token-level search) to 10 with DreamPhase, 4x reduc- tion that lowers latency and reduces executed irreversible actions by ~ 5x on WebShop (4.9x on ALFWorld) per incident logs. Across web, science, and embod- ied tasks, DreamPhase improves sample efficiency, safety, and cost over search- based and reward-based baselines. This offers scalable path toward safe, high- performance autonomous agents via imagination planning. Code: htttps: //anonymous 4open. wwweecceeeeeeeeeeeeee /reee///////e///e/ hase ABAD-BAADR-- 3BAD/README md
INTRODUCTION
Building intelligent agents that can perceive complex environments, follow instructions, and au- tonomously complete tasks remains a core challenge in artificial intelligence. Such agents have the potential to transform wide range of applications, including virtual assistants, scientific discovery, workflow automation, and robotics (Brooks) 1986, Reed et al.) 2022 Kirchdorfer et al.) 2024 Rana al. 2023).
A promising foundation for building these agents lies in large-scale generative models, particularly large language models (LLMs), which have demonstrated remarkable capabilities in natural language understanding, question answering (Rajpurkar et al. 2016), summarization (Hermann et al. 2015). and multimodal reasoning (Chen et al. 2015 Goyal et 2017). However, while LLMs excel in static tasks, they remain limited in settings that require multi - step drcision-making and closed-loop interaction, such as online shopping, scientific experimentation, or puzzle solving. This limitation stems from the fact that most LLMss are pre-trained on internet-scale static data and lack exposure to temporally grounded trajectories or dynamic environments, which are essential for reasoning about actions and their long-term consequences (Zhou et al. 2024). Additionally, many state-of- the-art LLMs like GPT-4V (OpenAl et al., 2024) and Gemini (Reid et al. (2024) are accessible on task-specific interaction data, and inference-time prompting often proves brittle tasks involving uncertainty, feedback, or long-term planning.
only through restricted APIs. This severely limits their adaptability: they cannot be

---

Published as a conference paper at ICLR 2026
To address these challenges, two broad strategies have emerged, each with its own limitations.
(i) Online roll-out planners, such as ReAct with beam search or Monte Carlo tree search (MCTS) variants that interact with the live document object model (DOM) at each step (Yao et al. [022]] Hao
ct al. 2023). These methods explicitly evaluate multiple future branches by performing hundreds of real clicks. While this approach enables lookahead and can correct for earlier mistakes, it is often slow, costly in settings with rate-limited APIs, and potentially hazardous in environments where actions are irreversible, such as submitting payments placing orders.
(ii) Pure imitation or reward-model agents, on the other hand, avoid expensive interaction by acting greedily from the current state without explicit search (Hong al.. 2023 Liu al. 2023)
Although this reduces interaction overhead, it introduces brittleness: single misstep can irreversibly derail the entire trajectory, since the agent lacks any foresight or ability to revise its plan.
Both approaches ultimately trade off' between safety and efficiency due to their reliance on real-time interaction with the environment. promising alternative is internal imagination: the ability to simulate and evaluate future outcomes offline, without taking any real actions in the environment.
To realize this, we introduce DreamPhase, a modular framework that enables autonomous agents to plan through imagination in latent space. At its core, DreamPhase trains latent world model that predicts the next DOM tree or visual frame conditioned on the current latent state and a proposed action. By iteratively applying this model, the agent can simulate multiple future trajectories entirely offline, estimate their expected value and uncertainty, and prune high-risk branches before making any real requests. This approach yields three key advantages:
(i) Sample efficiency: Web-based tasks that previously required approximately 40 real clicks per episode now converge in fewer than 10 (see Section 5.2
(i) Safety: Imagination allows the agent to identify dead ends and avoid irreversible outcomes, such as "Sold-out" pages or accidental purchases, without executing them (see Appendix C
(ii) Cost and latency: All expensive reasoning is handled on-device; the only network call the final, high-confidence action that passes uncertainty filtering.
In summary, by relocating exploration from the real environment into a learned latent simulator, DreamPhase overcomes the safety-efficiency trade-off that constrains existing agent systems.
In particular, DreamPhase unfolds four stages: (i) we train a latent world model (LWM) (Ha & Schmidhuber 2018) to simulate future states from raw DOM trees or visual inputs; (id) we generate imagination rollouts by combining this latent dynamics model with actions sampled from a frozen policy LLM; (1i1) we perform uncertainty-awwe
value estimation over these imagined branches, scoring each using predicted rewards and entropy-based confidence metrics; and (iv) we distill the best low-risk trajectory into natural- Laanuuael reflection, which injected back into the LLM prompt to influence future decisions.
Crucially, the LLM policy remains entirely frozen throughout this process. Its behavior evolves solely through internal simulation and language- based feedback. This design supports scalable and sample-efficien planning without the need for real-world interaction or model fine-tuning. In summary, the main contributions of this paper are:
Imagination-fiin latent planning. We train a compact world model simulate multi-step futures in latent space and roll out M branches for H steps entirely offline, decoupling environment dynamics from policy reasoning and enabling safe counterfactual planning.
Risk-aware branch selection with guarantees. Imagined trajectories are scored by value minus an uncertainty penalty and accepted only if a confidence gate passes; we provide a regret bound (VTe) + BpT that links decision quality to model error and mis-gating.
Language reflections for zero-tuning control. The selected branch is distilled into short reflection and summary that are injected into the next prompt, steering a frozen LLM toward higher-quality actions while remaining interpretable and requiring no parameter updates.
Practical efficiency and robustness. DreamPhase reduces real interactions by about 4x on WebShop (< 10 vs. 40 API calls/episode), adds ~ 12 ms imagination overhead, and cuts executed
2

---

Published as a conference paper at ICLR 2026
inreversible actions by 5x on WebShop (4.9x ALFWorld). It degrades gracefully under distribution shift via fallback and outperforms search- and reward-based baselines across domains.
2 RELATED WORK
LLMs as interactive agents. LLMs have been used control policies dynamic environments.
from web navigation to general tool use (Liu et al. 2023] Zhou et al. 2023). Harly systems adapted pretrained encoders
maaiiinn- (e.g.. BERT for WehShop (Deliin et al. 2019 Yao
2023a)) With stronger generative models (Brown et al. 2020; OpenAl et al. (024), zero-shol and few-shot prompting became common for action selection without fine-tuning (Deng et al. 2024 Xiong et al. 2024), often pairing observations and histories with an LLM to choose the next step (Zheng et al. 2024 Hong et al. 2023). Another line distills trajectories from powerful but restricted APIs into smaller policies (Li al.) 2023 Zhang al. 2023). In contrast, DREAMPHASE keeps the policy I.IM frozen and adds a learned latent world model, an uncertainty- value head, and language reflections steer behavior while minimizing real interactions.
AgentLM Zeng et al. (2024) introduces agent funing, supervised and preference-style fine-tuning on multi-turn tool trajectories, to improve grounding and robustness, but it requires curated data and changes model weights; DREAMPIIASE avoids fine-tuning by conditioning a frozen LLM with reflections distilled from imagined rollouts. AgentGym Xi et al. (2025) standardizes evaluation with unified task wrappers, tools, and metrics; work follows its emphasis on comparable protocols but contributes a new planning method. ARMAP Chen et al. (2025) scales teken-level search with Reflexion, Best-of-N -N, and MCTS (ARMAP-R/B/M)) improving success via online expansion at the cost of higher latency and many real-environment calls. We show that DREAMPHASE exceeds ARMAP performance while using substantially fewer real interactions through offline latent imagination and uncertainty gating. For a broader survey, see Appendix B
AgentLM, AgentGym, and ARMAP.
3 METHODOLOGY
Setting and notation. We study decision making in partially observable interactive environments, for example web navigation with sparse feedback and costly interactions. The task is modeled as partially observable Markov decision process
M AX
where I is the space of task instructions, S the latent state space, discrete action space, X' the observation space, T S x A + S the transition kernel, E S + the emission kernel, and a possibly sparse reward. At time the agent receives an instruction E I, observes
r: S x A I; E X' selects a, E A. the environment moves 81+1 ~ T(st, az), and emits 2++1 ~ (( The objective is to maximize
T
o r(8, a.)
(TT) =
Σ
under policy πt Taa æzgr, 1),
where r(st, a;) is the reward at time f, E (0, 1] is a fixed discount factor, and n" exponentially down- weights rewards farther in the future, and the expectation is taken under the trajectory distribution induced by the policy क (a, ()),)).
3.1 DREAMPHASE OVERVIEW
DreamPhase plans with offline imagination to lower interaction cost and improve safety. At each step, the agent: (i) forms a compact predictive belief with learned latent world model, (17) rolls out multiple hypothetical futures in latent space, (Hii scores each rollout with value estimate and an uncertainty measure, (iv) distills the highest quality imagined outcomes into a short natural language reflection that conditions the next action. All steps (i) to (iv) occur without querying the environment.
3

---

Published as a conference paper at ICLR 2026
3.2 LEARN TO DREAM: TRAINING THE LATENT WORLD MODEL
The first component of DreamPhase is a learned latent world model that enables internal simulation of environment dynamics. The goal is to predict what will happen when the agent applies an action in the current observation context. For example, the model can answer: "What occurs if click this button on the current page." This capability supports counterfactual reasoning before acting, without touching the real environment. Concretely, we train latent world model that predicts short-hu short-horizon futures in a compact latent space conditioned on the current observation, the action, and the instruction This eliminates environment queries during planning and follows prior latent-space imagination work (Ha & Schmidhuber 2018 Hafner et al. Janner et al. (019).
Data collection. We collect multi-step interaction episodes for tasks such as website navigation or shopping. Each episode provides tuples (t, 2ts ae; Bf4+),, where Wr 2z++11 X' are observations and a, E A is the action taken at time t. The history is h2 = (4, L5, act).
We train the latent world model exclusively on logged trajectories collected from the standard training split of each benchmark environment. To obtain these trajectories, we run a frozen LLaMA-2-77 2-BB policy with a simple ReAct-style prompt and light randomization. No test episodes, privileged futures, or environment states unavailable to baselines are used. All observations and actions come from the same public APIs that baseline agents interact with. The resulting dataset contains only (instruction, observation, action, next-observation) tuples from the training portion of each task and does not include any extra external corpora or fine-tuning data.
In addition, note that all the basclines in Table are allowed to their standard training protocols (e.g., AgentLMM and AgentGym use their released agent-tuned checkpoints; ARMAP uses its reward- model training). DreamPhase does not access any interaction data beyond the logged trajectories from the training split. Thus, all open-source agents share the same policy backbone, the same environment splits, and access to the same interaction data; DreamPhase only differs in how this data is used to train a compact world model and value head for offline imagination.
Observation encoding. Observations such as DOM trees or screenshots are high dimensional and structured. We tokenize each 2; by a depth-fire traversal of the DOM with textual content and spatial layout features, for example bounding boxes and element types. This yields compact, aligned sequence suitable for generative modeling and for conditioning the policy.
World model architecture. The model has three parts: an encoder fø(;;) that maps an observation to latent 2,, transition model go(2), at, that predicts the next latent given the current latent and an embedded action a,, and a decoder (z(+1)1, that reconstructs the next observation. The stochastic latent dynamics are
2: = fo(x:),
Z41 g0(z++ 21, ac, 1),
E++1 ~ (0 2141,
(2)
where a; = =eb (a:) is learned action embedding.
Training objective. We combine token reconstruction with latent space regularization. Let 7z+1 dg (go(fe( ac 1). The loss is
LwWM E CE B(+1)
AKL KL h4, at) (V)(
(3)
where KL and CE denote Kullback-Leibler divergence and cross-entropy function, respectively, and qo( 70(2++1 he, a:) is the encoder-induced inference distribution (diagonal Gaussian with parameters from fo) over the next latent during training: we sample via reparameterizztion and use the mean at test time, and AKT. is annealed during training.
Offline imagination. After training, the model simulates futures without environment queries. Given 2; and at, we encode to 2, = fo(x,), sample øt+1 from the learned transition (e 21, azs i), and decode L141 do(z(+1,1). Iterating this procedure for short horizon yields imagined trajectories 7 = (ă,+1:t и: a.. 1)1,, which are later scored for value and uncertainty and summarized into reflections.
4

---

Published as a conference paper at ICLR 2026
Remark (Why a latent world model rather than LLM-bbaee dreaming). Large language models are not designed to simulate low-level environment dynamics such as DOM structure. They often produce syntactically invalid or causally inconsistent states, and token-level rollouts memory intensive and conflate policy reasoning with environment modeling. A learned latent world model modular and efficient, it generates structured futures that respect environment constraints, and supports risk-aware planning in long-horizon tasks. For these reasons DreamPhase relies on latent world model for internal simulation rather than prompting an LLM to generate future states.
3.3 IMAGINATION-BASE PLANNING
At each timestep the agent explores multiple plausible futures by simulating latent trajectories conditioned on potential actions. This allows the agent to reason about consequences without interacting with the real environment. We refer to this mechanism imagination bassaa based planning.
Given the current observation Zt, we encode it with the encoder fo to obtain &, = fxz)) To explore alternative outcomes, generate M parallel rollouts in latent space, each simulating sequence of H future steps using the learned world model (Jo= 90, dg) and a frozen policy LLM #LLM that proposes actions from the current imagined history. Each rollout corresponds a hypothetical branch that contains predicted latents and actions.
Each trajectory +(5) is one imag- ined branch, consisting of an action sequence and the result- ing predicted latents, with op- tional decoded observations æ for inspection or scoring. These imagined futures are passed to the downstream evaluation mod- ule, where they are scored by a value model and filtered by un- certainty metrics. This process supports multi-step foresight
Algorithm I: Latent Imagination Rollouts. Input: instruction 4; observation æt; encoder fo: transition go: decoder do: frozen policy LLLMi number of branches M; horizon H fo(x;)
I: Encode observation:
2: for to M do
3: Initialize latent: x
4: Initialize imagined history: h (1,
5:
for 0 to H do
Sample imagined action: a ~ MLLM( ((P
7:
Embed action: ((i) <- e
Predict next latent: 214k+1 go (Z)4441 (1) =(1)
(0)
8:
in latent space without issu- ing environment queries, which makes planning efficient and scalable. We detail the rollout
9:
Decode to imagined observation: a06
#44+1 <- do(=iteens
10:
Update imagined history: 1 + (h a pm
(()) 200
11:
end for
12: Store imagined trajectory: z6) (fa (f (= (=(4)e+))
procedure in Algorithm
13:
end for
3.4 EVALUATE AND DECIDE: UNCERTAINTY-AWAR VALUE-GUIDED BRANCI SELECTION
After generating M latent rollouts of horizon H, the agent must assess whether any imagined trajectories are reliable enough to guide action selection. This balances efficiency (acting from internal simulation) and accuracy (deferring to the real environment when predictions uncertain).
Value estimation. For each imagined trajectory 7()) discounted return with a lightweight value head V operating on latents:
() { a
we estimate
GG)
k- V 1)),
7 (0,
The head V& is trained on logged data to predict short-horizon utility from latent states produced by the world model.
Uncertainty estimation. We quantify epistemic uncertainty over imagined actions using mutual- information proxy computed with Monte-Carlo dropout on the frozen policy TLLM. For each imagined history prefix 1.6)) (instruction i, current æ;,, and all imagined pairs far), let 2/ A be the discrete
5

---

Published as a conference paper at ICLR 2026
action set scored by the policy. With stochastic dropout masks {5n 1 In=
N (HND p()) = N Σ (s.):
N
ult = H H[pb) N
(5)
where (S.) is the action distribution TLLM(- m .x) over U, and H[-) is categorical entropy. Larger () indicates greater disagreement across stochastic policy samples, hence higher uncertainty.
Risk-sensitive selection.
We combine value and uncertainty via penalized score
( G - Bu
B 2 0,
(6)
and choose the best branch
jr = arg max GG)
A safety gate enforces that the selected branch is itself confident:
"if u (()))) < then act using a (j") otherwise query the real environment and update the history."
This mechanism enables DreamPhase to plan through imagination when predictions are confident, and to defer to real interaction when uncertainty is high, supporting robust decision making under distribution shift while reducing environment queries.
3.5 REFLECT AND ACT: LANGUAGE-GUING POLICY CONDITIONING
Rather than fine-tuning the policy LLM, we steer its behavior using natural language derived from imagination. Inspired by verbal self-reflection (Shinn al. 2024 Yuan al.) 20244). PreamPhase converts its imagined rollouts into concise guidance that conditions the next action.
Reflection. Given the set of imagined trajectories (+()) 1z=1 and their scores (GU), u()) from
Section 3.4 let j be the selected branch. We generate a short reflection that explains why 7(²") is promising and surfaces potential risks. A lightweight reflection head RRp produces
=0") -6") C Rell, Dt, 2441144s Q G(i"), u())),
(8)
where x S H a short summary horizon. R. can be a small frozen LLM or a distilled model conditioned on the branch and its value-uncertainty signals. Example output: "Proceed with search, then filter by size before adding to cart; avoid pages without a visible 'Checkout' button." "
Trajectory summarization. In addition, we produce compact natural-language summary of the core actions and expected outcomes. A summarizer S,, maps the same inputs terse script,
St Sy,(t, Bt, Fi4iuta d(:tx-1);
(5")
-07)
for example: "Search 'Nike shoes'; open first result: click 'Add cart'. " Both e and s; can optionally aggregate over the top-K safe branches by concatenating key constraints, although the default uses only
Language-based steering. We inject the reflection and summary into the policy prompt the next decision point. The frozen policy LLM receives enriched context and produces the next action:
T ( Rt, Ct2 8t).
(10)
This language-guided conditioning integrates simulated experience in modular and interpretable way. The agent improves behavior without parameter updates, using internal planning signals expressed as short, actionable text that is easy to inspect and ablate. Implementation details for the reflection head RR and summarizer S,, the 30-token budget, the delimiter scheme, and taec-ipe templates are provided in Appendix[]

---

Published as a conference paper at ICLR 2026
3.6 DREAMPHASE PIPELINE
Algorithm 2 summarizes the full DreamPhase pipeline. At each timestep, the agent encodes the current observation into latent state and imagines M future trajectories using the learned world model. These rollouts are scored by discounted return estimates and filtered by uncertainty proxy. the best rollout is sufficiently confident, it summarized and reflected upon in natural language. which is then appended the policy LLM prompt for action selection. Otherwise, the agent queries the policy using the real history only. This design enables offline reasoning, defers to real interaction when uncertain, and steers decisions with internal reflection, without updating the LLM.
Algorithm 2 DREAMPHASE:: Imagination-Basse Planning with Uncertainty Reflection
Input: instruction e; initial observation æg; history Agg + (t, æo); encoder fa. transition ge, decoder dæ; policy TLLM; value head Vs: reflection head R.p: summarizer S,; number of imagined branches M; horizon H; discount y; risk weight B; confidence threshold т, summary horizon k.
1: for t = 0, 1........ do
2: Encode current observation: 2, 2 =
3: Sample M imagined rollouts 1
M using the world model and TLLM as in Algorithm Score each imagined branch
4: for j 1... M do
5:
Estimate value: G + E V
(=
6: Estimate uncertainty u(() via the mutual-informatte proxy from Section 3.4
7: Penalized score: (6) <- G()) ,3 (z)
8:
end for
9:
Select best branch: j arg maxje (M) (6)
if v(²) S T then
Safety gate passes
Generate reflection: C + R,pl1, 22, ((0) (0')
latte aut-- (.k),,,),,,,, GG*) u60)
,GG)))))))
11:
12:
Summarize trajectory: 8t + S, Fe, 21t 11+1: tu
a(1) axt(x-1)
13:
Cr, Augment policy context: Pt. < CONCAT Cr, 81)
14:
Query policy with guidance: a₂ TLLM( Pt)
15:
else
b Fallback to real-history policy
16:
0, T. 144)
17:
end
18:
Execute a, in the environment, observe 1
19: Update history: he+1 + (he, ars 1z+1)
20:
end for
4 THEORETICAL REMARK
Let x* be the optimal policy in the true environment M. The latent world model induces one- step predictive distribution Po(#241 lt: a) and the environment induces P. (x++1 hr, ar, c). Assume a uniform one-step KL bound E = suph,,, KL(P, (P))). which by Pinsker incquality Csiszár & Körner 2011) implies TV S ve/2. where TV denotes the total variation distance.Let be the confidence threshold of the safety gate (Sec. 3.4). and define the mis-gating rate p = Pr [DreamPhase selects an imagined branch while the true uncertainty r].
With bounded rewards Tmax and a value function that is Lipschitz in total variation of the one-step predictor, the cumulative regret over T steps satisfies
T-1
Regret,-
Σ (ri PP)] 1=
CVTE BpT,
where C > depends Fiax and the smoothness constant, and B Tmax bounds the per-step loss under mis-gating. A full derivation is given in Appendix A
The V term captures error from model approximation; BpT accounts for occasional acceptance of unreliable imagined branches. When small and p is rare, regret grows sublinearly.
7

---

Published as a conference paper at ICLR 2026
Table Evaluating results on eight different tasks. Within the open-source block, bold denotes the
best method and underline the second best.
Algurithms
SeiWorld BabyAI
MAZE
Wordle
TextCran
Tool Beather
TODOList
BIRD
Avi
Closed-sourced Models Agents
DeepSeek Chat
DeepSeck-AAl 2024
16.8
45.6
4.0
24.0
23.0
70.0
75.0
13.5
34.0
Claude-3-Haiku
Antaropie 2024
0.8
1,9
4.0
16.0
0.0
55.0
13.5
19.6
S-nnne Annhroopic 2024
2.8
79.3
0.0
36.0
38.0
65.0
80.0
17.0
39.8
GPT.3.5
Ouyany et
12.5
GPT-4-Turbo
penAT 4723
14.4
72.8
68.0
88.0
77.0
80.0
95.0
16.0
63.9
Open-source Llama-2-7B as the Agent
Agentl M
1.5
0.5
12.0
4.0
4.0
0.0
15.0
5.0
5.3
AgeniGym
38.0
82.7
12.0
12.0
64.0
25.0
70.0
9.0
39.1
ARMAP-M
51.2
81.5
12.0
17.0
59.0
35.0
72.0
10.5
42.3
DREAMPHASE (5075)
14.0
34.0
62.0
13.5
50.1
Table 2: Cross-backbone performance WebShop. SciWorld (seen/unseen), and Game-of-24, Two backbone groups per row. Bold best, underline second best.
Llama70B
Llama8B
Algorithm
WebShop
SW-seen
SW-unseen
Game24
Arg
WebShop
SW-seen
SW-unseen
Game24
Sampling
52.0
53.9
50.6
41.5
56.4
24.5
20.6
25.9
Greedy
50.4
42.2
$7.7
23.8
28.4
ARMAP-R
56.5
59.6
56.7
16.0
47.2
58.3
31.2
28.0
6.0
30,9
ARMAP-B
62.0
57.3
57.0
19.0
48.8
59.3
35.7
28.1
11.0
33.5
ARMAP-M ICLDOOOS L000S
66.8
58.2
55.9
24.0
51.2
60.2
32.5
24.9
9.0
31.7
DREAMPHASE (Durs)
68.4
59.2
58.6
28.0
53.6
61.8
36.7
29.7
12.0
35.1
Mistral7B
Phi3.8B
Algorithm
WebShop
SW-seen
SW-unseen
Game24
Avg
WebShop
SW-seen
SW-unseen
Game
Avg
Sampling
17.7
18.4
17.1
1.0
13.6
34.7
10.0
7.6
2.0
13.6
Greedy
37.2
21.1
19.6
1.0
19.7
42.4
9.5
6.5
2.1
15.1
ARMAP-R ICLA 2025
54.1
21.7
19.7
24.4
53.3
7.2
4.0
18.5
ARMAP-I
54.4
ARMAP-M M
58.2
23.4
28.9
$3.7
24.3
10.0
29.1
DREAMPHASE (ours)
60.5
33.5
22.9
30.7
55.5
32.8
24.1
13.0
31.4
5
EXPERIMENTS
We evaluate DREAMPHASE across diverse agentic environments and backbones. We first outline the evaluation protocol, then present two main results tables: (i) diverse-task comparison across closed- and open-source agents (Table T). and (1i) rross-backbone results on WebShop, SciWorld (seen/un- seen), and Game-of-24 (Table Implementation details, byper-parameters, dataset descriptions, and bascline implementations appear Appendix
Evaluation protocol. For each task, report mean success (%) over 5 random seeds; standard deviations for Tables I and 2 are reported in Appendix D The Avg column is the unweighted mean across environments within each table.
Episode budgets and decoding settings follow prior work and are detailed in Appendix
5.1
MAIN RESULTS
5.1.1
DIVERSE TASKS ACROSS AGENTS (TABLE
Tasks. SeiWorld Waag al. 2022a), Baby AI (Chevalier-Boisss
al. 2019), MAZE (Abdulhai
ct al. 2023). Wordle (Abdulhai et al. 2023). TextCraft (Prasad et al. 2023). BeathWetther (Ma et al.
2024
TODOList (Ma al. 2024), BIRD (Zheng et al.) 2023).
Agents. We include closed-source references (DeepSeek-Cii) Chat (DeepSeek-AL Al 2024). Claude-3.
Haiku/Sonnet (Anhhropic/ 22024), GPT-3.5/4-T /urbo (Ouyang et al.) 2022 OpenAl, 2023) and open-
source agents (AgentLM (Zeng et al. 2024). AgentGym (Xi et al. (025). ARMAP (Chen et al.)
2025)). For the open-source block (including DREAMPHASE in this table), standardize the policy
8

---

Published as a conference paper at ICLR 2026
backbone to LLaMA-2-7B (Touvron al. 2023 and match decoding and episode budgets for fairness; closed-source results are reported as returned by their APIs.
Results. Table D shows that DREAMPHASE attains the best average within the open-source block while keeping real interactions low (Appendix (..2). Note that our open-source comparisons standard- ize LLaMA-2-7B backbone, which is substantially smaller than the closed-source references (e.g.. GPT-4-Turbo, Claude-3) and trained with con-prorriettry data; higher absolute scores from those APIS are therefore expected, yet DREAMPIIASE achieves performance comparable these com- mercial models. Under the matched 7B setting, DREAMPIIAS consistently outperforms search-bases baselines. with the largest gains on manipulation-/tool -eavo tasks where the uncertainty gate prevents risky actions when confidence is low.
5.1.2 CROSS-BACKBONE BENCHMARKS (TABLE
Tasks. WebShop Yao et al. 2023a), SciWorld (Wang et al. (reported as SW-seen and SW-unseen), and Game-of-24. We also report ALFWorld in Appendix E
2022a) with seen and unseen task graphs
Backbones and Agents. We evaluate LLaMA3-70B, LLaMA3-8B 8B, Mistral-7B, and Phi-3-8B. Baselines include ARMAP with Reflexion, Best-of-N, and MCTS (denoted ARMAP-R / ARMAP-B ARRMP-MM,, plus Sampling (temperature 1) and Greedy (temperature 0) decoding. For supervision of the value head Vo, we generate preference-style pairs with LLaMA3-70BB 7---70-IIInstee (Dubey et al. 2024]: we avoid commercial APIs reduce cost and improve reproducibility.
Results. Table 2 shows that DREAMPHASE achieves the best or second-best success on nearly all backbones and tasks, with especially strong gains on Game-of-24. On SW-unseen (novel goals and step compositions), DREAMPHASE outperforms all baselines across multiple backbones, indicating robustness distribution shift (Appendix[H). Ablations over latent dimension dz, horizon H. branches K, risk penalty 8, and confidence gate are provided in Appendix G and latency of latent rollouts in Appendix J
We also report computation
5.2 INTERACTION BUDGET AND LATENCY ON WEBSHOP
We quantify real environment interactions as API calls issued to WebShop (HTTP requests and DOM mutations that change page state). We evaluate Llama-8B policies the standard WebShop split, batch size 1, A A100-80GB. Latency is measured as policy forward time plus search/planning overhead.
We count only environment-affectin calls (page loads, form submissions, cart updates, DOM actions with side effects). Cache hits, retries, and latent-only computations are excluded. Latency numbers use identical hardware and decoding settings for both methods.
Table 3: WebShop interaction and latency comparison (mean s.e., 1-000 episodes). DreamPhase uses ~ 4x fewer real API calls and ~ 3x lower per-step latency.
Method
Policy
Aug API calls/ /epl x fewer vs. ARMAP-M Per-step latency (mns)
Success (%)
ARMAP-M (token-level search)
Llama-88
39.8 1.1
250
60.2 + 0.6
DREAMPHASE (latent imagination)
Llama-8B
9.3 0.4
4.3x
ss 84
61.8 +: 0.6
As observed in T DreamPhase attains its performance with substantially fewer real interactions: under 10 API calls per episode average, versus about 40 for ARMAP-M. The latent imagination overhead is small relative to the policy forward pass, yielding lower per-step latency. Fewer real calls also reduce the chance of unwanted side effects, since more planning happens offline.
* * * Beyond interaction counts, DreamPhase lowers executed irreversible actions by ~5x on WebShop (and ~4.9x ALFWorld); see Appendix C
5.2.
QUALITATIVE COMPARISON
Section 1.2.1
contrasts ARMAP and DREAMPHASE on a Game-of-24 instance. ARMAP commits
early and returns an incorrect value, while DREAMPHASE imagines multiple futures, selects a high-value cow-uncertannty branch, and executes the correct sequence (11 7) x 3) 24.

---

Published as a conference paper at ICLR 2026
Qualitative Rollout on Game-of
Input numbers 7, 9, 11 Baseline (ARMAP) Trajectory Failure
+1+9 9 20
(remaining: 3,
2. - = 13
(remaining: 3, 13)
+
(remaining: 16)
Outcome: result is 16, not 24. Baseline halts without valid solution.
DreamPhasee Trajectory - Success Imagination phase (latent world model)
Branch I: (11 9) (7 3) 16
(low value)
Branch (11 7) * (9 3) 24
(high value, low uncertainty)
Three other branches score I 0.4 value and/or high entropy
Selected branch (reflection injected)
1. 7 =
(remaining: 3,9,
9-3
(remaining: 4, 6)
3. 4 6 24
(remaining: 24)
Outcome: DreamPhase produces a correct expression (11 - 7) * (9 3) 24.
CONCLUSION
We presented DreamPhase, a modular agent framework that plans through internal imagination. By simulating futures in a learned latent space and scoring them with value and uncertainty estimates, DreamPhase acts only when a high-confidence trajectory is found. This greatly reduces real-world interactions, lowering consistent gains across web navigation, scientific reasoning, and puzzle solving, all without fine-tuning the policy LLM. By combining latent imagination, uncertainty-awar gating, and language-based reflections, DreamPhase provides a scalable path toward safer, more sample-efficient agents.
and risk while maintaining strong performance. Our experiments show
REPRODUCIBILITY STATEMENT
We have taken steps to ensure our results are reproducible. All model and algorithmic details, training procedures, hyperparameters, evaluation protocols, and metries are specified. An anonymized GitHub repository contains the source code and configuration files, and pre-trained checkpoints. All datasets used our experiments are publicly available.
LLM USAGE STATEMENT
L.M used only for grammar and wording edits; no generation ideas, methods, analyses, results, or citations. Authors reviewed all edits and accept full responsibility.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
Anthropic. The claude 3 model family: Opus, somnet, haiku, https://wwwcch..atttettet Com/ de fel.aspo....aaaooooooooooooooo33 pdf
Card Claude
2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.00003,, 2022.
R. Brooks. A robust layered control system for mobile robot. IEEE Journal on Roboties and Automation, 2(1):14-23, 1986. doi:
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, and /2005
et al. Language models are few-shot learners, 2020.
14165
Xinlei Chen, Hao Fang. Tsung-Y- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015. URL
https://arxiv. org. abs 1504. .03255
Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, and Chuang Gan. Scaling autonomous agents via automatic reward modeling and planning. In The Thirteenth International Conference on Learning Representations, 2025, URL https: htRL::ttt:////otmm /ttp:////peeeeeeeeee net wwr.m/ormmmmmmmmmm Wom wowU0>EweO
Maxime Chevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: platform to study the sample efficiency of grounded language learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https //openreview.oo hwt.:aaweeett.oormmmeeeeeeeoooo.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
DeepSeek-AL. Deepseck-v2 v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 2024.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The llama 3 herd of models, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stee Staa,, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi. Huan Sun, and Yu Su. your Ilm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06557 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations.

---

Published as a conference paper at ICLR 2026
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv, 2023.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and compre- hend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL /tttps://proceeeinngg newrips..eennrrrrccec.....s co/paper Files//aper/2015) olleaa/ffeec/005ccc1111322222210
ppff
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Lukas Kirchdorfer, Robert Blümel, Timotheus Kampik, Han Van der Aa, and Heiner Stuckenschmidt. Agentsimulator: An agent-based approach for data-driven business process simulation. In 2024 6th International Conference on Process Mining (ICPM), pp. 97-104. IEEE, 2024.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv, 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong. and Jie Tang. Evaluating ilms as agents. arXiv preprint arXiv: 2308.03688, 2023.
Agenthench:
Chang Ma. Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn LLM agents. CORR, abs/2401.13178, 2024. doi: 10.48550/RXIV..441.115
htttst
7/doi 2abll..or./.0.4555/rrx.x........ http://rxxx....1......
OpenAl. GPT-4 tecchncall ieco CoRR, ab/22333.000777 2023. doi: 10.48550/ARXX1.2300..057 URL
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, and et al. Gpt-4 technical report, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, Ne 2022, New Orleans, LA, USA, November 28 December /hash/
9. 2022. 2022. URL http: b1 (lerfaas3eee34443331444583333333 htt.a//3330..0440000ooooooooonn-
nips http:sspppppppppppppppppppppp fiirrrfiieese
12

---

Published as a conference paper at ICLR 2026
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal. Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. CoRR. abs/2311.05772, 2023.
doi: 10.48550AARXIV.3111.0777 URL btttps://do....
48550/arxiv wwt.x/i...11.... 2 05772.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100,000+ questions for machine comprehension of text, 2016. URL
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Chou-Chakra, Ian Reid, and Niko Suenderhauf.
Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. In 7th
Annual Conference Robot Learning, 2023. htth::t:::/ httttss://openrevieeww http://wwwwwww...e..eeeeeeee
1d=wMpOM00Ss74
Scott Reed, Konrad Zolna, Emilio Parisotto,, Sergio Gomez Colmenarejo, Alexander Novikov, and et al. A generalist agent, 2022.
06675
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2024.
Samuel Schmidgall, Rojin Ziaci, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agentclinic: a multimodal agent benchmark to evaluate ai simulated clinical environments, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexander Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 202 URL
battps://arxie www.vvioo./roooooooooooo wwt.///..... 03768
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021b. URL httpps::/opeeeee/a///
Hausknecht. (ALFW Jorld: Jord:
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv, 2017.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics. Association for Computational Linguistics, 2024.
Hugo Touvron, Louis Martin, Kevin Albert Stone, Peter Albert, Amjad Almahairi, Yassine Babaei, Nikolay Bashlykov, Dhruv Batra, Preetum Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. URL/httss! 7/arx //arxiv.or www. 2aap///3/..02222 Models include 7B/13B/70B parameter variants; we use the 7B 2-lama-2-7BB.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11279-11298 2022a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11173 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of -f prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022.
13

---

Published as a conference paper at ICLR 2026
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, et al. Agentgym: Evaluating and training large language model-based agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 27914-27961, 2025.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. arXiv, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv, 2022.
Shunyu Yao, Howard Chen, John Yang. and Karthik Narasimhan. Webshop: Towards scalable
real-world web interaction with grounded language agents, 2023a. URL htttt:///aaxxii
brg/aba/2201 01206
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023b. URL httphttts//aaxiia
org/ /2005. 10601
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mets and exploratory learning, arXiv preprint arXiv:2410.02055 020522, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4/st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 57905 57905-57925, 57923. PMLR, 21-27 Jul 2024a. URL https //proceedings. www.p/sssr.p.re.s.mmmmm
Sarres/v23550 v235/ wttpp////32yyymm222mm yuwn2uannd4. html
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv, 2024b.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for Ilms. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3053-3077, 2024.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a- juddjge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Sacnko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December nips cc/paper files/ files/paperr2023/ Abstract Datasets and
10 16, 2023, 2023. URL =bstraaeeecc.aaattaaaaaaaaaareer Benchmarks html
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13855 13854, 2023. URL SoLL:tttt///eebrrre
dev
Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. paaposser-ggentevvluuaaae (pae): Autonomous skill discovery for foundation model internet agents. arXiv preprint 1r11v:2412.13194, 2024.
14

---

Published as a conference paper at ICLR 2026
A REGRET ANALYSIS OF DREAMPHASE
In this appendix we derive the regret bound in Section 4
We restate the setting and assumptions, adapt standard model based RL arguments, and incorporate the effect of uncertainty gating errors.
A.
PRELIMINARIES
We work with the environment M defined in the main text and compare one step predictive dis- tributions over observations. Let IP, (2c+1 hrs 0z, e) be the true conditional distribution and
Po (x he, ac.t) the model induced by the learned world model. Assume uniform step KL bound
E = sup KLPP( his ar, ar, P| | lee az, ())
(11)
h..a:
By Pinsker, TV(P.,,o) & ve/2 uniformly, equivalently & distance is at most 22..
Gating error. Let r denote the fixed confidence threshold used by the safety gate in Section 3.4 Define the mis gating rate
=
Pr[u and DreamPhase still selects an imagined branch]:
(12)
that is, the probability that the agent erroncously trusts an unreliable imagined rollout.
Smoothness and bounded rewards. Assume rewards bounded by Mmax and the value function is Lipschitz with respect to the total variation of the one step predictive distribution. There exists > such that replacing P- he, at, by any Q( he, at, i) changes the optimal H Hmax step value by most TV (IP,, Q) per step.
A.2 REGRET DECOMPOSITION
Let Nt be DreamPhase's s action at time t. Define cumulative regret after T steps
T-1
Regrety
Σ (V."
V
(13)
where values are taken under the true environment. We decompose regret into (i) model approximation when acting on imagined rollouts and (iz) mis gating when the agent trusts high uncertainty branch.
(i) Model approximation term. When DreamPhase follows an imagined branch, the one step distributional error is bounded by TV s Vs/2. By the Lipschitz assumption and a telescoping argument over at most Hmxx steps,
( - EE, (V s CVTs,
Σ
Hmax 22
(14)
(ii) Mis gating term. On any step where an over confident imagined branch is used, the per step value loss is bounded by a constant B that depends Tmax and H Since such events occur with probability o per step, their contribution is at most B p...
A.3 FINAL BOUND
Combining the two terms yields
Regrety S CVT + BpT,
(15)
with C' = H. /2. When the world model accurate (s + 0) and mis gating is rare (p - 0),
regret grows sublinearly, DreamPhase approaches Bayes optimal performance.
15

---

Published as a conference paper at ICLR 2026
B FURTHER RELATED WORK
LLMs as Interactive Agents. Recent advancements have demonstrated the potential of LLMs as interactive agents capable of reasoning and acting within dynamic environments (Liu et al. 2023 Zhou al. 2023). Early approaches explored adapting models like BERT (Devlin al.
2019 to
structured decision-making settings, such as online navigation tasks Yao et al. 2023a With the
emergence of more capable generative models (Brown et al. 2020 OpenAl et al. 2024). research
has increasingly focused using zero-shot and few-shot prompting to control agent behavior without fine-tuning (Deng et al. 20224] Xiong et al. 2024). These agents typically act by processing
observations and histories through LLMs select the next action (Zheng al., 20244 Hong et al. 2023). Some efforts have also focused on training smaller policy models by distilling trajectories
from powerful but restricted APIs (Li et al. 2023 Zhang et al., 2023). Our work departs from this
line by learning a reward model directly from M.M-generated data, enabling downstream planning without needing fine-tuned policies.
Language Model-Based Planning. LLMs have also been employed for planning tasks, often through prompting techniques that combine reasoning and action generation. Methods such as ReAct Yao et al. 2022) integrate step-by-step thought chains with action prediction (Wei al. 2022) Tree- based deliberation strategies have further improved multi-step reasoning (Yao et al. 2023b Hao et al. 2023). However, most existing work focuses on generating text-level plans rather than structured trajectories within grounded environments. Moreover, these methods typically rely on black-box box models like GPT-4 for evaluation and planning, which limits interpretability and adaptability. By contrast, our approach introduces a modular system that learns a reward model from data and uses to supervise structured planning via imagination.
Learning via AI Feedback. Another line of research involves training models using feedback generated by other AI systems (Bai et al. [022]] Lee et al. 2023 Yuan al.) 2024b] Koh al.. 2024). These techniques often rely on prompting commercial models to generate preference data for finetuning. We draw inspiration from these methods but take a different route: instead of training generative model, we optimize lightweight evaluator to score trajectories with respect to task instructions. This reward model is then integrated into planning, improving performance without additional model updates.
Inference-Tinee Reasoning and Control. Inference-time planning methods provide an efficient alternative to training-inteen intensive pipelines. Approaches such as MCTS with language models (Silver et al. 2017 Koh et al. 2024) allow dynamic reasoning through rollout-based simulation. Other
strategies include speculative execution (Gu et al. (024). reflective tree search (Yu et al.) 2024), and sampling-based -baee strategies (Wang et al.] 2022b). Our method belongs to this family, offering two key advantages: (1) generalization across task domains including web agents, scientific simulations, and
mathematical games; and (2) compact reward model trained from synthetic data that avoids reliance
on commercial LLM APIs. We use a fine-tuned visual- languaa model (Lin et al. 2023) to evaluate trajectories in structured and scalable way.
C SAFETY EVALUATION BEYOND INTERACTION COUNT
We measure safety not only via gating diagnostics (AUROC/ECE) and interaction budgets as reported in Appendix H but also by logging irreveriible-action incidents during execution.
What we count. An irreversible action is any environment-affecinn write with persistent conse- quence, including (i) form submissions that change server state (e.g., checkout, account creation), (ii) purchases/cart checkouts, and (iii) destructive operations (delete/confirm). "Unintended" means inconsistent with the user instruction and task constraints
'We apply a deterministic ruleset keyed to domain endpoints (e.g., /checkout, /submit), DOM event types (e.g., submit t, c1 i ck on [type-submi it)). and confirmation dialogs. A 200-episode human audit found 0.94 precision in the automatic labels.
16

---

Published as a conference paper at ICLR 2026
Detection protocol. We run in sandbox/dry-run mode with (a) network stubbing for purchase endpoints; (b) DOM-level hooks capturing submit/click events with element metadata; and (c) server logs for state- cate-changing requests. We record three metrics per 1,000 episodes: (1) Critical incidents (unintended irreversible actions actually executed), (2) Near-misses (attempts blocked by confirmation dialog sandbox stub), and (3) Unsafe proposals (actions proposed by the policy but rejected by the safety gate; not available for methods without gate).
Table 4: Safety outcomes per 1,000 episodes (mean + s.e.). Critical counts unintended irreversible actions executed: Near-miss are blocked by confrmmtiions/anddoox Unsafe proposals are caught by DreamPhase's s gate before execution. Dream reduces executed incidents by ~5x on WebShop and ~4.99 on ALFWorld while surfacing risky proposals for inspection. Results on LLaMA-8B backbone.
WebShop
ALF'Wordd
Method
Critical
Near-miss
Unsafe proposals L
Critical +
Near-miss +
Unsafe proposals
ARMAP-M (token scarch)
7.1 1.2
18.4 2.3
0.9
7.8 1.5
DRFAMPHASE (aurs)
1.4 + 0.5
26.7 +: 2.8
61.2 + 3.5
0.8 + 0.3
12.6 = 1.9
44.9 3.1
Takeaways. Fewer real calls (Sec. (..)) translate fewer opportunities for harm, but incident reduction is not solely due to budget: DreamPhase's uncertainty gate prevents many unsafe proposals from being executed, yielding 5x (WebShop) and 4.9x (ALFWorld) drop in executed irreversible actions under identical hardware and decoding settings.
D STANDARD DEVIATIONS FOR MAIN RESULTS
Numbers are computed over random seeds; for each seed we evaluate fixed episode budgets per benchmark. The Avg column's 's std is computed as the standard deviation of the seed-wise average across benchmarks (not the average of per-benchmark stds).
We report the standard deviation (std) for each cell in Tables and 2
Notes. Std values reflect run-to-run variability from random seeds and dataset sampling. The Llama70B models show lower variance overall, while smaller backbones (Mistral7B, Phi3.8B) exhibit higher variability, especially on Sä-unseen. DreamPhase's variance is comparable or lower than ARMAP-M across backbones, consistent with its uncertainty- gating and reduced real-environment interaction count.
E RESULTS ON ALFWORLD
We evaluate ALFWorld (Shridhar et al. 2021b) using the standard and dev splits with 50-step cap, Llama-8B policy. batch size Success is the default environment metric. We also report the average number of real API calls per episode quantify interaction cost.
Discussion. Dream Phase improves success over ARMAP-M on both splits and reduces interaction cost by running rollouts entirely in latent space with uncertainty-awar gating. Lower API calls also reduce latency and side-effect risks in this long-horizon setting.
Why ALFWorld is challenging. Compared to WebShop, SciWorld, and Game-of-24 ALFWorld requires language grounding for navigation and manipulation in partially observable homes. Plans span many steps and include preconditions (open, piek up. place, heat, clean) that couple far-apart states. Rewards are sparse and largely terminal, so errors compound. The action space exposes many affordances per state, which creates a higher branching factor than click-only navigation or symbolic arithmetic. Scenes and templates differ across splits, the agent must generalize across rooms, object aliases, and layouts rather than memorizing flows. These properties make ALFWorld a strict test of long-hol ong-horizon reasoning under uncertainty.
Takeaway. ALFWorld stresses long-horizon planning, partial observability, and irreversible actions more than other benchmarks. DreamPhase is well-suited here because it plans in latent space,
17

---

Published as a conference paper at ICLR 2026
Table 5: Standard deviations corresponding to Table |
seeds).
Algorithms
Tool-Weuther SciWorld
BabyAI
MAZE
Wordle
TextCruft
TODOList
BIRD
Arg
Closed-sourved Models & Agents (std)
DeepSeek-CHi
1.5
0.9
Claude-3-Haiku
0.9
0.6
0.5
0.5
1.0
0.4
0.9
0.6
0.6
Claude-3-Sonne
1.1
0.8
1.8
0.4
1.4
1.5
1.0
0.8
0.9
GPT-3.5-Turee
0.8
0.9
1.6
0.6
1.1
1.3
1.2
0.7
0.8
GPT-4-Turbo
1.2
1.1
1.7
1.5
1.0
0.8
1.0
Open-source Models Agents (std)
AgentLM-7B
0.7
0.9
0.6
0.6
Agent. MM138
0.9
0.8
0.4
0.6
0.5
0.5
0.8
0.5
0.6
AgentLM-70B
0.7
1.0
0.5
0.6
0.8
0.8
1.1
0.7
0.7
AgentGym
1.3
1.8
1.2
0.7
1.1
1.4
1.3
0.7
1.2
ARMAP-M
DREAMPHASE (ours)
1.0
1.4
0.8
0.5
1.2
1.3
0.9
0.6
0.9
Table 6: Standard deviations corresponding to Table 2 space efficiency.
Two model groups are displayed per row for
Llama70B (std)
Llama8B (std)
Algorithms
WebsShop
SW-seen
Game24
Arg
Algorithms
WebShop
SW-seen
SW-unseen
Game24
Avg
Sampling
0.9
0.4
Sampling
Greedy
0.8
0.7
0.4
0.6
Greedy
0.9
0.9
0.8
ARMAP-R
0.7
0.6
0,7
0.5
0.6
ARMAP-R
0,8
0.9
0.8
0,5
0.7
ARMAP-B
0.7
0.7
0.7
0.6
0.6
ARMAP-B
0.8
1.0
0.9
0.6
0.7
0.6
0.7
0.6
ARMAP-M
DreamPhase
0.6
0.6
0.5
0.5
DreamPhase
0.8
0.7
Mistral7B (std)
Phi3.8B (std)
Algorithms
WehShop
Game24
Avg
Algorithms
WehShop
Avg
Sampling
1.2
1.1
0.3
0.9
Sampling
1.2
1.2
Greedy
1.0
0.3
0.8
Greedy
1.1
ARMAP-R
0.9
1.0
1.0
0.4
0.8
ARMAP-R
1.2
1.1
1.1
0.6
1.0
ARMAP-B
0.9
1.0
1.0
0.4
0.8
ARMAP-B
1.2
L.1
1.1
0.6
0.9
ARMAP-M
0.8
1.1
0.5
0.9
ARMAP-M M
1.1
1.2
1.2
0.7
DreamPhase
0.8
1.0
0.5
0.8
DreamPhase
1.0
gates uncertainty before executing risky manipulations, and therefore keeps real interactions low while maintaining success.
F IMPLEMENTATION DETAILS
F.1 TRAINING DATA FOR THE LATENT WORLD MODEL
For every benchmark in Table and Table 2, the latent world model and the value head are trained on logged interaction trajectories collected exclusively from the training split of the corresponding environment. We execute a frozen LLaMA-2-7B policy with a simple ReAct-style prompt and mild stochasticity gather trajectories of the form (I, (t, de, x1+1). The world model receives only these tuples and does not access any test episodes or privileged environment states. No external corpora, fine-tuned LLMs. or environment augmentations are used. Baseline agents are permitted use their released training pipelines; DreamPhase uses only the same environment APIs available to all methods.
F.2 FAIRNESS OF COMPARISONS
In all open-source comparisons (Table 1), we the policy backbone LLaMA-2-7B and match decoding settings, episode budgets, and environment splits across DreamPhase, AgentLM, AgentGym, and ARMAP. Baselines use their recommended training pipelines (c.g., agent tuning or reward-mode training). DreamPhase does not leverage any additional data beyond the logged training saining-split trajectories described above. This ensures that performance improvements stem from imagination- based planning and uncertainty-aware selection rather than from unequal data training advantages.
18

---

Published as a conference paper at ICLR 2026
Table 7: Experimental results on ALFWorld (Llama-8B). Columns report success on the standard and dev splits. DreamPhass matches or exceeds token-level search while using about four times fewer real interactions per episode.
Models
ALFWorld-std
ALFWorld-dev
Avg API calls / /ep.
Sampling
0.13
0.14
34.9
Greedy
0.18
0.30
33.7
ARMAP-R
0.22
0,35
41.5
ARMAP-B
0.30
0,45
39.8
ARMAP-M
0.31
0.46
46.7
DREAMPHASE (ours)
0.39
0.49
11.6
Table 8: Qualitative comparison of task characteristics. Horizons are typical ranges, action counts are approximate per state.
WebShop
SciWorld
Game-of-24
ALFWorld
Observation
DOM/ text 8-15 5-15
Text (lab)
Numbers
Text (embodied)
Actions per state Typical plan length
10-20
4-6
20-40
6-12
3-6
15-35
Reward density
Sparse terminnal
Spurse terminal
Dense terminal
Spurse terminal
Ineversible effects
Medium (purchases)
Low-Medium
None
High (manipulation)
Core difficulty
Navigation filters
Procedural constraints
Symbolic search
Nav manipulation, preconditions
F.3 TASKS
MAZE (Abdulhai et al., 2023) is a grid-based puzzle game where the agent observes its current position, the goal location, and nearby walls. The agent can move up, down, left, or right, receiving reward per step until reaching the goal. Success rate is measured with a maximum of 15 steps episode
Wordle (Abdulhai al. 2023) is a five-letter word guessing game. After each guess, feedback reveals which letters present and correctly positioned. Each step yields reward of -1 until the correct answer is found or attempts are exhausted. Success rate the evaluation metric, with a maximum of 8 guesses allowed.
(Wang et al. 2022a) evaluates scientific reasoning across 30 task types, such as conducting simple experiments using measurement tools. We use GPT-4-Turbo generate reasoning traces for golden paths of 22 task types and collect 1000 trajectories DreamPhase. Reward is used as the metric with maximum of 30 steps
SciWorld
Baby (Baby) CChevalierBoiisvertt et al. 2019) is a grid-world instruction-followine environment with 40 tasks. We adopt the AgentBoard implementation, which converts visual observations text and supports high-level actions such as "pick up key" or "open door." We annotate 400 trajectories
for DreamPhase. Reward is used the metric with a 20-step limit 41)
TextCraft (Prasad et al. 2023) is a text-only Minecraft-like environment where tasks involve crafting target item by composing multi-step recipes. The action space includes craft, get, and inventory. Rewards are given only for successful completion. We annotate 300 verified trajectories for DreamPhase. Success rate is reported with maximum of 20 steps
Weather (Ma et al. 2024) allows agents to query weather data (temperature, precipitation, air quality) using tool backed by the Open-Meteo API. The dataset is expanded to 343 queries via self- instruction and instruction evolution with GPT-3.5/4 3.5/. We annotate 160 trajectories for DreamPhase. Success rate is the metric, with a limit of steps
19

---

Published as a conference paper at ICLR 2026
Movie (Ma al. 2024) enables agents retrieve movie-related information (film metadata, cast, production companies) through 16 tool actions, using data from The Movie Database. We expand the dataset to 238 queries, annotating 100 trajectories for DreamPhase with GPT-4-Turbo, Success rate used as the metric with 12 maximum steps.
TODOList (TL) (Ma al. 2024 lets agents interact with a personal agenda using the TodoList API. The dataset is expanded to 155 queries. We annotate 70 trajectories for DreamPhase (combining GPT-4-Turbo and human annotations). Human review further refines annotations. Success rate measured with up to 15 steps.
BIRD (BD) (Zheng al. 2023) evaluates database- database-grounded text-to-SQL. ability. Agents must generate SQL queries that retrieve correct answers from database. From the 9428 available problems, we select 3200 for instruction tuning, adding GPT-4-Turbo reasoning traces to 2000 DreamPhase. Success rate is measured, and BD is a single-turn task
WebShop (Yao al., 2023a) is a simulated online shopping environment. The agent must interpret a textual product query and select matching item from a web interface. We follow the AgentBench (Liu et al. 2023) protocol and report performance on the validation set using the default string-match reward function.
SeiWorld (Wang et al. 2022a) offers a text-based interactive world for completing science ex- periments. Agents must navigate multiple rooms, manipulate objects, and follow procedural steps grounded scientific logic. We evaluate performance on both tasks seen during instruction synthesis and novel ones held out for generalization, which we denote by SW-seen and SW-unseen in this section, respectively.
Game-of-24 24 is symbolic reasoning benchmark where the goal is combine four integers using basic arithmetic operations (+, -. /) to reach exactly 24. For example, the numbers 3, 5, 7, and 11 yield 24 via the expression (7 -7-3) 3) x (11 5). Following Yao et al. (2023b), test 100 difficult puzzles (indices 901-1000), measuring success as the fraction of correctly solved problems. For this, we follow prior work by allowing up 100 samples per instance.
ALFWorld ALFWorld (Shridhar et al. 2021b) is a text-based embodied household bench- mark built TextWorld and aligned with ALFRED scene layouts. Each episode provides natural-language goal (e.g., "Put a pan on the dining table"). The agent observes partial tex- tual description of the current room, visible objects, and inventory, and issues templated actions such as go to <room>, open <containes <receptacle> heat/cool, and clean. Tasks require long-horizon navigation and manipu- lation with preconditions (e.g., open before take, place before heat), sparse terminal rewards, and irreversible effects, making error recovery difficult. Scenes, object aliases, and surface forms vary across homes and splits, stressing generalization rather than memorized flows. Following prior work, we evaluate on the std and der splits with a 50-step budget and report success rate (goal achieved within budget); also track real environment API calls per episode to quantify interaction cost.
take <object>, put <objects on/ in
F.4 LARGE LANGUAGE MODEL CONFIGURATION
We utilize variety of open- open-source source LLMs to assess the generalizability and robustness of our approach. These models, accessible via Hugging Face, are compatible with the VLLM library (Kwon 2023), making deployment and experimentation straightforward. Below is list of the models used along with their Hugging Face repositories:
Llama70B:
1-70 -n0B-InnttructtAW00-1
. Llama8B:
1-8B- Instruct
Mistral7B:
Mistral-7as 7B- Instruct-v0.3 http:///ugginffec...o/sssttaaaat
Phi3.8B:
5
com/A1 http::/wwww...aaaaaaaae///a///aa
20

---

Published as a conference paper at ICLR 2026
VILA3B:
5-3b
These models are selected to ensure a broad evaluation across architectures and parameter scales.
F.5 ENVIRONMENT SETUP.
We construct our experimental environments by building upon established implementations from prior work (Liu et al. 2023) Song et al. 2024] Yao et al., 2023b Shridhar et al., 2021a; Schmidgall et al. 2024). For tasks such as WehShop and ALF-orlld, we leverage Docker infrastructure provided by AgentBench (Liu et al. 2023) and integrate various planning strategies including Reflexion, Best-of-N, and MCTS. Other environments such as SciWorld, Game-of-24, and AgentClinic are initialized based on the corresponding setups introduced in Song et al. (2024). and Schmidgall et al. 2024), respectively.
Yao al. (2023b),
F.6 PLANNING STRATEGY CONFIGURATION.
We evaluate multiple planning algorithms under consistent budget for trajectory exploration. On WebShop and SciWorld, we cap the number of explored trajectories at 10 to balance computational cost and performance. For Game-of-22 we increase the limit to 100 following Yao al. (2023b). To reduce the branching factor during search, only the top 10 action candidates proposed by the LLM are considered at each decision point. Additionally, we constrain each trajectory to maximum of 10 steps.
For Reflexion, a limit of trials is applied across all tasks. The stopping condition based on task-specific thresholds: when the reward from generated trajectory exceeds the threshold, the iteration halts and the corresponding result accepted. If no trial surpasses the threshold, the last trial is used in WebShop and Game-of-24, whereas the first is chosen in SciWorld.
F.7
The typre-parameters used in our method are summarized in Table and ??.
Table 9: Fixed hyper-parameters used for all reported results; 3 trades off value against epistemic uncertainty; is the confidence gate below which the agent trusts imagination.
Hyper-parameter
WebShop
SciWorld
Game-of-24
Latent dimension
256
256
128
Horizon H (imagined steps)
5
6
4
Branches K (per decision)
8
10
12
3 risk penalty (Eq. 9)
0.35
0.30
0.25
Uncertainty gate World-model Ir (Adam)
0.20
0.25
0.15
2 x 10
2 10
X 10
Batch (world-model)
32
64
Value-head Ir
x 10
1 10
5 10
Temperature (policy sampling)
0.8
0.7
Max real interactions per episode
50
60
30
Note. The main text uses for the number of imagined branches per decision; the "Branches K" row in Table SableeCooresooe corresponds to M.
G ABLATION STUDIES
Unless noted, we fix all hyper-parameters Table 9 and vary one knob at time. We report task success (points, higher is better), the fallback rate (fraction of steps where the safety gate defers to real interaction), and imagination overhead in milliseconds per decision step an A 1000-80GB (batch 1). We use M for the number of imagined branches and for the horizon. The latent dimension is d: (the "t" column in Table 9 We use Llama-8B as the backbone model.
21

---

Published as a conference paper at ICLR 2026
Table 10: Latent world-model architecture and training hyper- paremeerssssssss for the encoder fo. transition 9o, and decoder dg-
Component Hyper-parameter
WebShop
SeiWorld
Game-of-24
Encoder (or.. -.
Encoder type
Transforerr encoder
Transformer encoder
Transformer encoder
Layers
8
Model dimension
512
512
384
Attention heads
8
FFN dimension
2048
2048
1536
Latent dimension di
256
256
128
Instruction conditioning
prepended instruction tokens
same LayerNorm GELU 0,10
same LayerNorm GELU
Normalization
LayerNorm
Activation
GELU
Dropout
0.10
0.10
Action embedding a. enca((a)
Action encoder
mean-pooled token embeddings 128
same 128
same 64
Action embed dim [c.]
Transition (RSSM) go( (2, Ze+1
Deterministi state dim
512 GRU
512 GRU diagonal Gaussian same
384 GRU
Recurrent
Stochastic latent prior
diagonal Gaussian 2-layer MLP (GELU)
diagonal Gaussian same
Prior / posterior beads
Decoder do(=)-1.4) Ri+1
Decoder type
Transformer decoder
Transformer decoder
Transformer decoder
Layers
Model dimension
512
512
384
Attention heads
8
FFN dimension
2048
2048
1536
Cross-atiention memory
latent state tokens (s, h, Yes first e 2 steps of selected hranch
same Yes same
same
Weight tying with encoder embeddings
Yes same
Decoding scope (for reflection)
Optimization (world-model only)
Optimizer/ hetas
Adam, (0.9, 0.999)
Adam, (0.9. 0.999)
Adam, 0.999)
Learning rate
2x 10-1
2x10-
1x Lx10-1
KL weight schedule Ası.
lincar warmup target over 10k steps 32
15k steps 32 1.0/ 1 10 bf16
5k steps
Batch size (world-model)
64 1.0/ 1x10-
Grad clip/ weight decay Procision
1.0 1x 10
bf16
bl16
Notation. Triplets shown as (W/S/G) correspond to (WebShop / SciWorld / Game-of-24). For example, H applies per domain in that order.
Table 11: Ablation on latent dimension de.
d, (W/S/G)
WebShop
SciWorld
Game-of-24
Fallback (%)
Overhead (ms)
64 / 64 / 64
59.4
34.2
28.4
24
10.4
128 / 128 / 128
60.8
35.7
29.4
22
11.3
256 / 256 / 128 (Default)
61.8
36.7
29.7
20
12.0
256 / 256 / 256
62.1
37.0
30.0
20
13.2
512 / 512 / 512
61.6
36.5
29.2
21
16.0
Analysis of Table 77 Performance improves from 64 - 256 due to richer latents; gains flatten at 512 with a small overhead increase. WebShop and SciWorld prefer d.=256, while Game-of-24 peaks at 128, consistent with its simpler dynamics. Fallback decreases representation quality improves.
Analysis of Table 12
Short horizons under- plaa; very long horizons add compounding model error and uncertainty. The per-domain sweet spots line up with Table 9 (H=5 WebShop, 6 SciWorld, 4 Gfmee-of22)) Fallback rises with H as uncertainty accumulates.
Analysis of Table More branches provide modest gains that saturate around M E [12, 16). Fallback drops slightly (more chances to find a safe plan). Overhead seales near- -ineerly with M, consistent with Appendix l We default to M in [8, 12 for the best latency/quality trade-off.
22

---

Published as a conference paper at ICLR 2026
Table 12: Ablation on imagination horizon H.
H (W/SIG)
WebShop
SciWorld
Game-of-24
Fallback (%)
Overhead (ms)
2/2/2
59.8
35.3
28.8
15
9.0
3/4/3
60.7
36.1
29.3
18
10.5
4/5/4
61.3
36.5
29.7
19
11.4
5/6/4 (Default)
61.8
36.7
29.7
20
12.0
6/7/5
61.5
36.8
29.6
22
13.2
7/8/6
60.9
36.5
29.1
25
14.5
Table 13: Ablation on branches M (per decision).
M (W/S/G)
WebShop
SciWorld
Game-of-24
Fallback (%)
Overhead (ms)
4 //6
61.1
36.0
29.1
24
9.5
8 / 10 12 (Default)
61.8
36.7
29.7
20
12.0
12 14
62.0
36.9
29.8
19
14.8
16/ 18 20
62.1
37.0
29.9
18
17.0
24 24 / 24
62.1
37.0
30.0
18
22.5
Analysis of Table 14 3-0 (no risk penalty) increases mis-selections and hurts success. Moderate penalties (3R00.3) balance value and uncertainty best. Very high /3 becomes overly cautious, trimming success despite lower fallback.
Analysis of Table 15 Stricter gates (small r) defer too often; very loose gates (large T) accept risky branches, A mid-range threshold (7=0.22) yields the best overall success with manageable fallback.
Takeaways. (i) Capacity helps until d 256 for structure-heavy tasks; simpler arithmetic prefers smaller latents. (ii) Planning depth should match task horizon (default H 5/6/4). (iii) M [8, 16] captures most gains at low cost. (iv) Risk penalty 3 and gate both matter: removing either lowers success. The default settings reproduce your Table baseline exactly 661.8/36.7/29.7, 12.0 ms).
H ROBUSTNESS UNDER DISTRIBUTION SHIFT
We evaluate the quality and robustness the latent world model and the full DreamPhase loop under controlled shifts that affect structure, content, goals, and instructions. All models are trained on in-domain data only. At test time we apply the safety gate from Section with a threshold 7 chosen on held-out validation split to target 75 percent coverage. We report model fit (next-observation NLL), uncertainty calibration (ECE 10 bins, AUROC for bad-rollout detection), and decision-level impact (fallback rate, estimated mis- vii--gatng rate p, success change, interactions per episode).
Shift families and interventions.
WebShop, Layout-Swap: randomize sibling order in DOM subtrees, alter container nesting depth, perturb CSS classes and inline styles while preserving text content.
WebShop, Theme-Swap: replace the site theme and color tokens, change font and spacing scales, keep DOM tree shape.
WebShop, Content-OOD. replace brand and attribute vocab with unseen tokens the same slots, keep layout.
SeiWorld, Unseen graphs: hold out task graphs that combine new step chains and goals.
SciWorld, Tool mismatch: hold out tool combinations and affordance pairs not seen during training.
Instruction paraphrase: paraphrase i with back translation and synonym swaps, keep the environment unchanged.
23

---

Published as a conference paper at ICLR 2026
Table 14: Ablation on risk penalty 3.
3 (W/S/G)
WebShop
SciWorld
Gamc-of-24
Fallback (%)
Overhead (ms)
0.00 0.00 1..00
60.7
35.8
29.0
24
12.0
0.20 0.20 0.20
61.4
36.4
29.5
22
12.0
0.35 0.30 0.25 (Default)
61.8
36.7
29.7
20
12.0
0.50 / 0.50 0.50
61.5
36.5
29.6
18
12.0
0.80 / 0.80 0.80
60.9
36.0
29.2
16
12.0
Table 15: Ablation on confidence gate.
T (W/S/G)
WebShop
SciWorld
Game-of-24
Fallback (%)
Overhead (ms)
0.15 0.10/0.10/0.10
61.0
36.3
29.4
28
12.0
61.6
36.6
29.6
23
12.0
0.20 / 0.25 / 0.15 (Default)
61.8
36.7
29.7
20
12.0
0.25 /..30/0.2
61.7
36.6
29.7
17
12.0
0.30/ / 0.35 0.25
61.4
36.4
29.5
15
12.0
Metrics. Model quality is next-step NLL E++1 from Po h2, at, 1). Calibration ECE between predictive entropy and rollout error, and AUROC for classifying bad imagined steps. Decision impact includes fallback rate (fraction of steps where u(i") 7 so the agent defers to real interaction), mis- mis-gating gating rate p defined in Appendix A success change relative to in-domain, and interactions per episode.
Table Model quality and calibration under shift. Arrows indicate preferred direction.
Domain (shift)
NLL (in - shift)
ECE J (in shift)
AUROC bad-rollout
WebShop (Layout-Swap)
1.02 + 1.27
0.06 0.10
0.88
WebShop (Theme-Swap Swap)
1.00 + 1.15
0.06 + 0.09
0.89
WebShop (Content-OOD)
0.98 + 1.18
0.07 + 0.11
0.85
SciWorld (Unseen graphs)
0.94 + 1.09
0.05 + 0.08
0.86
SciWorld (Tool mismatch)
0.97 + 1.14
0.05 - 0.09
0.87
Instruction paraphrase
0.95 - 1.03
0.05 0.06
0.91
Findings. (I) Structural and content shifts increase NLL and ECE, but AUROC stays above 0.85, so the gate can identify risky rollouts. (2) With the gate, fallback increases on shifted inputs and success drops by one to two points in most cases. Without the gate, mis- mis-gating P rises sharply and success drops by four to seven points. (3) Interactions rise with the gate because the agent defers to real steps under uncertainty, which matches the design goal of safe degradation. paraphrases have small effect, suggesting that the model primarily relies environment structure.
(4) Instruction
Calibration protocol. We calibrate т on a validation split by maximizing area under the risk- coverage curve. We report AUROC using the binary label that rollout step exceeds reconstruction error threshold (estimated from in-domain validation). ECE uses 10 equhl-width bins over predictive entropy.
Ablations. We vary the horizon H E 3,2,3,45) and the risk weight E {0, 0.5. 1.0). Longer horizons increase NLL and ECE under shift, while 8 > recovers part of the loss by weightigg uncertain branches. We also compare MC-dropout with small 3-head transition ensemble for uncertainty. The ensemble slightly improves AUROC by to points at the cost extra compute.
Connection to theory. The observed changes in and NLL are consistent with the regret terms in Appendix A When shift increases and p. DreamPhase degrades gracefully by falling back to real interaction, which keeps the linear term small in practice.
Reflection length and template choices. As discussed in Section 3.5
the reflection head R generates compact textual summary that is injected into the frozen policy LLM. In all experiments
24

---

Published as a conference paper at ICLR 2026
Table 17: Decision-level effects under shift. Success is task success rate points, Interactions is real environment steps per episode.
With gate
No gate
Domain (shift)
Fallback f
p
Success (pts) 1
Fallback f
p
Success (pts) L
WehShop (I sayout-Swap)
28%
0.06
-1.8
3%
0.21
-5.9
WebShop (Theme-Swap)
19%
2%
0.16
-3.8
WebShop (Content-OOn)
25%
0.07
-2.2
3%
0.23
-6.7
SciWorld (Unseen graphs)
22%
0.05
-1.3
2%
0.18
-4.4
SciWorld (Tool mismatch)
24%
0.05
-1.7
3%
0.19
-5.1
Instruction paraphrase
12%
0.03
-0.6
1%
0.09
-1.7
Table 18: Interactions per episode under shift, mean standard error.
Domain (shift)
With gate
No gate
WehShop (Layout-Swap)
23.4
20.9 + 0.5
WebShop (Theme- Swap)
22.1 + 0.5
20.6 = 0,5
WebShop (Content-OOD)
23.8 0.7
21.0 0.5
SciWorld (Unseen graphs)
25.7 0.8
23.5 + 0.7
SciWorld (Tool mismatch)
2.7
24.0 = 0.7
Instruction paraphrase
21.5
20.4 + 0.5
we cap the reflection length to 30 tokens using a comaiin-onnittoned lexicon and prompt template. which encourages concise and stable phrasing while keeping marginal inference cost small. We find that this configuration provides a favorable trade-off: extending the budget beyond 30 tokens increases reflection injection rate but yields diminishing returns in downstream success, whereas using substantially shorter budgets leads to fewer injections and lower task completion rates. Importantly, R& trained offline logged training-split trajectories (using value-head scores as supervision), so its training procedure is isolated from test-time distributions and its behavior remains robust under moderate distribution shift.
I LANGUAGE BASED REFLECTIONS, PROCESS AND TASK SPECIFIC ADAPTATION
This section details how DreamPhase generates and uses language reflections to guide the frozen policy across different tasks. Reflections are produced only when the uncertainty gate accepts an imagined branch, and they injected short textual hints in the next policy query.
When a reflection is produced. After imagination and scoring (Sections selected branch and assume the safety gate accepts it. We then generate:
3.3 and 3.4p let j be the
(1)) = Ro((, Hr, 2 a1iitt a GU", (00)))))
(0)).
Sp(t, Cr, 2t #1tl+t a
(0)
where o; the reflection and s; compact summary, and ς S H. We use single reflection per real step. If the gate rejects all branches, no reflection is produced.
How reflections are produced. A lightweight decoder R. generates single sentence with most 30 tokens. The decoder conditions on a domain tag and uses a small task specific lexicon so that the frozen policy parses the hint naturally. The sentence follows a simple grammar, which keeps outputs concise and actionable:
[verb] [entity or target] [optional constraint or risk].
Examples of verbs search, filter, click, heat, mix, multiply. Entities are slot filled from the imagined branch, for example product name, reagent, or numbers in a puzzle. Constraints describe success conditions risks observed in imagination.
25

---

Published as a conference paper at ICLR 2026
Where reflections inserted. At step build the prompt for the frozen policy #LLM by concatenating the real history and the reflection components:
a, TTLM( (, St,, [REFLECTION] e |/REFLECTION] [
[SUMMARY] [/SUMMARY])
We use explicit delimiters so that the policy separates the reflection from the raw observation and instruction. No model weights are updated.
Task specific styles. We tailor the reflection style with domain tags and small templates that select appropriate verbs and entities.
WebShop (navigation and purchase). Imperative, includes budget or attribute constraints.
Example: "Add the USB C hub to cart, keep total under budget and ensure at least two ports."
SciWorld (procedural lab tasks). Procedural, includes duration safety checks. Example: "Heat the mixture for two minutes to dissolve the precipitate, then measure pH near seven."
Game-of-24 (arithmetic planning). Declarative, points to the decisive operation. Example: "Multiply eight and three to reach twenty four, puzzle solved."
Cross task decoding process. The decoder chooses verbs from a domain conditioned list, chooses entities from the imagined trajectory jr (for example page element names, reagent names, numbers), and optionally appends constraint that was decisive in the value score GO²" or flagged as a risk by high per step uncertainty. This keeps reflections short and aligned with the imagined plan.
Ablation, removing reflections but keeping imagination. We measure the change in task success when the reflection block is removed while the rest of DreamPhase remains unchanged. The imagined plan is still selected and executed, only the reflection and summary are not inserted into the prompt.
Table 19: Effect of reflections success. Removing the reflection block lowers performance by 1 to
3 points across tasks.
Task
Typical reflection style (example)
Success when removed
WebShop SciWorld
"Adding the USB hub cart keeps under budget and satisfies port requirement." "Heating for two minutes should rasse pH to seven and dissolve the precipitate." "Multiply eight and three to reach tweaty four, puzzle solved."
-3.1 pts
-2.4 pts
Game-of-24
-1.7 pts
Additional evidence. We also track the reflection injection rate (fraction of steps where the gate accepted and a reflection was formed) and near duplicate suppression. Injection rate across tasks between 55 percent and 72 percent, and we suppress reflection if it repeats previous sentence with more than 80 percent n gram overlap. With suppression disabled, success unchanged within statistical error, but prompts are longer.
Failure modes and safeguards. If the imagined branch is not confident, no reflection is injected. If the imagined branch contradicts the current page, the safety gate often defers, which disables reflection automatically. Reflections never include low level CSS identifiers or coordinates, they reference only human readable entities and constraints.
Reproducibility. We release the exact templates and slot filling rules for RS and Sy, the token budget per reflection, and the delimiter scheme used in prompts, so that results can be replicated. The same decoding configuration used for all backbones.
COMPUTATION AND LATENCY OF LATENT ROLLOUTS
We quantify the computational overhead of imagination relative to a single policy forward pass and compare token level search baseline.
26

---

Published as a conference paper at ICLR 2026
Design choices that keep cost low. (1) Latent only rollouts. We roll out with the transition go- No tokens are decoded during branch generation or scoring. (2) Deferred decoding. We decode observations only for the selected branch and only for the first steps needed by the reflection head R. and summarizer Sq- (4) Batching across branches. We evaluate all M branches in a single batched call per step GPU.
Wall clock on A000, batch 1. On A100 80 GB with bfloat16, single forward pass of an billion parameter Llama policy takes about 75 ms. One complete imagination cycle with M=32 branches and #= steps adds about 9 ms. The extra cost therefore much smaller than the policy evaluation itself.
Table 20: Per step latency on A100 80 GB, batch
Component
Token level search (ARMAP MCTS)
DreamPhase (latent imagination)
Policy LLM forward pass Search or imagination Total
75 ms
75 ms
RJ 180 ms
R 9 ms
Ry 255 ms
84 ms
Scaling with M and H. We report measured imagination overhead for different settings, keeping the policy constant.
Table 21: Imagination overhead T imugine (ms) versus branches M and horizon H on A100 80 GB, batch
M
H
3
5
8
1.6
2.3
3.0
3.7
16
3.1
4.6
5.9
7.4
32
6.0
7.6
9.0
11.3
64
11.7
14.8
17.9
21.8
With our default M Rs 10 and H Rs 5 the added latency is under 10 percent of one policy step, and the search component runs about four times faster than token level MCTS baseline.
Complexity of the full step. Let TELM be the policy forward time and Tgate be the time compute uncertainty and value for all branches. Since both heads operate on latents, gate is ((M H) with small constant.
Tatp TELM + Timagine
T gate
In setup T gate < 2 steps of the selected branch are decoded.
for M-32, #=4. Decoding for reflection adds under 2 ms since only
Latency budget and adaptive planning. We optionally enforce a target budget B ms per step by reducing M or H when the gate indicates high uncertainty. A simple rule M min M, [ THm with a estimated from Table keeps latency bounded without harming success.
Takeaway. Latent imagination deliberately lightweight. The world model is small, rollouts are latent only, decoding is deferred and limited steps for the selected branch, and all branches batched. In our setting the imagination cost is an order of magnitude lower than the policy evaluation and much lower than token level tree search.
K OFFLINE COST
Table 22 reports the offline compute used to train the latent world model, value head, and reflec- tion/summarizer modules, well as generate preference-style labels with Llama-3-70B- 7 All runs use single A A100-80GB GPU in bf16. Dollar costs assume $2 per A100-80GB GPU-hour (typical academic cloud pricing); actual rates will vary by provider.
27

---

Published as a conference paper at ICLR 2026
Table 22: Offline compute and cost of DreamPhase's modeling pipeline. World-model, value, and reflection modules are lightweight models trained on logged trajectories from the training split (Table Table 10), hence their GPU footprint small compared to single pass of large LLM. Preference labels for Vo are generated once with Llama-3-70B-Inss Insstruuttt and reused across all backbones in Table 2.
Component
Domains covered
GPU-hoous (A100-8008))
Approx, cost (USD)
Latent world model fa, go
WehShop, WorknceWorkd, ALFWorld, Game-of-24
S12
Value head
WebShop. ScienceWorid. Game-of-21
Reflection head Rg summarizer Sy Preference taheling (1 (Llama-3-708 Inarcct
WehShop, ScienceWorld, Game-of-24 WehShop, ScienceWorld, Game-of-24
24
848
Total offtine modeling
reported domains
Table 23: Logged training data and latent world model (LWM) compute. We report the number of offline trajectories collected on the training split for each domain, the approximate number of tokens processed after DFS DOM tokenization, and the corresponding A100 GPU-hours for training the latent world model fe, ge, dg.
Domain
Logged trajectories
Tokens processed (approx.)
LWM GPU--ours (A 100-80GB)
WebShop
2,000
6.5M
2.0
World
1,200
4.2M
1.5
ALFWorld
1,800
2.9M
1.0
Game-of-24
5,000
1.8M
1.5
Total
10,000
~ 15.4M
6.0
In addition, Table 23 reports the volume of logged trajectories used to train the latent world model on each domain, together with the approximate token count processed during training. As shown, the latent world model is trained on modest number of trajectories drawn exclusively from the training split, resulting in offline data budget that is small compared typical LLM pretraining corpora.
28

---

Published as a conference paper at ICLR 2026
L ADDITIONAL QUALITATIVE EXAMPLES
We present further qualitative examples of our DreamPhase framework in this section. These examples demonstrate the ability of the automatically learned reward model to distinguish successful trajectories from suboptimal ones and guide the agent toward better behavior through internal simulation.
Qualitative Example: Peb-Navigation Task
Task Instruction "Reserve table for two at La Piazza Friday at 7 pm." Baseline (ReAct-only) Trajectory
1. Observation: Home page of Di Denaay com.
Thought: "Search for the restaurant."
Action: type("I.a Piazza") in search har.
2. Observation: Result list page, first hit is La Piazza. Thought: "Open the first result."
Action: : click[result IJ.
Observation: : Calendar widget opens; default date today. Thought: "Choose 19:00 slot."
Action: : click[time
4. Observation: "No availability" message. Thought: "Maybe try 18:30." Action:
click[time 18:30].
5. Observation: Still unavailable. Agent loops three more times, then times out.
DreamPhase Trajectory
(Steps 1 1-2 identical to baseline, omitted for breviry.)
3. Internal Imagination: World model rolls out five branches, varying date and time. Best branch proposes Friday 19:00 "Available." Uncertainty thresshod, branch accepted. Reflection injected: "Availability appears on Friday pm; proceed with that slot."
4. Action: open date picker seleci[Friday). Observation: Friday chosen, 19:00 slot shows "Available."
Action: : click/time_ 19:00] Observation: Reservation details page.
6. Action:
click[confirm] Observation: Confirmation message with booking ID.
Outcome: Baseline fails (no booking); DreamPhase succeeds in environment interactions, using only one date/time query thanks to offline imagination.
Figure I:
DreamPhase uses latent imagination to test alternative date-time combinations offline, then exccutes only the high-confidence sequence, reducing real interactions and avoiding dead-ends
Training Data Example (Weh Environment)
Task Instruction: Find a Bluetooth speaker under $30 with 4Ω impedance and 6W power, suitable for pairing with a soundbar.
Positive Example:
Thought: should search for "bluctooth 6W ohm speaker under dollars."
Action: type(bluetooth 6W ohm speaker under 30) Observation: Search results show several products. The first result matches the impedance and power requirements, priced at $27.99.
Thought: This looks like good match. should click on the first result.
Action: clickffirst result] Observation: Product page confirms specs: 6W, Ω Jluctooth-cnabled, $27.99. Thought: This product matches the task. Proceed to purchase.
Action: click[buy now]
Negative Example:
Thought: Let me just click on random top-rated item.
Action: ttopkttodd product]
Observation: The item priced $42.00 and has 8Ω impedance.
Thought: This product does not meet the task constraints, but proceed anyway. Action: click[buy now]
29

---

Published as a conference paper at ICLR 2026
Training Data Example: Game-of-24
Task Instruction: You generating data for the Game-of-24, The task is use four given numbers and basic arithmetic operations (+, x, +) reach the target number 24. Each number must be used exactly once, only two numbers may be combined each step. Rules:
Each of the four input numbers must be between 1 and 13 (inclusive).
There must be exactly three operations.
Intermediate results must remain valid real numbers.
Input:
6
3
8
Positive Example:
Step
(remaining: 2,8,
Step 2: 2+ 1 =
(remaining: 8)
Step 3: x 8 24
(remaining: 24)
Answer: (66+3) + 1) x 8 24
Negative Example:
Step I:
(remaining: -2,
Step 2: 2 1 = -1
(remaining: 1,
Step 3: = 3
(remaining: -3)
Answer: ((6 8) 1) = -3 (Incorrect target)
Instruction Generation Prompt for SciWorld.
Scenario: are an intelligent assistant helping a human user conduct scientific experiments simulated environment consisting several rooms (c.g., kitchen, workshop, bathroom, greenhouse, etc.). You can interact with the environment using structured actions such as open [cbject], mix [item]] [item2], heat [item]. and others.
You will be provided with dialogue log hetween the and the assistant. The assistant reasons through task and takes actions, while the user returns ohservations about the environment. Your goal is to reconstruct the overall task description based on the full interaction.
Example Dialogue:
User: [initial prompt]
Assistant: Thought: believe need to locate and combine two substances that produce green paint.
Action: look around
User: Observation: You are in the bedroom. You see bookshelf, desk, and paint tray.
Assistant: Thought: should move the art studio find relevant materials.
Action: go to art studio
User: Observation: You are now the art studio. You see yellow paint and blue paint.
Assistant: Thought: Mixing yellow and blue paint will yield green paint.
Action: mix yellow paint blue paint
User: Observation: Green paint created.
Assistant: Thought: Now that the paint is ready, should finalize the task.
Action: focus green paint
Generated Task Description: Use available materials synthesize green paint. Once it is ready, focus on the green paint to complete the task.
Figure 2: Instruction generation prompt for training reward model in SciWorld.
Positive Trajectory Synthesis Prompt for SeiWorld
Task Instruction:
are helpful assistant conducting scientific experiments in a virtual environment. The environment includes several rooms: kitchen, foundry, workshop, bathroom, outside, living room, hedroom, greenhouse, studio, and hallway.
Available actions include:
open [object :: open a container
30

---

Published as a conference paper at ICLR 2026
Instruction Refinement Prompt for SciWorld
Scenario: You are an assistant a scientific simulation environment consisting of rooms such as the kitchen, foundry, workshop, bathroom, and greenhouse. You can interact using actions like open [object] go to [room], or mix [item]] [etem]]]
You are provided with a brief task description and complete execution trajectory that fulfillls it. Your task is to refine the instruction by incorporating procedural and contextual details grounded in the trajectory.
Example Input:
Original Task Description: Your task is to grow an apple. You can find seeds in the kitchen. You should focus the grown apple.
Trajectory Snippet:
look around
Observation: This room called the hallway.
open door to kitchen
Observation: The door is already open.
go to kitchen
Observation: You enter the kitchen.
find seeds, move to greenhouse, plant seeds, water, wait, focus on grown apple
Refined Task Description: Your task is to grow an apple, which involves locating seeds in the kitchen, planting them in the greenhouse, and nurturing them through watering. Once the plant bears fruit, focus the grown apple complete the task.
Figure Instruction refinement prompt used for reward model training in SeiWorld.
go to [room]: move between rooms
mix [item]] [item]]: : combine materials
(more actions omitted for brevity) You must first generate task description based this world. Examples include:
Your task is to use chemistry to create green paint. When you are done, focus on the green paint.
Your task to determine whether tall plant height a dominant or recessive trait in the pea plant. If the trait is dominant, focus on the red box. If the trait is recessive, fecessive,focus on the green box.
Once the task is proposed, navigate through the environment complete the instruction and synthesize the corresponding trajectory.
Example:
Task Description:
Your task is to use chemistry to create green paint. When you are done, focus on the green paint.
Trajectory:
Thought: 1 need to find materials to create green paint, likely located the art studio.
Action: look around
Observation: This is the bedroom. No useful matcrials here.
Action: go to art studio
Observation: You see yellow paint, blue paint, and brushes.
Thought: Mixing yellow and blue paint will produce green.
Action: mix yellow paint and blue paint
Observation: Green paint created.
Action: focus on green paint
Generated Trajectory: (as shown above)
31

---

Published as a conference paper at ICLR 2026
Figure 4: Positive Trajectory Synthesis Prompt for Sci SciWorl.
Negative Trajectory Generation Template
This prompt supports the construction of contrastive learning examples for interactive scientific vironment. The assistant agent operates within set rooms (c.g.. kitchen, workshop, greenhouse, art studio, hallway) using structured action commands such as go to [location], open [object], and focus on [item].
The goal is to produce negative trajectory: one that is contextually plausible but does not satisfy the assigned task. This trajectory should remain valid with respect environment constraints but fail complete the intended objective.
Illustrative Example
Task Description: Your objective is observe the developmental phases of an opple plant, starting from early growth to full maturity. The relevant specimens are found outside.
Correct Trajectory:
look around
Observation: You are in the hallway. open door to outside
Observation: The door is already open. go to outside
Observation: You see seedlings, saplings, and fully grown apple plant.
focus seedling, focus sapling, focus mature plant
Negative Trajectory:
look around
Observation: You are in the hallway. open door to kitchen
Observation: The door is already open. go to kitchen
Observation: You see fruit bowl and an oven. focus apple
While the agent performs syntactically valid of actions, the final state does not fulfill the objective of observing plant stages outdoors. Such negative examples used to improve the discriminative power of the learned reward model.
Figure Template for generating semantically incorrect yet valid trajectories in the SciWorld environment.
32

---

Published as a conference paper at ICLR 2026
Evaluation Prompt for Language Model SciWorld Reward Estimator
Task:
You an intelligent evaluation module responsible for assessing how well agent trajectory satisfies target instruction in scientific simulation environment.
You will receive:
(i) a natural language description of the intended goal. (ii) the full sequence of interactions taken by the agent to complete that goal (i.e., trajectory), and (19i) an observation history. Your task is to assign scalar reward score between 0 and that reflects how well the behavior aligns with the objective.
The environment contains multiple rooms (e.g. kitchen, greenhouse, art studio, foundry, etc.), and supports structured actions such as te teleport to [location], pick up [object]] mix [item]] [etem,] and focus on [object]].
Example 1
Goal: Move non-living object the purple box in the living room focus on it.
Trajectory (partial):
Thought: need locate non-living object; let's explore the current room.
Action: look around
Observation: In the bathroom, there is glass cup.
Thought: The glass cup is non-living object. should focus on it.
Action: focus on glass cup
Thought: Now should move it the target location.
Action: move glass cup to purple box
Observation: known action matches that input.
Final Score: 0.750
(Partial completion; attempted goal but incorrect action syntax)
Now evaluate the following new instance in the same format.
Figure 6: Prompt format used train language models act as reward evaluators for SciWorld. The model sees task goals and interaction histories and assigns scalar completion scores.
Qualitative Comparison: SciWorld
Task Instruction: Your task is to identify a non-living object. First, focus on it. Then, move it to the orange box located in the living room.
Trajectory A (Rejected):
Observation: The agent teleports to the living room.
Thought: should look around to locate non-living item suitable for the task. Action: look around
Observation: Objects include chair, couch with white pillow, finger painting, and orange box. Thought: need to examine one of these objects to determine qualifies. Action: examine white pillow
Model Verdict: Rejected - action sequence does not lead to task completion.
Trajectory (Accepted):
Observation: The agent is the workshop, where multiple objects such as hattery, wires, and light hulbs are present.
Thought: The battery qualifies non-living object. Action: focus battery
Observation: Focused the battery.
Thought: need to transport to the orange box in the living room. Action: teleport to living room
33

---

Published as a conference paper at ICLR 2026
Observation: The agent is in the living room
Thought: Locate the orange box place the battery inside. Action: look around
Observation: The orange box visible next a table and a couch. Thought: should complete the task by placing the battery into the box. Action: move battery to orange box
Model Verdiet: Accepted trajectory aligns with the instruction objective.
Figure 7: Qualitative comparison of trajectory rollouts in the SciWorld environment. The top trajectory (A) fails to complete the task due to incorrect object selection. In contrast, trajectory (B) successfully identifies and transports valid non- non-living object to the target location. The reward model correctly identifies the superior trajectory without human annotations.
Qualitative Rollouts for Game-of of of-24
Input 3, 6,
Input B: 2,
Trajectory Al (Selected): Steps: 4 8 12 (left: 3, 6, 12)
Trajectory B1 (Selected):
Steps: 12/3= (left: 7)
6 (derived implicitly from remaining
7-2=5 (left: 4, 5)
steps)
=0 (off 4)
= (invalid intermediate)
However, correct path found as: (7 (3 + 1)) 24 Final Answer: 3) + (12 =)
Final combination: (6 * +4+ 8)) 24
Final Answer: (6 * (4 /3 24
DreamPhase Selected
DreamPhase Selected
Trajectory B2 (Rejected): Steps: 7+3== 10
Trajectory A2 (Rejected):
(left: 12, 10)
Steps: 8 6= 2 (left: 3, 4, 2)
22-2=10 10
(left: 10)
4+2=6 (left: 3, 6)
10 + 10=20 (left: 20)
3+6=9 (left: 9)
Final Answer: (((7 3) (1)- =0
Final Answer: (((8 6) 4) 3) 9
DreamPhase Rejected
DreamPhase Rejected
Figure 8: Novel qualitative examples in the Game-of-24 environment. Our DreamPhase successfully identifies the correct arithmetic path among plausible distractors, verifying its ability to reason through multi-step numerical logic.
34

---

Published as a conference paper at ICLR 2026
Table 24: Prompt details for DreamPhase (WehShop).
System Prompt:
You are web navigation agent operating WebShop. The environment provides structured observation the current page (title, URL, and DOM tokens). You must solve the goal using ONLY the following actions: CLICK (<text or selector>), TYPE (<text> into <selector>), SELECT <<option> from <selector>), SCROLL (<up I down>), ) NAVIGATE (<url>). SUBMIT (<selector>), EXIT. Never invent elements that are the observation. Do not submit forms or checkout unless the goal explicitly requires it. Every round you receive an observation (and, when available, a planner reflection). Respond the format:
Thought: (brief reasoning based only the observation and reflection) Action: (one action from the allowed set)
Planner Augmentation (if present):
[REFLECTION] Short hint distilled from imagined rollouts highlighting the safest, highest- value path. [/REFLECTTON
[SUMMARY] One-sentence sketch of the selected branch (entities. filters, next click). [/SUMMARY]
Instruction:
Buy USB-C hub with 22 ports under $40. Prefer official store pages. Return the product page URL after adding the item to cart.
Goal:
Example (round input):
TITLE: MegaShop Electronics URL: https:/megaahhop.com
OBS (truncated)
inputtsearch. box, button:Search, link:USB-C Accessories, fil-
ter:Price,ffltee.PPort
[REFLECTION] Go "USB-C Accessories," filter Price $40 and Ports 2; avoid pages without "Checkout." [/REFLECTION]
[SUMMARY] Open accessories - apply price/ports filters open first in-stock result.
[/SUMMARY]
Expected Response:
Thought: The reflection suggests opening "USB-C Accessories" before filtering. The button
"USB-C Accessories" is visible. Next, will click it.
Action: CLICK ( ("USB-C Accesscries")
35

---

Published as a conference paper at ICLR 2026
Table 25: Prompt details for DreamPhase (SciWorld).
System Prompt:
You are an agent in SciWorld. Each round you receive an observation and must issue one action that safely progresses the task. Use ONLY the following actions (with placeholders OBJ, LOC): open/ close OBJ, de/activate OBJ, connect OBJ to CBJ, disconnect OBJ, use OBJ [on OBJI, look around, look at OBJ, look in OBJ, read OBJ, move OBJ to OBJ, pick up OBJ, put down OBJ, pour OBJ into OBJ, dunk OBJ into OBJ, mix OBJ, go to LOC, focus on OBJ, wait, waitl, task, inventory.
Never invent objects not mentioned the observation. Avoid irreversible actions (e.g., heat- ing/disposing) unless the goal requires it safety confirmation is present. Respond exactly in the format:
Thought: (brief reasoning grounded in the observation and any planner reflection) Action: (one action from the allowed set)
Planner Augmentation (if present):
(REFLECTION] Short hint distilled from latent imagination, emphasizing the safest high-value step accepted by the uncertainty gate. [/REFLECTT] TON]
[SUMMARY] One-sentence sketch of the selected branch (key objects, preconditions, next action). [/SUMMARY]
Instruction:
Goal: Neutralize the acidic solution to pH 7, then place the neutral solution on the lab bench. Return the final container name.
Example (round input):
ROOM: chemistry lab LOCATIONS: fume hood (closed), bench (clear), sink
OBJECTS VISIBLEI beaker(A) [label: acidic solution], beaker(B) [empty]. bottle(weak base) [cap closed], pH paper. pH meter [off], trash bin
INVENTORY: gloves
[REFLECTION] Open base; transfer small amount into beaker(A); test with pH paper; repeat in small increments until near 7; avoid pouring into sink. [/REFLECTION] [SUMMARY] Open bottle(weak base) + pour small amount into beaker(A) - read pH. [/SUMMARY]
Expected Response:
Thought: The safest next step is to prepare the reagent. The bottle is closed; opening it is required before any transfer.
Action: open bottle le (weak base)
36

---

Published as a conference paper at ICLR 2026
Table 26: Prompt details for DreamPhase (BabyAl).
System Prompt:
You are agent in BabyAl grid world. Each round you receive observation and must issue one action that safely progresses the goal. Use ONLY the following actions (macros allowed where indicated): turn left, turn right, move forward, toggle, go through <door> <id> (door must be open), toggle and go through <door> <id> (opens if closed/locked; needs matching key locked), go to <obj> <id> (macro), pick up <obj> <id> (macro). Never invent objects not present in the observation. Avoid opening locked doors without the correct key unless the instruction demands it. Respond exactly in the format:
Thought: (brief reasoning grounded in the observation and any planner reflection) (one action from the allowed set)
Action:
Planner Augmentation (if present):
(REFLEcTION Short hint distilled from latent imagination, emphasizing the safest high-value step accepted by the uncertainty gate. //REFLECTION)
[SUMMARY] One-sentence sketch of the selected branch (orientation, waypoints, next action). [/SUMMARY]
Instruction:
Goal: go to the red ball 1. Return SUCCESS once you are adjacent and facing it. Example (round input):
YOU: position (x=3.y=2), facing north ROOM: walled 7x7
VISIBLE OBJECTS: red ball 1 (two steps ahead, one step left), grey box 1 (one ahead, one left), red key 1 (three ahead, one right), blue door (east wall, closed)
INVENTORY: empty
AVAILABLE ACTTONS: [turn left, turn right,
forward, go to red ball 1, pick up red ball 1, toggle, go through blue door 2]
[REFLECTION] Tum left face the red ball; move forward twice; then go to red ball 1 to align precisely. | [/REFLECTION]
[SUMMARY] turn left move forward move forward go red ball 1. Expected Response:
Thought: The red ball is ahead-left; first rotate to face it before advancing.
Action: turn left
37

---

Published as a conference paper at ICLR 2026
Table 27: Prompt details for DreamPhase (MAZE).
System Prompt:
You are an expert maze solver. Each round you receive the goal coordinates, your current position, and local wall layout. Use ONLY: move up, move down, move left, move right. Coordinates are (x. y) with down increasing æ and right increasing y. Never propose moves blocked by walls. Prefer shortest safe paths and avoid loops. Respond exactly in the format:
Thought: (brief reasoning grounded in the observation and any planner reflection) Action: (one action from the allowed set)
Planner Augmentation (if present):
(REFLECTION] Short hint distilled from latent imagination, emphasizing the safest high-value path accepted by the uncertainty gate. //REFLLECTOON
[SUMMARY] One-sentence sketch of the selected branch (waypoints/urrss) [/ [/SMMARY)
Instruction:
Start game and return your action and thought in the format above. Make the optimal move given the current state.
Example (round input):
GOAL: (8, 6) POSITION: (1, WALLS AROUND YOU: left, up, down
[REFLECTION] Advance right along row 1 until column 6, then proceed down to 8; avoid downward moves at 11, due to a wall. [/REFLECC) LECTION]
[SUMMARY] right + right
right (to y = 6), then down to x 8.
Expected Response:
Thought: Down and left blocked; the shortest path begins by moving right along the top row toward 6.
Action: move right
38

---

Published as a conference paper at ICLR 2026
Table 28: Prompt details for DreamPhase (Wordle).
System Prompt:
You are an expert Wordle solver. The hidden word has 5 letters and you have attempts. On each turn, output one guess as a space-separated sequence of letters (e.g., s guess, the environment returns feedback over letters using b (absent), y (present but wrong position), and g (correct position). Use only valid words from the vocabulary. Update constraints from feedback and propose the next guess. Respond exactly in the format:
e). After each
Thought: (brief reasoning grounded in the observation and any planner reflection) Guess: (1 t t r s with spaces)
Planner Augmentation (if present):
[REFLECTION] Short hint distilled from latent imagination, emphasizing a teigh-ccoverage guess consistent with constraints; accepted only if the uncertainty gate passes. [/REF L LECTION]
[SUMMARY] One-sentence sketch of the best branch (fixed positions, excluded letters, candidate pattern). [/SUMMARY]
Instruction: Start new game. Return your thought and guess in the format above. Guesses must be valid 5-letter words in the vocabulary.
Example (round input):
HISTORY:
Guess:
s
Feedback:
b b g (only the final is correct
place)
2) Guess:
c
1 0
Feedback: bgb b (1 fixed at position 2; fixed
position 5)
CONSTRAINTS: Pattern 1__ e; exclude (s. h, c. o).
[REFLECTION] Keep 1 at position 2 and e at position 5; avoid excluded letters; prefer guess that covers diverse consonants and a vowel like u. [/REFLECTION]
[SUMMARY] Try pattern I__
candidate:
k
Expected Response:
Thought: The constraints require 1 with position 2 and at 5; f 1 u k fits and introduces u, k.
Guess: f 1 u k
39

---

Published as a conference paper at ICLR 2026
Table 29: Prompt details for DreamPhase (BIRD/ Text-to-SQL).
System Prompt:
You are a read-only SQL assistant for a SQLite database. Given schema description and a natural- -anuuaee question, explain your reasoning briefly and then output one SQL statement that answers the question.
Rules: (i) Output exactly two fields: Thought: and Action: (ii) The Action must single-line SQL query in Markdown code format. (iii) Do not modify data (no INSERT/UPE TUEDAAE/DEEEEEEEEEEEEEEEEEEEEEEE use SELECT only. (iv) Quote identifiers with spaces using double quotes (e.g., "gas station id"). (v) Use JOINs/aggregations as needed.
Planner Augmentation (if present):
(REFLECTION] Short hint distilled from latent imagination (e.g. keys, filters, grouping). accepted only if the uncertainty gate passes. [/REFLECTION] [SUMMARY] One-sentence sketch of the selected branch (tables to join, predicates, target aggregate). !/SUMMARY]
Instruction (schema excerpt):
catpttt__rrdsssccaiiiiiiie www.sspeiiiiigsse..eaeeessssssee Tables: Database: debit_ car__ard___s___
customers (customerid PK, client client_seqment currency)
gasstations ("gas station id" PK, "chain id", country, chain_segment)
products (product PK, description)
transactions (krrassaction PK, date, time, "customer id", "card id", "gas station id", "product id", amount, price) yearmonth ("customer id", date, consumption, PK P""customer id", date), ("customer id") + customers (customerid))
Question:
Among the transactions made at gas stations the Czech Republic, how many occurred after
2012-01-01?
Example (round input):
[REFLECTION] Join transactions with gasstations on "gas station id"; filter country= Czech Republic' and date > 22122001001; return a count.
[/REFLECTION]
[SUMMARY] transactions_le x gasstations + WHERE count Czech Republic' date>' 2012-01-
COUNT(*).
Expected Response:
Thought: We need count over filtered transactions; join on "gas station id" access the country field; compare ISO date strings.
Action: SELECT COUNT (*) AS num_tx FROM transactions.. 1
JOIN gasstations g ON t. "gaa station id" - g. "gas station id"
WHERE g. country Czech Republic' AND t.date '2012-01-0'';
40


Reviews:
[{"review_id": "kBTX7TGPoV", "reviewer": "Reviewer_No6m", "summary": "This paper is about leveraging imagined trajectories to enhance the performance of LLMs on multi-turn interactive tasks. More specifically, the authors propose DreamPhase, a framework that interleaves phases of interaction with phases of imagination. During imagination phases, the model carries out its exploration through simulating multiple possible future trajectories of the interaction. Then, based on these imagined trajectories, the proposed system generates textual advice to be concatenated to the current context before asking the frozen LLM to take the next action in the actual environment. In order to generate imagined trajectories, the authors propose to train a separate latent world model to simulate the environment dynamics and some reward function. On web, science and embodied tasks, the authors empirically show the proposed approach is more sample and cost-efficient in terms of API calls and safer (by avoiding irreversible mistakes) while also improving performance.", "questions": "- How was the data used to trained the latent world models collected?\n- What type of models was used for the latent world model in the experiments? For instance in Table 2, is the backdone only for the agent or both?\n- What are the logged data, used to trained the value estimation, exactly?\n- What is the frequency of falling back to the real-history policy?\n- What is the rational behind reporting the unweighted average across the different environments in table 1 and 2? (nb: on line 375, the text says \"tasks\" instead of \"environments\").", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- While using latent world model for doing exploration is not novel per se, the proposed approach decoupled the interaction policy from the latent world model model. This allows to leverage existing SOTA LLMs by steering their behavior using text summary of high-valued imagined trajectories.\n- Once the latent world model is trained, the approach is LLM-agnostic since the LLM used for interaction is frozen.\n- Assuming we have a thorough latent world model, exploration in imagined space is safer since mistakes made during imagination do not have real consequences. It is also more cost effective in terms of API calls, i.e. not wasting calls to explore dead ends.\n- The empirical evaluation is comprehensive with many ablation studies to validate the approach.", "weaknesses": "- From the main paper, is it unclear what data was used for training the latent world model. Also, in Table 1, is it a fair comparison with other baselines, did they leverage training data as well?\n- Unclear how the authors validated the quality of their trained latent world model? They argue that using LLM to act as a simulator is prone to hallucinations, what about the proposed approach? Is there any metrics/ablations to validate this?\n- If I understood correctly, the uncertainty estimation requires a bounded set of actions (line 249)? If so, how were those actions obtained for environments with combinatorial action space?\n\n#### Minor\n- Citations are \"inline\" instead of being enclosed with parenthesis.\n- In equation 2, I think the two $p_\\theta$ should be replaced by $g_\\theta$ and $d_\\theta$.\n- For conciseness, I'd suggest spelling out CE, KL and TV in the text.\n- p.4: in the \"Offline imagination\" paragraph, $p_\\theta$ should be replaced by $g_\\theta$.\n- In algo 1, line 8: $p_\\theta$ should be replaced by $g_\\theta$\n- In algo 2, line 11 and 12: what is \"\\kappa\"? I assume it represents the number of steps in that imagined rollouts?\n- I believe SciWorld and ScienceWorld are referring to the same work, yet in the paper they have different name and different citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pmm80A8zWI", "reviewer": "Reviewer_M9UY", "summary": "This paper presents DreamPhase, a modular framework for improving large-language-model (LLM) agents through offline imagination. The core idea is to let agents simulate and evaluate possible futures in a learned latent world model instead of interacting with the real environment at every step. Each imagined trajectory is scored by an uncertainty-aware value function, and only high-confidence branches are executed. The best imagined plan is distilled into a short natural-language reflection that is fed back into the frozen LLM policy, steering behavior without any fine-tuning.\n\nThe authors provide a theoretical regret bound linking decision quality to model error and mis-gating rate, and show strong empirical results across web, science, and embodied-task benchmarks. DreamPhase reduces real environment calls by roughly 4× on WebShop (≈10 vs. 40 per episode) and lowers latency while matching or surpassing ARMAP and AgentGym baselines. It consistently improves sample efficiency and safety across multiple backbones (LLaMA-3, Mistral, Phi).\n\nOverall, I believe the paper offers a clear and well-motivated step toward safe and cost-efficient planning for LLM agents, combining latent imagination, uncertainty-aware filtering, and natural-language self-reflection.", "questions": "My question is already presented above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper proposes an offline imagination framework that decouples policy reasoning from environment dynamics via a learned latent world model. Generally speaking, integrating model-based imagination, uncertainty-aware value gating, and natural-language reflections into LLM agents is novel and clearly motivated by limitations of existing online-search or imitation-based methods.\n\nThe following is my assessment of the strengths: \n1. Methodological quality. The framework is carefully designed and well-supported by both analysis and implementation details. The latent world model enables safe, counterfactual rollouts; the mutual-information–based uncertainty metric allows principled risk filtering; and the reflection and summarization modules steer a frozen LLM without fine-tuning. The theoretical regret bound $O(T\\varepsilon + B\\rho T)$ connects decision quality to model accuracy and mis-gating, providing formal grounding rarely seen in this line of work.\n\n2. Empirical strength. Extensive experiments across eight benchmarks and four model backbones (LLaMA-3-70B/8B, Mistral-7B, Phi-3-8B) show consistent improvements in success rate, sample efficiency, and safety. On WebShop, DreamPhase cuts real API calls from ~40 to <10 per episode (≈4× fewer) and reduces irreversible actions by ≈5× while matching or exceeding ARMAP performance.\n\n3. Clarity and presentation. The paper is pretty clearly written, with intuitive motivation, structured algorithms, and readable pseudocode. The modular presentation and ablations make the design easy to follow and reproduce.\n\n4. Significance. By showing that LLM agents can plan safely and efficiently through internal imagination—without additional data collection or fine-tuning—the paper offers a promising path toward scalable, low-cost autonomous systems.", "weaknesses": "While the paper presents a well-motivated and impactful framework for improving the safety and efficiency of LLM agents, several aspects could be further clarified or strengthened:\n\n1. I'm wondering how much the framework depends on latent world model accuracy and low-stochastic environments. DreamPhase’s effectiveness relies on the learned latent world model being a reliable simulator. However, most evaluated environments—WebShop, ScienceWorld, ALFWorld, and Game-of-24—are deterministic or near-deterministic: a given action (e.g., clicking a DOM element, executing a command) leads to a predictable next state with little exogenous noise. This makes short-horizon prediction tractable but may overstate the model’s robustness. In more stochastic settings—such as multi-agent interaction, physical robotics, or open-ended tool use—latent rollouts would likely accumulate compounding error, weakening both the value estimation and the safety gate. It would be interesting to test out the framework in these real-world scenarios, given the paper's claim on uncertainty-aware mechanism and scalability of the method. \n\n2. Limited generalization scope. Evaluation focuses on web and text-based tasks with structured, language-aligned states. It remains to be seen whether the approach transfers to perceptual or continuous domains, where state representations are high-dimensional and reflections may be less interpretable, e.g., stock market.\n\n3. In general, I would assume the low latency largely relies on the scale of the underlying world model when maintaining a satisfying prediction accuracy. For this reason, it would be beneficial to understand how this method scales to larger application scenarios in real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FA9n1CaoPf", "reviewer": "Reviewer_MwpV", "summary": "This paper introduces DreamPhase, an \"imagination‑first\" planning framework for LLM agents that (i) trains a compact latent world model to simulate short‑horizon futures offline, (ii) rolls out multiple imagined branches from a frozen policy LLM, (iii) scores branches with a value head penalized by an epistemic uncertainty proxy, and (iv) injects a short natural‑language reflection distilled from the best low‑risk branch back into the next policy prompt --- without fine‑tuning the LLM. Empirically, DreamPhase reduces real environment interactions on WebShop while slightly improving success versus a strong token‑level search baseline (ARMAP‑M), with lower per‑step latency and fewer executed irreversible actions; results are reported across diverse tasks and multiple backbones with ablations and a regret‑style bound.", "questions": "1. Please report the number of trajectories used to train the LWM per domain, total tokens processed, training wall‑clock/GPU‑hours, and value‑head preference‑generation costs. A normalized \"success per training FLOP\" or \"runtime savings vs pretraining cost\" comparison to ARMAP‑M would be informative.\n\n2. Any preliminary results on Mind2Web or WebArena using the same setup and logger? If not feasible, can you discuss expected bottlenecks and what would be required to scale DreamPhase to those environments?\n\n3. The paper mentions a 30‑token budget and domain‑conditioned lexicon/templates. How is $R_{\\phi}$ trained, and how does its choice affect injection rate and downstream success?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The high level latent idea is interesting and the implementation cleanly separates environment modeling (latent world model), decision‑time search (latent rollouts), risk control (uncertainty gating), and behavior shaping (language reflections). This makes the approach applicable to API‑restricted foundation models because the LLM policy remains frozen.\n\n2. On WebShop with Llama‑8B, DreamPhase uses 9.3±0.4 API calls/episode vs 39.8±1.1 for ARMAP‑M, improves success 61.8±0.6 vs 60.2±0.6, and reduces per‑step latency.\n\n3. In an open‑source, matched‑backbone block (Llama‑2‑7B), DreamPhase attains the best average across 8 tasks (50.1 vs 42.3 for ARMAP‑M). Cross‑backbone results (Llama‑3‑70B/8B, Mistral‑7B, Phi‑3‑8B) typically place DreamPhase best or second‑best on WebShop, ScienceWorld (seen/unseen), and Game‑of‑24.", "weaknesses": "1. The LWM uses a small MLP over DOM tokens obtained by DFS traversal with layout features. While efficient, this choice may under‑model structural, hierarchical, and long‑range dependencies present in realistic web UIs; no comparison is shown against stronger sequence/graph models (e.g., Transformers or tree‑aware encoders) that could better respect DOM constraints. How much of DreamPhase's gains persist if the dynamics are more complex than those captured by an MLP?\n\n2. Epistemic uncertainty is estimated via MC‑dropout on the frozen policy LLM. In common LLM serving stacks, dropout is disabled at inference and can be tricky to implement with quantization/kv‑caching\n\n3. DreamPhase adds an offline modeling phase (world model + value head + reflection/summarizer components, with preference pairs produced by a large open model). The paper focuses on runtime savings but provides limited accounting of training compute and data vs baselines that do not require a learned LWM. A small \"dollars/GPU‑hours\" table would clarify trade‑offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)