Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Authors: Xiao Yu
Baolin Peng
Michel Galley
Hao Cheng
Qianhui Wu
Janardhan Kulkarni
Suman Nath
Zhou Yu
Jianfeng Gao
Venue: ICLR 2026 Poster (2026)
Keywords: AI agents
reasoning
planning
reinforcement learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon, interactive tasks such as web navigation and computer/phone-use. Inspired by literature on human cognition, we argue that current AI agents need ``vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting - in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning. In stage 1, we introduce Reasoning with Simulations (ReSim), which trains the agent to generate structured reasoning traces from expanded search trees built from real experience gathered through environment interactions. ReSim thus grounds the agent's reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its reasoning. In stage 2, we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the agent's simulation and decision-making ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one realistic benchmark (AndroidWorld) demonstrate that (1) ReSim effectively infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning-intensive tasks. Together, these results highlight the central role of simulation in enabling AI agents to reason, plan, and act more effectively in the ever more challenging environments.

Published as a conference paper at ICLR 2026
MINA-MIND: LEARNING TO SIMULATE FROM EXPERIENCE FOR BETTER AI AGENTS
Xiao Yu': Baolin Penga Michel Galley?, Hao Cheng?, Qianhui Wu? Janardhan Kulkarni Suman Nath", Zhou Yu', Jianfeng Gao² 'Columbia University, NY Microsoft Research, Redmond (xy2437, 2y2461) co umbia iimia.... aaboia..edi edu (baolinpeng, jfgao) @microsoft.com coo
ABSTRACT
Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon interactive tasks such as web navigation and computer/phone-use. Inspired by literature we argue that current Al agents need "vicarious trial and error" "rror"-the capacity mentally simulate alternative futures before acting in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents integrate such simulation into their reasoning. In stage I, we introduce Reasoning with Simulations (RESIM), which trains the agent generate structured reasoming traces from expanded search trees built from real experience gathered through environment interactions. RESIM thus grounds the agent's reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its reasoning. In stage we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the agent's s simulation and decision-makinn ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one realistic benchmark (AndroidWorld) demonstrate that (1) RESIM effectively infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning- intensive tasks. Together, these results highlight the central role of simulation in enabling Al agents to reason, plan, and act more effectively in the ever more challenging environments.
human cognition,
INTRODUCTION
Recent advances in language models have unlocked impressive reasoning capabilities in domains such mathematics and programming (Shao al. 2024 Jimenez et 2024). However, many emerging applications unfold in complex environments that require multi-step reasoning, such as web navigation (Zhou et al. 2024b Deng et al. 2023), deep research (Gou al. [025a Du et al. 2025), and computer/phone-ute tasks (Xie et al. 2024 Rawles et al. (022)). Success in these domains depends not only on the ability to decompose goals and reflect on past progress, but also on Al agents' ability to construct accurate world models that capture the structure and dynamics of increasingly complex environments (Shao et al., 2024 Jimenez al. 2024).
Insights from human cognition indicate why such ability to model and simulate complex environments critical. Neuroscience research (Tolman 1948) Daw et al., 22055 Daw & Dayan 2014 Bennett 2023) highlights the emergence of the neocortex mammals to engage "vicarious trial and error": mentally simulating possible futures, evaluating their consequences, and selecting advantageous actions without directly experiencing each option. This ability greatly enhanced adaptability and becision-making, which we argue is equally essential for reasoning in long- mogghorizonn AI agent tasks.
turning point in intelligence, enabling early
Work done during internship Microsoft Research; Project Lead

---

Published as a conference paper at ICLR 2026
DeepSeek- R1 performance
Sokobon
V-Q
ALFWorld
m Coooon Saar (0-10000))))))) (0-100M
V=
Smidaiion Score
0.6
ky inpoonmentt
V-Q
0.4
V-
A2P0
0.24
Let's
Stage 1: ReSim Training
Stage 2: Gyna-GRPO
(a) Simulation ability v.S. performance
(b) Dyna-
We find the performance of strong reasoning models is heavily affected by its ability to simulate in different environments (left). We introduce Dyna-Mind, a two-stage training framework to integrate and improve simulation ability of AI agents (right).
Figure 1:
Empirical evidence supports this view. In Figure la we observe that while strong reasoning models such as DeepSeek-Ro RI can simulate and solve structured environments like Sokoban, their performance drops sharply in more complex domains such as ALFWorld-borl in simulation accuracy and overall task success (also see [Section 4.1.2p. Initial attempts to address this limitation, such as Dyna- Think (Yu et al.) 2025b), integrate simulation into reasoning through distilling simplified traces and adding auxiliary next-state prediction tasks. However, these methods rely on the strong capability of reasoning models to directly generate synthetic simulation data, which can embed errors and biases.
To overcome this limitation, we present Dyna-Mind, an improved two-stage training framework to teach (V)LM agents to simulate the environment by directly learning from real experiences. In stage 1 training, we propose Reasoning with Simulations (RESIM) to algorithmically construct reasoning traces using expanded search trees obtained from real environment interactions, and train a policy model using these reasoning traces. In stage 2 training, we further improve the policy and its simulation ability using online reinforcement learning (RL.). We introduce Dyna-GRPO, novel algorithm that utilizes both outcome rewards and intermediate states from rollouts to improve the simulation ability of the policy. Extensive experiments on two widely used synthetic benchmarks (Sokoban and ALFWorld) and one realistic benchmark ( (AndroidWorld) show the effectiveness of each stage of the framework. Our results indicate that (1) RESIM'S reasoning traces effectively teach Al agents to simulate; and (2) Dyna-GRPO, by leveraging both outcome rewards and intermediate interactions, learns better policies for long-horizon, planning-intent tasks. These findings highlight the importance of world simulation ability for reasoning in
tasks.
2 RELATED WORK
(V)LM as decision making agents The use of (visual) language models as autonomous agents has been explored in a wide range of applications such as interactive game playing (Wang et al. Fong al.) 2025), computer, phone, and browser uses (Xie al. 2024| Zhou et al. 2024b Rawles et al. 2025), software engincering (Jimenez et al., 20244] Yang et al. 2024), and more. Early works include reactive agents (Yao et al.] 2023b) that directly prompts an (V)LM to make decisions on immediate observations without simulation or planning approaches, hindering performance on
complex Long-horizon tasks. Recent advances include: (1) search-bases methods (Yao et al., 2023a
Zhou et al., 2024a Koh et al.) 2024) Yu et al. 2023 2025a) that augments (V)LM agents with algorithms such BFS, DFS, and MCTS; and (2) hierarchical, multi-agent methods (Zheng et al. 2024; Agashe et al. 2024 2025 Liu et al., 2025 Gou et al., 2025b) that orchestrate multiple specialized agents to complete long-horizon tasks. While these methods show improvements, they often introduce substantial overheads during inference, such as requiring additional interactions with the environments or designing complex heuristics to orchestrate multiple agents. We focus on enhancing a single (V)LM agent by integrating simulation into its reasoning via training.
Training (V)LM agents Early methods in training (V)LM agents mostly rely on supervised learning (SFT) with human annotations data synthesized by otate-of-the-art (reasoning) models
(Zeng et al. 2023 Chen et al., 2024 Zhang et al. 2024 Xu et al. 2025). Recently, many methods
such Feng et al. 2025 Wang et al. 2025b); Wei et al. 2005a leverage reinforcement learning
"Code available wttptttpp... /tttpa///gthub..... htwpiiichuccocoojaaaaaa wwwm///ssnyu//lyynn...
nd
2

---

Published as a conference paper at ICLR 2026
(RL) with verifiable rewards to directly train agents to complete tasks by prompting them to reason before taking actions, following the success of DeepSeck-R1 (DeepSeek-AL et al. (2025a). However, it remains unclear whether extensive reasoning is necessary for all scenarios (Shojaee et al. 2025), and what aspects of such reasoning is essential for long-horizon tasks (Yu et al. (025b). In this work, specialize in integrating and improving the simulation ability of (V)LM agents during reasoning. and show that planning with world simulation is crucial for long-horizon tasks.
World models and Dyna algorithms Beyond task completion, real-world interaction data contains rich information that can be used to help decision making. Early examples include Dyna algorithms (Sutton 1991), which combine model-based and model-free methods to efficiently learn optimal policies. Given set of real-world rollout data, Dyna (1) separately train world model using these rollouts; (2) perform additional simulated rollouts with the world model; and (3) update the policy using both real and simulated rollouts. Applications of world model training have been explored in work such as Chae et al. (2025); Gu et al.] (2025), facilitating search algorithms such as MCTS to improve performance; and applications of Dyna include Deep Dyna-Q Q (Peng al.. 2018), Switch- DDQ (Wu et al. 2018), and more (Zou et al.) 2020. Yu et al.] (2255)). However, these approaches either result modular systems (a separate policy and world model) or require accessing state-of-the-arr reasoning models (e.g.. DeepSeek-RII Our work does not rely strong reasoning models, and focuses on integrating and improving simulation as part of an agent's reasoning process.
3 DYNA-MIND
Research in human cognition (Daw et al. 2005 Daw & Dayan 2014| Bennet 2023) as well as in games like chess, go, and othello (Schrittwieser et al. 2020: Li al.) 2024 Nanda et al. 2023 Chae ctt al. 2025) suggests that strong agents implicitly store and use a (compressed) representation of the world to enhance their preision-making. This perspective highlights two key questions in existing approaches to improve (V)LM agents for long-horizon tasks: (1) how to synergize world simulations with reasoning; and (2) how to improve the simulation ability help improve the policy.
To address these questions, we introduce Dyna-Mind, two-stage training framework to teach (V)LM agents to plan with simulations during their reasoning and improve their task performance. We detail these two training stages next in Section 3.2 and Section 3.3 respectively.
3.1 NOTATION
Completing tasks in complex, realistic environments is typically formulated as a Markov Decision Process of (S, A. T, R). In the generic setting of multi-step tasks, an agent ma receives an instruction and observation from the environment ~ S at time step 1, generates an action de ~ To(-(s)), and transitions to the next state 81+1 ~ T(se, at). This process repeated until the task is completed or until reaching a maximum number of steps, upon which terminal reward rT ~ R(ST, @T) is provided based on whether the task is completed successfully or not. In the context of simple text games such as Sokoban Schrader (2018), state 8, can represent the complete game state, and an action ar is one of "left", "right", "up". "down" " (after some reasoning process). In more complex environments such as Android World (Rawles et al. 2025). state 8; is the current screenshot of the android device, and an action a, can be "tapping on a coordinate (x.y)", "swiping up", "swiping down", etc. We note that since aim to train agents to generate simulations within their reasoning process, any text that represents simulation is always part of the response az [] Any variant of the symbol represents real states from environment interactions, unless explicitly stated otherwise.
3.2 REASONING WITH SIMULATIONS (RESIM)
To enable an agent simulate during its reasoning, first construct imitation learning data where the reasoning process consists of explicitly planning with simulations. Different from prior work
Technically, any input the agent from environments is an observation (as in POMDP) instead of state. However, to simplify notation we used to generally denote the agent's input from the environment.
As action plan/final action are always extracted from model response, we use a (by slight abuse of notation) to denote either the full response or the extracted executable action. Distinctions are made clear in context. Example model response for each benchmark is provided in Table A4 Table A5 and angure A
3

---

Published as a conference paper at ICLR 2026
ReSim Data Collection/inferenee
(estil((ReSim)
ReSim
ReSim
(b
80
$
RoSim 4
ReSim d
ReSim
$0
combine into
(VLL
4g
high
curinioCurentlly we are at Sul Lot's plan:
(N)
11
Maybe we can try di Let's think. This
8/2
This lead to $1 Alter that, perhape of Value: Altematively, maybe nij .... Value: en
lead to ii After that, maybe di o.. Value: o Maytoo can try a5? Let's
a
ReSSm
ReSim
aj
ReSim
4/4
m
So, the best plan og, di... bocause..... <plans oie di diplano
+
cactions oj clactions
ReSim
!1)
(3)
ST
(N)
low value
astimatto valuo search tree
2
simulation guided reasoning
N trajectories
Figure 2: RESIM integrates simulation into reasoning (a, (aSm) by using expanded search trees built through real environment interactions (left). RESIM then trains an agent to directly generate such simulation-guided reasoning trace of ReSim without any algorithm support (right).
such Yu al. (0225)) that leverages superior LLMs such DeepSeek-RI which already shows world modeling capability in its reasoning traces (see Section 4.1.T for more details), we construct simulation-guics guidedd reasoning traces using search trees built from real environment interactions.
RESIM Data Collection To construct reasoning data with rich simulations. we leverage algorithms such as depth first search (DFS) to construct search trees based on environment interactions, and then use an (V)LM to aggregate the entire search free into single reasoning response a ReSin for later training. Specifically, given a state 8, RESIM first uses a rollout model wo to generate rollouts from up to depth d. This rollout model can be specialized/fnnttneed LLM (see (Sction 4.1) simply prompting a generic LLM (see Seeciioncinn 4.2). Then, RESIM uses a value function Ve to provide an estimate of the quality of each of the partial rollouts, where the V. can be implemented finetuned value model (see[Section 4.1p or using LLM-as-a-junn judgee (see Section 4.2). Finally, we use a generic (V)LM to aggregate all these rollouts and their values into single response a ReSim by prompting the (V)LM to 1) first independently summarize each partial rollout, which contains ground- truth future states info information from the environment; and 2) then aggregate all these summaries into a coherent response conditioned on the current state and previous h actions and states, and choose the best plan and the next immediate action for exccution. The final chosen action from a ItSim is then executed in the environment, and this process is repeated until the task is solved until a maximum number of steps reached. We illustrate this process in Figure Left and Alloriihhm 3, We note that since RESIM essentially converts real search trees into single reasoning trace, is not limited to (1) agent-environmen interactions; (2) specific search algorithms used in this work. We believe other domains such agent-user--nnvioomee applicable, which we leave for future work.
either
interactions other algorithms such as MCTS are also
RESIM Distillation Since each response a ReSim encapsulates an entire search tree in its reasoning, directly use a ReSim as the training target given input to teach the model to perform simulation- guided reasoning without any algorithm support. We illustrate this in Figure Right. Specifically, 8r,, a7 Resm produced by RESIM ReSim given the current state st
given collection of trajectories = (so, a ReSim 81, a inference, we use SFT to train the model to directly generate each a as well as a maximum history of previous actions and states in the trajectory (i.e., the same input used by other inference methods such as REACT).
ReSim
3.3 DYNA-GRPO
While RESIM provides principled way to synergize simulation with reasoning, it is computationally expensive and relies on multiple modules (a rollout model, a value function, and a (V)LM to aggregate the search tree into single response) to construct training data. Additionally, such offline training may limit models' generalization ability to new tasks. To address this, we propose DYNA-GRPO, a modification of GRPO (Shao al. 2024 to further improve the model's simulation ability during
4

---

Published as a conference paper at ICLR 2026
Policy Improvement f
Simulation Improvement
80
Group
Imean
Group
Comp
Group
Comp.
Legend:
fukure
Figure 3: DYNA-GRPO iterates between policy improvement (left) and world model improvement (right), optimized by GRPO. During policy improvement, we perform grouped policy rollouts with GRPO. During simulation improvement, we perform both policy rollouts and simulation refinement rollouts (see Figure 4) and trains the model directly generate an improved policy as well to better perform simulation refinement when provided with future-states information.
online RL without using any search or additional modules. The standard GRPO objective JaRPo is:
min (po (9) A((a)) clipp(p ( DxL To
GT
su(als) where po(a) neat s) is the importance sampling ratio, , is the KL regularization coefficient, and
AGRPO is the episode-level advantage function (Wang al. 2025b Feng et al.
2025):
A(o, = ())) R(((1)) mean([[[(=0)))]
T Σ t=1
R(7())
R(st, az),
where G' the group size, R(-) is the reward provided by the environment, with R(s,, a,) = -0.1 for non-terminal steps and R(sr, (sr,ar)= 10.0 or R(sr, ar) 0.0 for terminal steps when task succeeded or failed, respectively.
However, RL algorithms such as GRPO aim optimize policy only using scalar rewards Ry but do not provide any direct training signal on refining the reasoning process or world model simulations. We propose DYNA-GRPO
address this, by additionally incorporating future state(s) information S1+1, St+2. ... as textual signals to help improve the model's response a Ta (-1s;) during RL training. Since textual signals cannot be directly "optimized", we propose SIMROLLOUT instead prompt the underlying model to refine its simulation in To(-(s;) utilizing real future state(s) St+1, St+2. ... during RL rollouts. Then, during optimization we train the policy both directly generate the refined action and also improve its "simulation refinement" ability (DYNA-GRPO). We detail these two modifications below.
Rollout
SIMROLLOUT In simulation refinement rollout (SIMROLLOUT), at each state 8 we first sample response T-(ss));; then extract the final chosen plan {āz, āz,- âd} up to depth d from and execute them in the environment obtain ground truth next-states {5(41, $(42.... ... %aaai and finally prompt To again refine its re- sponse given these real future states a refine To s (s, a 4/4 02 +
refine refine ag
SimRollout
a refine
80
81
① add back
Inference
Go
plan 4ug
staa We illustrate this rollout process Fig ure and provide the pseudo-code in Algorithm 2 We note that this different from methods such
Figure 4: SIMROLLOUT generates refined action per state 8; using real environment interactions
Reflexion (Shinn et al. 2023). which performs reflection at the end of the episode utilizing suc- cess/failure information, and is also not intended for any training purposes. Empirically, we find the resulting refine indeed improves the policy's simulation and performance (see Appendix D.4).
*Since d is determined dynamically by the model, we capped it 5 ensure training stability.
5

---

Published as a conference paper at ICLR 2026
DYNA-GRPO Training To utilize refined tra- jectories from SIMROLLOUT during RL, we fol- low Dyna algorithms to improve the model's policy and simulation ability iteratively. Specif- ically, DYNA-GRPO iterates between (1) sim- ulation improvement where models learn from refined policies that use future states information from SIMROLLOUT to improve its simulation ability; and (2) direct policy improvement where models are trained on standard rollouts without future-stat state access, allowing to better integrate simulation ability into decision-making. We il- lustrate both training processes in Figure 3 and detail the overall algorithm in Algorithm
Algorithm DYNA-GRPO
Require: policy zo, environment 7, group size G Require: hyperparameeers G.N, Ke,,,
1:
for N training iterations do
2:
// simddation improvement
3:
for we steps do
4:
#see (Mgoritho 2
5:
(r)) (iciiee +- SimRollout(To. GG/2)
6:
{7) 4 Rollout(To. T./))
7:
Update Te with (RPO( (7) (r'))
8:
Update 5o with GRPO (T (Tatine) using Arefin
end for
10:
// policy improvement
11:
for steps do
12:
{+) + RRollou((To. T.G)
13:
Update Te with GRPO((T))
14:
end
end for
During simulation improvement, for each task we (1) first perform SIMROLLOUT with group size of G/2. collecting refined trajectories with and without future-state information removed: T' {sa, 40 afefine 81, .1 afeline ...) and Frefine = {xgeline aafrnn s1 .qetine, .afetine, ... }: (2) then perform standard rollouts with group size of G/2; (3) combine these standard rollouts т with refined trajectories 7' into a single group of size G and perform GRPO on this combined group; (4) finally utilize Treime to also improve the model's 's simulation refinement ability, using the following modified ad advantage reward refinements that both correctly solves the task and improves upon (the mean reward of) standard policy rollouts which does not access future states:
return
ArefineT (T refinee
f 1.0, if (i)
Trefine is correct and R(Trefime)
(i)
max R, Freline)
0.0, otherwise
G/2 R(7(1)) is the mean reward of the standard policy rollouts (line 6 of Algo where R G72 Li=l
rithm : Rrefine 6/2 G/2 R(T(() refine) is mean reward from SIMROLLOUT (line of Algorithm 1).
During policy improvement, we perform standard policy rollouts without future state information,
optimized by GRPO using episode-level advantage (Feng al. 2025 Wang et al. (0255).
4 EXPERIMENTS
We first evaluate Myna-Mind on two "synthetic" environments (Sokoban and ALFWorld) that require efficient planning for successful task completion. These lightweight environments allow us to provide detailed analysis of the different reasoning styles as well as different RL algorithms. Then, we extend our methods to a more complex and realistic environment (AndroidWorld).
4.1 TEXT GAMES
Benchmarks Sokoban (Schrader 2018) is a grid-world game where the agent needs to push boxes to target destinations while avoiding obstacles, and successful task completion requires spatial planning to avoid deadlock situations. ALFWorld (Shridhar et al. [021] is text-based embodied environment where the agent needs locatc/interact with objects to complete houschold tasks using natural language instructions. To evaluate the agent's s generalization ability, we construct training set, an in-distribution (ID) test set, and an ouitof-diitributtion (OOD) test set. For Sokoban, we use training set with 6x6 room layouts with I box and destination; ID test set with different 6x6 room layouts than training; and OOD test set with 8x8 room layouts with box and destination. For ALFWorld, we directly use the official training, ID, and OOD test splits from Shridhar et al. (2021).
Baselines setup To evaluate RESIM, compare against (1) ReACT based prompting methods with models such as GPT-4o OPpeAA 2024), Claude-3.7 (Anthropic 2025). DeepSeek- V3 (DeepSeek-Al et al. 2025b), and DeepSeek-RI RI DeepSeek-AL et al. 2025a); and (2) training methods that distill the reasoning traces from strong policy models such as DeepSeek-R.. To evaluate stage DYNA-GRR PORPO training, we compare against other popular group-based RL algorithms such RLOO (Kool et al.

---

Published as a conference paper at ICLR 2026
Table I: Performance on text game environments such as Sokoban and ALFWorld. "Gen. Token" denotes the average number of tokens generated per turn relative to that of Qwen2.5-7B-Instree All training in stage-1 and stage-2 are based on Owen2.5-7B-Insttuue All results are averaged over runs. Our methods are highlighted in gray.
Sokoban
ALFWorld
Method
Gen. Token
ID
OOD
AVG
ID
OOD
AVG
R RFACT(Qwen2.5-77 (-7-
1.0x
25.8
35.4
REACT(Qwen2.5-32B Instrrutt))
2.7x
36.7+41
36,2+13
REACT(GPT-40)
1.5x
37.8=10
51.3=21
BEACTT(Cladde-37-SSomm
2.3x
70.3
16.
REACT(DeepSeekkVV
2,5x
57,0=15
55.2+10
REACC(DeepSee-RRI
14.5x
96.62n3
62.5a05
RESIM
2.0x
96.4101
87.7.11
Dyna- Think
DID(RI)
24.2x
74.0+14
57.5=12
65.8-19
63.2+15
56.7+1a
58.9+13
Dyna-Mind Stage I (SFT)
DVSTILL(V3)
2.1x
49.211.
34.4213
41.8_11
58.911
56.711
57.8112
(ISTILL(RI)
24.0x
72.5+29
57.0=19
64.8-15
59.4+15
54.2+19
56.8+35
DISTILL
(SIM) (RESI)
2.0x
71.9211
55.5216
63.7=19
78.9
69.3g13
74.1
Dyna-Mind Stage 2 (RL)
(RETILL(RESIM)) RLOO
2.2x
78.1+15
65.1=13
71,3=09
85,9+13
85,4=20
85,5+20
D GRPO
2.1x
19.
67.82m6
73.1
87.0g
87.1
87.0k1a
DISTILL GYRA
DYNA-GRPO
1.9x
82.5
70.1-11
77.1-12
92.5
89.1
90.8
2024). Overall, also compare against Dyna-Think Yuu al. which similarly uses two-stage training (DIT and DDT) to improve model's 's simulation ability.
2019) and GRPO (Sha et al.
2025b
Dyna-Mind setup To instantiate RESIM, we use Qwen2.5-233 1ann2.5-32B-Imme Instrcct (Qwen et al. 2025 as rollout and value function models, finetuned on rollouts obtained by using DeepSeck-V. V3 (see Appendix D.3 for more details) and use DeepSeek-VS V3 V3 as the LLM to aggregate the search tree into single response. For Sokoban, we use
5, b 16, Darin 2; for ALFWorld, we use d = b,6 24, bunin = 4. We note that all models used by RESIM are by themselves much weaker than other models such as DeepSeek-RR as well as RESIM itself. Since DeepSeek-RI and RESIM have higher success rate than DeepSeek- V3, to isolate improvement from better reasoning from simply training with more (diverse) data, we thus only used trajectories where all methods correctly solved the task for stage training. This results in total of 207 trajectories in Sokoban and 200 trajectories in ALFWorld from each method (DeepSeek-R1, DeepSeek- V3, and RESIM) in the subsequent stage training.
To instantiate DYNA-GRPO, we continue training the best model from stage distillation. To ensure fair comparison, we use identical hyperparameters for all methods (RLOO, GRPO, and DYNA-GRPO), when applicable. For DYNA-GRPO, we use ny 10 and πス = 10 for Sokoban and nt 10 and n₂ 20 for ALFWorld. For more setup details, please see Appendix D.5
4.1.1 MAIN RESULTS
In the upper section of Table we first evaluate RESIM'S performance against other strong reasoning models such as DeepSeek-RT. Then, we compare different training methods to integrate/improve the simulation ability of the policy model. In Table we first find that RESIM achieves near- pear-perfect performance on Sokoban (96.4% success) and a strong performance on ALFWorld (87.7% success), significantly outperforming all other methods. On Sokoban, we find strong reasoning models such as DeepSeek-RR -R also achieves near-perfect performance, which we attribute to R1's ability to correctly simulate Sokoban game states (but not ALFWorld) during its reasoning process (see SScciio 1.2 for empirical results). In contrast, RESIM utilizes ground -trut simulations from search trees, and hence was able to achieve strong performance in both environments.
In stage training, we find DISTILL(RESIM, achieves a similar performance to DISTILL(RI) on Sokoban but significantly outperforms both DISTILL(V3) and DISTILL(RI) ALFWorld. Addition- ally, since RESIM constructs reasoning traces consists almost entirely of only planning via simulation (see Figure Left), DISTI.I. outputs / /xx less tokens average compared to (RS)
7

---

Published as a conference paper at ICLR 2026
Table Measuring simulation ability of different models across different training stages. We report the average success rate and the simulation ability (Sim Score E [0, 1)) averaged across all trajectories. We also report the correlation coefficient between the success rate and the simulation score.
Sokoban
ALFWorld
Method
Success
Sim Score
Success
Sim Score
CEAACT(QQen2.5-7BB I-I--ttttt
25. 11,8
0.2 -044)
35.4
0.18 -0.40
REACTT(eeppeekk-V.
57.0#16
0.54g =0an)
55.2
0,35 m0.68
REACTT(eepSeek-RI)
96.6102
0.93, =0.06)
62.5=as
0.36 =0.00
RESIM
96. 402
1.00
87.7
1.00
Dyna-Think Think
DIT(R ()DDT(())
74.0
0.62, =0.44
63.2
0.36 =076
Dyna-Mind Stage (SFT)
DISTILL(RI)
72.5+29
0.6le -0.75)
59,4+115
0.34g -0.7)
(RESIL
71.9+15
0.62 .0.7%)
78.9 #211
0.37 -0.74)
Dyna-Mind Stage 2 (RL)
DISTILL(RESIM)) GRPO
79.1
0.62 -0.55
87.0#32
0.38 -0.48)
DISTILL(RESIM) DYNA-GRPO
82.5
0.67
92.5 #0a
0.43 =0.55
These results indicate that that strong performance from RESIM can be learned by SFT, and that the ability to model and simulate the environment is crucial for long-horizon,
tasks.
In stage 2 training, we continue from the best model (DISTILL(RESIM)) with online RL. In Table we find that DYNA-GRPO improves upon both GRPO, RLOO, as well as Dyna-Think, while maintaining a similar output token length compared to its base model DISTILL(RESIM)) This indicates that DYNA- is effective at improving the model's simulation ability during online RL training (also see [Sectinn 4.1.2 for empirical results), and that improving such simulation ability helps improve task performance. For more detailed ablation study comparing the effect of search algorithm used by RESIM as well as DYNA-GRPO, please see Table Al and Appendix D.I
4.1.2 MEASURING SIMULATION ABILITY
Dyna-Mind aims to integrate and improve the simulation ability of agents. To measure this simulation ability, we evaluate the Simulation Score (Sim Score) of different models and the Spearman Correlation Coefficient (rs) between sim score and success rate. Given a state 8t and generated response ~ Te (-]s;). we evaluate the simulation score of a, by 1) first prompting an LLM to extract the final action plan (ā), . ... ,..) and the natural language description (i.c., simulation) of the corresponding imagined next-states ($t+1. $1+2, ... Setd) from the response a,; 2) then execute the action plan in the environment to obtain ground truth next-states {$1+1, St+2. ... sttd): ; 3) finally, prompt an LLM to judge (Zheng et al.) (023) the correctness of these simulated next-states à; by comparing them against the ground truth 8,, returning a score E [0, 1]. Finally, we averaged the score for each turn to obtain overall simulation score for the trajectory. To ensure a fair judgment, we used a different LLM from all of our experiments (Qwen3-22) 235B-A222 22B-Innttruct (Qwen Team 2025
For judgment prompts, please see Appendix D.6
We present the results in Table across both Sokoban and AorWorld due to its perfect simulation ability (by construction), whereas DeepSeek-RR R1 struggled in ALFWorld as it struggles to model the environment layout; and 2) both (RESILL(RESMM) and DYNA-GRPO improve the simulation ability alongside task performance compared to their baselines. These results show that our methods helped improve the simulation ability of the model beyond simply improving task performance.
Table 2 we find that 1) RESIM maintains its strong success rates
4.2
ANDROIDWORLD
Next, we extend our Dyna-Mind to AndroidWorld (Rawles et al. 2025) a highly challenging benchmark that evaluates the agent's ability control and complete tasks on virtual Android device.
Benchmarks AndroidWorld (Rawles et al. 2025) provides fully functional Android environment that requires the agent to interact with Android' 's GUI to complete tasks across 20 real-world Android apps. Since tasks in AndroidWorld are parameterized by task types (116), construct training
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance on AndroidWorld. All training are based on Qwen2.5-VL--7B322--mms We exclude Dyna-Think Think since (most) VLMs cannot predict images, as required by DDT(Î) training. "Gen. Token" denotes the average number of tokens generated per turn relative to that of Qwen2.5- 7B-Instruct. All results are averaged over runs. Our methods are highlighted in gray.
AndroidWorid
Method
Gen. Token
ID
OOD
AVG
REACT(GPT-40)
1.0x
5.1
1.0x
5,3+02
BEACCTQQwe2..5-V1-722--
1.1x
19,5204
RESIM
2.1x
34.4
Dyna-Mind Stage I (SFT)
00w.77BB00 -nnstruct
1.0x
13.1=04
8.6=02
10.8
DISTILL- DISTILL-7B(RESIM))
2.1x
21.1
10.2106
15.7
DISTILL-92B(RESIMM)
2.0x
32.8=04
15.6
24.2200
Dyna-Mind Stage 2 (RL.
DISTILL-92B(RESIM)) + GRPO
2.1x
35.3
20.3+06
27.8 8
DISTILL- (RBESI)) + DYNA-GRPO 1.9x
40.7
22.9
31.8+1.
with 81 task types with in total 1946 tasks, an ID test set with 128 different tasks from the same task types, and an OOD test set with 128 tasks from the remaining 35 held-out task types. We use maximum number of 15 steps and the screenshot-only modality input. We provide an example task and action in Appendix E.
Baselines setup Since our methods consider end-to-end training, we compare against models that are capable of directly generating executable actions given GUI screenshot, and exclude modular
systems such as Gou et al. (2025b): Agashe et al. (2025). We thus mainly compare against (1) REACT based prompting method with
VL--BBLL (Bai et al. 2025). and GPT-4o; and (2)
distillation from Dowe2.55VL-772.
To evaluate stage 2 DYNA-GRPO, we compare against GRPO
following Section 4.1 We exclude comparison against Dyna-Think in this experiment, because DDT(7) trains the model predict next-state (in this case, screenshot images), which cannot be implemented using most VLMs as they can only generate text.
Since AndroidWorld is highly challenging and compute-intensive environment (each episode average takes 15-20 minutes complete), do not perform any rollout/value function training for RESIM. Instead, we directly prompt Qwen2.5-VLL-72 VL-72B as the rollout model, prompt GPT-4o as a judge to approximate the value function, and also GPT-4o the VLM aggregate the rollouts into single response in RESIM. We use d = 1, 16, Irain = 4 for RESIM, and a total of 128 trajectories for distillation/stage training. To instantiate DYNA- OYNA-GRPO, generally followed the same recipe as SSeectinnn 4. but used less training steps (60) as AndroidWorld highly compute-intensive and time-consuming. For more details, please see Appendix E.2
Dyna-Mind setup
4.2.1
MAIN RESULTS
Results We present the results in Table In general, we observe similar results compared to Section 4.1.1 First, we find that RESTM inference significantly improves performance, and that the improved performance can transferred to Qwen2.5-VL-77 and 32B via (RESIL Next, in both training stages of Dyna-Mind, we find improved performance in both ID and OOD test sets compared to baselines, including awen2.5-VL-725 and even RESIM. These results highlight the effectiveness of method to improve agent's performance in complex environments.
Error Analysis Compared to synthetic text games (Section 4.1.1) where RESIM achieves near- perfect performance, we find RESIM struggles in Android despite improvements compared to baselines. After analyzing trajectories produced by RESIM, find performance is bottlenecked by the rollout model 20wen2.5-VL-7233 mainly due to: (1) incomplete understanding of some GUI
We were unable to reproduce the reported performance of more recent GUI models such as UI-Tars1.5 Qin 2025), and hence focus using Qwen2.5-VI. for simplicity. Please see Appendix F.3 for details.

---

Published as a conference paper at ICLR 2026
interfaces and certain button functions, and (2) inability to recover after making multiple mistakes. We believe methods to improve the foundation model's capability could mitigate these problems (Wang et al. 2025a
Qin et al. 2025), which we leave for future work.
CONCLUSION
In this work, we propose Dyna-Mind synergize reasoning with simulations for autonomous AI agents, We empirically show that an agent's ability to model and simulate the environment strongly correlates with its ability to correctly reason and complete long-horizon planning- intensive tasks. We introduce Dyna-Mind, two-stage training method to explicitly teach (V)LM agents to integrate and improve such simulation part of their reasoning. In stage training, we propose RESIM to train model to simulate future states by learning to predict an expanded search tree in their reasoning. In stage 2 training, we propose DYNA-GRPP to further refine the agent's ' reasoning and simulation ability using online RL. Empirical results three benchmarks show that (1) RESIM effectively teaches Al agents to simulate; and (2) DYNA-GRPO, by leveraging both outcome rewards and intermediate interactions, learns better policies for oong-horizon, planning-intension tasks.
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang. Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human, 2024. URL
0ts/2211008815
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: compositional generalist framework for computer use agents, 2025. URL https/
7/arxiv. http://v.rg/ab///////aoa/e 2504 00906
Anthropic. Claude 3.7 Sonnet and Claude Code. claude- 3-0--33-7------ 2025. Accessed: 2025-05-13.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https::/aaxxin org/ abtp 13923
M.S. Bennett. A Brief History of Intelligence: Evolution, AI, and the Five Breakthroughs That Made Our Brains. HarperCollins, 2023. ISBN 9780063286368. URL http::///ookk...ooppp
comm/books1ld-yyyyEAAAAAA
Hyungjoo Chae, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation, 2025. URL
2410. 13232
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models, 2024. URI
abw./////340 12881
Nathaniel D Daw and Peter Dayan. The algorithmic anatomy of model-based evaluation. Philosophi- cal Transactions of the Royal Society B: Biological Sciences, 369(1655):0010477,, 2014.
Nathaniel D. Daw, Yael Niv, and Peter Dayan. iincetaintyy-base competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8: 70144117 2005.
URL httpt/htt//aappp semanti www.aanttsscchcccooo ar org Cotp/s oowpchoooo....rroooooosssss 6385268
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan,
10

---

Published as a conference paper at ICLR 2026
Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du. Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, and et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025a. URL
org. /agg//bb///// 2501 12948.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Junlong Li, Junxiao Song, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, and et al. Deepseek-v v3 technical report, 2025b. URL httthttt//aaatt
org/ abs 2412.4111111 19437
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.
abs/ 2206/ 06070
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL 2506. 11763
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. 2025. Utt.ttps::/ htt.:::ttt///tttt //axiiv../vvvvaa//aa org/abs////eeooooroggggggg 2505. 10978
Boyu Gou, Zanming Huang. Yuting Ning. Yu Gu, Michael Lin, Weijian Qi. Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song. Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agent-as- auddee,, 2025a. URL httpt:ttp///tt..tt. waw.iiivivv.....v...aaaaaaaaa... org abs/
2256..2500
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025b. URL httthhtts///aiiii
org/ abs 2410.440 05243
Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xic, Peng Qi. Huan Sun, and Yu Su. Is your Ilm secretly a world model of the internet? model-base planning for web agents, 2025. URL ptttps://arxiv... www.iii.orrra.oo. abs/
2411. 06559
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: blmebased ontut- safeguard for human-ai conversations, 2023. URL https: http://aaxxvvv...go.. 2312.06674
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL
https::/ wwtp://////aa//a///oo// 06700
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, URL
11

---

Published as a conference paper at ICLR 2026
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get baseline for free!, 2019. URL
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on synthetic task, 2024. UtLpttts::
wttp 2210.22111... 13382
Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, and Fei Huang. Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pe, 2025, URL httpps://aaa/
orq/abs/ 2502 14282
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models, 2023. /tt//t/s////aarrrrrr//////22...0
OpenAl. New and improved content moderation tooling. https: https://ppenai.cooo http:///pppeaaa..mmmeeeee.
Accessed 2025-05-13.
2022. Accessed:
OpenAl. Hello GPT-4o. httpa///t/aa//////eaaac.///////a
2024. Accessed:
2024-09-28.
OpenAl. Introducing GPT-4.1 in the api. httpss//openaa..omm/nneex//// Accessed: 2025-09-17.
2025.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang- Yu Su. Deep dyna-q: Integrating planning for compleennnn dialogue policy learning, 2018. URLhttt
//arxiv. http:/bs///8000 06176
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang. Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL https: http://axxvv.oo.g.oo htt.://gg/b////00000 /2501 1 12326
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //tp:v/axivvvvo o abs/
Qwen Team. Qwen3 technical report, 2025. URL
Christopher Rawles, Sarah Clinckemailie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell- Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarkiin environment for autonomous agents, 2025. URL UKL/httesr//arrrr..rr//////22222
Max-Philiph B. Schrader.
gok-sokoban.
gym- sokoban 2018.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 (1839):6046009 December 2020. ISSN 4687. doi: 10. 1038/44 00/8//41186--00 1586-000--02 120-03051-4 URL httt:://xx.dd
doi wtw..ddi...oo..g.. www.o0.g/000...... 1038/118000 s 586 020- 02000-3051-4
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li. Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
2402. 03300
12

---

Published as a conference paper at ICLR 2026
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shumyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https: /ttp:///aaxvv... wttp://..rr///s//2333.ooo
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity, 2025. httphhttt/aaaattt wwwpxxxw.wrg/ab///22022 06941
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021.
URL htttttt/p//aaattt wttp://wwww...p//////........ 03768
Richard S. Sutton. Dyna, integrated architecture for learning, planning, and reacting. SIGART
Bull., 2(4):160-163, July 1991. ISSN 0163-5719. doi: 10.1 00.1144/122344112277, URL https
1tt/:/10..115/1220 122377
Edward Tolman.
Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar. Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
URL wttphhtps///attttt
6291
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu. Yuyan Wang. Jixuan Chen, Yuxiao Ye. Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang. Heng Wang, Diyi Yang, Victor Zhong, Flood Sung. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025a. URL httpt:ttt::/aaaa
org/ / abs 2500 09123.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang. Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Goutlieb Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in abs/ I1m agents via multi-turn reinforcement learning, 2025b. URL httts://arxxiv/
2504 20073
Yuxiang Wei, Olivier Duchenne, Jade Copet. Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing Ilm reasoning via reinforcement learning on open software evolution, 2025a. /ttp:sttt/////rx.v.ooo
2502. 18449
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. Webagent-rl:: Training web agents via end- to-end multi-turn reinforeement learning, 2025b. ///
16421
Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. Switch-based active deep dyna-q: Efficient adaptive planning for task-completion dialogue policy learning, 2018. URL
https://arxiv: http///rxxvv..g/abb....
www./org//....oooooooo wtt.oo////1111... 07550
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024.
2404. 07972
Yiheng Xu, Zekun Wang. Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2025.
URL http::tp//// //tp:////vvvvvvi
org/ /.ds/2411. 04454
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: aggent-computer interfaces enable automated software engineering, 2024. URL htt.hhttss://aee wwtp://wwwwwwwwa/x///eeeooae/ 15793
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a.
[tLttss:///azxx org. attp//bb.222.... 10601]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https://axxiii
hog/aba/2210. 03629
Xiao Yu, Maximillian Chen, and Zhou Yu. Prompt-based monteccaroo tree search for goal-oriented dialogue policy planning, 2023. URL httppsss/aaxx...rr//////2......
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mets and exploratory learning, 2025a. URL
https: wwwaaaiiivvv... org/ abs /ww.///////. 2410.11000. 02052
Xiao Yu, Baolin Peng. Ruize Xu, Michel Galley, Hao Cheng, Suman Nath, Jianfeng Gao, and Zhou
Yu. Dyna-think: Synergizing reasoning, acting, and world model simulation in ai agents, 2025b.
URL httlsttss:///xii http://///xxxvvv.o/aaassse/////s org/ abt///bs////// 00320
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang. Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for Ilms, 2023. URL
wwtp://///aaaaaaaa.a/ /2310
12823
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models empower agent systems, 2024. URL htttpt:::////xx
www. abs 2409. 03215
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded, 2024.
abs/ 1ttp://....r./ba/2.......1.. 01614
Lianmin Zheng, Win-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi I. Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging alm-as-a-judge with mt-bench and chatbot arena, 2023. URL
abs/233 05685
Andy Zhou, Kai Yan, Michal Shlapentokh-Rode -otthema,hhRRRhhman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024a. URL https
waariivorrgaaaaa.. abtp///s./2221..../ 04406
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo. Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024b. URL 2307 13854
Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yur Yun Nie, and Dawei Yin. Pseudo dyna-q: A reinforcement learning framework for interactive recommendation. In Proceedings the 13th International Conference on Web Search and Data Mining, WSDM '20, pp. 816-824, New York, NY, USA, 2020. Association for Computing Machinery. ISBN URL https:://dol.org///1100
9781450368223,
doi: 10. 10.1145/3336191.3377 33711800.
3336191 3336191.337180 3371801
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE
This work used LLMs as general-purpose writing assistants to improve the grammar and clarity of the paper. We did not use LLMs generale any research ideas, automate experiments, analyze results.
B ETHICS STATEMENT
Generally, while most methods and models are not designed for unethical usage, there often potential for abuse in their applications. Autonomous Al agents can be used for variety of tasks such as automating information gathering, software development, computer/phone-use and more. In this work, we proposed our Dyna-Mind framework to enhance the simulation ability and hence performance Al agents. However, since AI agents are fundamentally task-agnostic, it is possible to use them for unethical tasks such scamming disseminating false information
the internet.
We believe developing guardrails such as safety filters (OpenAl [022] Ina et al. 2023) are highly valuable for AI agent research. We do not condone the Dyna-Mind its constituent methods for any unlawful or morally unjust purposes.
C ADDITIONAL ALGORITHMIC DETAILS
In Algorithm 2 we provide the pseudo-code for SIMROLLOUT. On high level, SIMROLLOUT aims to generate refined response at a given state s, with better simulation content compared that of the original response. Specifically, SIMROLLOUT first performs normal inference to generate response; extracts the plan (ā1, 02, ... â.) from 0 using the "<plan><</plan>" tags (see TTbll A4]for for example response with such tags): executes the extracted plan in the environment and obtain the actual next-states {8t+1, S1+2, ... St+a]; and finally, prompts an LLM to refine the original response based the actual next-states, using the prompt in Table A3 The resulting refined response at is then used as the next action a,, and this process repeated until the task is completed or maximum number of steps is reached.
<e(-(s))
Algorithm 2 Simulation Refinement Rollout (SIMROLLOUT)
Require: policy Te, environment T, group size G
1: repeat the following G times:
2: 7' () Teemue {} 0, Sn T
3: while not done and < Imax do
4:
5:
{āi,..-,
extract extract_plan(()
6:
// imppooee action using next-state information
7:
(s1+1,'.. 8t+u } t (T(st, â1), .T...TTSssa T(8t4n-1,, 0n)
8:
spefine <- refinmment prompt(a St, a, {st+1, 01, ... sttn)) see Table A.
9:
refine t T (sttfen
10:
update episode buffer
11:
T' + -' U (8), aretine] // learn improved policy
12:
Treine f- Teeine U {strine, arefine) 3 // learn to refine simulations
SC-1 + T refine)
14:
t+t+1
15:
end while
16: return 7', -
D ADDITIONAL DETAILS ON TEXT GAMES
D. MORE ABLATION STUDIES
In Table Al provide more detailed ablation study of Dyna-Mind for Sokoban and ALFWorld. For the second-stage DYNA-GRPO training, we consider replacing Agrine with the standard GRPO advantage (denoted " A ") and removing DYNA-GRPO (denoted as " DYNAA-GRO"), We
15

---

Published as a conference paper at ICLR 2026
note that removing DYNA-GRPO reduces Algorithm to standard GRPO. For the first-stage RESIM, we consider replacing DFS with BFS (denoted as "- DFS++FS") as well as removing DFS entirely (denoted as *- DFS"). We note that removing DFS entirely reduces DISTILL(RESIM) to simply DISTILL(V3).
D.2 EXAMPLE TASKS AND ACTIONS
Sokoban (Schrader 2018) is a grid-world game where the agent needs to push boxes to their des- tinations while avoiding obstacles. Valid actions in Sokoban are up, down, left, and right. As an example, we provide an example imput state and generated action in Table A4 ALFWorld (Shrid- har al. 2021) a text- based embodied environ- ment where the agent needs to locate/interact with objects to complete embodied household tasks us- ing natural language instructions. Valid actions in ALFWorld are dependent on what's available the current state. We provide an example input state and generated action in Table A5
Table A1: Ablation study on Dyna-Mind
Method
Sokoban
ALFWorld
Dyna-GRPO
82.5 %
92.5%
Arefiin
80.8%
89.2%
SimRollout
79.1%
87.0%
(RESIL)
71.9%
78.9%
-DFS+BFS
71.1%
78.9%
DDS
49.2%
58.9%
D.3 RESIM IMPLEMENTATION DETAILS
We provide a pseudo-code for RESIM in Algorithm 3 Instruct as rollout and value function models using DeepSeek-V3's V3'ss rollouts. Specifically, we first use DeepSeck- V3 to generate 256 rollouts using tasks from the training set. Then, to train the rollout model, simply perform SFT training function, we use the trained policy model to generate the same 256 rollouts, repeated over 3 times, and compute (sx)) the probability of successfully completing the task from 8, across all trajectories that contains discounted by the number of remaining steps needed in the current trajectory:
For text games, we finetune Qwen2.5-32B-
correct rollout for each task. To train the value
V(s)) 7 fmax
Σ
1/7 is successful), where T = (TI, 72, ... 18t E Ti}
T
7ET
where y is the discount factor and fmax is the maximum number of steps trajectory. In both environments, we used 0.95. Finally, we finetune a separate Qwen2.5-32B- 5-3---Innttttdct as the value function by adding linear value head to the model architecture, and perform MSE loss training on the computed (s²) across all states from all trajectories.
Since Sokoban and ALFWorld environments are fast, these rollouts were completed within hour. For complex environments such as Android we directly prompt pretrained VLMs such as Qwen2.5-VL-722 and GPT-4oo as rollout and value function models (Section 4.2).
D.4 SIMULATION REFNNEMEN PERFORMANCE
To empirically show that (V)LMs are capable of leveraging next-state information to improve their action, we evaluate the performance of SIMROLLOUU compared direct prompting (REACTT). We report the result in Table A2
In general, we find that 1) all models showed improved task success rate when provided with next- state information; and 2) stronger models such GPT-4o and GPT-4.1 (OpenAl 2024 2025) shows larger improvement compared to weaker models such as S.ve22.5-7B-Inntruce because correcting its own mistakes is requires triviaal reasoning ability, which is more difficult for weaker models such as Owen2.5--B-Instruuc to achieve. Overall, this result indicates that world modeling eиror (c.g., especially for tasks such as ALFWorld) remains significant bottleneck for (V)LM agents reasoning ability in long-horizon tasks.
believe this
D.5 ADDITIONAL TRAINING DETAILS
To instantiate DYNA-GRPO. continue training the best model from stage I distillation. To ensure a fair comparison, we use identical hyperparameters for all methods (RLOO, GRPO, and DYNA- GRPO), when applicable. We use batch size of 8 tasks per batch, group size of G learning
16

---

Published as a conference paper at ICLR 2026
Table A2: SIMROLLOUT performance on Sokoban and ALFWorld. We show that when provided with ground-truth next-state information (SIMROLLOUT), models achieve better performance compared direct prompting (REACT).
Base Model
Method
Sokoban
AL.World
Dwwe2..5-7B-Instenn
REACT
25.8
35.4219
SIMROLLOUT
30.0
39.1
GPT-40-2024-11-20
REACT
37.8+1a
51.3=2.
SIMROLLOUT
41.4112
64.8
GPT-4.1
REACT
67.9
54.4=2.
SIMROLLOUT
71.1+13
67.9.20
Algorithm 3 RESIM
Require: policy ny. value function V,, environment T, (V)LM M Require: hyperparameters b, d, mmax: buain
1: + ( 0, 80 T
2: while not done and 4max do
3:
{r'} + sample rollouts using To starting from for max steps
4:
{r' Ti=1 deduplicate
5:
{v estimate value (V=(s(a)))
6:
1 subsamppe rollouts
7:
T" t A max; e²
8:
{T')
M==l + {T" subsample buain - rollouts from the rest of {r 1=1
bmii
9:
Il agggeggate rollouts into single reasoning response
10:
{plan')
==1 <- summarize (M(r', ; ot vt)) Amn
Onnin
RESIM aggregate M (8t, {plan' )=1)
12:
// next step
S1+1 + T (( RESIM)
14:
T TU T RESIM)
++++1
16: end while
return
rate of le-6, and 300 training steps in total for both Sokoban and ALFWorld. For DYNA-GRPO, we
use ny = 10 and n. 10 Sokoban and ny 10 and ntz = for ALFWorld. All training are performed on top of Qwen2.5-7B (Qwen et al. 2025) using 8xH100.
D.6 SIMULATION SCORE PROMPTS
To evaluate the simulation ability of a model тy. we use LLM-as-a-judge (Zheng et al. 2023) to measure the correctness and quality of the simulation generated by me at each turn given jectory. Specifically, for each
To(-ss;), we first prompt an LLM to extract the final action plan (ā), 02, ... âa) from 0, and the corresponding natural language description of the next-states ($t41, $142, &(ta) a from the response We present the prompts used for Sokoban and ALF. World Tables A6 and A8 respectively. Then, we execute the action plan in the environment to obtain ground truth next-states (st+1, S142, St4d). Finally,
prompt an LLM to judge the quality of the plan by comparing "imagined" next-states generated by To against the ground truth states, using prompts in Tables A7 and A9 This results in a score [0, for each turn in the trajectory, which is then averaged across all turns to obtain an overall simulation score for the entire trajectory.
17

---

Published as a conference paper at ICLR 2026
Task: on my schedule
Simple Calendar Pro? Answer titles only.
T-B
cobeecationes
The sdrvan aa aga the schoball for Coskkr 28 (Thundug) slowing one event Youl wit Bue" viible this
huigee
cinsonakiss
Ounobr (TT6)
<thinie
Dairy ler me took at the cument buption Freme ihy. colikedion Ouanter 20 cwknnar showing the ocheriile for Oolater 26 inteeed Chtober Fiieed sinelatt to Octiter 28 the eventa achodulis for
Let's tark about a few possibll plams
Mayla
dlisiing the foowart
somer
parks This action may load to scrsen displaying the sc schedule for Gizzber 27 (hiiday) which Maybe dlisking the schedliee
furthrr Desaber thins, This action lnd to the soncen ploplinggg the saboduae for Oukiter which nor moving sloser Octsbor
thene stmuuattona, diicdin farward smow icon the top right comer
28, whichis the kinger maki the most efficien opiioo..
rlodi paate
uswe" "ergurentt" ("oodeen" "click", "woerdnat"" (1009, s97g))
*tool
fonward ioon the rghtt Ocluber 28 the sollotlle won.
ruvigate dlserr
80
Sy
01+1
Figure A1: Example task, input screenshot, and output generated by model trained using Dyna-Mind. For clarity, we directly rendered the proposed action in 0++1 (click at 1023,387) in green on 8ç-
E ADDITIONAL DETAILS ON ANDROIDWORLJ
E. EXAMPLE TASK AND ACTIONS IN ANDROIDWORLD
In this work, we use the dockerized environment provided by Android to evaluate and train all methods. We use the screenshot-only modality. In Figure A1 we present an example task, input screenshot 8, from AndroidWorld, as well as an example output 0 generated by models trained using Dyna-Mind. For more details AndroidWorld please refer to Rawles et al. (2025).
E.2 ADDITIONAL TRAINING DETAILS
To standardize training and evaluation, we use the dockerized version of AndroidWorld and adapt the action space provided by Rawles et al. (2025).
To instantiate GRNPO we continue training the best model from stage distillation. We followed 4.1 and used batch size of tasks per batch, group size of G 8, learning rate of 1e-6. Since Android World is highly compute-ines intenteve and time-consuming to run, we perform total of 60 training steps for RL training, using ny and n₂ 8. All training are performed on top of Dwee2.55VLL7B--Ing
and -iren..--VL-32--Innss denoted as "DiSTILL--B" and "DISTILL--2B" in Table respectively.
(Bai et al. 2025) using 16xH100,
E.3 OTHER IAPLEMENTATIOO DETAILS
In this work, focus end-to-end training (SFT RL), and hence selected VLMs capable of directly interacting with android's GUI interface. This include models such as Qwen2.5-VL. (Bai al. 2025) and Ul-Tars (Qin et al.) 2025). While these models have undergone specific finetuning on mobile control tasks, at the time of the work we were unable to find evaluation scripts that supports using these models on AndroidWorld, To our best effort, we utilized the official mobile-use prompts provided by the respective repositories, as well as prompts from recent work such as (Gou al. 20256). However, we were unable fully reproduce the reported performance, especially for Ul-Tars 1.5. At the time of this work, we find similar concerns has also been raised publicly com heemm/yytedanccce / /UI TARS com UI - Twrs/aars/00 OUI Tars Tar--iarrssauaaa issues 155] https: / http//tttttttbb/./ com/ /UI -
(e.g., https: /tte:://gtttttt
83 Inttps://githun
issues /21). To this end, we focus on using Qwen2.5-V6 VL for consistency with other experiments
conducted in the rest of the paper.
18

---

Published as a conference paper at ICLR 2026
Table A3: Prompt used by SIMROLLOUT to refine the agent's 's original response given actual next-state information. The next-state information is obtained by 1) extracting the final chosen plan from the agent's ss response (e.g., left, left, up in Sokoban), and 2) executing the plan in the environment to obtain the actual next states.
Prompt
...iiiiiin some text
# Current observation
(current observation]
# Example response and feedback
To help you reason and plan better, have explored plans for the current step and obtained the following feedback from the environment:
## Example response
(agent original response]
## Ground truth feedback
(actual next _ettul_next_____
# Back the current step
Now, the environment has been reset back to the current observation/current step. It's your turn to refine the example response based on the ground truth feedback. You should think about:
Correctness: is the example response aligned with the feedback? did the feedback reveal some incorrect/ineffective actions in the example response?
Progress: did the the environment feedback show positive progress towards solving the task? Note: the example response may hallucinate incorrect outcomes different from the ground truth feedback. You should avoid coming up with similar hallucinations in your response.
If you think the example response is correct and has made progress, no revision is needed and your should **directly output the example response verbatim*** Otherwise, you should modify the example response's thinking process/plan/action to be consistent with environment feedback. Specifically, you should:
1. Incorporat all relevant details from the feedback** into the example response and then **improve its accuracy and progress** Be detailed when adding information from the feedback into the response.
The final refined response should **read as a standalone answer, without saying its a revision referencing the feedback section** ** The output should look like the final authoritative response that faithfully incorporated relevant details from the feedback.
Now, refine the example response. Keep the same tone and format as the example response. Ensure the final plan/action is logically consistent with your (refined) thinking process. DO NOT generate anything else.
19

---

Published as a conference paper at ICLR 2026
Table A4: Example state in Sokoban and our model's ' generated action
Type
Content
Input
/| ....oiiiiin some text about sokoban game rules
# Current Step
Your current observation is:
Col 0 Col 1 Col 2 Col 3 Col Col
Row 0 # #
Row #
#
Row 2 # P
Row # X
# Row # #
Row 4 # 0
Currently, the player is at (2, 2); boxes are at (3, 2); unsolved targets are at (4, 3). Next to the player, there is a floor above the player at (1, 2), box below the player at (3, 2), a floor left of the player at (2, 1), wall right the player at (2, 3).
Your admissible actions are ("up", "down", "left", "right").
Now it's your turn make a move (choose ONE action only for the current step). You should first reason step-by-ster about the current situation - observe the posi- tions of boxes and targets. plan path push box toward a target, and avoid traps like corners or walls. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've 'v finished your reasoning, you should choose a plan as well admissible action for current step. The plan needs to be presented as a se- quence of admissible actions within <plan> </plan> tags (e.g., <plan>up, right, up</plan>). The action needs to presented within <action> </action> tags (e.g.. <action>upp<actioon>))
Output
<kay, let me look the current situation. The player at (2, 2), with a box at (3, 2) and an unsolved target at (4, 3). Let's think about a few possible plans.
Maybe we can try moving "down". This action will lead to the state where the player is at (3, 2), pushing
box to (4, 2). Is this a good next step? Let's think: After "down", maybe do "left". This results in the player moving to (3, 1), with the box remaining at (4, 2) and the target at (4, 3). After that, maybe "down", which leads to the player moving (4, 1), with the box still at (4, 2) and the target at (4, 3). The player now has a box to the right and a floor above. Discounted success rate if continued further: around 80%.
- After "down", maybe do "left". This results in the player moving to (3, 1) with the box still (4, 2). After that, maybe "down", which leads the player moving to (4, 1) with the box and target unchanged. Next, try "right", which moves the player (4, 2) and pushes the box onto the target (4, 3), solving it. Discounted success rate if continued further: around 100%.
Based on these simulations, "down" is the best action for the current step. This is because the second plan (down -> left -> down right) achieves discounted success rate of 100% in 4 steps.
</think>
<plan>down, left, down, right</plan> <actionsdown<<actionn>
20

---

Published as a conference paper at ICLR 2026
Table A5:
Example state in ALFWorld and our model's generated action
Type
Content
Input
Il ....omitting some text about alfworld rules
Your task is to: put some book on sidetable.
Your admissible actions of the current situation are: ['go bed 1' "go to diningtable I' "go to drawer go to drawer 2' go to drawer 3' "go to drawer 4' "go to garbagecan go to sidetable 1 "go to sidetable 2' "inventory" "look").
Now it's your turn take an action.
You should first reason step-by-step about the current situation, and think about plans to explore/solve the task. To plan well, you should think about the plan's s future outcomes and whether it is making progress not. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning. you should choose plan well admissible action for current step. The plan needs to be presented as sequence of admissible actions within <plan> </plan> tags (e.g.. <plan>go desk take mug from desk I</plan>). The action needs to be presented within <action> </action> tags (e.g., <action>go to desk <</action>).
Output
<taink>Okay, let me look at the current situation. Currently, am in the middle of room with several objects around me, including two sidetables (sidetable 1 and sidetable 2), where my task is to put a book on one of them. Let's 'ss think about a few possible plans.
try Go diningtable 1. This action may lead arriving diningtable 1, where see a creditcard 1, a book 3, a cdplayer 2, a laptop, pencil, three pens, nutshell, and a vase. Is this good next step? Let's think:
Maybe
After Go to diningtable 1, maybe I can take book 3. This results in picking up the book 3 from the diningtable. Discounted success rate continued further: around 80%.
After Go diningtable 1, maybe I can inventory myself. This may reveal that nothing is being carried, so taking anything is impossible. Discounted success rate if continued further: around 10%.
Alternatively, maybe we can try Open drawer 1. This action may lead to opening drawer and seeing nothing inside. Is this a good next step? Let's think: After Open drawer 1, maybe "examine drawer 2" reveals a combination lock and keychain inside. Discounted success rate if continued further: around 40%.
Based on these simulations, going the diningtable has the highest discounted success rate (80%) as book 3 may be found. This achieves the highest success if followed by "take book 3", achieving 80% success rate. </think>
<plan>go to diningtable 1, take book 3</plan> <action>go to diningtable ></action>
21

---

Published as a conference paper at ICLR 2026
Table A6: Sokoban prompt to extract plan and imagined observation from an agent's 's response
Prompt
// ...oiiiiing some text about sokoban game rules
# Extraction/parsing rules
Your task is parse the response and extract the following information, IF present.
1) simulation branches
definition: one sequence of actions the agent planned solve the puzzle
example: // ./..iiiiiing some text 2) discounted success rates
definition: the (discounted) success rate of the simulation branch, or some numeric evaluation of how much progress that branch makes towards the goal. example: ....oiitting some text
3) final chosen branch
definition: the simulation branch/plan that caused the agent's final decision for the current step.
example: Based these simulations, "up" is the best action for the current step. This is because after "up". the player can proceed with "left" and "up" again, which achieves a discounted success rate of around 90% in 3 steps.
example output: l"up". "left", "up"]
note: The agent chose "up" as the next action. However, need to find the ENTIRE branch/plan that caused the agent's current decision, which is ["up". "right", "down"] in this case.
note: if the agent did not explicitly mention which branch is chosen, you should choose the branch in the response with the highest discounted success rate. final imagined observation
definition: the imagined observation after executing the final chosen branch.
example: After "up", "left", "up". the player pushed the box to (4,4). Now, the player is at (4, 3), with the box target below at (4, 4). The player has floor above (2, 4)... The target is This the best branch according to the discounted success rate. So the next action should be "up". example output: The player pushed the box to (4,4). Now, the player is at (4, 3), with the box target below at (4, 4). The player has floor above at (2, 4)... The target is note: DO NOT include the action sequence in this field. Only keep the deseription of the aaayer/oxes/harget//waaa position AFTER the last action in the final chosen branch. note: // .mmiiiing some lext
# Your task
Your task is to output a JSON object in the following format: <json>
"extracted branches": : ...//.. ..miiiiiiggg some text ].
cextracted_brannches..
{ "actions": ["action 1", "action 2", ..,, "action n"]. # the ENTIRE branch/plan that caused the agent's current decision
"extracted_
"iisssooeevvation.. "detailed, comprehensive description of the imagined observation AFTER executing the entire action sequence above.".
"discounted, -sccuntedddccsessssseeee successsratte (..(a number between 0 to 100. -1 the agent did not mention the discounted success rate)
</json>
# Input response
(imput (input_agent response)
# Your task
Now, parse the response and output the JSON object enclosed by <json> and </json> tags. DO
NOT generate anything else.
22

---

Published as a conference paper at ICLR 2026
Table A7: Sokoban prompt to evaluate the quality of the next-states imagined by an agent in its reasoning process, using the actual next-states as references.
Prompt
.miitinn some text about sokoban game rules
# Evaluation rules
Provide an overall score between 0.0 and 1.0 based on the following two dimensions. Start with a score of 0.0, and add points to the score if the criteria are satisfied. Add 0.0 if a criteria is not satified. DO NOT deduct points if a criteria is not satified.
correctness (max 0.3 points. if exceeds 0.3, cap at 0.3)
in the imagination description, the coordinates of the player are correct; add 0.1 point
in the imagination description, some of the mentioned boxes and targets have correct coordinates; add 0.05 point
in the imagination description, all mentioned boxes and targets have correct coordinates; add point
in the imagination description, all mentioned walls and empty spaces have correct coordinates; add 0.05 point
2) progress (max 0.7 points. exceeds 0.7, cap it at 0.7)
in the reference observation, if the task is completely solved (all boxes targets); add 0.7 point
relative the current observation, if the reference observation shows major progress (unsolved boxes are moved much closer to targets, task close to be solved); add 0.5 point
relative the current observation, the reference observation shows minor progress (unsolved boxes are moved a bit closer targets); add 0. 1-0.3 point, depending on how much progress is shown
relative the current observation, if the reference observation shows no meaningful progress; assign 0.0 point for this dimension
in the reference observation, if the task is no longer solvable (e.g., of the boxes is pushed into a corner and cannot be moved anymore); assign 0.0 point for this dimension // ....onitiing some text
# Your output format
Your task is to output JSON object in the following format: <json>
"correctness analysis": "which correctness criteria in the evaluation rules are satisfied, and which are not.", ", # no more than 50 words
"correctness score": 0.0-0.3, score for the correctness dimension
"progress analysis": "which progress criteria in the evaluation rules are satisfied, and which are not.", # no more than 50 words
"progress score": 0.0-0.7, # score for the progress dimension "score": 0.0-1.0 () total score; add the correctness score and progress score
</json>
# Current observation
(current_ obs)
# Agent imagined observation after some actions pagent_imaginne Lettineeddde_____eeeeeeeee cext_actions. _______dd____ and_obs} # Reference observation after some actions
[actual _cxt__obs
# Your task
Now, provide evaluation analysis and according to the evaluation rules above. Output the
JSON object enclosed by <json> and </json> tags. DO NOT generate anything else.
23

---

Published as a conference paper at ICLR 2026
Table A8: ALFWorld prompt to extract plan and imagined observation from agent's response
Prompt
// ...omiiiing some about sokoban game rules
# Extraction/parsing rules
Your task is parse the response and extract the following information, IF present. // .m.iiiiin
some text
3) final chosen branch
definition: the simulation branch/plan that caused the agent's final decision for the current step. example: Based on these simulations, "go to countertop 1" the best action for the current step. This is because this followed by "go countertop 2" leads to high chance of finding mug. Therefore, next action for the current step should "go to countertop 1".
example output: ["go to countertop 1", "go to countertop 2"]
note: The agent chose go to countertop 1" as the next action. However, we need to find the ENTIRE branch/plan that caused the agent's current decision, which is ["go countertop 1", "go to countertop 2"] in this case.
note: if the agent did not explicitly mention which branch is chosen, you should choose the branch in the response with the highest discounted success rate.
final imagined observation
definition: the imagined observation after executing the final chosen branch.
example: After "go shelf 1", "take pencil from shelf 1" results in successfully picking up pencil. This is the best branch according to the discounted success rate. So the next action should be "go to shelf 1".
example output: The agent successfully picks up a pencil.
note: DO NOT include the action sequence in this field. Only keep the description the imagined observation AFTER the last action in the final chosen branch.
note: In general, you should gather the most comprehensive and detailed description found in the response (i.e., especially try to include any mention of what objects is present). If this description scattered across multiple places in the response, MERGE them into a single, continuous description.
# Your task Your task is to output a JSON object in the following format: <json>
"extracted branches": ..///..omittinn some text }.
"extracted_fine chosen_branche { "actions": ["action 1", "action 2", "action n"], # the ENTIRE branch/plan that caused the agent's 's current decision
"last observation": "detailed. comprehensive description of the imagined observation AFTER executing the entire action sequence above.".
"discoonted_suucee" success rate": (a number between 0 to 100. -1 if the agent did mention the discounted success rate)
</json>
# Input response
cinput_agent_respee
# Your task
Now, parse the response and output the JSON object enclosed by <json> and </json> tags. DO NOT generate anything else.
24

---

Published as a conference paper at ICLR 2026
Table A9: ALFWorld prompt to evaluate the quality of the next-states imagined by an agent in its reasoning process, using the actual next-states as references.
Prompt
o.miitiing some text about sokoban game rules
# Evaluation rules
Provide an overall score between 0.0 and 1.0 based on the following two dimensions. correctness (max 0.3 points. if exceeds 0.3, cap at 0.3)
in the imagined observation, it is near identical to the reference observation; add 0.3 point in the imagined observation, key object(s) required by the goal are found, and they are also present in the reference observation; add 0.2 point
the imagined observation, relevant location(s) required by the goal are visited, and the description is somewhat aligned with the reference observation; add 0.1-0.2 point, depending how much the description aligned with the reference observation. in the imagined observation, key object(s) required by the goal are found, but these key object(s) are *NOT* present in the reference observation; assign 0.0 point
in the reference observation, it shows nothing happened; directly assign 0.0 point for this dimension
2) progress (max 0.7 points. if exceeds 0.7, cap it at 0.7)
in the reference observation, if the goal is completely solved (all required items are found/moved/heated//et
or the correct location, goal is achieved); add 0.7 point
relative to the current observation and action history, if the reference observation shows major progress (i.e., objects required by the goal are found); add 0.5 point
relative to the current observation and action history, if the reference observation shows minor progress (i.e., objects related to the goal are found, locations relevant the goal are visited); add 0.1-0.3 point, depending on #how useful this information is, beyond what was already known in the current state and action history*.
relative to the current observation and action history, if the reference observation shows meaningful progress (nothing happened); assign 0.0 point for this dimension // ...omiiting some text
# Your output format
Your task is to output JSON object in the following format: <json>
"correctness analysis"; "...", # no more than 50 words
"correctness score": 0.0-0.3, score for the correctness dimension "progress analysis": "...", # no more than 50 words "progress score": 0.0-0.7, 7, # score for the progress dimension
"score": 0.0-1.0 # total score; add the correctness score and progress score
</json>
# Action history
The current goal is to: faask_dsccrpption [action_ [ration_histrr]]
# Current observation
(current_ obs)
# Agent imagined observation after some actions
pagent_imaginnn Lettenedd_ex____eeeee next_actions. ___ss__dd____ _nd_obs)
# Reference observation after some actions
[actual _ext__obss
# Your task
Now, provide evaluation analysis and according to the evaluation rules above. Output the
JSON object enclosed by <json> and </json> tags. DO NOT generate anything else.
25


Reviews:
[{"review_id": "pSTFNRlODR", "reviewer": "Reviewer_z3zA", "summary": "This paper aims to enable a model to make decisions directly through internal simulation during inference, rather than by directly experiencing each option. The authors propose Dyna-Mind, a two-stage framework to achieve this goal. In the first stage, a search tree is constructed to build reasoning trajectories, and a VLM is trained to distill actions based on generated trajectories. In the second stage, reinforcement learning further refines both the decision-making strategy and the model’s simulation capability. Experimental results show that Dyna-Mind outperforms weaker LLMs and previous approaches, including Dyna-Think, and surpasses DeepSeek-R1 in more complex environments.", "questions": "1. Please address the concerns raised in the weaknesses.\n\n2. The definition of $a^{refine}$ in Line 274 is not sufficiently clear, and it is uncertain whether it is identical to the definition given in Line 275. Do the authors first generate $\\tau'_{\\text{refine}}$ and then remove the future-state information from it to obtain $\\tau'$?\n\n3. In Algorithm 1, the method requires different kinds of rollouts, such as $\\tau$, $\\tau'$, and $\\tau'_{\\text{refine}}$​, but without providing an ablation study for using different rollouts in the training. What would be the impact on performance if one of them were removed?\n\n4. What is the reason for the use of distillation? In Table 1, it seems that both ReAct and RESIM can also be evaluated on the OOD test set. What is the reason for not providing these results?\n\n5. In Table 1, what would be the performance if RESIM (without distill) were combined with Dyna-GRPO? Or the Distill(R1)+DYNA-GRPO?\n\n6. In the AndroidWorld experiment analysis, the paper attributes the suboptimal performance to the rollout model (Qwen2.5-VL-72B). It would be beneficial to provide more concrete evidence supporting this claim or modify the method to demonstrate this limitation better.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a novel idea that enables the model to simulate the environment during reasoning rather than performing actual physical simulations.\n2. The paper achieves better results not only on Sokoban but also on the other complex environments compared to other LLM methods.", "weaknesses": "1. This paper presents a comprehensive algorithm with several new proposed components, including ReSim, ReSim distillation, and Dyna-GRPO. However, it is not clear what the motivation is for the design of these components. Overall, the proposed architecture is a bit complicated without clear explanations. For example, ReSim first proposes a tree search-based method to simulate and generate trajectories. However, in Dyna-GRPO, there is another SimRollout approach to do the same thing. In addition, it is also unclear to use three different types of trajectories. Including an ablation study for each component would be more convincing.\n\n2. In the experiments, comparisons should also be made with other search-based methods mentioned in the Related Work section, such as ExACT. \n\n3. The training process requires several steps, including constructing tree search reasoning in ReSim and generating Rollout/SimRollout in Dyna-GRPO, which increases the training cost. The paper should provide the training computational cost compared to other methods. In addition, the inference computational cost should also be provided for a more complete evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4YR4HAOPB", "reviewer": "Reviewer_Tcgq", "summary": "This paper introduces Dyna-Mind, a two-stage training framework for teaching (V)LM agents to simulate environment dynamics during reasoning.\nIn Stage 1 (RESIM), the model learns to generate reasoning traces grounded in real environment rollouts by converting search trees into structured simulations.\nIn Stage 2 (Dyna-GRPO), an online reinforcement learning phase further improves both simulation and policy quality using feedback from intermediate and final outcomes.\nExperiments on Sokoban, ALFWorld, and AndroidWorld show that Dyna-Mind enhances long-horizon planning and reasoning compared with existing baselines.", "questions": "The paper proposes a well-motivated and technically coherent framework that bridges reasoning and simulation. Although ablation and data-quality analyses are missing, the idea is novel and the results are promising for advancing long-horizon agent research. Moreover, pleaser refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Presents a clear two-stage design that mirrors the human cognitive process of “mental simulation before action.”\n2. RESIM effectively links reasoning traces with real-world dynamics, providing grounded supervision.\n3. Dyna-GRPO adds an elegant reinforcement step that combines intermediate and outcome feedback.\n4. Evaluation covers synthetic-to-realistic benchmarks and uses diverse metrics (success rate, simulation quality, token cost).\n5. Writing and motivation are strong; results convincingly support the claimed improvements.", "weaknesses": "1. Ablation analysis is incomplete: it is unclear how much gain comes from (a) using real future states vs. (b) stronger aggregation LLMs or A-refine components.\n2. The RESIM data-selection process for choosing “best plans” is under-specified, criteria for simulation quality and filtering are not explained.\n3. Minor presentation issues: inconsistent decimals in tables, unclear “1.0×” unit in Table 1, and a few typos (e.g., planing-intensive 2 planning-intensive).\n4. Lack of discussion about computational cost and scalability of RESIM when applied to larger real-world environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D666Icmh1g", "reviewer": "Reviewer_K5cm", "summary": "The authors present Dyna-Mind, a training framework that teaches a VLM to learn a simulation of the underlying environment and incorporate it into learning. A two-stage training framework is presented, including a method for training agents to produce reasoning traces called Reasoning with Simulations (ReSim), and an online reinforcement learning algorithm called Dyna-GRPO.", "questions": "There is this notion of the “simulation ability of the policy” and it’s not clear what that means. Typically in RL the policy is not simulating any part of the environment, it only outputs an action.\n\nIt would be useful to make an explicit distinction between the environment model and the policy, just as in Dyna-Q. It seems that the LLM is generating entire rollouts, including actions and states? So the environment model and policy are represented by a single LLM?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work is decently situated in the field. The math is extensive. Dyna-GRPO is an interesting implementation of model-based RL. Results are clear and well presented. A relatively large number of environments are used.", "weaknesses": "The paper is slightly confusing at times, there is slight misuse of the notion of “policy” (see comments below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dECOtEJ8Ry", "reviewer": "Reviewer_J1B4", "summary": "This paper presents Dyna-Mind, a two-stage training framework designed to equip LLMs with the ability of ***vicarious trial and error***, i.e., the capacity to mentally simulate alternative futures before acting, in order to perform better in complex, long-horizon interactive tasks. In Stage 1, the model learns to generate structured reasoning traces from search trees built on real environment interactions, grounding its reasoning in faithful world dynamics and enabling it to anticipate future states. In Stage 2, the model is further optimized via online RL, leveraging both outcome rewards and intermediate feedback from real rollouts to enhance its simulation and decision-making abilities. Experiments on Sokoban, ALFWorld, and AndroidWorld show that RᴇSɪM effectively instills simulation capabilities, while Dyna-GRPO improves policy learning for long-horizon, planning-intensive tasks. Overall, the study underscores the pivotal role of simulation in enabling AI agents to reason, plan, and act effectively in complex environments.", "questions": "About $S_{t}^{\\text{refine}}$, since it consists of $d$ state-action pairs, could its length pose an issue for the LLM input? Specifically, is there a risk that the sequence could become too long for the model to handle efficiently? Is the model performance sensitive to $d$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel conceptual contribution: the introduction of ***vicarious trial and error*** as a cognitive-inspired mechanism for AI agents is well-motivated by human reasoning literature.\n\n\n2. Clear and systematic framework: the two-stage design, i.e., RᴇSɪM + Dyna-GRPO, provides a coherent pipeline from reasoning grounding to online policy refinement.\n\n\n3. Empirical validation: the experiments on both synthetic, i.e., Sokoban, ALFWorld, and realistic, i.e., AndroidWorld, benchmarks demonstrate solid and consistent performance gains.\n\n\n4. Interpretability: the structured reasoning traces in RᴇSɪM make the agent’s decision process more transparent and analyzable.", "weaknesses": "1. High computational cost: the proposed pipeline appears quite complex, involving rollouts, value function updates, search tree aggregation, and RᴇSɪM distillation, all of which may be resource-intensive. Consequently, it remains unclear whether the method can scale efficiently to large environments or applications.\n\n\n2. Ablation clarity: more detailed ablation studies would help clarify the contribution of each component, particularly the individual modules within RᴇSɪM and Dyna-GRPO, to the overall performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)