VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
Authors: Kangrui Wang
Pingyue Zhang
Zihan Wang
Yaning Gao
Linjie Li
Qineng Wang
Hanyang Chen
Yiping Lu
Zhengyuan Yang
Lijuan Wang
Ranjay Krishna
Jiajun Wu
Li Fei-Fei
Yejin Choi
Manling Li
Venue: NeurIPS 2025 poster ()
Keywords: Visual States
World Modeling
Multi-turn RL
VLM Agents
Primary area: deep_learning

Abstract: A major challenge in training VLM agents, compared to LLM agents, is that states shift from simple texts to complex visual observations, which introduces partial observability and demands robust world modeling. We ask: can VLM agents build internal world models through explicit visual state reasoning? In this work, we architecturally enforce and reward VLM agent’s reasoning process via reinforcement learning (RL), formulating the problem as a Partially Observable Markov Decision Process (POMDP). We demonstrate that structuring agent’s reasoning into StateEstimation (“what is the current state?”) and TransitionModeling (“what is next?”) is critical by studying five reasoning strategies. Investigating how agents should ground visual states and represent these internal beliefs, we reveal the optimal representations are task-dependent: Natural Language excels at capturing semantic relationships for general tasks, while Structured formats are essential for high-precision manipulation. These insights motivate our approach to reward shaping and credit assignment. We leverage a WorldModeling Reward to densely rewards the agent’s turn-by-turn state predictions, while our Bi-Level General Advantage Estimation (Bi-Level GAE) enables turn-aware credit assignment. Through such world model reasoning, we enable a 3B model to achieve performance of 0.82 on a set of five diverse agent tasks, nearly 3× improvement over its untrained counterpart (0.21) and surpassing proprietary reasoning models like GPT-5 (0.75), Gemini 2.5 Pro (0.67) and Claude 4.5 (0.62). All experiments are supported by our VAGEN framework, a scalable system for training and analyzing multi-turn VLM agents across diverse visual environments

VAGEN: Reinforeing World Model Reasoning for Multi-Turn VLM Agents
Kangrui Wang'*
Pingyue Zhang", Zihan Wang Yaning Gao Linjie Li Qineng Wang'
Hanyang Chen Chi Wan'
Yiping Lu Zhengyuan Yang" Lijuan Wang'
Ranjay Krishna Jiajun Wu
Li Fei-Fei
Yejin Choi Manling Li
Northwestern University ²University of Washington *Stanford University *Microsoft
University of Ausconsin-Madisse "University of Illinois Urbana- Champaain
http://mllllab.notttveeeee...ce
kangrui wang@northwesteen. edu, manling 11 iinorthvestern.cc edu
Abstract
A major challenge in training VLM agents, compared to LLM agents, is that states shift from simple texts to complex visual observations, which introduces partial observability and demands robust world modeling. We ask: can VLM agents build internal world models through explicit visual state reasoning? In this work, we architecturally enforce and reward VLM agent's 's reasoning process via reinforce- ment learning (RL), formulating the problem as a Partially Observable Markov Decision Process (POMDP). We demonstrate that structuring agent's reasoning into StateEstimat
("what is the current state?") and FransitionModee ing ("what is next?") is critical by studying five reasoning strategies. Investigating how agents should ground visual states and represent these internal beliefs, we reveal the optimal representations are task-dependent: Natural Language excels at capturing semantic relationships for general tasks, while Structured formats for high- sech-preeiioon manipulation. These insights motivate our approach to reward shaping and credit assignment. We leverage WorldModell ing Reward to densely rewards the agent's 's cum-by-turn state predictions, while our Bi-Level General Ad- vantage Estimation (Bi-Level GAE) enables turn-aware credit assignment. Through such world model reasoning, we enable 3B model to achieve performance of five diverse agent tasks, nearly 3x improvement counterpart (0.21) and surpassing proprictary reasoning models like GPT-5 (0.75), Gemini 2.5 Pro (0.67) and Claude 4.5 (0.62). All experiments are supported by our VAGEN framework, scalable system for training and analyzing multi-turn VLM agents across diverse visual environments.
essential
0.82
its untrained
Introduction
The core challenge in multi-turn agentic tasks is accurate interpretation and tracking dynamic environ- ments, which becomes significantly harder when the agents sense the world through vision rather than text. Vision-language Model (VLM) agentic tasks inherently complex due the challenges in understanding visual states, which often are partial and noisy Observations, fundamentally reframing the problem from Markov Decision Process (MDP) to more challenging Partially Observable Markov Decision Process (POMDP). Within a POMDP, the agent is no longer simply acting, but first estimating the state of the world from observations. The bridge connecting what an agent sees what needs to know is the central focus of our work: the internal World Model [1]. It functions as a cognitive engine, maintaining an internal belief over the true state based on visual observations. While VLMs agents have been proposed for visual agentic such as like games [2-4], embodied
"qqual contribution. "ooreeponndigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Environment Interaction
Agent Internal Belief 4
Environment Interaction
Environment
stocking cube
goel
8
Red cube position: (17. 170. 20).
cobservations
move ahead right
Green sube pusition: 1-8, 56, 20).
POMDP
Purple cube position: (:16, -75, 20) <lobsonvctions
Sp=11
Agent Internal Belief
pickt? 170, 201
<observatio>> There is plate on table right.
creasoning>
a
First. need pick the res cube. the cubo. </ressooingg>
eneszonings Fins, I should mow forword to
ag
the table. Then, con move the right be in front of the taple. <frensening>
csrediction>
Agent Internal Belief 4
Red cubo position: (-8, 56, 100).
Green sube position: [-8, 56, 20), Purple cube position:: (16, 75, 20) </prodiction>
1++
cprediction- I will be claser to the tabis c/predictions
Examples in PrimitivoSkill
Examples in Navigation
Figure 1: Overview of the VAGEN Framework. Within a POMDP, VLM Agents build internal world mod- els through explicit visual state reasoning over the StateEstimation P(s or) and transitionModeling P(5e41 lot, 81, āz), where & denotes the internal belief of the state, â represents the internal belief of the action, and of is the parsed executable action, detailed in Table 4.
AI [5-7], and computer use [8-10], current approaches in such multi-turn agentic tasks often lack explicit internal world modeling strengthen visual state reasoning. This lead us think: How can we effectivvey teach VLMs to build internal world models through explicit visual state reasoning?
To address this, reward VLMs via reinforcement learning (RL) to encourage their think- ing process to be structured into StateEstimation (describing the current visual state) and TransitionModel1 inn (predicting next state), which are the essential two components of WorldModeling [1]. We focus on multi-turn RL to optimize the entire interaction trajectory [11- 16, 14, 17], that the agent can maintain and update internal beliefs interactions unfold. To validate this principled reasoning structure, we systematically compare five reasoning strategies via controlling format reward in RL: NoThink [18], FreeThink [19, 20] as implicit visual state reasoning, and explicit reasoning through StateEstimati on via cobservation! tokens, FransitionModes via <prediction: ctin>> tokens, and their combination, Worl VorldModeling. Our findings indicate that incorporating explicit visual state reasoning like Statefstimation on and commitiioMModlll inn into VLM's 's thinking process during RL training can enhance task performance. Notably, the full reasoning strategy, World sorldModelingg ing, achieves an overall performance of 0.76, yielding better results than FreeThink (0.67) while clearly surpassing NoThink (0.28).
Building on the insight that VLM agents might make better decisions when reasoning about what they now (StateEst timtitinn and what they will see next question lies in representation: for an agent to "think" about the visual world, what is the optimal representation for its internal monologue? To translate pixels into mental model it can reason upon, we explore a spectrum of representation possibilities and our findings point to a crucial design principle that the choice of representation is not universal but is dictated by the task's demands. Natural language excels in a general-purposs tasks by providing the robust and inherent semantic- centric representation, allowing the VLM to use the vast pre-trained knowledge. However, its inherent ambiguity make it unsuitable for high-precision tasks such as robot manipulation, where structured formats providing exact coordinates is essential. As a third alternative, symbolic representation offers a more abstract representation. Surprisingly, this abstraction does not produce a more generalizable solution. Instead, it proves to be the least effective method in our experiments, creating grounding problem where the models struggle to connect abstract symbols to raw visual input without targeted training.
iowiittiiooMMMnnnn... foundation
Having established what VLM agents should reason about and in what ways they can reason, the remaining question is how effectively optimize such reasoning via reward shaping and advantage estimation. To this end, we introduce a turn-level WorldModel 1MMdeeiig Reward, dense reward derived from an LLM-as-a-Judee [21] framework, which evaluates the accuracy of the agent's explicit state descriptions and predictions against ground-truth states. Further, credit assignment in multi-turn settings can be optimized through our proposed Bi-Level General Advantage Estimation (Bi-Level GAE) via encouraging credit propagation in each single-step world modeling. Standard GAE [22] methods, which compute advantages token-by-token a backward manner from the end of a trajectory. would lead to unstable reward propagation because the sparse end-of-trajectoo travel across long horizons. To address this, we first compute advantages at the turn level to assess whether a VLM's 's response in single-step world modeling is generally effective, and then propagate
signal must
2

---

these signals to the token level, providing each VLM generated token with Line-graines advantages to optimize its generation process. Our results demonstrate that our VAGEN- approach, with WorldModeling Reward and Bi-Level GAE, consistently outperforms the VAGEN-Base without these mechanisms, leading to improved reasoning quality and higher task success rates by providing the dense reward with the adapted delivery mechanism.
We present VAGEN, a scalable training framework that decouples environment setup from model training, enabling efficient experimentation and algorithmic extensibility. Together, this work establish a principled pathway for developing VLM agents capable of building intern world models through explicit visual reasoning.
2
Build Internal World Models via Visual State Reasoning in Multi- Colti-TurnBan RL
2.1 Problem Formulation
We frame the multi-turn VLM agentic tasks as Partially Observable Markov Decision Process (POMDP) [23], represented by the tuple (S, O, P, R, 7). Here, 8 denotes the environment state space, O is the space of observations perceived by the agent, and A is the space of actions. At each turn f, the agent produces an action a E A.. In response, the environment transitions from state 8 to a new state S1+1 according to the state transition function P(s++1|$;, a,) and emits scalar reward r; R(s,, az). The agent then receives a new observation ot+1 E O, which is a partial view of the new state, sampled from ((-(s++)) The agent's 'ss objective is to learn policy To that maximizes the expected cunulative discounted return over a trajectory, maxg TF,,, P.Q EE, ?"rr], where 7 E [o. 1] is the discount factor. In the VI.M agent setting, the policy To is parameterized by a VLM that takes in images and textual descriptions as observations, and outputs language token sequences as actions. A summary of important notations can be found in Table 4. A detailed discussion of our POMDP formulation and its motivation can be found in Appendix A.I.
2.2 Visual State Reasoning an Internal World Model
Existing RL frameworks for VLM reasoning [24] are primarily based on single-turn optimization, which limits their ability to capture evolving interaction context. To better address the demands of multi-turn agentic tasks, we optimize the entire interaction trajectory, inspired by [11, 25]. We propose to maintain an internal belief of its interaction that evolves over time, and we design a training framework that integrates world modeling into multi-turn trajectory optimization for visual state reasoning, as shown in Figure 1. Our implementation is based on ver][26].
Reasoning Trajectory Rollout for Multi-Turn VLM Agents. Each trajectory begins with an initial state So, observation Ou and goal g. The observation or is visual images observed at turn for VLM agents, which we normally input to VLMs together with corresponding textual prompts. The VLM agent then generates an action de using the current policy my. To reason about visual states and build internal beliefs, the generated action de is a sequence of text tokens including both reasoning tokens 2, and executable action at:
a₂ (Ze, of)
After of parsed and executed, the environment produces reward r; as the feedback and transitions to new state $1+1, providing new observation 0++1 to the agent. This process is repeated over T turns to collect a trajectory 7 (su, Og. au, "u......s-11. or 0r- 11, ST, 0r).
Visual State Reasoning as World Modeling, Instead of just acting to what the VLM agent sees, we explicitly reason about visual states build internal belief of the environment. In agentic tasks, especially those defined as POMDPs, the agent's visual observation o, offers a view of the true world state st. To act effectively, the agent should interpret what the current visual observation actually represents and predict future state changes. We train the VLM agent to do this by explicitly generating its reasoning tokens 2 as building up an internal belief of structured world model [1], including a state model that interprets the present (StateEstimation and a transition model that predicts the future TrransitioModeell into a state belief & that approximates the hidden true state st, highlighting the central challenge of POMDPs compared to fully observable MDPs. Meanwhile, Transit: tioooodellmnnn reasons over the belief of the potential next state $1+1, allowing the agent to simulate internally how its action at turn I will transition the states. This predictive step is crucial for planning of a multi-turn trajectory. We implement multiple reasoning strategies, ranging from minimal (NoThink, FreeThink) to structured
In detail, StateEstimation grounds visual observations o
3

---

world modeling (StateEstimation, TransitionNodell. WorldModeling), by shaping the reasoning tokens 2, differently to capture and update internal beliefs about visual states.
1. NoThink: we train the VI.M agent generate only executable action aj, and the output action token a, is <tink></thiikk>asseee
</ </answer>,
2t 0
2. FreeThink: we train the VLM agent to produce any form of natural language reasoning, allowing visual state reasoning to emerge without a predefined structure. The agent generates action tokens as <think>2, </think> <answer>a </answr>>
+ 0, and is natural language tokens.
3. StateEstimation we train the VLM agent explicitly verbalize the current state belief & given the visual observation o,, to approximate the true underlying state 8, and help better predict the executable action a,. The action token a, is formatted as <thinn>><obeervation &, </ /bbservation> <reasoning> å, </reasoning> </think> Sanswer>a<</ansvee
2t (8t, az), learning to approximate & 8t
4. comsitionModelll we train the VLM agent to explicitly simulate in its internal belief space about the state 81+1, which helps reason over the executable action of with best expected reward. The output action token o₂ is structured as <thinnk<<easoning âe </reasoning> <prediction>$(41 </prediction> </think> Kanswer>a<</ansker
2 (ā,, 8(+1), learning to approximate $441 + 81+1
5. WorldModeling: is required both describe the current state and predict the next state: Sobervaiiis </observation> <reasoning> à, </reasoning> <prediction> 4411 </prediction> </think> <answer>af</answer
4 ($t ae $141), learning to approximate & + $t. $1+1 + 81+1
To encourage strict adherence to such a structured action format, we incorporate a format reward foormatt during training, following the strategy of DeepSeek-RI [20]. The reward design is detailed in Section 2.3.
Policy Optimization. Once trajectories are collected, we begin the optimization phase using an actor-critic approach. The actor's policy अe is updated using the Proximal Policy Optimization (PPO) objective [27]. Denote F token sequence converted from the trajectory т with an encoder E, (0) (F Fss)) the probability ratio between the current and old policies, and let E<g denotes {u
the prefix of token i. The PPO loss is defined as:
JPPO (0) = E. Mloss Σ M! min (u, (0)A,, clip u; (0), 1,,,,,,,,))
where .!less is a mask that is 1 for action tokens and 0 for observation tokens, A, is the per-token advantage and is clipping hyperparameter.
Ad Advantage and Value Estimation. Concurrently, the critic parameters á of the value function V updated by minimizing the squared error between its predictions and the target values Y;:
JCitiic (x)
Σ M (Vo(Fect1) -Y,)²
Σ, M MIONs
The actor and critic updates require computing per-token advantages A, and target values Y. In our VAGEN-Bas Base setting, we use Token-Level Generalized Advantage Estimation (GAE) [22]. At each token index we apply per-token KL penalty that encourages the current policy to frozen reference policy Feef The penalty is scaled by a coefficient 3 > 0. For intermediate
stay close
For simplicity, we represent the belief of next state in t-th turn as 8 Please note that in the turn of i + 1, we re-generate state belief' 6.41 given the parallel observation o.- rather than using the predicted state belief' in the t-th turn.

---

(a) Sokohan
(h) aro
(c) Navigation
(d) PrimitiveSkill
(e) SVG Reconstruction
Figure 2: Examples of visual states from five environments used in our study.
action tokens, reward r; is set to the KL penalty. At the final action token I, reward ry is set as the sum of KL penalty and total trajectory return I?(T) - R(st, az).
We then calculate the comporal-differcee (TD) error using discount factor y and the GAE parameter A:
6, = r, 7V -
4
This recursion is initialized at the end of the sequence with A, 81, j denotes the index of the next action token after token (skipping the observation tokens). The target value for the critic update is defined as
Y; A, + (Fcc).
The iterative process of trajectory collection, advantage estimation, and policy update continues until convergence. We present detailed illustration of our multi-turn agent RL framework in Algorithm 1.
2.3 Reward Design in Different Environments and Tasks for VLM Agents
To systematically analyze the learning dynamics and visual reasoning capabilities of VLM agents, we developed an evaluation suite featuring five distinct agentic tasks (Figure 2). These tasks were chosen cover wide range of challenges, including diverse visual state representations and action spaces: Classic Grid Puzzles (Sokoban and Frozenl Lake), Embodied 3D Navigation (Navigation), Detailed Object Manipulation (PrimitiveSkill), and Abstract Geometric Reconstruction (SVG Reconstruction).
Sokoban [28]: In this classic puzzle, the agent must push all boxes target locations. The visual state is a 2D grid, and the action space is discrete (up, down, left, right).
FrozenLake [29]: The agent navigates 2D grid to reach goal while avoiding holes. The visual state and discrete action space are similar to Sokoban. We disable the "slippery" setting for determinism.
Navigation [7, 30]: In this 3D embodied task, the agent follows instructions to find an object, perceiving the world through first-person view and using discrete actions (e.g., moveahead).
PrimitiveSkill [31-34]: The agent controls a Panda Arm to perform complex manipulation, using hybrid action space (e.g.. pick(x,y, The agent must ground objects in the third-person 3D scene to list of coordinates.
SVG Reconstruction [9]: The agent's goal is to generate SVG code that replicates target image using an onen-ended text action space.
Reward Strategy and Metrics. For the SVG task, the reward R(s,, a;) is dense similarity between the generated image Igen and the target Turget, computed weighted combination of DreamSim and DINO scores. Task performance is reported as the average DreamSim E., [g(T)) and DINO E.~D[h(7)) similarity of the final output. For the other tasks, R(s,, at) scaled binary reward (e.g., {0, 10}) indicating whether the trajectory completes the objective, and performance is measured by the average success rate (T))] where f(r) {0, 1). We modified some environments form original setting to enable efficient RL training, full implementation details are provided in Appendix A.4.
'Due to Gemini 2.5 Pro's safety policy, subset of evaluation responses for PrimitiveSkill and Navigation blocked; reported results based solely the unblocked test cases.
5

---

Table 1: Reasoning internal world models given visual states, including both StateRatimation and cramsitionMoellingg can significantly improve the RL performance. Test success rates are reported for Sokoban, FrozenLal ak, Navigation and PrimitiveSkill. Test Dino and DreamSim scores are reported for SVG Reconstruction task. Best performance is in bold. We gray out results without controlled variables (i.e., same size and architecture), shown for reference only. VAGEN Full with world modeling enhancement on reward shaping and credit assignment will be introduced in See so we also gray out, shown as a reference.
Model/Method
Sokohan Lke
Navigation
PrimitiveSkill
SVG
Overall
Base Common Average Place Stuck Drawer Align| Average Dino DreamSim Average
Open-Source Models
Owen2.5-V1.-79 [35]
0.18
0.44
0.72
0.75
0.73
1.00
0.50
0.00
1.00
0.44
0.89
0.62
0.76
0.51
Owen2.5-VL-78 [35]
0.13
0.14
0.28
0.39
0.34
0.00
0.00
0.00
0.83
0.55
0.27
Qwen2,5-VL-33 [35]
0.14
0.14
0.22
0.27
0.24
0.00
0.00
0.00
0.00
0.00
0.80
0.27
0.54
VLM-RI-3B [24]
0.13
0.13
0.31
0.34
0.33
0.00
0.00
0.00
0.00
0.00
0.81
0.28
0.55
0.23
VAGEN: Multi-Tumo RL
with World Model Reasoning for Visual States
(Backbonc: Qwen2.5-VV
VL--3B)
FreeTbink
0,57
0.68
0.67
0.67
0.67
1.00
0.63
0.00
1.00
0.66
0.91
0.64
0.78
0.67
KoThink
0,57
0.09
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.89
0.62
0.76
0.28
StateBetination
0,56
0.68
0.78
0.69
0.74
0.00
0.00
0.00
0.00
0.00
0.92
0.64
0.78
0.56
in
0.41
0.76
0.67
0.59
0.62
1.00
0.63
0.63
1,00
0.82
0.89
0.64
0.77
0.68
WorldModel ing
0.61
0.71
0.78
0,80
0.79
1.00
0.88
0.88
0.88
0.91
0.90
0.65
0.78
0.76
VAGEN- Full
0.79
0.80
0.81
1.00
0.88
1.00
0.91
0.79
RL Baselines with World Model Reasoning Strategy (Backbonc:
Qwen2,5-VL-3B)
Vanilla-PPO PPO
0.18
0.21
0.32
0.25
0.29
0.00
0.00
0.00
0.00
0.00
0.83
0.44
0.64
0.26
GRPO w/ Mask
0.20
0.57
0.88
0.81
0.85
0.00
0.00
0.00
1.00
0.25
0.92
0.66
0.79
0.54
Tum-PPO w/ Mask
0,38
0.70
0.78
0.84
0.81
0.00
0.00
0.00
1.00
0.25
0.89
0.64
0.77
0.55
Proprietary
Models
GPT-5 [36)
0.70
0.77
0.75
0.81
0.78
1.00
0.63
0.00
1.00
0.66
0.95
0.75
0.85
0.75
[37]
0.60
0.78
0.81
0.75
0,78
1.00
0.63
0.00
1,00
0.66
0.92
0.71
0.82
0.73
o4-mini [37]
0.44
0.75
1.00
0.50
0.00
0.90
GPT-4o [38]
0.43
0.54
0.75
0.69
0.72
0.50
0.63
0.00
0.88
0.50
0.91
0.69
0.80
0.60
Gemini 2.5 Pro [39]
0.58
0.78
0.63
0.63
0.63
0.63
0.63
0.00
0.75
0.50
0.93
0.78
0.86
0.67
Gemini 2.0 [40]
0.28
0.61
0.50
0.63
0.56
0.75
0.13
0.00
0.25
0.28
0.93
0.74
0.84
0.51
Claude 4.5 Sonnet [41]
0.31
0.80
0.67
0.67
0.67
0.63
0.50
0.00
1.00
0.53
0.95
0.81
0.88
0.64
Claude 3.7 Sonnet [42]
0.25
0.69
0.48
0.47
0.47
0.63
0.13
0.00
1.00
0.44
0.94
0.77
0.85
0.54
2.4 What Can We Reason About Visual States?
Off-the-Shelf VLMs struggle to solve multi-turn agentic tasks. We benehmark 7 models including proprietary models and open-source models in Table 1. We use different reasoning strategies to prompt models, which are detailed in Appendix B.1. Notably, VLMs that are trained to solve non-agentic tasks, such as VLM-R1, I, does not show multi-turn agent task advantage. Instead, most models struggle on these tasks, with the best-performing GPT-5 reaching 0.75 out of 1 in overall score. Particularly, no model succeeds on the PrimitiveSkill Drawer task. These results indicate a significant capability gap in current VLMs to reason about complex multi-turn visual agentic tasks.
World Modeling can improve visual reasoning via multi-turn RL. To examine whether explicit reasoning about visual states improves performance, we train the Qwen2.5- VLL-3B model with five reasoning strategies using VAGEN-Bast Base (Section 2.2). As shown in Table 1, FreeThink consistently outperforms NoThink, particularly in embodied environments like Navigation and PrimitiveSkill, which indicates the importance of explicit reasoning in multi-turn decision-making tasks.
Among different reasoning strategies, StateEstimatin and specific strengths. StateEstimation performs well in Navigation tasks, where understanding current observations is the key. In contrast, Transitions wwweeiiiiooeeeeliii lng achieves strong results in Primi- tiveSkill, where predicting future states is crucial for manipulation. However, each strategy alone may lead to reduced performance in tasks where model's prior is less aligned with the task structure or state complexity.
show task-
Ono the other hand, the combined mance across all tasks, as the trained model achieves a substantial improvement over its untrained counterpart (+0.55), and even outperforms all proprietary models despite smaller scale. These results demonstrate that explicitly visual states reasoning is crucial for VL.M agents. the following studies, we use WorldModel ing as the general visual reasoning strategy.
ionoodelingggn strategy results in strong and stable perfor-

---

Table 2: Examples of how visual converted into internal beliefs using natural language, symbolic and
structured representations. Visual State
Natural Language
Symbolic
Structured
"player: 'box 1..I.
"The player at the upper-lell, the box to the right the player, the target below the player"
P.O. X
'target' [2,0].
#
'wall": [0.2]
Table 3: Performance comparison across different visual state representations. shows that the representations are task-dependent.
PrimitiveSkill
Visual State Representation
Sokoban
FrozenLake
Place
Stack
Drawer
Align
Average
Natural
Natural-Lanaguag Lanaguage
0.61
0.71
1.00
0.88
0.88
0.88
0.91
Structured
0.28
0.63
1.00
0.88
0.88
1.00
0.94
Symbolic
0.49
0.49
Existing RL methods are inadequate for multi-turn VLM agents. We also compare our VAGEN framework with different RI. baselines. Vanilla PPO [11, 25] fails due lack of observation token masking. For image/text-to-text models, learning from observation tokens is fundamentally incorrect as image tokens are not part of the model's generation process. For image/text-Be tagee/eexxtto-mmme/eeeet models, learning from observation tokens might also be problematic because: (I) observation tokens not generated by the agent's own policy, and (2) lengthy observation sequences can dominate the learning weight distribution. Group Relative Policy Optimization (GRPO) [43] with masking remains insufficient due to high trajectory diversity from scene change, requiring unaffordable sample sizes. Turn-level PPO with masking [13] underperforms because uniform advantage estimates for action tokens within a turn cannot capture individual token contributions to policy performance. These limitations motivate our VAGEN framework's 's design for effective VLM agent training.
3 How Can We Represent Internal Beliefs about the World?
To further understand visual state reasoning, we investigate how different visual state representations affect task performance. We consider three representations: Natural-Lanaguage, Symbolic, and Structured format, as shown in the example of Table 2. Specifically, during RL training, we prompt models use the WorldModeling reasoning strategy and require them output the specific format for the <observations and <prediction> fields. We conducted these experiments in three tasks: FrozenLake. Sokoban, and PrimitiveSkill. FrozenLake and Sokoban, we compare all three formats. The Natural- format consists of free-form textual descriptions. The Symbolic format uses environment-naiie native grid-base biiddbaeed symbols, and the Structured format requires the model to output dictionary containing dask-specifie information such as players', targets' and boxes' positions, which are detailed Appendix C. For PrilitiveSkill, we compare Natural- Natural-Lanaguaa and Structured formats.
Results and Insights. Our experiments reveal depe-depnndent trade-off in the choice of visual state representation. As shown in Table 3, the optimal format varies with the task nature. In moozenlake and Sokoban, Lanaguae outperforms Symbolic and Structured formats. This is likely because the model lack sufficient prior knowledge to interpret symbolic layouts effectively, and structured outputs, when derived from image-only input, are noisy due to limited grounding capabilities. In these environments, the flexibility and familiarity of capabilities gained from pretraining stage. For PrimitiveSkill, Structured slightly outperforms Natural- Lanaguage. It's probably because we provided structured object position list as prompts to the model, allowing the model to ground its understanding more precisely and facilitate more accurate next state prediction. Consequently, for our subsequent studies, we adopt Natural-Lanauuag as the default, general-po purppss state representation, while specifically employing the Structured format for the PrimitiveSkill task.
align better with the model's
7

---

Can World Modeling Help with Reward Shaping and Credit Assignment?
Recognizing the effectiveness of enforcing VLMs to explicitly reason as a world model on StateEstimation and TransitionWodellng further explore to explicitly leverage these signals to inform reward structures and optimize the advantage estimation for the reinforcement learning framework.
4.1 WorldModeling Reward
Reward shaping is common practice to guide the specific agent behavior. We aim to intro- duce a reward that supervises the agent's understanding of visual states. Specifically, we extract <observation> and spredictions fields from the agent' 's response, compare them with the ground- truth visual states, and give a reward based on the matching score.
Our initial attempt use CLIP [44] based image-text similarity to calculate the reward. However, we found CLIP to be insufficiently sensitive fine-grained spatial and geometric details, rendering the resulting reward signals unreliable.
To address this limitation, we adopt an LLM-as-a-Judge [21] approach. We try get text-based ground-truth information about the visual state from the environments. For example, in Sokoban, we obtain 2D positions of the player, boxes, and targets; for FrozenLake, we extract 2D positions of the player, target, and holes; for PrimitiveSkill, we derive object names and their coordinates; and for Navigation, we calculate relative distances and directions from objects to the player.
With this text-based state information available, we compute a WorldModeling Reward by assessing the alignment between the agent's reasoning (in <observation> and <prediction> and the ground truth states (s, and 8t41). This is achieved through a hybrid evaluation protocol where an L.M-as-a-Juddee either provides direct judgment first extracts structured information from the agent's 's text for a subsequent rule-based comparison (e.g., F1-score). The reward at each turn t is defined as 5
reason
3, llstatesartimetooo(Be..
B. Irramaitionttdeeinng (S Sc+1),
where T a generalized matching score (binary from direct judgment or a continuous, normalized score from rule-based metrics), and 3x, 8r are reward coefficients. Details on the LL---as-a-Jude prompts and evaluation protocols are available Appendix D.1.
4.2 Bi-Levell General Advantage Estimation (GAF)
The PAGEN-Base framework described in Section 2.2 reveals a key limitation when incorporating WorldModeling Reward: by aggregating all task rewards at the final token of a trajectory. it provides only trajectory-level feedback. This coarse signal is propagated backward via single GAE calcula- tion, making it difficult to assign credit for turn-specific successes or failures, which is especially critical for reinforcing step-by-step visual reasoning. To address this, we introduce Bi-Level GAE more granular credit assignment mechanism designed to deliver fine-grained, turn-level reward signals. As illustrated in Figure 3, this approach operates in two stages, introducing a turn-level discount factor Yumm for transitions between turns and a token-level discount factor "ioken for tokens within a single action.
Turn-level advantage estimation: We compute an advantage estimate for each turn in the trajectory. For given turn let ri be the total reward assigned to that turn (its composition will be detailed in Section 4.3). We define the turn-level TD-error 5him using the critic's 's value estimates Vo at the end of each action sequence:
stum
+ Yum V (F (Fgo.))
Here, Fsa, denotes the full token prefix of the trajectory up to and including the action oz For the final turn T the next state value (o(Tsar) considered zero. The turn-level advantage A is then calculated recursively using GAE:
A
8tumn + "hum A turn A
'Please Note 3, regenerate state belief and next state belief each turn.
8

---

This backward pass is initialized with AT = 8T T--1 where Aum [0, 1] is the GAE parameter for inter-turn credit assignment.
Token-level advantage estimation: After computing all turn-level advantages (A0mh T-11 perform second, inner GAE calculation for the tokens within each action dz- The reward for any given token 7, within the action a, is defined as its KL-penalty, = , The token-level TD-error and advantage are calculated for all tokens belonging to the action dg:
turn
KT. t,
Tioken V (F( Vo(Fe.c))
A.i
token
Stoken + 'okeen Atokon Alrken
The key step linking the two levels occurs here: the backward pass for the token-level advantages initialized by setting the advantage of the final token of action d, to the pre-computed turn-level advantage, A!m. This injects the turn-specific feedback at the end of the action and allows it to be propagated backward to tokens that generated it. The detailed procedure outlined in Algorithm 2 in the Appendix.
Observation token Action token Reward
Token-Level GAE
Aog
40
Ao.ln
1.0
Asu
An,a A
AmiL
Yuuus
Ysáuo
Vizucs
Bi-Level GAE
Yians
Arus
Asa0
Auu
AvLa
A
Vister
Visken
Figure 3: Token-Level GAE and Bi-Level GAE frameworks, Standard Token-Level GAE, where single, sparse reward at end of trajectory is propagated backward token-by-token. Our proposed Bi-Level GAE assigns rewards at each turn. first computes a turn-level advantage (purple arrows) before propagating that credit to the individual tokens within the turn (orange arrows), allowing for robust, hierarchical estimation for credit assignment.
4.3 VAGEN-Full Multi-Turn Reinforcement Learning Framework
By combining the structured reasoning strategies with our Bi-Level GAE mechanism, we introduce VAGEN-Full. In this setup, we train the agent using the grounding and world modeling. The turn-level reward r. used in the Bi-Level GAE calculation is defined as a composite sum:
ling reward to encourage explicit
Ft F
reuson
formal
+ R(s,, at).
where peassn is reward for the quality of the world model reasoning for visual states (&z and 61+1), format is the reward for adhering to the specified output structure (Section 2.4), and R(st, a.) is the sparse, task task-specific reward from the environment. The rest of the training pipeline follows the VAGEN-Base procedure (Section 2.2), with the standard GAE module being replaced by the Bi-Level GAE for advantage estimation.
Experiment Setup. We now compare VAGEN-Bass with VAGEN-Full across all tasks. The VAGEN- Base (Section 2.4) uses the forldModeeingg reasoning strategy along with format and task-specific rewards. VAGEN-Full builds on this and incorporates World Modeling Reward and Bi-Level GAE, with the reward coefficients 3, and 8., set to 0.5. For SVG Reconstruction. only Bi-Level GAE applied. Since SVG Reconstruction a multi-turn reasoning task without world dynamics, we only apply Bi-Level GAE configuration, and report VAGEN-BASE with Bi-Level GAE the VAGEN-FÜLE -UGLN results. Details regarding training time and token usage is given in Table 26.
Results and Insights. As shown in Table 1, VAGEN-Full achieves consistently better test-time perfor- mance across all tasks compared to VAGEN- Base WNrllModelingg This gap especially promi- nent in PrimitiveSkill: although both methods reach similar training accuracy (Figure 4), VAGEN-Full

---

significantly outperforms VAGEN-Bass on the test set. This suggests that CrateEstimattl and TransitionModel ing improve the agent's ability adapt to new scenes, leading better robustness and generalization ability.
4.4 Ablations
Sokoban
Frozenlake
SilnttteSkll-Auuraaa
Navigation-Auerage
Success Rate
Success Rate
Success Rate
Suczees Rate
Steps
Steps
Steps
Steps
Steps
VAGEN-Base
VAGEN-Base w/ Bi-Level GAE
VAGEN-Base w/ WorldModelling Reward
VAGEN-Full
Figure 4: Training success rate for VAGEN- -asee, VAGEN-Full and ablations. Since SVG reconstruction is multi-turn reasoning task without world dynamies, only apply the Bi-Level GAE configuration. In this section, we study the independent contribution of Bi-Level GAE and WorldModeling Reward to VAGEN. We observe interesting patterns from Figure
Bi-Level GAE alone provides significant but inconsistent gains. Its performance is highly sensitive reward sparsity and accuracy, which can lead to training instability in environments lacking dense and accurate intermediate rewards.
The WorldModeling Reward alone consistently improves upon the baseline by providing crucial learning signal for visual understanding. However, its effectiveness is limited by the coarse, trajectory-level credit assignment of standard RI..
VAGEN-Full is the most stable among all methods and performs generally well all tasks.
These observations verify that fine-graine fine-grained credit assignment (from Bi-Level GAE) and high-quality reasoning supervision (from the WorldModeling Reward) are both essential to effectively improve VLM reasoning. Additional results comparing models (by size and family) and methods are reported in Table 25. We also provide case studies and detailed analysis of agent behavior in Appendix E.
Related Work
Recent studies visual reasoning have explored visual perception and grounding [45-49] and causal tracing in VLMs [50], and also examined visual information flow in single-turn QA scenarios [51-55]. However, maintaining visual state reasoning across multiple interaction turns, like building an internal world model, remains an underexplored challenge. Concurrently with our work, world modeling has been applied to code generation [56]. Our research addresses this gap by examining how VLMS maintain and update internal beliefs of visual states consistently during multi-turn interactions, focusing on improving reasoning of the environmental dynamics across consecutive turns with RL. We provide additional related work on RI. and multi-turn agent training for LLMs and VLMs [5-7, 9, 12-14, 16, 20, 24, 40, 57-80] in Appendix F.
Conclusion and Limitations
We present a multi-turn RL framework that rewards reasoning to build an internal world model via explicit visual state reasoning over StateEstimation (grounding) and Transition iewiitioooooeein (prediction). This encourages VLM agents to explore, model transition dynamics, and update beliefs across turns. We introduce dense turn-level WorldModeling Reward that measures the accuracy of internal state simulations against ground truth, and Bi-Level GAE that first values an entire turn't 's reasoning before allocating credit to individual tokens, mitigating long-horizon credit assignment. VAGEN improves task performance and visual reasoning quality agentic benchmarks. Limitations include reliance on a specific model family and a finite set of environments; future work will broaden architectures and incorporate supervised finetuning for multi-turn visual understanding.
10

---

Acknowledgments and Disclosure of Funding
This work is in part supported by ONR N00014-23- 1-2355,, ONR YIP N00014-24-1-2111, ONR MURI N00014-24-1-2748, and ONR MURI NO0014 2-2114-2--.
References
[1] Eric Xing, Mingkai Deng, Jinyu Hou, and Zhiting Hu. Critiques of world models. arXiv preprint arXiv:2507.05166 2025. 2,
[2] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet- scale knowledge. In siirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL httpp:///penrevierr net/forumiid 1rcormmddddrr
[3] Oriol Vinyals, Igor Babuschkin, Wojeicch M Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi multi-agent agent reinforcement learning. nature, 2019.
[4] Lanxiang Hu, Mingjia Huo, Yuxuan Zhang, Haoyang Yu, Eric P. Xing, Ion Stoica, Tajana Rosing, Haojian Jin, and Hao Zhang. Imgame-hench: How good are llms at playing games?, 2025. URL https: //trxiaaaivvv//////////eaaaawava org/ htgg/bbs/265 15146.
[5] Guanzhi Wang, Yuqi Xie, Yumfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL
https://arxiv. www.vvoorg//bbee wwwg/b//g//goo//o/ee 16291. 10, 54
[6] Mohit Shridhar, Xingdi Yuan, -leeaAAle Yonatan Bisk, Adam Trischler. and Matthew Hausknecht. FLFWorld: Aligning Text and Embodied Environments for Interactive Learning. Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxix org/abs/ aba/ 2010.03768. 54
[7] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, al. Embodiedhench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.00 095600, 2025. 2, 5, 10, 54
[8] Shuyan Zhou, Frank F. Xu. Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Ncubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, 2024. URL https://openrreiieee
[9] Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. Starvector: Generating scalable vector graphics code from images and text, 2024. URI. https:
abgg/2bs/3122.... 11556. 5, 10, 54
[10] Kunato Nishina and Yusuke Matsui. Sygeditbench: A benchmark dataset for quantitative assessment of Imm's 's's svg editing capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 8142 8147, June 2024. 2
[11] Zihan Wang, Kangrui Wang. Qineng Wang, Pingyue Zhang, Linjie Li. Zhengyuan Yang. Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu. Kyunghyun Cho. Jiajun Wu. Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-tumn reinforcement lcarning, 2025. URL //arxivv//xxxv...../ org/ abs/ /504. 20073. 3,3,7
[12] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. 19446, 2024. 10, 54
[13] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alanc Suhr, Saining Xie, Yann LeCun, Yi Ma. and Sergey Levine. Fine-tuning large vision-language models as decision- making agents via reinforcement learning. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httph:://////ooo //tp:////pp/eeeeeeeee .et.
7, 54
[14] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine. Sainbayar Sukhbaatar, and Xian
Li. Sweet-rl: Training multi turn 1lm agents on collaborative reasoning tasks, 2025. URL https: abs/2533.
// //arxiv. org/ abs/ 22003 15478. 2, 10, 54.

---

[15] Lior Shani, Aviv Rosenberg. Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori. Hila Noga. Orgad Keller, Bilal Piot, Idan Szpektor, Avinatan Hassidim, Yossi Matias, and Rémi Munos. Multi-tum reinforcement learning from preference human feedback, 2024. URL https:/ /ataas//aaraaa aarxiv.iivvv../ www. org/abs/2405. 14655.
[16] Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le. Sergey Levine. and Yi Ma. Sit memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. URL https: //tp:///aivvvv//a org/abs aa/a//bb////5/ 17561.7761. 2, 10, 54
[17] Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators, 2025, URL https://arxiv org/ abs/2502 00640. 2
[18] Ming Li, Jike Zhong, Shitian Zhao, Yuxiang Lai, Haoquan Zhang, Wang Bill Zhu, and Kaipeng Zhang. Think not think: A study of explicit thinking in rule-based visual reinforcement fine-tuning, 2025. URL https://arre /arxivv. org/
16188. 2
[19] Jason Wei. Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoc V Le. Denny Zhou, et al. Thain-of-thoughn prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022. 2
[20] Dee Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025. URL https: /ttp/v...vvv//////////..//..aa./ wwtg/aa/g///oaeeoooeee /2001. 12948. 2, 4, 10, 54
[21] Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang. Wen Gao, Lionel Ni, and Jian Guo. survey on Ilm au-aee 2025. URL https://taaaet //apsii//////a/////aaeeaasaasma www.riiw.....ab...eee ha///bb///4/// 2411, 15594. 2,8
[22] John Schulman, Philipp Moritz, Sergey Levine. Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint (rXiv:5506002353 2015. 2,
[23] Karl Johan Áström. Optimal control of markov processes with incomplete state information i. Journal of mathematical analysis and applications,
1965. 3
[24] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma. Jiajia Liao, Qiaoli Shen, Zilun Zhang. Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vim-rl: A stable and generalizable rl-style large rision-language model, 2025. URL https: //arxivvvvv//////////.////aaa org/ aba/2504 07615. 3, 6, 10, 54
[25] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang. Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms reason and leverage search engines with reinforcement learning, 2025.
URL httpa: h/app::///rxxiv org/ /200 09516. 3,
[26] Guangming Sheng, Chi Zhang. Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. Proceedings of the Twentieth European Conference on Computer Systems. Association for Computing Machinery, 2025. URL attpp://ddon org/ 10. 3444//5663311.1. 0198903..666675... 3696075. 3
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://axxxxi org/ abs//7777.664// 4
[28]
Max-Philipee B. Schrader. gom-sokoban. https:///githi http://.......maaarroo/ggggg/ee/
2018. 5
[29] Mark Towers, Ariel Kwiatkowski, Jordan Terry. John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A standard interface for reinforce- ment learning environments, 2024, URL https: / //tr//axxxvvv www.oorrg//////////// 17032. 5
[30] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AL arXiv, 2017. 5
[31] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li. Tongzhou Mu. Nan Xiao, Arna Gurha, Viswesh Na- gaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025. 5
12

---

[32] Sanjana Srivastava, Kangrui Wang. Yung-Chieh Chan, Tianyuan Dai. Manling Li, Ruohan Zhang. Mengdi
Xu, Jiajun Wu, and Li Fei-Fei. ROSETTA: Constructing code-based reward from unconstrained language preference. Workshop on Continual Robot Learning from Humans, 2025. URL httpss//pppeeeeeeiiee htttforun?id-xuDPUne
[33] Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA). pages 7477-7484, 2022. doi: 10.1 (O0//CCCRA4663322222222296535
[34] Ayano Hiranaka, Minjune Hwang, Sharon Lee, Chen Wang. Li Fei-Fei, Jiajun Wu, and Ruohan Zhang- Primitive skill-based robot learning from human evaluative feedback. 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7817-7824, 2023. doi: 10.1109/R055555. 2023,103411 10341912. 5
[35] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile and beyond, arXiv preprint arXiv:2308. 12966, 2023.
model for understanding, localization, text reading.
[36] OpenAL. Introducing gpt-5, 2025. 6
[37] OpenAL. Introducing openai o3 and o4-mini, 2025. 6
[38] OpenAL. Gpt-4o system card, 2024. URL https: //trxiva.vvv////////// org/ wwgg/bs////aaa/ 21276.
[39] Google. Gemini 2.5: Our most intelligent ai model, 2025. 6
[40] Gemini Team. Gemini: A family of highly capable multimodal models, 2025. URL https://arxiv
org/aba/2311 11805. 6, 10, 54
[41] Anthropic.
Introducing claude somnet 4.5, 2025.
[42] The claude model family: Opus, sonnet, haiku. 2024. URL http:p/s/pppp..emnntcccccha....o CorpusID: 268232499,
[43] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https:
org/ /as///b///// 2202 03300. 7
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021. URL. btttps:/pproceedii: radford21a.. html. 8
mlr, press/v139)
[45] Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning, 2025. URL https:///arrtt /ttps//////av.//// org/ abs/2505. 23678.
[46] Shenghang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie, Eyes wide shut? exploring the visual shortcomings of multimodal lmss. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[47] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llavanext: Improved reasoning, ocr, and world knowledge, 2024.
[48] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang. Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 37: 87310-87356, 2024.
[49] Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2862-2861 2023. 10
[50] Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. Towards vision- vision-language mechanistic interpretability: A causal tracing tool for blip. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages
2023. 10
13

---

[51] Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. Understanding information storage and transfer in multi-modal large language models. arXiv preprint arXiv:2406.0422 2024.
[52] Clement Neo, Luke Ong. Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language language models, 2025. URL https: pestps:/aarie /ttp///aa/vv///////////eee/awava org/asg//////oeeooooooooooeo abs/2410. 07149.
[53] Omri Kaduri, Shai Bagon, and Tali Dekel. What's in the image? a deep-dive into the vision of vision language models, 2024. URL https: : htt////rrvvvv///// Saarg//bb//// /2411 17491.
[54] Hugo Laurengon, Leo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models? In The ehirty- Annual Conference on Neural Information Processing Systems, 2024. URL btttps///oppe /ttp:////ppeeeieeeeeeeee net wow.mmidddddr.I11/222..
[55] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba: Visual state space model. In The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /oppereveeeeeeeeee wwt///eunnnieee htte://ooommiiiZtttlRRBL
10
[56] Meta. Cwm: An open-weights I1m for research on code generation with world models, 2025. 10
[57] Marwa Abdulhai, Isadora White, Charlie Victor Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. LMRL gym: Benchmarks for multi-turn reinforcement learning with language models, 2024. URL https: /tap::://ppmaes http////ppeeeeeeeeee/ee/ee/eeeee ew. net/forum feammmddd-ccille dcBcMMMrrRbb.. 10, 54
[58] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer Fl-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravee, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022. URL ht /wtxxv/xxvvvv////////////////v org/ abs/2204. 008622. 54
[59] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. 54
[60] William Chen, Oier Mees, Aviral Kumar, and Sergey Levine. Vision-language models provide promptable representations for reinforcement learning. arXiv preprint orXiv:2402.02651, 2024. 54
[61] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-grn policy optimization for Ilm agent training. arXiv preprint arXiv: 10978, 2025. 54
[62] Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025. 54
[63] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs. et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024. 54
[64] Pengxiang Li, Zechen Hu, Zirui Shang, Jingrong Wu, Yang Liu, Hui Liu, Zhi Gao, Chenrui Shi, Bofei Zhang, Zihao Zhang, et al. Efficient multi-tumn rl for gui agents via decoupled training and adaptive data curation. arXiv preprint arXiv:2509. 233666 2025. 54
[65] OpenAL. Openai 01 system card, 2024. URL https: wwwaaaiwwvw...aaaaaaaaaae htt.b////2..... 16720. 54
[66] Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbch Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner. Eric Undersander, and Tsung- Yen Yang. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. In International Conference on Learning Representations (ICLR), 2025. alphabetical author order. 54
[67]
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li. Zhengyuan Yang. Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self evolution in Ilm agents via multi-tun reinforcement learning. 2025. URL https: / /arxivivvvv org/ absgg//b////000 20073. 54
14

---

[68] Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310./9921, 2023.
[69] Arth Shukla, Stone Tao, and Hao Su. Maniskill-hab; A benchmark for low-level manipulation in home rearrangement tasks. CoRR, 0bss/2412.132111111 2024. doi: 10.48550AARXI.2442..222 URL https: wwwpg/....855//// 000.444555/rraa arwivvi/arii..o 2412. 13211. 54
/ //doi. aooo..rrg/00
[70] Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural 2020. 54
information processing systems, 33:3008-300
[71] Zhiqing Sun. Sheng Shen, Shengcao Cao. Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint wrXiv2339.145223 2023. 54
[72] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalic Mackraz, Devon Hjelm, and Alexander T Toshev. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023. 54
[73] Zihao Wang. Shaofci Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.000 0600000..........00 2023. 54
[74] Yufei Wang. Zhanyi Sun. Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vim-f: Reinforcement learning from vision language foundation model feedback. In Proceedings of the 4 International Conference on Machine Learning, 2024. 54
[75] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai. Yifei Li. Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). Association for Computational Linguistics, 2024, URL https: h/ta:////caaaaaa///a/a// or org/2028 acl.24..cll--00 acl-long.. 510/. 54
[76] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mohile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. 54
[77] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. -o-eeviis:::: Advancing generalized multimodal reasoning through modal formalization, 2025. URL https://taxcett org/ abs / /503. 10615. 54
[78] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang. Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu. Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma. Ya-Qin Zhang. Lin Yan, Mu Qiao. Yonghui Wu, and Mingxuan Wang. Dapo: An open-source I1m reinforcement learning system at scale, 2025. URL https: / arxiv. wwwpi.oorgaaa..aaaea..e...oowete abs/2503. 14476. 54
[79] Hengguang Zhou, Xirui Li. Ruochen Wang, Minhao Cheng. Tianyi Zhou, and Cho-Jui Hsieh. RI-zero's "aha moment" in visual reasoning 2b non-sft sft model, 2025. URL https://arxiv. arg/abs/253.. 05132. 54
[80] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodci, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferenees. arXiv preprint vrXiv:I909.0833 2019, 10, 54
[81] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 19256. 2024. 18
[82] Stephanic Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, volume 36, pages 50742-50768. 2023. URL aattps://arxtt org/ abaa 221./3206000 09344. 23
[83] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
15

---

Appendix
Table of Contents
A World Modeling for Visual State Reasoning via Multi-Turn RL
17
A.1 Problem Formulation: VLM Agent Training under POMDP Formulation
17
A.2 Multi-Turn Reinforcement Learning in VLM Agentic Tasks
17
A.3 VAGEN Framework
A.4
Environments and Tasks for VLM Agents
A.5 Reward Assignment
22
A.6 Evaluation Metrics
23
B What Can We Reason About Visual States?
24
B.I Bottleneck of Off-the-Shelf VLMS on Agentic Tasks
24
B.2 Reasoning in Multi turn RL Training
29
B.3 Learning Dynamics of RL Baselines
35
C How Can We Represent Internal Beliefs about the World?
35
C.1 Symbolic Representations
36
C.2 Structured Representations
37
D How to Improve Reasoning over World Models?
37
D.1
World Modeling Reward via LLM-as-a-Judge.
37
Mitigating Reward Hacking via Structured Evaluation Repetition Penalty
45
D.3 Bi-Level General Advantage Estimation (GAE) Details and Additional Results
46
E Case Study
47
E.1 World Model Reasoning Enhances Spatial Understanding and Multi Step Planning
47
E,2 Response Convergence and Reduced Exploration
49
E.3 Reward Hacking and Over-optimization.
52
F Extensive Related Work
G Summary of Findings
54
16

---

Table 4: Summary of important notations used in the VAGEN framework.
Notation
Symbol Description
Observation
The raw visual observation at turn t.
State
81
Ground-trr -ruth environment state at turn t.
State (Belief)
81
VLM's belief of state s, at tum 4.
Action
The action tokens at dz (ze, ai), augmented with reasoning tokens se. VLM's helief of action a; at turn t.
Action (Belief)
ag
Action (Executable)
a
The parsed, executable component of ai.
Reward
Ft R(s,, ax) where R is the reward function. Rollout (su, 4o, ru, $r; or))
Scalar reward
Trajectory
Tokenized Trajectory
Token sequence VLM training, concatenating encoded observations and actions.
Agent Policy
We
Policy parameterized by a VLM with parameters
Critic / Value Func.
V4
State-value function parameterized by 4: the i-th token. Fayt tokens i-j. Feyi prefix to i - 1. Fr.st the i-th token of the f-th turn.
Token Index
Trajectory Reward
R(T)
Sum of rewards over trajectory, E, R(s;, a:).
Advantage Estimate
As: advantage for token é; A advantage turn t; Alkes advantage for
token turn f.
Temporal- Cemporal-Difference Error 8
d; TD-error for token dmm. TD-error for turn s TD-error for token tum
Discount Factor
7
E [0, 1]; "Noken: within-turn discount; "Nom: across-turn discount.
Appendix
A
World Modeling for Visual State Reasoning via Multi-Turn RL
In this section, we detail our training pipeline, algorithm and system.
A.1 Problem Formulation: VLM Agent Training under POMDP Formulation
We choose to formulate our problem a POMDP setting rather than a fully observable MDP for two main reasons.
Inherent Partial Observability in Environments. Many of the environments we study, such as the Navigation task, does not satisfy the assumptions of an MDP. In these environments, the agent does not have access to the complete, true state of the environment (sy) at each timestep. Instead, it receives an observation (e.g., a first-person visual scene from its current vantage point) which constitutes only a partial view of 8t. To gain more comprehensive understanding of the environment and to locate targets critical information, the agent must actively explore, such rotating its viewpoint or moving to a new location. This necessity to explore and gather information is a hallmark of POMDPs.
Methodological Congruence with POMDPs. We optimize policies over the full sequence of observations, actions, and reasoning steps by applying teacher-forcing strategy during updates. This allows the model to leverage the entire trajectory as context for current decisions, ensuring consistency between rollout and update and making our method especially effective under partial observability.
A.2 Multi-Turn Reinforcement Learning in VLM Agentic Tasks
Our multi-turn reinforcement learning framework for VLM agents is shown in Figure 5. At each turn t, the VLM agent receives the current observation oe (comprising visual image and an optional textual prompt). Based on o and its interaction history, the VLM generates a structured output a,. This output a; includes an explicit reasoning component 2, (e-g., <think><observatiion>.. <prediction>... seriiionn.......bbee
and an executable action component of (e.g., <answer> Cansver>..K/ansseer </answer>> The executable action of parsed and sent to the environment. The environment then transitions to new state, providing the next observation ot+1 and scalar reward r'.. This cycle repeats for N turns
17

---

to form trajectory. The VLM's parameters are updated using reinforcement learning, specifically Proximal Policy Optimization (PPO), based on the collected trajectories. We leverage world modeling for visual state reasoning and design training framework that integrates it with multi-tum trajectory optimization, as shown in Figure
Oz
Og
Ta
VLMs
Output
Grounding
Modaling
<Cansio
< </anc>>
Multi-tur Interaction with Environment
Figure 5: Multi-turn RL framework for VLM agents.
Trajectory Buffer
Reward
Environment
Action
Environment
cobservation-Thee the dining <think>
Visual Reasoning
Training Loop
Format
right. </abaeraatio <reasoning> forward to get closer to the table. Then, I can right
Observation
Policy Update
table. </reasonings sprediction>I will move coolee</predietion </think
Rollout Loop
PPO
You home and perform navigation taaks according to instructions.
Bi-Level
conevernnoveshedd
noveright</answee
Advantage
VLM-Agent
Estimation
Figure 6: Overview of the VAGEN Framework. The framework operates in dual-loop structure: inner Rollout Loop data collection and an outer Training Loop for policy optimization.
The detailed training algorithm for our multi-turn RL framework is presented in Algorithm 1.
The detailed reasoning templates given Table 5,
A.3 VAGEN Framework
To support our experiments, we developed VAGEN, a framework built upon VeRL[81]. VAGEN specializes in multi-turn reinforcement learning (RL) training for Vision-Language Models (VLMs), integrating various RL algorithms and environments. We adopt emv-as-service design, which decou- ples training from interacting with environments, enhances scalability.
A.4 Environments and Tasks for VLM Agents
Sokoban The action space and hyperparameters are given in Table and Table 7.
Table
Action space for Sokoban environment.
Name
Description
Up
Move the agent cell upward on the grid.
Left
Move the agent one
the the grid.
Right
Move the agent one cell to the right on the grid.
Down
Move the agent cell downward the grid.
18

---

Algorithm VAGEN-Base: Reinforcement Learning for VLM Agents
Input: Actor VLM Te, Environment, Critic Va, Reference VLM Tmf Hyperparameters: Discount factor ?y GAE coefficient X, PPO clip range e, KL penalty coefficient
Phase 1: Trajectory Collection
1: Initialize trajectory T = J: Sample initial state and observation so, Og from Environment
2: for t O,..., - do
3: Sample action de To [o(-)Teai autoregressively, where Tica, is the history.
4: Parse executable action of from 96-
5: Execute of in Environment to get 8t-1, oc+1 and task reward R(se; ae).
Append (8t, 0c, 0t R(8,, a+)) to trajectory T.
7: end for
8:
Construct training sequence = [E(oo): ((n):: ... :(or- E((rr Store old policy probabilities Tau (F. Fee)) for all generated tokens
9:
Phase 2: Advantage Estimation (Token-Level GAE)
10: Let ". be mask that is 1 for action tokens and 0 for observation tokens.
11: Calculate total trajectory task reward R(T) ΣT- R(s,, az).
12: for cach action token index
0,...
do
13: Calculate KL penalty: KL = 8 KL (T<(())<))) (F)))
14: Initialize token reward
KL.
15: end for
16: Add trajectory reward to final action token: 1 + 1/71-1 + R(T).
17: Initialize advantage A|+| = O.
18: for i = |-|- 1, 0 (backwards) do
19:
Find the least index j of an action token where >
20: Compute TD-error:
rı + (V((T<)) Vo(T<)). (Assume (a = 0)
21: Compute advantage: A, = 6, + 7AAy.
22:
Compute return for critic update: G, A, + Vo(Fer).
23:
end for
Phase 3: Policy Update with PPO
24: for each action token index = 0,..../5! do
25:
Compute probability ratio: u0) sa(7; r.s)) (ou(F//Tes)
26: end
27: Compute PPO objective:
(0) JPP
.ul E; M. a min (1 -,,
28: Compute critic loss:
JLmmic (6) = Σ M. E, M (Vo(Fes) - -
29: Update parameters 0 and & using their respective gradients.
Table 7: Hyperparameters for the Sokoban environment.
Name
Value
Description
dim room
(6, 6)
Dimensions of the Sokoban grid
max_ max_steps
100
Maximum number of steps allowed per episade
num_boxo boxes
1
Number of boxes be pushed targets
min actions_too Loctonsttt__uueee succeed
5
Minimum number of actions required to solve the puzzle
max actions per step
3
Maximum number of actions the agent can take per turn
turns
3
Maximum number of turns the agent interact with the envi- ronment
FrozenLake The action space and hyperparameters are given in Table and Table 9.
19

---

Table Reasoning stralegy formats.
Strategy
Format
NoThink
<answer>a(</ansee>>
FreeThink
<think>s;<thhnnk>
<ansver>df<aansee>>
StateEstimation
<think>
cobervvatiopssccobeervvttt
treasoningg(ns/rraassninng
</think>
<answer>d<<ansser>
TransitionKodeling
<think
<reasoingghic<rresonnig>>
sprediction0&411/ppeiiiiioo
</think>
/nswer>uu<<ansser>>>
WorldModeling
<think>
<olarrattionn::kaaerrrtttoo
<reasoniiggarc/roassonnng
speediction33:::4ppreeiicii
</think>
<answer>a<</ansve>>>>
Table 8: Action space for the FrozenLake environment.
Name
Description
Up
Move the agent one cell upward on the grid.
Left
Move the agent cell to the left the grid.
Right
Move the agent one the right on the grid.
Down
Move the agent one cell downward on the grid.
Table 9: Hyperparameters for the FrozenLake environment.
Name
Value
Description
desc
None
Environment map layout (if None, randomly generated)
is slippery
False
Whether the frozen tiles are slippery
size
4
Size of the square grid
max_ aetaaacttttcreeaeee_ _pe_sttppep_ppppp
3
Maximum number actions the agent can take per turn
min actions to succeed
5
Minimum number of actions required to reach the goal
turns
3
Maximum number of turns the agent interact with the envi- ronment
Navigation The action space and hyperparameters given in Table 10 and Table 1l.
20

---

Table 10: Action space for the Navigation environment.
Name
Description
MoveAhead
Move forward by some distance
MoveBack
Move backward by some distance
MoveRight
Move rightward by some distance
MoveLeft
Move leftward by some distance
RotateRight
Rotate to the right 90 degrees
RotateLeft
Rotate to left by degrees
LookUp
Tilt the camera upward by 30 degrees
LookDown
Tilt the camera downward by 30 degrees
Table 11: Hyperparameters for the Navigation environment.
Name
Value
Description
resolution
255
Resolution of the rendered images
down sample ratio
1.0
Ratio for down-sampling images
fov
100
Field view angle degrees
multiview
False
Whether to use multiple camera views
actions aetioonsppr__ttee per_step
Maximum number of actions the agent take per turn
success threshold
success_threshold
1.5
Threshold considering task successful
step_length
0.5
Distance traveled in a single movement action
turns
4
Maximum number of turns the agent can interact with the envi- ronment
PrimitiveSkill The action space and hyperparameters are given in Table 12 and Table 13.
Table 12:
Action space for the PrimitiveSkill environment.
Name
Description
pick(x, y.
Grasp an object located at position (x, y. the robot's workspace
place(x. y.
Place the object currently held by the robot's 's gripper at target position (x, y, 2)
push(xl, y1, z1, x2, y2, 22)
Push an object from position (x1, y1, zl) to position (x2, y2, z2)
Table 13: Hyperparameters for the PrimitiveSkill environment.
Name
Value
Description
max actions per step
2
Maximum number of actions the agent can take per turn
max turns
3
Maximum number of turns the agent can interact with the envi- ronment
SVG Reconstruction The action space and hyperparameters given in Table 14 and Table 15.
Table 14: Action space for the SVG Reconstruction environment.
Name
Description
SVG Code
Open text format allowing for the specification of SVG markup code
21

---

Table 15: Hyperparameterss for the SVG Reconstruction environment.
Name
Value
Description
datasct_name name
harvector/svg-icons-oooi
Datasct used for SVG examples
max_turns
2
Maximum number of turns the agent interact with the envi- ronment
A.5
Reward Assignment
Sokoban The reward structure for the Sokoban environment is presented in Table 16.
Table 16; Reward structure for the Sokoban environment.
Reward Type
Value
Description
Success reward
10
Awarded when all boxes are placed on target locations
Failure penalty
-0.1
Applied each step when the task is not completed
Box placement reward
1
Granted for each box pushed onto target location
Format reward
0.5
Provided at cach turn encourage visual state reasoning
Grounding reward weight
0.5
Weight applied to StateEstimate reward
World modeling reward weight
0.5
Weight applied to irannitionMMdell.. ing reward
FrozenLake The reward structure for the FrozenLake environment is presented in Table 17.
Table 17: Reward structure for the FrozenLake environment.
Reward Type
Value
Description
Success reward
10
Awarded when agent reaches the goal position
Failure penalty
-0.1
Applied cach step when the task is not completed
Format reward
0.5
Provided each turn encourage visual state reasoning
Grounding reward weight
0.5
Weight applied to StateEstimation reward
World modeling reward weight
0.5
Weight applicd to TransitionModell reward
Navigation The reward structure for the Navigation environment is presented in Table 18.
Table 18: Reward structure for the Navigation enviromment.
Reward Type
Value
Description
Success reward
10
Awarded when agent reaches the goal location
Failure penalty
-0.1
Applied each step when the task is not completed
Format reward
0.5
Provided at each turn to encourage visual state reasoning
Grounding reward weight
0.5
Weight applicd to StateEatimation reward
World modeling reward weight
0.5
Weight applied to TransitionModel11. reward
PrimitiveSkill The reward structure for the PrimitiveSkill environment is presented in Table 19.
22

---

Table 19: Reward structure for the PrimitiveSkill environment.
Reward Type
Value
Description
Success reward
10
Awarded when the manipulation task is completed
Failure penalty
-0.1
Applied each step when the task is not completed
Stage-based reward
(stage + 1) x 2
Granted upon completing key manipulation subgoals (where stage is the highest successfully completed stage)
Format reward
0.5
Provided each turn to encourage visual state reasoning
Grounding reward weight
0.5
Weight applied StateRatimation reward
World modeling reward weight
0.5
Weight applied to TramsitionModelling reward
SVG The reward structure for the SVG Reconstruction environment is presented in Table 20.
Table 20: Reward structure for the SVG Reconstruction environment.
Reward Type
Value
Description
Image similarity
Variable
Weighted DreamSim [82] and DINO [83] scores measuring sim-
ilarity between generated and target images
Format reward
0.5
Provided each
encourage visual state reasoning
Grounding reward weight
0.5
Weight applied to StateEstimation reward
World modeling reward weight
0.5
Weight applied to Transit t onModeling reward
DreamSim weight
5.0
Scaling factor applied DreamSim similarity scores
Dino weight
0.0001
We only use DreamSim score for reward
A.6 Evaluation Metrics
We employ a range of metrics to evaluate agent performance across our diverse task environments. For trajectory-basee evaluation, we define several key functions over trajectory
f(r) (0, 1): A binary success indicator function that equals if the trajectory success- fully completes the task, and otherwise
g(T) [0, 1]: The DreamSim similarity score between the target image and the final generated image in trajectory
h(r) E [0, 1]: The DINO similarity score between the target image and the final generated image in trajectory r
Metrics for Task- Tasp-Compl Environments For Sokoban, FrozenLake, Navigation, and Primi- tiveSkill environments, we use the average success rate over dataset D of test trajectories:
Success Rate [],-
A trajectory is considered successful (f(r) 1) when the agent completes the specific task objectives for each environment.
Metries for SVG Reconstruction For the SVG Reconstruction task, where success is measured by the similarity between generated and target images rather than discrete goal achievement, report:
DreamSim Score = Er~D [g(T)]
(2)
DINO Score E,-Dlh(r)
These complementary metrics capture different aspects of visual similarity:
DreamSim Score: Measures perceptual similarity between images using a pre-trained diffusion model [82].
DINO Score: Based on self-supervised visual features from a small version of the DINO model [83].
23

---

B
What Can We Reason About Visual States?
B.1 Bottleneck of Off-the-Shelf VLMs on Agentic Tasks
To understand the capabilities and limitations of existing VLMs agentic tasks, we evaluate several state-of-the-art off-the-shelf VLMs across our proposed tasks. The prompts used for evaluating these models follow the FreeThink reasoning strategy format: <think>i,</thin <answer> oj </ 'answer>.
Sokoban Prompt for evaluating off-the-shelf VLMS
System
You are Sokoban solver.
Sokoban Quick Guide
Goal: Push all boxes targets. Symbols (If image provided there no symbols):
Wall I 1- Floor 0 Target X Box I P You Box on Target You on Target
Rules:
Push boxes (can't pull).
Avoid walls.
Actions you can take: Left, Down, Right, Up.
You can take up to 3 action(s) at a time, separated by You should first give your reasoning, and then your ansver. Your response should be the format <think> <thinkt...<thinnneee </tthhhiiibanseerrr..... </aaserrrr..//./>>>>>>
e.g. Sthinn<creassning"TTh box is one step below me, and the target is two steps below me, need to go down then push the box down to the target. wtt.:///wnnnnggb///aaaaaaaaeeeee Downsa/aa
User
[Initial Observation]:
<image> Decide your next action(s).
You take up 3 action(s) You should first give your reasoning, then your answer. Your response should be in the format of: <think>... <athin/ </answerr> Assistant
time, separated by
<think>Since there are no boxes or targets visible, the first step is move the character (P) to the wall to see if it will push the wall and open path. This might also help in accessing more space to maneuver. </think> <answer>Left, Up, Right</ansuer>
reat the
FrozenLake Prompt for evaluating off the-shelf VLMs
System
You FrozenLake solver. FrozenLake Quick Guide
Goal: Reach the goal (G).
Symbols (If image is provided there are no aymbols):
Frozen I 0 Hole G Goal P Player X Player fell into hole Player
goal
Rules:
1. Avoid falling into holes.
2. Frozen tiles are slippery, you move perpendicular to your intended direction.
Actions you can
Left, Down, Right, Up.
You can take up to 3 action(s) at a time, separated by You should first give your reasoning, and then your answer.
24

---

Your response should be the format of:
<think>.. </think><nnswr>>... </ansswer////
e.g, <think>I can see the target is on my down left, I should go down then Lttp://nnnnnaaseerrooooaaaaaaa
left to reach the User
[Initial Observation]:
<image>
Decide your next action(s).
You take up 3 action(s) time, separated by You should first give your reasoning, then your answer. Your response should be in the format of: h/chau.....ihiinnnssee........
Assistant
<think>There is present the bottom right, which likely the goal. Since the player is near the present and the goal is usually at the bottom right, I should move downwards to reach it, wtt.//htii Down, Down</ansver>
of the ...
Navigation Prompt for evaluating off-the- thelf VLMs
System
You are home robot and perform navigation tasks according to instructions. Actions you can take: moveahead, moveback, moveright, moveleft, rotateright, rotateleft, lookup, lookdown.
moveahead: Move forward by some distance moveback: Move backward by some distance
noveright: Move rightward by some distance
moveleft: Move leftward by
distance
rotateright: Rotate the right by degrees rotateleft: Rotate to the left by 90 degrees
lookup: Tilt the camera upward by 30 degrees
lookdown: Tilt the camera downward by 30 degrees
Rewards:
Format correct: +0.5
Achieve the human instruction: +10.0
The instruction will be provided with each observation. Look at the image carefully and navigate to complete the instruction. Hinta:
1. You can take multiple actions at a time, in most cases, you find the target object is far away from you, you can call moveahead, moveleft and move right multiple times.
be stuck, you lookdown if there's 's any object above or below you, you can also rotate to see there's any object behind you. Example: Round 1:
2. If you find yourself
image_1
<think>I can see the garbage can in the upper left corner of image, next to the kitchen sink. To move there, we can forward-left, but since there's 's kitchen counter directly ahead, we should go left first. Following the strategy, can go by first moving leftward. </tinkk cansver>moveleft, coveleft</answer> Round 2:
Env lnv_feedback: Last action is executed auccessfully.
image_2
that by moving leftward, are getting closer to the garbage can. Now, the garbage can in front me, slightly to the left. And there's a large area ahead of us. Following the strategy, I can by first moving forward then moving leftward, Leftward,<tthink>
<think>From the secene,
25

---

Sanswer>moveathead, moveahead, aawpaaalw...alllllaa.aooeeeeaaaa Round 3: Env :ev_feedback: Last action is executed successfully. image_3
<think>From the image to our front-left. Moving leftward might blocked but can see that there is still space in front me to get closer to the garbage can. Following the strategy, we can take about two steps forward then one step left to reach the garbage can. fan.</think>
the garbage is very close to us, still
<answer3movoahead, moveabed,
Round 4:
Env_feedback: Success
You can take up to 5 action(s) at a time, separated by ',' You should first give your thought process, and then your answer. Your reaponse should be the format of: <think>... <thinio..<<thhinbaaaaa </thnnn.e... <//assser>> e.g. <think>I can see from the sight the target object is right in the top left of will move forward, then move left to access st.s/tiink-annseerooveeeeee atw.:e/iaee..eeaaaaaaaaaa.eeeeee
User
[Initial Observation]
<image>
Human Instruction: I need to dispose of some trash properly. Please navigate that object and stay near
Decide your next action(s).
You can take up to 5 action(s) at a time, separated by ',' You should first give your thought process, and then your answer. Your response should be in the format of: <..as//ve///r>
Aesistant
<think>From the image, I can see the trash can is in the bottom-right corner. need to navigate towards it. will first move forward and then the right reach it. wtt.://iinkkmmsse//oeeeeeeeeee rest the turns
rotateright</
PrimitiveSkill Prompt for evaluating aff-the-shelf VLMs
System
You assistant controlling a Franka Emika robot Your goal to understand human instructions and translate them into a sequence executable actions for the robot, based on visual input and the instruction.
Action Space Guide
You can command the robot using the following actions:
1. pxck(x, 2) # To grasp object located at position(x.y,2) in the robot's workspace.
place the object currently held by the robot's gripper at the target position (x,y,2).
2. place(x, y,
3. (xs, y1, z1, x2, y2, z2) # To push an object from position (x1,y1,21) to 222,
Hints:
1. The coordinates (x, y, 2) are in millimeters and are all integers.
2. Please ensure that the coordinates are within the workspace limits.
3. The position is the center of the object, when you place, please consider much higher when placing
the volume of the object. It's always fine
an item.
4. We will provide the object positions to you, but you need to match them
to the object in the image by yourself. You're facing toward the negative
26

---

x-axis, and the negative y-axis is your left, the positive y-axis is your right, and the positive z-axis up.
Examples:
round1:
imagel
Human Instruction: Put red cube on green cube and yellow cube on left target Object positions:
[(62,-55, 20), (75,33,20) (-44, 100, 20), (100, -300,-43,0000 -3,00), (100, 43,01 00,
Reasoning: I green cube on my right Since I'm looking toward the negative x axis, and negative y-axis is to my left, (62, 20) would be the position of the red cube, (75,33 175,33,20 20) would be the position of the green cube and (-44,100, 20) is the position of the yellow cube.
from the picture that the red cube is my left and
near me.
Also the (100,-43, (100, 43,0) would be the porition of the right target. need to pick up red cube first and place it on the green cube, when placing, I should set much higher. Anwer: pick(62, -5,
would be the position the left target, and
round2:
image2
Human Instruction: Put red cube green cube and yellow cube left target Object positions:
[(75, 37,5,33,50) 775,33,20, (-44, 100, 20), (100,-43,00) (100, 43,
Reasoning: Now the red cube is on the green cube, so I need to pick up the yellow cube and place it on the left target.
Anwer: pick(-44, 100, 20) (10.
You can take up to 2 action(s) at a time, separated by You should first give your thought process, and then your answer. Your response should be in the format of: <think> h/titt.....//hhinnaaaaaseee <<answe>r.
e.g. e.g. <think>I need pick the red_cube_ppo at (10,20,30) place it on the green_ block_block_po at
60,,60,40).... <atthnnkkansssrrppcckk 200) pplace 60, 70) < /nsswer>>>> User
[Initial Observation]
<image>
Human Instruction: Please align the cubes in the y-axis, which means the x-coordinates of both cubes should be 0 (+-10mm)
=_workspace__imie _imiit: (-100, 150)
j_vorkspace_limis (-200, 200)
2. ..orkspace_limit: (10, 200)
Object positions:
[(129, -119, 20), (108, 124, 20)]
Other information:
No other information needed Decide your next action(s).
You can take up to 2 action(s) at a time, separated by I. You should first give your thought process, and then your Your response should be the format <think>... "think>..<<thhinkccaam </thinkkaaansseerr........... </answer> Assistant
<think>First, need to pick up the red cube position (129, -119, 20) and place it position (0, -119, 20) </thinb>xnnseerppcct2222 -119, 20) Iplace(O, -119, </aswrr>
rest of the turns
27

---

SVG Reconstruction Prompt for evaluating off the-shelf VLMs
System You precise SVG code generator.
SVG Quick Guide
Goal: Transform the provided image into precise SVG code that replicates the image.
Process:
1. First analyze the image carefully, identifying distinct visual elements
2. Identify colors, dimensions, positions, and relationships between elements
3. Generate accurate SVG code that reproduces the image, you path
for better shape
Rewards:
Overall visual similarity: +5.0
Structural accuracy: +10.0
Example:
<think>I can see the image contains a red circle and a blue rectangle. The circle is positioned at the top-left, while the rectangle is the bottom-right. </think> </thhii
<answer><svg viewBox="O 100 100" xmlne="http://wu. /0rp//0000000//0/////g"""""/
<circle cx="25" cy="25" r="15" fill="red" /> <rect x="60" y-"60" width-"30" height-"20" fill-"blue" /> </svg></answer>
You can take up action(s) at a time, separated by You should first give your thought process, then your answer Your response should be in the format of: <think>.. </thn//aaaniinnsssrr....... </ansserr e.g. <think>I
the image contains red circle and blue rectangle. The circle is positioned at the top-left, while the rectangle is at the bottom-right. </tignt
<answer><avg vieuBox="O 0 100 100" malns="htpp://ww u3. w0.orr//.0////////////// <circle cx="25" cy="25" r="15" fill="red" /> <rect x="60" y="60" width-"30" height="20" fill-"blue" />
</svg></answer>
User
[Initial Observation]:
<image>
Please carefully observe the image, generate SVG code that reproduces as accurately as possible.
Decide on your SVG code.
You
up action(s) You should first give your thought process, and then your answer. Your response should be in the format of: <//aaser.....//sser>>>>>
time, separated by
Assistant
<think>Based on the image, can see circular shape with letters "kr" inside it. The letters are positioned in the center of the c/tile.//thine <answer><avg vieuBox="O 0 100 100" xmlne="httpp:/uuu..d http://www.3..og//////o/oooo <circle cx="50" cy="50" r="40" fill="none" stroke="black" stroke-width="?" />
org/2000/svg">
<text x="50" y=" font-size="40" font-family=AAriaal! fill="bled">ar</txx>>> </svg></answer> rest of the turns
The information of evaluated VLMs with corresponding hyperparameters provided in Table 21.
28

---

Table 21: Off-the-shelf VLMs for evaluation.
Model Name
Model ID
Provider
Max Tokens
Temperature
VLM-RI-3B
-ww.::Ww--------22----- -L----5VV---B--LL Math-0305
omlab
150/400
0.7
Qwen2.5-VL-3B
Qwen
150400
0.7
Qwen2.5-VL.-7B
Qwem/\Wwe2.5-VVL77B.
Qwen
1504000
0.7
Qwen2.5-VL-72B
Qwen
150/000
0.7
GPT-4o
gpt-4o
openai
150/400
0.7
Gemini 2.0
gemini-2.0-flash
gemini
1504400
0.7
Claude 3.7 Sonnet
claule-377sommet-2225500
claude
1504000
0.7
B.2 Reasoning in Multi-turn RL Training
Following the training methodology described in Section 2.2, we use reinforcement learning to train Qwen2.5-VL-3B across all reasoning strategies presented Table Our experiments are conducted on servers equipped with 8xH100 GPUs, 104 CPUs, and 1.7TB of memory. Each server can run two experiments at the same time, with each training session requiring approximately 4-8 hours to complete.
The training hyperparameters used in our experiments are detailed in Table 22.
Table 22:
Multi-tum RL training hyperparameters.
Parameter
Value
Description
Rollout Phase
Top-p
0.95
Nucleus sampling parameter for action generation
Temperature
0.7
Sampling temperature for controlling randommess
Update Phase
Advantage Estimator
masked_gar gue
Generalized Advantage Estimation with masking
Model
Qwen/Qwen2.5 -L---VVLL----tteee
Pre-trained model used for initialization
Critic Model
Pre-trained model used for critic initialization
"token
Discount
token- advantage calculation
KI. Penalty Coefficient (8)
0.001
Coefficient for Kl. divergence penalty in PPO objective
Actor Leaming Rate
Ie-6
Learning rate the actor network
Critic Learning Rate
le-5
Learning rate for the critic network
Train Batch Size
Total batch size training
PPO Mini Batch Size
32
Mini Mini-hatch size for PPO updates
We also release the prompts used in the multi-turn RL training process. For different reasoning strategies, the structures of the prompts are largely identical, with minor differences response formats, To avoid redundancy, we provide only the WorldModeling prompts here. For FreeThink prompts, please refer to Appendix I. For other reasoning strategy prompts, please see our provided code or infer them from the existing prompts.
Sokoban Training Prompt for WorldMode ing Strategy
System
You are Sokoban solver.
Sokoban Quick Guide
Goal: Push all boxes targets. Symbols (If image provided there are no symbols):
Wall I- Floor 1 Target X Box I P You Box on Target You on Target
Rules:
Push boxes (can't pull).
2. Avoid walls.
Actions you can take: Left, Down, Right, Up.
You can take up to 3 action(s) at a time, separated by
29

---

You should first give the description of your observation, then your reasoning, then predict the next state, finally your answer. Your response should be in the format of: http://:wwa....aaaaaaaaaaaaaaaaa
htt.:p/po.....oooooooooooooooaa. <//nnsuer>>>>>>> e.g. <chinae<caberrvationnnnn box below the player and target below the boxx/bberratioon-tteaaonnnn need to go down then push the box down to the crggt//eaaciigggppediiccoaaa.
player will be above the box, the
target and box will be at the same
User
[Initial Observation]:
<image> Decide your next action(s). You take up action(s) You should first give the description your observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: <think>oobsevattion o w/w.bbeeeettttim/arrooo.o....... <//eessoonn/ / hrt.a//aw.....cooooooooo/ooooaaa <//nnserr> Assistant
time, separated by
player is at the bottom of the screen, and there is box to the right of the player. The target is to the left of the box. wtt.://bbrvaatioo/etaaaniinn to complete the wtt.e///oonnggeereiittiiaaaa the target, moving up, down, and to the left. //pprddcioon<//bbbl> <answeer Down, Left</answer> rest of the turns
player needs push the box the target player will push the box
FrozenLake Training Prompt for WorldModeling Strategy
System
You are FrozenLake solver.
FrozenLake Quick Guide
Goal: Reach the goal (G).
Symbols (If image is provided there are no symbols):
Frozen | 0 Hole G Goal P Player x Player fell into hole Player
goal
Rules:
1. Avoid falling into holes.
2. Frozen tiles are slippery, you move perpendicular to your intended direction.
Actions you take: Left, Down, Right, Up.
You can take up to 3 astion(s) at a time, separated by You should first describe the observation, then your reasoning, then predict the next state, and finally your Your response should be the format wat.::/awa.......eerereerraaaaa. "hrrllcc.....ccpppaccttooo////at oaaaraicccaacccaaaaaaaaaaaeaaear </ansrer e.g. <thinbbxabaerrvatiionnnhee player is the above the targett//oberratttooorrrreerrrrr should go down then left reach the
farett//eassnnnn--ppppocccoooooo player will reach the
User
[Initial Observation]
<image>
Decide your next action(s).
You can take up to 3 action(s) at time, separated by
30

---

You should first describe the observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: httpa//eeeeeeeaaa..ooooeeeearrre Assistant
</reasoonn////gggg /appa///o...pppppocoooooo///aaaa </answer> <
<think>>oobervattionnie player is on the right side of grid. wt/b///raatiioneerrsnnnneeeeeeee which is indicated by the position the grid. httad////aaaonnnggpreeiiiiio player will to the left or http://peedctttooo/mmmmmamaueeee Left, Down</answer> rest of the turns
player is on the right side of the grid,
Navigation Training Prompt for WorldModeling lig Strategy
Systemm
You robot perform navigation tasks according instructions.
Actions you can take: moveahead, moveback, moveright, moveleft, rotateright, rotateleft, lookup, lookdown.
moveahead: Move forvard
distance
moveback: Move backward by distance
moveright: Move rightward by some distance moveleft: Move leftward by some distance
rotateright: Rotate to the right by 90 degrees rotateleft: Rotate to the left by 90 degrees
lookup: Tilt the
upward by degrees
lookdown: Tilt the camera downward by 30 degrees
Rewards:
Format correct: +0.5
Achieve the human instruction: +10.0
The instruction will be provided with each observation. Look the image carefully and navigate to complete the instruction.
Hints:
1. You can take multiple actions at time, in most cases, if you find the target object far away from you, you call moveahead, moveleft and move right multiple times.
2. If you find yourself seems to be stuck, you can lookdown to see if there's 's any object above or below you, you can also rotate to see if there's any object behind you. Example: Round 1: image_1 <chink>-obaervatioon/lee is garbage can in the upper left corner of the image, next the kitchen sink. To there, since there's kitchen counter directly ahead, we should go left first. htt.e///arratiiaeeaaaaannnnaLLa. moving leftward.
go forward-left, but
the strategy, I can go by first ooffuuarrdd/eessonneeeeeerittaaa will be infront of the gabbge//peedcttion<///hitt <answer>moveleft, moveleft</ansver> Round 2:
Env Env_feedback: Last action is executed auccessfully. image_2
<hhink-oosearatioo/Poom the secene, see that by moving leftward, getting closer to the garbage can. Now, the garbage can is front me, slightly to the left. And there's 's a large area ahead of the strategy, I can go by first moving forward then moving leftward. </rasonnng-ppreeitttoom will be closer to the garbage</prediction>>//a/
us.
<ansver>moveahead, moveahead, moveahead, moveahedd Round 3: Env fee_bacbbcck: Last action is executed successfully.
31

---

image_3
<hhinnoobserrationn2oa the image we can see the garbage can is very close to us, still to our front-left. Moving leftward might be blocked but i can see that there is still space in front of me to get closer to the garbage <w../obbrrtttoooorrsssonnngoreee
the strategy, can take about two
steps forward then step left reach garbage hon.s//raasnnng<<ppeiiccct
will reach the sarbage</predicciinn<//hia KansverPmoveahead, moveahead, moveleft</amswerr
Round 4:
Env fnv_feedback: Success
You can take up to action(s) at a time, separated by ',' You should first give your thought process with the your observation, reasoning, and prediction of next state, then your answer Both the observation and prediction should describe what you expect the environment.
Your response should be the format of: http:/rwaaaaa...../ooo/aaaiaiiia
<thinvk o h/w.../aaooaiiiiirrrrraaaai...eo <//eassoonn///
e.g. <thini>><oserratiionn> am the <aansee.....///a/se.. of bedroom. There is bed to the left, a desk with lamp the right, and closet straight ahead. The target object, a book, appears to be on the desk. book. I'1l turn right and move forward. </resonngg<<prediccioon standing in front of the desk. The desk has lamp, computer, and several books on it. The target book is within reach on the right side of the desk.
need to move toward the desk to reach the am now
moveahead, / moveeahead//nnswer
User
[Initial Observation]
<image>
Human Instruction: am looking for a luminous device on my desk to read books at night. Can you navigate to that object and stay close?
Decide your next action(s).
You take up 5 action(s)
time, separated by ','
You should first give your thought process with the your observation, reasoning, and prediction of next state, then your answer. Both the observation and prediction should describe what see or expect to in the environment.
Your response should be the format of: <thint>-bbervttinn htt.://wwwwwa..........aaaaaaaaa httpaa//w...cppciiccoaaoaaaaaaaa <//nnserrr>>
Assistant
luminous device the desk the right of the chair. To reach it, should move forward and then turn
<hiink--observviioo/TTeeer
right.</obsert w/barrvttimxxrresonnnaallll
the strategy, I can go by first moving forward then turning right to reach the luminous
device. </ressoning<<prddctiion will be close to the luminous device< http://pppeciiiooc///aaaaaaaeeee rest the turns
moveahead, moveright</answer
PrimitiveSkill Training Prompt for WorldModel ing Strategy
System
You AI assistant controlling Franka Emika robot Your goal is to understand human instructions and translate them a sequence executable actions for the robot, based on visual input and the instruction.
Action Space Guide You can command the using following actions:
1. pick(x, z) # To grasp an object located at p2sition(x,y,2) in the robot's 's workspace.
32

---

2. (xace(x, y, To place the object currently held by the robot's gripper at the target position (x,,,2).
3. p y1, z1, x2, y2, z2) # To push an object from position (x1,y1,21) to (x2,
Hints:
The coordinates (x, y, 2) are in millimeters and are all integers.
2. Please ensure that the coordinates are within the workspace limits.
3. The position is the center of the object, when you place, please consider much higher when placing
the volume of the object. It's always fine
an item.
4. We will provide the object positions to you, but you need to match them to the object in the image by yourself. You're facing toward the negative x-axis, and the negative y-axis is to your left, the positive y-axis is to your right, and the positive z-axis is up.
Examples:
round1:
image1
Human Instruction:
on green cube and yellow cube left target
Object positions:
(66,,-55,
(-44, 100, (100, -4, , (100, 43,00,43,00 43,0)]
Reasoning: I can see from the picture that the red cube is my left and green cube is my right and near Since I'm looking toward the negative x axis, left, (62,-55, 20) would be the position of the red cube, (75,33, 20) would be the position of the green cube and (-44,100,20) 20) is the position of the yellow cube.
negative y-axis to my
Also the (100,-43,0) would be the position of the left target, and (100,43,0) would be the porition of the right target.
need to pick up red cube first and place it on the green cube, when placing, I should set much higher.
Anwer: (62, 20) (pl.cce7,,33,35551
round2:
image2
Human Instruction: Put red cube green cube and yellow cube on left target Object positions:
[(75,33,50) (75,33,20) (-44, 100, 20), (100,-43,00 (3,0)
Reasoning: Now the red cube on the green cube, so need to pick up the yellow cube and place it on the left target. Anwer: (ick,-44,100, 20) (10,c(100,,-33550
You can take up to 2 action(s) at a time, separated by You should first give your thought process with reasoning and prediction of next state, and then your ansver.
Your response should be in the format of: <thinnt>
</think>><ansuer>.... </ansver>>
red cube is on the left at (100, 100,00000 the green cube is the rigatt/oseervatoooorresoooo cube (100, 40) and place it (80, 100, 60) /tt.e//aaian.peeeccttttoott.eeee will be at (80, 100, 60) /tet:///ppeeiiiiooo///hiia
<thinkk-
need to pick the red
executing this action, the red cube
Sanswer>pick1100,p 100, 40) (80,ce 100, 60) </answer>
User
[Initial Observation]:
<image>
Human Instruction: Please put the apple in the drawer and close the drawer.
Avorkeppace_iiiit (-500, -100)
y_workspace_limit. (-600, 200)
2_ wwokkspace_limit (10, 650)
Object positions:
33

---

[(-127, 192, 36), (-500, -537, 200)] Other information:
('drawer_ope) _ereeoopee_tieeee value': opt_o_eeenea: 263)
Decide your next action(s)...
You can take up to 2 action(s) at time, separated by 1.
You should first give your thought process with reasoning and prediction of next state, and then your answer.
Your response should be in the format of: <think>><bbervattonn> w/waaaaataooooaeaaaaaoeeeaaittia </reasoonn <prediction" htrpa///a...oooooooooooo///aaaaa Assistant <thanie"-oberrvvtioon/Tooo apple is on the ground, right side of the wwweer//oobbrrrttioo<<raaaaooo
</answer> <//anss
need to pick up the apple first, as it is on the floor. After placing the apple in the drawer, I will close the drawer. wtt.///saningggreediitiiaaoa/aaa
apple will be placed in the drawer and the drawer will be closed. </peddctiiaooetiiibbeeeaaeeeeeee 36) Iplace -537, 500) (-500, -537, 500, -500, -537, 650) </ /answer>> rest of the turns
192,
SVG Reconstruction Training Prompt for Worl dMode ing Strategy
System
You are precise SVG code generator.
SVG Quick Guide
Goal: Transform the provided image into precise SVG code that replicates the image.
Process:
1. First analyze the image carefully, identifying distinct visual elements
2. Identify colors, dimensions, positions, and relationships between elements
3. Generate accurate SVG code that reproduces the image, you cam use path
for better shape
Rewards:
Overall visual similarity: +5.0 Structural accuracy: +10.0
Example:
<think><observattonnn
image containing red circle positioned
the top-left of the canvas, approximately coordinates (25, 25) with radius units. There also a blue rectangle at the bottom-right area, sized about 30x20 units and positioned at coordinates (60,
60) wt//bbbrattiabxeessonnnggaaaaaae
my observation, need to create an
SVG that precisely matches these elements. The circle appears be slightly too far right, so should adjust x-coordinate 20 instead 25. The rectangle could benefit from being slightly wider wider.</rean s/reesooingg-ppeeiitttii resulting SVG should similarity score to improve approximately 0.98, the modified positions and dimensions will better represent the original seaphic.//peedccioon//hhiii
implementing these adjustments, the closely match the original image. expect the
<answer><avg viewBox="O 0 100 100" halns="http://wur wwtpo/.///o//////////// <circle cx="20" cy="25" r-"15" fill-"red" /> <rect x="60" y="60" width-"35" height="20" fill-"blue"
</svg></answer>
You can take up to 1 action(s) at a time, separated by You should first give your thought process with the your observation and reasoning, then predict Both the observation and prediction should describe what you see or expect to see in the environment.
state, and finally the answer
Your response should be in the format of:
34

---

<thinn>><obsrrvationn http:///aaarrrataooooooo.eeeeare </bbbbrrtttooo/ttrraoiii........ </resooonng> <prediction> ... httpp///pp.cccccoco/////aaaaaaaa e.g, <think><obeervationn
</answeer>>>>>
can see an image containing a red circle positioned at the top-left area of the canvas, approximately at coordinates (25, 25) with radius of 15 units. There is also blue rectangle at the bottom-right area, sized about 30x20 units and positioned coordinates (60, 60). voberrationn<reesnniggaaaee on my observation, I need to create an SVG that precisely matches these elements. The circle appears to be slightly too far right, 25. The rectangle could benefit from being slightly wider. s/rresoningxxprrdccttoooo//
should adjust its x-coordinate to 20 instead of
implementing these adjustments, the resulting SVG should more closely match the original image. expect the similarity score to improve to approximately 0.98, as the modified positions and dimensions will better represent the original graphic.</predicee </preddiciiinnx//hiitt
<answer><svg viewBox="O 100 100" malne="http://wur. ww....//0000/////"""
<circle cx="20" cy="25" r="15" fill="red" /> <rect x="60" y="60" width="35" height="20" fill="blue" /> </svg></answer>
User
[Initial Observation]:
<image>
Please carefully observe the image, and generate SVC code that reproduces it as accurately possible.
Decide on your SVG code.
You can take up to 1 action(s) at a time, separated by You should first give your thought process with the your observation and reasoning, then predict Both the observation and prediction should describe what you see or expect to see in the environment.
state, and finally the <think><observationn> .4/bbservatiomm/rrramoiiig
Your response should be in the format of:
</reasoning . http://awwwatttooooooo..eeeaaaaa prpedictioon...ppreeiiiiii Assistant
can see an image of the letters "kr" inside circle. The circle is centered at the top of the image. The letters are inside the circle, with the circle having consistent thickmess around the perimeter.</ofe <//berratiion<<raasiiinin.. circle with consistent thickness around it and the letters "kr" inside. The circle should be centered and have specific size, and the letters should be placed inside the circle.</reasen < htt.r//asooingggpprddttttttttlla SVG will need to accurately depict the circle, the letters "kr", within the circle. The circle's size and position should be precise, and the letters should be correctly placed inside. inside.</predie htttee/e/eeda.iinn/tiiiiiii
SVG should accurately represent the
their positioning
<answer><avg viewBox="O 0 100 100" milns="htpp///uu u3. wrg.//00///g""' <circle cx="50" cy="50" r="25" stroke="black" strche-width="1" fill="none"
/>
<text x="50" y="50" font-size="24" fant-family='sann-errrf "mxt-ae aligmment weggeent-baselinneeeeeee "oateoo"oooaal""///ttttt </svg></answer>
rest the turns ...
B.3 Learning Dynamics of RL Baselines
C
How Can We Represent Internal Beliefs about the World?
In this section, we release details of different visual state representations. To further understand visual state reasoning, we investigate how different visual state representations affect task performance by prompting models to use the WorldModeling reasoning strategy and requiring them to output specific formats for the <observation> and <prediction> fields.
35

---

Sokoban
Frozenlake
Primitive -Avergee
Navigation Average
SVG
Suozen Rate
Success Rate
22
Aournag Sccare
Steps
Steps
Steps
Vanilla-PPO
GRIO w/ Mask
Turn-PPO wf Mask
VAGEN Base
Figure Training rates for RL Baselines and VAGEN-Base
explore different visual state representations across our environments: Sokoban and FrozenLake support three representation formats (Natural-Lanaguage Symbolic, and Structured), while Prim- itiveSkill supports two formats (Natural- Laniruua and Structured). For all other tasks, we use Natural- visual state representation as the default. Examples of Natural Natural-Lanagun Lanaguage visual state representations for all tasks can be found in the prompts provided Appendix B.2.
The different visual state representation formats are summarized in Table 23:
Table 23: Visual representation formats across diffcrent environments.
Environment
Format
Type
Description
Natural-Lanaguaa
Text
Free-form textual descriptions of the game state
Sokoban
Symbolic
Grid
Environment-native grid-bused symbols representing game entities
Structured
Dictionary
JSON-like dictionary containing positions of game objects
Natural-I Natural-Lanaguage
Text
Free-form textual descriptions the game state
FrozenLake
Symbolic
Grid
Fnviromment-native grid-based symhols representing game entities
Structured
Dictionary
JSON-like dictionary containing positions of game objects
Natural Natural-Lanagen Lanaguage
Text
Free-form textual descriptions of the manipulation state
PrimitiveSkill
Structured
Dictionary
JSON-like dictionary containing 3D coordinates of task- relevant objects
C.1
Symbolic Representations
For symbolic representations, we use environment-specific symbols to represent different entities in grid format.
Sokoban Symbolic Format uses the following mapping:
Wall Floor Target
Target X Box
You
Box on Target | You on
example of the symbolic representation:
###**#
#_P__#
#__O_
###__#
FrozenLake Symbolic Format uses the following mapping: Frozen
Hole
G Goal
P Player X Player fell into hole Player goal
example of the symbolic representation:
G
0
36

---

_P--
C.2
Structured Representations
For structured representations, we use dictionary- based formats containing precise positional infor- mation of relevant objects.
Sokoban Structured Format contains player, box, and target positions along with grid dimensions. An example of the symbolic representation:
posiition: (4, 2), box_ box_positions: [(3, 2)],
target_positions: [(3, 1)], grid_size: (6, 6)}
FrozenLake Structured Format contains player, target, and hole positions along with grid dimen- sions.
An example of the symbolic representation:
(player._position (3, target_position: (3, 2), hole_positions: [(1, 2)], grid_size: (4, 4)}
PrimitiveSkill Structured Format contains 3D coordinates task-relevant objects. An example of the symbolic representation:
(red_cube: (100, 100, 40), green_cube: (200, 200, 60)}
These different formats allow to investigate how the choice of visual state representation affects the model's ability to reason about and predict state transitions across different types of environments.
D How to Improve Reasoning over World Models?
D.1 World Modeling Reward via I.L--asaaJuudg
To enhance visual state reasoning capabilitics, we implement an Ludgae-a-Judde framework that eval- uates the quality of the agent's 's StateEstimatinn and TransitionModell reasoning. The judge model used our experiments is GPT-4. nano, which evaluates whether the agent's descriptions and predictions accurately match the ground truth state information.
Our evaluation system uses task asek-specific prompts for both StateEstimatior (current state ground- ing) and Transith ti onModel ing (next state prediction) assessment. The prompts are structured to guide the judge model through step-by-step reasoning before providing binary YES/NO evaluation.
Sokoban StateEst imation Evaluation
Evaluate whether the description accurately captures the key position relationships in the Sokoban game state. Answer YES if the directional relationships correct, NO if they contain directional errors.
# Context
You are evaluating whether the description correctly identifies the directional relationships between: The player and the box(es)
2. The box(es) and the target(s)
The description doeen't need to be perfectly precise mention every detail it just needs have the directional relationships (Up, Down, Left, Right).
Example:
37

---

or Groundtruth Current State Information: ['box0 is the and to the the player', 'targeto above and on the left side the player', "target0 is above and on the left side of box0'] State Description: The player is below the box and the target is below the box.
<think>The state description contains spatial relationship information, further analysis. According the ground truth data, box0 at same row and to the left of the player, target0 is above and on the left side of the player, target0 is above and on the left side of box0. The description states 'The player is below the box and the target is below the box. The target
player actually at the same row as actually above the box (not below). Both directional relationships are incorrectly
(not below),
Example:
Groundtruth Current State Information: ['box0 is above on the right side of the player', 'targetO is above and at the same column as the player', 'target0 is above and the left side of box0']
State Description: The box is above the player and the target is to the left of the box
<think>The state description contains spatial relationship information, do further analysis. According to the ground truth data, box0 is above and the right side of the player, target0 is above and at the same column the player, target0 above and the left side of box0. The description states 'The box above the player
to left the box. It correctly identifies that the box is above the player (box0 is above and on the right side of the player). It also correctly identifies that the target is to the left of the box (target0 above and the left side box0). Both key directional relationshipa described. </thinio<<nsseer/BB////m
the target
accurately
# Groundtruth Current State Information: {state_inf inf.roaaitiiinniicc...
State Description: lantuaal
Think step by step:
1. Relative Relationship Requirements:
Must describe at least one relationships BETWEEN entities (player-box, player-target, box-target)
Absolute positions like "player is the left side" insufficient
Need relational descriptions like "player is left of target"
2. Essential Relationships to Check
Player-Target relationship (highest priority)
Player-Box relationship Box-et relationship
3. Equivalent Expression Recognition "box is above player" "player is below box" "target left of Must recognize these as identical spatial relationships. Absolute position is not alloved
"box right of target"
Your should be in the format of Shhint....<<thhiikkaaaaaas <think>..
38

---

Sokoban Transiti cwmnsiiioModeliing Evaluation
Evaluate whether the prediction correctly anticipates the key position relationships that will exist the Sokoban state. Answer the predicted directional relationshipe are correct, or NO if they contain directional errors.
Context
are evaluating whether the prediction correctly identifies the directional relationships that will exist after the move:
1. The future position of the player relative to the box
2. The future position of the box (ex(es) relative the target(s)
Important:
The Prediction Comes First
Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task is to check if the prediction correctly anticipated what actually happened.
If the box target are same position, this prediciton seen as success immediately (YES)
Example:
Groundtruth Next State Information: ['bor0 is above and the right side the player', 'targetO above and on the side of the player', 'targetO is above and on the left side of box0']
Next State Prediction: The player will be to the left of the box, and the box will be the right of the target.
<think>The prediction state contains spatial relationship betueen player and target, do further analyeis. According to the ground truth data, box0 above and on the right side of the player, target0 is above and on the left side of the player, targetO is above and the left side of box0. The description states 'The player will be the left of the box, and the box will to the right of the target. It correctly identifies that the player is to the left of the box (since box0 is on the right side of the player). It also correctly identifies that the box is to the right of the target (since targetO is the left side of box0). Therefore, this description correctly identifies relationships. htt.iipp../thhiinnaaeeeeee //thhnbbemmser>2S5///eeeee
key directional
# Groundtruth Next State Information:
(state_inf intate_innfrmaaiindddde.t
# Next State Prediction: (natural laagalllaagaagaaeeaaaaaae
Think step by step:
Relative Relationship Requirements:
Must describe at least one relationships BETWEEN entities (player-box, player-target, box-target)
Absolute positions like "player is the left side" insufficient
Need relational descriptions like "player is left of target"
2. Essential Relationships to Check
Player-Target relationship (highest priority) Player-Box relationahip Box-Target relationship
3. Equivalent Expression Recognition "box is above player" "player is below box" "target left of box" "box is right of target"
Must recognize these as identical spatial relationships. Absolute position is not allozed
39

---

if contains errors. Your should be the format of
<think>... ..onk>...////... h/tp:////////aaas/////////a/a/aa or
<think>.. <think".../thinieee htta://////aaaaaaaaa////////aaaa
Lake StateEst inteesstttaiii on Evaluation
Evaluate whether the description accurately captures the key position relationships in the FrozenLake game state.
Anawer YES if the directional relationahips correct, NO if there errors.
# Context
You evaluating whether the description correctly identifies:
1. The directional relationship between the player and the goal (MUST Have)
2. The directional relationship between the player and the hole (if present)
The description doesn't 't need to be perfectly precise the correct directional relationships between the player and target (Up, Down, Left, Right), and between the player and hole if applicable.
just needs have
Groundtruth Current State Information:
iirate__informatiin_ddcc.
# State Description: (natural_ Inatural_ hawgugggeegggggggggpgppa description)
Think step by step:
1. Player relationship with Goal
Goal (Target) NUST include in state description, without target the description is automatically wrong (NO)
If there is no direction between player goal, "player is right
to the target", the description is automatically wrong (NO)
This takes highest priority over all other comsiderations
Equivalent Expression Recognition
"goal is above player" "player is below goal"
"target is left of box" "box is right of target"
Must recognize these as identical spatial relationships. Absolute position is not alloved
3. Simple Judgment Rule
If player at goal -> YES If direction aligns with needed movement >> YES Otherwise NO
Your answer should be in the format of h/tp...//chhinnnnsseeeesss.. <think>... http.//.///iiiaaaaeeee/////////
or
FrozenLake TransitionModell Evaluation
Evaluate whether the prediction correctly anticipates the key aspects of the next FrozenLake state.
Answer YES if the prediction accounts for directional relationships and potential holes,
Context
You are evaluating whether the prediction correctly identifies:
40

---

1. The position relationship between the player and the goal after the prediction
Important:
The Prediction Comes First
Remember:
The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your what actually happened.
check the prediction correctly anticipated
The prediction doesn't need to perfectly describe every aspect of the next state it just needs correctly anticipate the directional relationships (Up, Down, Left, Right) address any dangers from holes.
# Groundtruth Next State Information: {state_ings www.ionmtoooonnicc......
Next State Prediction: Lasuural_lnngugeee dtrpriiggeddecripiiollll
Think step by step:
Player relationship with Goal
If player is already at the goal position, the prediction is automatically correct (YES)
Goal (Target) MUST include in prediction state, without target the prediction is automatically wrong (NO)
If there is no direction betseen player goal, like "player is right to the target", the prediction is automatically wrong (NO)
This takes highest priority over all other considerations
2. Directional Correctness
Evaluate the predicted movement direction aligns with the relative position between player and goal
For example, if player is left of goal, moving right is correct
**CRITICAL: Recognize equivalent expressions the spatial relationshipe*
"player is above target" "target is below player"
"player is left of target" "target is right of player"
These are the SAME relationship expressed from different perspectives
3. Simple Judgment Rule If player at goal -> YES If direction aligns with needed movement >> YES
Otherwise -> NO
Your answer should be the format h/tnn//....//nkkkkawwsss//////// <think>... hiimm/...../hua//mmseee////////
or
Navigation StateEst inntreEsttiatiii Evaluation
Evaluate whether the description effectively communicates the spatial relationship between the agent and target object, even if the exact directional terms differ.
Answer YES if the overall spatial understanding is correct, NO if it fundamentally misunderstands the spatial layout.
# Context
You are evaluating whether the description effectively conveys where the target object located relative the agent. exact directional terminology (left, right, ahead, etc.) may differ between the state information and the description, but the important factor is whether the description would lead to correct navigation.
41

---

Groundtruth Current State Information: (state_inforration_.ccc.
State Description:
langural_ paspaaaeeaggggeeeeeeeippppoolll
Think step by step:
1. Check if the description contains spatial relationship between agent and target object
If no spatial relationship mentioned, answer
2. If spatial relationship exists, check if the predicted direction is consistent with the target direction "ahead/forward" "ahead" "left" "left"
"right" "right"
Combined directions like "forward-left", "forward-right" are acceptable if they include the correct primary direction
3. The prediction correct if mentions moving toward the target in direction that reasonably aligns with groundtruth direction
Your ansver should be in the format of <think>... hatt.///thhhiinasseeeeeSSS answer>> <think>... <think?...</thinnee htt..://hhiiiioaaaeeeett/////// s answer>>
Navigation Transi toonnsiitioModdeeinnn Evaluation
Evaluate whether the prediction effectively anticipates how the agent would navigate toward the target object, if the exact directional terms differ.
Answer YES if the overall navigation plan is reasonable, or NO if misunderstands or did not mention the spatial layout.
Context
are evaluating whether the prediction effectively anticipates how the agent would move to reach the target object. The exact directional terminology (left, right, ahead, etc.) may differ between the state information and the prediction, but the important factor is whether the prediction would lead auccessful navigation.
# Important: The Prediction Comes First Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task to check if the prediction correctly anticipated what actually happened.
# Groundtruth Next State Information:
{state taaate__nnfrmaaton.dcctt
Next State Prediction:
(natural_ www.gaaanngencrriiiiiiee
Think step by step:
First, check
prediction explicitly uses EXACT directional terms
that appear in the groundtruth state: "ahead", "left", "right", "up", "down"
Terms like "move towards", "closer to", "near", "approaching", "in front of" "by", "at" DO NOT qualify "Will be the beft/right/ahead" "Will move left/right/forvard" qualify
If no exact directional match to groundtruth is present, conclude with NO immediately
42

---

or 2. If explicit direction words exist, verify that they EXACTLY match the target object's 's direction the groundtruth:
If target is "ahead", prediction must specify "ahead", "forward", "slightly left", OR "slightly right" (special case: we allow slightly left/right for ahead targets)
If target "right", prediction If target "left", prediction must specify "left"
specify "right"
3. Even if the prediction mentions intermediate objects correctly, it MUST explicitly state the correct final direction to the target object
4. The prediction cannot substitute object references for directions (saying "move to instead "move right")
5. Remember that the prediction was made BEFORE the groundtruth state was determined
Your should be the format of <think>...<<thinnne htt.:.//.....oaaseer///////a/ee <think>...
PrimitiveSkill StateEstimatinn Evaluation
Compare the description of the current state with the groundtruth current state information.
Answer YES if the description reasonably matches the current state information, ND if doean't.
Context
You are evaluating whether an agent's description accurately reflects the actual state. The description should capture the meaningful relationships and positions relevant for decision-making the task.
Important evaluation criteria:
1. If the description includes coordinates, they don't 't need to be exact matches with the groundtruth
2. For coordinate values, consider them correct if they are within these error tolerances:
For and coordinates: within +10 or 10 units of groundtruth For coordinates: within +10 -10 units of groundtruth
The overall spatial relationships and object identifications should be correct
4. If the description includes dict-ffrmatted state information, that's good but not required
Groundtruth Current State Information: (state ingaats_innommaiinn.icccc
# State Description:
Think step by step:
1. Identify the key objects and their positions in the groundtruth information
Check the description correctly identifies these objects
3. For any coordinates mentioned, check if they are within the acceptable error range
4. Determine if the overall spatial understanding is correct, even if specific numbers differ slightly
5. coordinates the description differ from groundtruth but are within the error tolerance, consider them correct
43

---

Your should be the format of
<think>.. ..ank>...<//.......... ///iiiioaasseree///////ssseeee or < http:///////////aaaa//////////// <think>..
PrimitiveSkill TransitionModelling Evaluation
Compare the prediction of the next state with the groundtruth next state inf onmormaiio.
Answer YES if the prediction reasonably matches the next state information, or if it doean't.
Context
You are evaluating whether an agent's prediction of the next state is accurate. The prediction should capture the meaningful changes and relationships that will result from the planned actions.
# Important: The Prediction Comes First
Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task check if the prediction correctly anticipated what actually happened.
Important evaluation criteria:
1. If the prediction includes coordinates, they don't need be exact matches with the groundtruth
2. For coordinate values, consider them correct if they are within these error tolerances:
For and coordinates: within +10 or -10 units of groundtruth For coordinates: within +10 -10 units of groundtruth
3. The overall predicted movements and resulting spatial relationships should be correct
4. If the prediction includes a dict-formatted state information, that's good but required
Groundtruth Next State Information: (etate_informatte) int.rooiiooooonicc.....
Next State Prediction: (natural. language anngugge_descrppttoee
Think step by step:
1. Identify the key objects and their positions in the groundtruth next state information
2. Check the prediction correctly anticipates these object positions
3. For any coordinates mentioned, check if they are within the acceptable error range
4. Determine if the overall predicted movement and resulting state is correct, even specific numbers differ slightly
5. If coordinates in the prediction differ from groundtruth but are within the error tolerance, consider them correct
Your should be the format of <think>.. ... Biiink>.<</thiiii h/th:////..iiinnsseee/YY.. ansserr or <think)...<thhnn>> w/w.iiibbaasweree////aaaaaa
<think>...
Text-based State Information To provide ground truth information for the judge model, we extract and format state information differently for each environment:
Sokoban: We extract the positions of players, boxes, and targets, then convert their relative positions into natural language sentences. For example:
boxx is above and at the column as the player',
44

---

'targetO is above and at the same column as the player',
'target0 is above and at the same column as box0']
FrozenLake: Similarly, we convert positional relationships into descriptive sentences:
['target is above and at the same column the player', 'hole0 is above and at the same column as the player', 'hole1 is above and the right side of the player', 'hole2 is above and on the right side of the player',
'hole3 above and on right side of the player', 'hole4 is at the same row and to the right of the player']
Navigation: We dictionary-based information directly from the environment:
("target_obj_typo 'Toaster', prrrget_distance_to__laaaaaa 2.59, pirpettdddrectio_toooppoo
'ahead', vvisible, objjettss [('tyee': 'Cabinet', "direction, to to_player': 'left', toraancc_tt___lrer... 0.94), ('type': 'Drawer',
'left',
'distance_ta ttt_toooop'r: 1.18), ('type': 'CounterTop', "direction_to_playeee 'left', _iistance_to_player" 1.33), ...]}
PrimitiveSkill: We provide coordinate-based position information:
("1etz_targe___ositiion"" (80, -100, 0),
"righz_target..osttiirr (80, 100, 0),
'red_ cred__ubee_piie coabe__osition: (83, -102, 20). 1 beceen_cube_posttiine (-47, 95, 20)}
D.2 Mitigating Reward Hacking via Structured Evaluation and Repetition Penalty
To address reward, we developed and implemented a composite reward mechanism that combines structured evaluation with penalty for historical repetition.
The method consists of the following key steps:
1. Natural Language to Structured Reasoning Conversion: Instead of relying on a holistic, qualitative judgment from the LLM on the agent's natural language reasoning, we first use LLM to parse and convert the reasoning into structured dictionary. This dictionary explicitly captures key entities and relations within the reasoning.
2. Quantitative Evaluation with F1 Score: The structured dictionary is then programmatically compared against ground-truth dictionary representing the optimal reasoning for the given state. By calculating the F1 score between these two dictionaries, we obtain a precise and quantitative measure of the correctness and completeness of the agent's 's reasoning. This approach is significantly more robust than relying on vague text similarity or an LLM's overall impression.
3. Penalty Mechanism for Low-Entropy and Repetitive Behavior: We observed that agents can fall into pattern of over-exploiting previously successful reasoning, repeatedly generat- ing the same high-reward sentences even in inappropriate states. This behavior reflected in noticeable decrease in the entropy of its outputs. To counteract this, introduced penalty mechanism:
We maintain max-heap dynamically track the most frequently generated sentences in the agent's 's history.
When evaluating response from the agent, perform a dual-condition check:
45

---

(a) The response sentence matches one of the top frequent sentences in the heap.
(b) Its corresponding Fl score is below a predefined correctness threshold.
Only when both conditions are met do we conclude that the agent is blindly repeating an incorrect answer and apply negative penalty.
In our Sokoban experiments, we set the score threshold 0.7. response was identified as both repetitive and incorrect (FI < 0.7), a penalty of -0.1 was applied to the reward. This mechanism forces the agent to explore more diverse and state-relevant reasoning pathways.
D.3 Bi-Level General Advantage Estimation (GAE) Details and Additional Results
This section provides the detailed pseudo-code for Bi-Level GAE algorithm, which is shown in Algorithm 2. The key modification from the base RL. algorithm in Phase 2, where we introduce bi-level advantage estimation process that operates both turn and token levels. To explore the effectiveness of Bi-level GAE, we also experiment adding Bi-level GAE to Freethink baselines, and results in Table 24 shows that Bi-level GAE is able to serve general plug-in strategy for RL training. Results on different model families and sizes are shown in Table 25, training time and token usage given in 26.
Table 24: Bi-level GAE helps FreeThink RL baselines, and can serve as general mechanism in RL training for VLM agents. Group Model
Task
Method
Test Success Rate
FrozenLake
FreeThink
0.39
Qwen2.5-Vi L- 3B
Frozenl Lake
FreeThink Bi-L-evel GAE
0.62
Sokoban
FreeThink
Qwen2.5-VL. 3B
Sokoban
FreeThink + Bi-Level GAE
0.43
Table 25: Results on different model families and sizes Sokoban task.
Group Model
Method
Test Success Rate
VAGEN BASE
0.61
Qwen2.5-VL
VAGEN FULL
0.79
VAGEN-BASE
0.63
Qwen2.5-VL 7B
VAGEN-FULL
0.92
PAGEN-BASE
0.36
InternVL3-22
VAGEN-FULL
0.39
Table 26: Training efficiency for primary experiments (approximate).
Task
H100 GPU Hours (Approx.)
LLM-as-Judge Token Cost (Approx.)
FrozenI.ake
40
2.3 x 107
Sokoban
40
2.3 107
Navigation
30
6.0 x 10
ManiSkill
30
4.6 x 106
SVG
N/A
46

---

Algorithm 2 Bi-Level General Advantage Estimation (GAB)
Input:
Full trajectory token sequence f, organized by turns t = 0..T
Per-turn rewards To,.... PT-1-
Critic Vg. Actor =e, Reference policy Tiet
Hyperparameters: "huna Aumn; Thaken; Atoken
Output:
Token-level advantages A for each token in each action de.
Stage 1: Turn-Level Advantage Estimation
1: Initialize turn-level advantage AT +- 0.
2: for t= T - ......, 0
(backwards) do
3: Define value at current turn boundary: V2 +- Vi((Fa))
4: if ( T- then
Define value next turn boundary: V441 +
6:
else
7:
Define value at next turn boundary: Ve41 + Vo(
8:
end if
Compute turn-level TD-error: 61 4- Thmm Ve+1 V. Compute turn-level advantage: A +- 67mm + Yum Yum A(41-
10:
11:
end for
Stage 2: Token-Level Advantage Estimation
12:
for 0......
(forwards through turns) do
13:
Let J [8(a))] be the number of tokens action a..
14:
// Initialize the advantage of the final token of the action with the turn-level advantage.
15:
Let F. be the final token of action
16:
Calculate its KL penalty: r1, -8. KL (To((IF((e [
17:
Compute its TD-error ginken f- r Tikken (F ( Vo(Fa((.-1)). Set final token's 's advantage: A +- 0rplynn + Altr.
18:
19:
for i J ...... ... (backwards through remaining tokens in the action) do
20:
Let Tı,; be the current token.
21:
Calculate KI. penalty: r2, + -3- KI. [To(-)FZ(ua)) Teal TTd(-
Compute token-level TD-error: 6,,3 +- ri, + Tokke Vo(F<((.41)) - Vol Vo(F<(u)))
23:
Compute token-level advantage: Alahen gioken Takeo Auhen 1,8+1
end for
25:
end for
E Case Study
In this section, we present a comprehensive case study examining the behavioral patterns and learning dynamies of multi-turn VLM agents across distinct environments. Through systematic analysis of agent responses during different training phases, we investigate how world model reasoning capabilities evolve and identify key phenomena that emerge during the learning process.
E.1 World Model Reasoning Enhances Spatial Understanding and Multi-Step Planning
Our analysis reveals that incorporating explicit world model reasoning into multi-turn VLM agents significantly improves their spatial understanding and planning capabilities across different environ- ments.
The integration of StateEstimation and TransitionModeling steps enables agents better understand relative positions between objects. Sokoban (Figure 8 (c) and (d)), agents trained with visual reasoning demonstrate improved ability to identify spatial configurations between the player, target, and box. This allows them to effectively navigate around obstacles while maintaining progress toward objectives, contrasting with untrained agents that often make suboptimal moves without proper spatial consideration.
World Model reasoning enables sophisticated multi-step planning. In Navigation tasks (Figure 8 (a) and (b)), agents develop the ability to not only identify target objects and their positions but also recognize blocking objects that impede direct paths. This leads to:
47

---

(d)
dook, cas nee compuet
cihestriction oudtant There
plate rechit.
sate s1 an,
281 onta 100s
explore
dockk
need more fanward
plaper below boo and
right and sbove
201, puppl cubo ar.33
posiiioe
722 Pupple cubr
pooseringpteee the olamn
vootieennnn oloterratione
rigtt
the player-hune chutuccturnmnn vervoctnn
player
the player neods
tont green matee
reger
rosae to the and move
iritis stere-cinanntiie
meeee to the right
reae forword to pet clown
needs
the conastniie
sprad calore The player
to the left, then more down to
tinget-cisaion
opreciation dedeeasooonndeeee esncating
action cate
place on top ohf grean
spreciction cubo
mowelin rodatarngtt
deak lyppellcioonnn sittinae
movellh cleewweee
and mose Bioward dieck
revergght eeeeaneeee
rigtt op
tasko Streseniop
forward then taai
soont move
target box
above target, the
cable rnpresictionnntiie
and the perple cate will
toee 167.
222 Purde cubee
pualiioe:
placet-55
penaemppiatiiicc 488,,
posntonn
with Atbes/iottee
desk can see computer
ethilionnciiee aeseeernsate www.eaaaasssnnooooo There
player and target
the belowe
cabe s1135,
grees calos
prailion
other the deak cos.oinaatiooon
right
clock. shreakd
nggtt www.e/taaaaaaiiin
read move fowwrd
rihht reshit.
plaps didssereatio
the burget
box
plager degershuttenn cottraaeeeeen selereer
needa to mowe right to puon
above
tht,
go nod cale is on the
pootiie: (22,
grfeett reed pick the
dleck skubas, la on the other the
Since atamn
tngaRiss,, rehoadd mese
foowantiny adoser
move to posh the box
nown
bea
right wetessnnniiiritteee
purple sute
top the
check shere
dickk sundddctooo---tiieeeaa
sure scteec retato right
swwrrostterghh.
move
right cuurelict berechh totee
answer moseatsedd
the right to be foet off
cncl mess ferwar and
newwiighenasssee
then then
tarpet arree-niaatt minasonnnnn suasoninniiiieeeee engiggpreedi
srpoeer-tppp rightgghhtttttooo..o.gt Mycharoee
the larget and box will
new claveiia toressieeee
above player will
foet sianed inversstieeie
3RtpliceiSS, 167.
actien the pursie calee
and the grears
esucuting thiy
placu-55 -72.
281 281-/pr cietiiiiinnnnn tit
22 100),
222
fece posiiiins
(:5
poadderr
dite aiiia The
etbiiic There
with vestie alam oteck.
to the right muchill
dining
plapen abowe boc and
stayer and targat
the below
green cutoo (16,
locale alam
then the
the
plogas above
182 201 aff-ioteerraee snseatttion
atrutdi
mesadien Rles, siheedd mese
ming The player needa to
move to posh hoo
frant pursle cubs
lift
dbokk
cossdiltinn loroe retats
mesus fnrword sleakk
table, Ther. mave
table
unci will mesee forwand and
right pordiiiil
down the
tarpet ciassormpppieee
down then down
williene move
past the bas down to the
targes, the
above
setion
thesscringe
Adter catcating
dlock rede--torddttoo
anoterndoww, Sight Prowws
come
tarpet bus
will be fronk moroiiittion
movebett.
recwer geersssienn
192 201pp4nsc144
Figure 8: Comparison of VAGEN-Bas Base and VAGEN Full across Navigation, Sokoban, and PrimitiveSkill environments, from left to right. Within environment, the left column shows VAGEN-Ban Base results while the right column displays VAGEN-FAN results. The improvement of VAGEN-Full shows enhanced reasoning ability in understanding environments such as their spatial information.
Accurate spatial relationship identification between targets, obstacles, and agent position
Recognition of blocking constraints that prevent direct paths
Multi-step sequence planning that accounts for both immediate and longer-term goals
48

---

E.2 Response Convergence and Reduced Exploration
We observe a notable pattern of response convergence across different environments, reflecting the fundamental learning dynamics described in the main text. As shown in Cases I through 4, agents exhibit dramatically different response behaviors between early and late training phases, with this phenomenon occurring both with and without our TransitionModell1 Reward as a fundamental aspect of the learning process.
During early training, agents demonstrate strong exploratory tendencies with diverse response patterns. In Sokoban tasks, early responses show varied reasoning approaches, creative problem-solving strategies, and rich descriptive language when analyzing spatial configurations. Similarly, Navigation tasks initially clicit detailed environmental descriptions and multi-facete planning considerations, reflecting the high entropy characteristic of early-stage learning.
As training progresses, agents develop more streamlined and consistent communication patterns. Late-stage responses become notably more concise and structured, with standardized phrasing that primarily varies in directional or specific action tokens while maintaining clear, efficient expres- sion. This evolution toward templated sentence structures occurs across different environments and reasoning qualities, with entropy steadily decreasing as documented quantitatively in the main analysis.
This convergence represents a fundamental aspect of the learning process, where agents naturally optimize their communication patterns for task effectiveness while exhibiting the reduced exploration behavior characteristic of policy convergence in reinforcement learning.
Sokoban Early Training
World Modeling Transition Modeling
State:
['box0 is at the same row and to the right of the player', 'targetO is at
the same and to the right of the player', 'targetò is at the same place as box0']
Response:
The player will move down, then to the right, pushing the box to the target.
State:
and the same column as the player', 'target0 above and at the same column as the player', 'targeto is above and at the same column as box0']
['boxO
Response: Push the box down and then the right. These actions would bring the box closer to the target.
World Modeling State Estimationn(rrundigg
State:
same column the player', 'targetO above and on the left side of the player', 'targeto is above and on the left side of box0']
['box0 above and
Response:
The player is above the box, and the player is below the target, and there is a wall between the player and the target. The box above the player
State:
['boxO is above and the right side of the player', 'target0 is above and the right side of the player', 'targetO above and the right side box0']
Response:
From the image, the player is at the bottom of the wall, with the box directly in front of the player the right and the target directly above the box.
49

---

Sokoban Late Training
World Modeling Transition Modeling
State:
['boxO at the eame row and to the left of the player', 'targetO is above and on the left side of the player', 'target0 is above and at the same column boxO']
Response: The player will move to left, then push
left the target.
State:
['box0 is at the same row and to the right of the player', , 'targetO is above and the right side of the player', 'targetO is above and the column as box0']
Response:
The player will move to the right, then push the box right to the target.
World Modeling State Estimation/Crounding State:
['box0 is above and at the same column as the player', 'targetO is above and on the right side of the player', 'targeto is above and the right side of box0']
Response:
The player is below the box, and the box is below the target.
State:
['boxO below and on the left side player', 'target0 below and on the left side of the player', 'targeto is at the same row to the left of box0']
Response: The player is the right of the box, and the box to the right of the target.
Navigation Early Training
World Modeling Transition Modeling State:
("target_ojj_type" 'Laptop',
sttee_ddddstacc__tolaeeee"" 2.58,
"targe__direccionntto_ll.. to_playe'; 'right', vrieible_objecte// [('type': 'Bed',
piirection_to_plaeerr 'ahead', 'distance_to._ tttpncce_t__ppayrr.......a.. 1.22), ('type': 'Pillow',
'direction_to totilloe__t__paerr 'left',
'distance_to toppayerr 1.35), ...]}
Response:
I can laptop the desk. I will move forward and right to get closer to laptop.
State:
"target_obj_type'' 'Laptop',
sirreet_dddstacce_o_oleeyye 2.31,
bbarge_ddreectio_tto_olgog 'ahead', "visible_objects" [('type': 'Drawer', 'ahead', lattance_to_plaeer'. 0.85), 'distance
{'type': 'Television',
Eatrection_to_playee 'ahead', ceattance_to_playeee 1.3), ...]}
50

---

Response:
My next move will to approach the table where the laptop is placed, The laptop is placed on the right side of table in front of me.
World Modeling State Eatimation/Groundng
State:
("target_obj_type' 'CellPhone',
"target_distance_tt torpseeerttpppaerrr. 2.59,
prrpet__direciioontoooplaaarr 'ahead', 'visible_oojects"" [['type': 'GarbageCan', siirection_to_playerr 'ahead', calstance_to_playee 2.2), ('type': 'CD',
direction_to. : 'ahead', ...]}
'distance_ to.yeerrr 2.55),
Response:
There is table with cell phone on it in the middle of the room. The cell phone is the table, close the center of the room, right beside trash can.
State:
('target_obj_typp 'Statue', "target_distance_tt__llss 2.54, htrpeetddireccioooooooa.eaeeee 'ahead', 'visible_objects' [('type': 'Drawer', Catrection_to_playeee 'ahead', 'distance_r 'distance_to_playye totroee_t_pppprr 1.99), ('type': 'Drawer',
ddirctionn toriioerrt_ppayer 'ahead', 'distance_ to_ player' 1.99), ...]}
Response:
There is a statue on table the room. need move forward in order reach the statue. The floor is smooth, so I can move forward without obstacles.
Navigation Late Training
World Modeling Transition Modeling State:
('target_obj. :y:e'' 'DeskLamp's 1.71,
"target_distance_to__l_a
iatprrectioo_tt_paeerr.""" 'right', 'visible_objects'' [('type': 'Chair', ddirection_to_playree 'ahead', ceistance_to_playye 1.08),
('type': 'Laptop',
'ahead', cisstance_to_playee 1.19}, ...]}
Response:
I will forward-right reach the lamp.
State:
"'target_ojj_type" 'Pot', cetttance_to_plaeeeeeee
faarget_distance so_player': 2.54,
direetioddreetion__oo to_ to_player': 'left', 'visible_obeect'" [('yyee': 'Bowl',
'direction_to_pll to_player': 'ahead', tdistance_to_playee 0.88),
51

---

('type': 'Potato', ciirection_to_playeee 'ahead', 'distance_ cestancc_to_player 1.2), ...]}
Response:
will move forward-left to reach the pot.
World Modeling State Eatimation/Groumding
State:
("target_obj_typee 'Bowl', "target_distance_to_pp 1.52, 'ahead', 'visible_object [('type': 'Book', prrrectioo__o_plaeeeee 'right', 'distance_ to..aceeeeeppppyrr.......... 1.18), {'type': 'Floor', patrection_to_player" 'back', ttistanc_tt__playee 1.3), ...]}
Response:
There is a round container on the floor. To reach the round container, I need to move forward.
State:
("target_obj__ype'' 'GarbageCan' pegpott_ddstaancet__ppapps 1.71,
'ahead', 'visible_objects" [('type'; 'GarbageBag', iiirection_tto_laaarrr 'ahead', 'distance,_ torpacceet__pp_arrr...._ 1.42), ('type': 'GarbageCan', Eiirection_to_plaeerr 'ahead', tdistance_t__playee 1.71), ...}}
Response:
There is a trash can in front of the table to the right. To reach the trash can, I need move forward-right
E.3 Reward Hacking and Over-optimization
We observe interesting patterns of reward optimization behavior, particularly in spatially sensitive environments like FrozenLake where agents learn to adapt their responses to the LLM-as-a-Judge evaluation mechanism. As demonstrated in Cases 5 and 6, agents develop strategic approaches to maximize reward signals, representing a form of reward over-optimization that exemplifies the reward hacking phenomenon described in the main text.
The most notable optimization pattern involves agents converging toward generic, broadly applicable responses that incorporate multiple directional possibilities and general spatial relationships. Rather than providing highly specific, state-dependent analysis, agents learn to produce consistent reasoning templates that reliably satisfy the reward mechanism's evaluation criteria without necessarily reflecting deep, statc-specific reasoning. For instance, responses like "The player will be at the position of the gift" emerge universal patterns that can be applied across diverse state configurations, effectively balancing reward acquisition with computational efficiency while potentially sacrificing reasoning specificity.
This reward hacking behavior becomes more pronounced under Bi-Level training configurations, as highlighted in the main analysis. The enhanced optimization method and turn-aware reward structure enable agents to become more sensitive reward patterns and develop sophisticated strategies for consistent reward acquisition. The Bi-Level GAE's expanded exploration capabilities and improved credit assignment allow agents to efficiently identify and adapt to the evaluation criteria, demonstrating the system's 's remarkable effectiveness in learning woward-mxximizin behaviors.
52

---

However, this optimization may sometimes diverge from the intended reasoning improvements. illustrating the complex dynamics between reward design and genuine capability enhancement in LLM-based evaluation systems.
Frozenl.ke Early Training
World Modeling Transition Modeling State:
('target is at the same row and to the left of the player', 'hole0 is at the same place as the player', 'holel is below and on the left side of the player', 'hole2 is below and on the right side of the player']
Reaponse:
The player will either go left or right to avoid the hole and continue moving to reach the icy blocks.
State:
and on the right side the player', 'hole0 above on the right side the player']
('target
Response:
The player will be in the bottom center after moving right, and then move down to the center of the gift.
World Modeling State Estimation/Groudding
State:
('target below and on the left side player', 'hole0 below on the left side the player']
Response:
The player is standing player, and there is goal the left corner of the player.
block, there is hole directly in front of the
State:
('target is above and on the left side of the player', 'hole0 is above and on the left side of the player', 'holel is below and the left side of the column as the player']
player', 'hole2 below and at the Response:
The player is now at the right side of the grid, adjacent to a hole. The goal (G) is on the left side of the player.
FrozenLake Late Training
World Modeling Transition Modeling (Hacking) State:
['target is at the place the player', 'hole0 is below and the right side of the player']
Response:
The player will be at the position of the gift.
State:
('target above and on the right side the player']
Response:
The player will be at the position of the gift.
World Modeling State EstimationnGroumdin (Hacking)
State:
('target is below and on the right side of the player', 'hole0 is above and on the right side of the player', 'hole1 is below and on the right side of the p]ayer']
Response:
The player at top left, the gift is at the bottom right, and there are holes at the top and bottom right corners.
53

---

State:
('target at same row the left of the player']
Response:
The player is in the center of the grid, the gift is on the left side, and there is hole the right side.
F Extensive Related Work
both LLMs and VLMs{80, 70, 58, 20, 79, 65, 40, 12, 14, 13, 24, 71, 74, 77, 16, 75, 59, 78], with approaches ranging from human feedback [80, 70, 58, 65, 20, 40, 71]. to rule- based reward functions [12, 13, 74, 24, 77, 14]. For multi-turn RL training, [13] applies PPO to VLMs similar to ours, but adopt rajecttor-bbase optimization strategy that better supports POMDP scenarios by leveraging historical context for visual reasoning. Concurrently, [12] and [14] developed hierarchical RL frameworks for LLM alignment in text-only environments. Our work proposes Bi-level GAE, which is similar to the hierarchical design of [12], but with totally different optimization strategy.
RL for LLMs and VLMs. Recent studies have explored
Multi-turn Agent Training for LLMs and VLMs. Multi-turn interaction fundamental to agentic tasks for LLMs [12, 67, 63, 14, 57J. For VLMs, this challenge extends to the more complex domain of maintaining consistent visual state representations across interactions [13]. Previous research has explored various approaches including prompting techniques [5, 73, 76] and fixed LLMs/VLMs with additional adapters [72, 68, 60]. Our work extends to a more fundamental level by investigating the integration of multi-round visual state reasoning with reinforcement learning. This approach demonstrates generalizability across diverse benchmarks, including navigation(7], manipulation instruction following[6], image reconstruction [9], and collaborative tasks[66] where VLMs increasingly utilized as core reasoning engines [74]. Concurrently with our work, several frameworks for training multi-turn VLM agents emerged: GiGPO [61] (verl-agent) targets long-horizon credit assignment, AReaL [62] offers a fully asynchronous, scalable RL system for large paasoning/agentie models, and DART [64] improves GUI agents via decoupled training and adaptive data curation. Our work focus on building internal world models through explicit visual state reasoning in multi-turn VLM agent training.
G Summary of Findings
Our investigation into reinforcing visual state reasoning for multi-turn VLM agents, facilitated by the VAGEN training framework, has yielded several key findings regarding the efficacy explicit reasoning, optimal state representations, specialized RL techniques, and observed agent behaviors during training. These insights are erucial for developing more robust and capable VLM agents that can understand and interact with dynamic visual environments. Below is the summary of our findings.
Finding 1: Explicit Visual State Reasoning is Crucial for Multi-Turn VLM Agents
Vanilla VLMs struggle with multi-turn agentic tasks requiring visual state understand- ing. Integrating explicit visual state reasoning steps-specifically StateEstimation and iraansitionMddelingg -into the VLM's thinking process during RL training significantly enhances task performance. The combined MorldModeling iing strategy, particular, demonstrates strong and stable performance, enabling trained open- nource VLM to outperform its un un-trained counterpart and even surpass benchmarked proprietary models.
54

---

Finding 2: Optimal Visual State Representation is Task-Dependen
The choice of representation for visual states during explicit reasoning significantly impacts performance.
Natural Language: Performs consistently well, especially when structured information must be inferred from raw visual imput.
Structured Formats: Excel in manipulation-heat tasks (e.g., PrimitiveSkill) where object-centric state abstractions are readily available.
Symbolic Representations: Proved less effective due to the model's limited prior interpretability from visual input.
Finding 3: RL with World Modeling Rewards and Bi-Level GAE Enhances Reasoning Quality and Task Success
To specifically improve visual state reasoning, VAGEN incorporates:
Turn-level World Modeling Reward:
An LLM-as-a-Judge assesses the accuracy of the VLM's explicit state descriptions and predictions, effectively supervising reasoning.
Bi-Level General Advantage Estimation (GAE): Estimates advantages at both turn and token levels, providing finer-grained reward signals and improving credit assign- ment.
This approach consistently outperforms Base RL, leading to improved reasoning quality, higher task success rates, and better generalization.
55

---

Finding 4: Emergent Reasoning Patterns and Challenges
Beyond quantitative measurements, we qualitatively analyzed how agents learn to reason:
Reasoning Stability Varies by Task: While reasoning in tasks like Navigation and PrimitiveSkill (and often Sokoban) appears relatively normal and beneficial with ex- plicit rewards, tasks like FrozenLake show more erratic reasoning patterns, potentially correlating with its lower performance and the difficulty of its visual state reasoning.
Potential for Reward Hacking: Instances of "reward hacking" were observed, par- ticularly with certain reward configurations (e-g.. Bi-Level world modeling some contexts). Agents might learn to generate reasoning- like text that satisfies the reward mechanism (c.g., format reward, or even LLM-judge for simple cases) without gen- uinely reflecting deep understanding or accurate future prediction. This suggests the LLM-as-a-Judge mechanism, while an improvement, is not infallible and can gamed.
Bi-Level GAE Double-Edged Sword: While Bi-Level GAE can improve credit assignment, its interaction with World Modeling Rewards might sometimes allow for more "divergent" or less grounded thinking the reasoning reward itself can be easily hacked. Well-designed, hard-to-game reasoning rewards are crucial for Bi-Level GAE to be consistently beneficial; otherwise, it might amplify the effects low-quality process-reward.
Convergence to Standardized Phrasing: Regardless of initial hacking or reasoning quality, agents across different environments tend converge towards using more uniform, templated sentence structure for their reasoning and actions over prolonged training, which varies only in some action tokens. And we also observed huge decrease of entropy during training. This pattern may reflect learned efficiency for tasks with discrete action spaces, but may also signal reduced diversity or creativity in reasoning.
Rule-Based Filtering Potential Mitigation: For simpler forms of reward hacking where reasoning outputs fail basic semantic checks (e.g., not mentioning valid actions in grid world), simple rule-based filtering before reward assignment could be pragmatic interim solution.
These observations underscore that while explicit reasoning and rewards are beneficial, the design of these rewards must be robust against exploitation, and continuous monitoring of reasoning quality is essential.
56

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our problem scope and contributions in Sec 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. 6 for limitations.
57

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not provide any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose our method and experimental settings in Sec 2, and 4, with details provided in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
58

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We opensource our data and code in htttt::// www.ccowmoooooeeo.oooooccoooc.ea nu/
VAGEN.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ctwps.cccppbllcc//eeecc htpppuulc/guuiieeeeeeee guwdes/ /CodeSubmis ssi asionPooicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
59

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the details of the experimental settings in appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to limited computational resource, we do not provide error bars. We provide detailed experiments analysis in supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report computational resources need for each experiment in appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
60

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ttttp:://neuris... ct/pcc//p//....c cctpublic/Etllieecccccccoaeocece wtw.etttiiiiiiiiiiieeecccaaaiaie cswucdelulliiec line??
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
61

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All miird-party assets used in this work are properly cited and credited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
62

---

Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https hhteurrpp.cc//Cnfffeeeccee/////e for what should or should not be described.
63


Reviews:
[{"review_id": "reHrEngwYe", "reviewer": "Reviewer_HHiJ", "summary": "This paper introduces Visual Reasoning Reinforcement Learning (VR-RL), aiming to strengthen visual state inference, and proposes the Bi-Level GAE method, which computes advantage estimates at both the turn level and the token level. Experimental results demonstrate that, compared to baseline RL methods, this approach significantly improves task performance and the quality of reasoning.", "questions": "Questions:\n1. In Table 1, Grounding‐WorldModeling performs worse than WorldModeling on PrimitiveSkill. Could you provide some explanations and a failure analysis?\n2. Table 1 presents the results of several “off-the-shelf VLMs.” In some tasks—particularly FrozenLake and SVG—these VLMs even outperform the Grounding-WorldModeling RL, despite having received no training. Does this suggest that the chosen tasks are too simple to showcase the advantages of RL, or that the success rate of this RL algorithm is not yet high enough?\n3. The paper notes that structured representations, when inferred from \"image-only input, are noisy due to limited grounding capabilities\" in tasks like FrozenLake and Sokoban. Are there proposed architectural modifications or training paradigms that could improve the VLM's ability to autonomously generate accurate structured visual states from pixels, mitigating this performance degradation?\n4. Visual Reasoning RL, incorporating the visual reasoning reward and Bi-Level GAE, leads to \"more stable training and enhanced performance,\" with Base RL's performance sometimes declining in later stages while Visual Reasoning RL continues to increase. Could you explain what are the mechanistic reasons for Base RL's decline in performance during later training stages, and how does Bi-Level GAE specifically counteract this, leading to greater stability and sustained improvement?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides empirical evidence that incorporating explicit visual state reasoning into the VLM's thinking process enhances task performance.\n2. The introduction of Visual Reasoning RL, which includes a turn-level visual reasoning reward and Bi-Level GAE, is a novel contribution that effectively addresses the complexities of credit assignment and reward propagation in multi-turn visual tasks. \n3. The study systematically investigates and provides insights into the optimal choice of visual state representations, highlighting that natural language is generally effective, while structured formats are superior for manipulation-heavy tasks where precision is key and object-centric abstractions are available.\n\nWeaknesses:\n1. The paper notes a restricted model architecture as a limitation, suggesting that the findings might not generalize to all VLM families without further investigation.\n2. The experimental scenarios are relatively simple, the set of evaluation metrics is limited, and the evaluation methodologies and choice of algorithms also exhibit certain inherent limitations.", "was_revised": true, "final_justification": "The authors have provided clear and sufficient clarifications that address my initial concerns. In light of these responses, I find the work more convincing and thus raise my score to 4.", "rebuttal": "We sincerely thank Reviewer HHiJ for the detailed and constructive feedback. We are encouraged that the reviewer finds our **contributions novel** and **the empirical evidence compelling**.\n\nThe reviewer's main concerns relate to the generalizability of our model architecture, the complexity of the experimental scenarios, and the need for deeper analysis on certain results and mechanisms. We believe our clarifications and the new experimental results provided below will address these points.\n\n---\n\n### **1. (Regarding Weakness 1) Generalizability of the Model Architecture**\n\n**TL;DR**: We ran new experiments on a different model family (InternVL3). The results show that our Visual Reasoning RL method generalizes and provides better performance compared Base RL.\n\nWe appreciate the reviewer's point regarding the potential limitation of using a single model family. We chose Qwen2.5-VL because it is a powerful and widely used open-source VLM in RL research, allowing for a fair and controlled comparison with existing work. As the reviewer noted, we acknowledged this in the limitations section of the paper.\n\nTo explicitly address this concern, we conducted new experiments on the **InternVL3-2B** model for the Sokoban task.\n\n| Method | **Test Success Rate** |\n| :--- |   :---: |\n| Base RL | 0.360|\n| **Visual Reasoning RL** | **0.389** |\n\nThe results demonstratea that the Visual Reasoning RL generalizes to a different VLM and achieves higher performance compared to Base RL.\n\n\n### **2. (Regarding Weakness 2) Simplicity of Scenarios, Limited Evaluation, and Choice of Algorithms**\n\n**TL;DR**: We argue that our tasks are challenging, as evidenced by the poor performance of existing powerful VLMs. While the reviewer’s comment that “the set of evaluation metrics is limited” and “the evaluation methodology and choice of algorithms exhibit inherent limitations” essentially reiterates points that we explicitly acknowledged in our limitations section, we agree this deserves clarification. We explain below why our current evaluation is sufficient and how our framework supports richer analysis. We also conducted new experiments with additional RL baselines to directly address the concern regarding algorithm selection.\n\nWe thank the reviewer for encouraging us to clarify the complexity and scope of our evaluation.\n\n* **Task Complexity**: We respectfully disagree that our scenarios are simple. As shown in Table 1 of our paper, a highly capable model like **GPT-4o achieves an overall score of only 0.6**, which is far from the upper bound of 1.0. This indicates that the tasks are sufficiently challenging to differentiate even state-of-the-art models.\n\n* **Evaluation Methodology**: Regarding the concern on evaluation methodology, we want to emphasize that this is one of the points we explicitly listed in our limitations section. We focused on a small set of fundamental and widely recognized metrics in the main paper due to space constraints and to keep our analysis clear. Nevertheless, our evaluation is already extensive, covering **5 diverse tasks**, spanning **2D and 3D states**, and involving varied action spaces (**discrete, continuous, open-ended**), and evaluation metrics (**success rate, DINO score, DreamSim score**). Importantly, **our framework supports a wide range of metrics (e.g., format reward, grounding rewards, world-modeling rewards, entropy loss, response length, valid actions, effective actions, etc)**, which we will include in the appendix of the camera-ready version as auxiliary analyses.\n\n* **Choice of Algorithms**: Similarly, the point about the limited choice of algorithms is also one of the limitations we explicitly acknowledged. To further address this, we have now added experiments with additional RL baselines:\n\n| Method            | Overall Performance |\n| :---------------- | :-----------------: |\n| Vanilla-PPO       |         0.26        |\n| GRPO (w/mask)     |         0.54        |\n| Turn-PPO (w/mask) |         0.55        |\n| **Ours**          |       **0.82**      |\n\n*FrozenLake and Sokoban trained on 300 steps*\n\nThese results demonstrate that our proposed algorithm significantly outperforms the additional RL baselines, further validating the effectiveness of our design choices.\n\n\n### **3. (Regarding Question 1) Failure Analysis of Grounding-WorldModeling on PrimitiveSkill**\n\nThis is an excellent question. In the PrimitiveSkill task, we give a list of object coordinates as the additional information. We hypothesize that the \"Grounding\" step, which involves translating these precise coordinates into a natural language description, can introduce noise. This linguistic abstraction is less useful when precise textual information is given. The agent can directly perform \"WorldModeling\" that leverages the structured input to predict the next state. This makes it inherently more robust for such precision-critical tasks. This insight is validated in Section 4 of our paper, where we show that structured representations are indeed better for PrimitiveSkill. \n\n### **4. (Regarding Question 2) Off-the-Shelf VLMs Outperforming RL**\n\nWe thank the reviewer for raising this important question about the performance comparison between off-the-shelf VLMs and our RL approach.\n\n1.  **Task Complexity is Adequate**: As discussed previously (Regarding Weakness 2), we have explained that the tasks are challenging. Off-the-shelf VLMs do not solve them perfectly.\n2.  **\"No Training\" is an Unverifiable Assumption**: The statement that off-the-shelf VLMs have received \"no training\" for the tasks we present here might not be accurate. Proprietary models like Claude and GPT are extensively fine-tuned, often with methods like RLHF, and their training data and processes are not public. They might also have been pre-trained or finetuned on the task we choose.\n3.  **Our RL Methods on a 3B-Model Show Better Overall Performance**: Both Base RL and Visual Reasoning RL on a 3B-model outperform proprietary VLMs in overall scores. Proprietary VLMs used in our paper are often assumed to be significantly larger than 3B.\n4.  **New Experiments with Extended Training Steps**: To further showcase the potential of our method, we extended the training on FrozenLake from 150 to 300 steps. The results show that our trained agent now surpasses the best-performing proprietary model evaluated for this task in the paper.\n\n| Model | FrozenLake Success Rate |\n| :--- | :---: |\n| Claude 3.7 Sonnet | 0.69 |\n| Qwen 2.5 VL 3B Untrained| 0.14 |\n| **Qwen 2.5 VL 3B Trained  with Visual Reasoning RL** | **0.71** |\n\n### **5. (Regarding Question 3) Improving Generation of Structured Visual States**\n\nThis is a very insightful question about a key challenge. We propose several training paradigms and architectural modifications to mitigate performance degradation from noisy grounding:\n1.  **SFT**: We can first collect a dataset of scene images paired with their ground-truth text-based structured states. Fine-tuning the VLM (either the full model or just the vision tower) on this data would strengthen its ability to generate accurate structured representations for specific scenarios.\n2.  **Integrating Better Vision Encoder**: We can incorporate existing fine-grained image understanding approaches like GLIP, RegionCLIP, and FG-CLIP into the VLM architecture. These models are explicitly trained for enhanced grounding and could provide more robust object-centric features.\n3.  **Scaffolding and Prompting**: Techniques that guide the VLM's internal processing, such as prompting it to generate a \"cognitive map\" or spatial representation of the scene before outputting the final structured state, could improve accuracy.\n\nReferences:\n *Li et al., \"Grounded Language-Image Pre-training\", CVPR 2022.*\n *Zhong et al., \"RegionCLIP: Region-based Language-Image Pretraining\", arXiv 2021.*\n *Xie et al., \"FG-CLIP: Fine-Grained Visual and Textual Alignment\", arXiv 2025.*\n *Yin et al., \"Spatial Mental Modeling from Limited Views\", arXiv 2025.*\n\n### **6. (Regarding Question 4) Stability of Bi-Level GAE vs. Base RL's Decline**\n\nThis question gets to the core of our method's advantage. The performance decline in Base RL during later training stages is a problem in sparse-reward, multi-step tasks.\n*   **Why Base RL Fails**: In later training stages, the agent has already mastered the \"easy\" trajectories. For more challenging cases that require precise, long-horizon reasoning, Base RL provides only a single reward at the very end. If the agent executes several correct turns but makes one mistake at the final step, the entire sequence of correct actions is penalized along with the final error because the trajectory-level advantage is negative. This noisy credit assignment discourages the necessary exploration to solve hard cases and leads to training instability.\n*   **How Bi-Level GAE Provides Stability**: Our Bi-Level GAE directly counteracts this. By computing advantages at both the trajectory level and the turn/token level, it provides **turn-level rewards**. Even if a trajectory fails at the end, our method can assign positive rewards to the intermediate turns that were executed correctly. This fine-grained feedback allows the agent to learn progressively, reinforcing correct sub-policies even when tackling difficult problems."}, {"review_id": "7vWLAV4V09", "reviewer": "Reviewer_77zp", "summary": "The paper tackles the challenge of training VLMs to act as effective agents in multi-turn tasks like Sokoban agme. The obstacle as the authors mention is the VLM's difficulty in reasoning about changing visual states over time. To address this problem, the paper proposes \"Visual Reasoning RL\". They show that making the VLM explicitly reasoning about its environment significantly boosts performance, inluding grounding and worldmodeling. The paper also explores the way to represent the visual states. It finds that natural language is effective for most tasks. There are two techniques introduced to specifically improve the quality of visual reasoning (a turn-level visual reasoning award, and Bi-level general advantage estimation). The experiments demonstrate that this approach could improve task success rates and the quality of the reasoning.", "questions": "The questions are in the section \"Strengths And Weaknesses\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The work is complementary to existing RL-trained reasoning VLMs. It extends the current one-turn reasoning VLMs (usually tackling math problems) to support multi-turn reasoning tasks like Sokoban.\n2. The experiment is conducted on many environments/tasks, such as Sokoban, FrozenLake, Navigation, ... The ablation shows the effectiveness of VLM doing reasoning, the grounding and the worldmodeling. it also compares with a few baselines such as Qwen2.5-VL and close source models (e.g., GPT-4o), showing that the proposed framework is performant.\n3. The Bi-Level General Advantage Estimation breaks a long, single optimiation problem into a series of smaller, more manageable ones, making it possible to train the multi-turn reasoning VLMs.\n\n## Weaknesses\n1. The proposed visual reasoning framework is mainly tested on constrained, game-like environments (grid worlds, simple 3D navigation) with known coordinates. The success of Grounding and WorldModeling relies on the fact that the states in these tasks can be described with language. This may not hold in cluttered and dynamic environments with unpredictable elements.\n2. Training efficiency could also be discusses, as the trajectory in this setting will be long and LLM-as-judge is used. \n3. It is unclear that the method could be scaled up or transferred to other VLMs (other than Qwen-VL).", "was_revised": true, "final_justification": "My concerns have been addressed. I'll maintain the current score (5).", "rebuttal": "We sincerely thank Reviewer 77zp for the excellent review and insightful feedback. We are greatly encouraged that the reviewer recognizes our work as complementary to existing research, praises our comprehensive experiments, and highlights the novelty and importance of the Bi-Level GAE method for enabling multi-turn visual reasoning.\n\nThe reviewer’s main questions are related to the applicabitliy of our method to more complex environments, training efficiency, and its scalability to larger size and generalizability to other VLMs. We believe our clarifications and the new experimental results provided below will address these concerns.\n\n---\n\n### **1. (Regarding Weakness 1) Generalizability to Complex Environments**\n\n**TL;DR**: We chose controlled environments to enable fundamental and deep analysis. Our framework is directly applicable to most current agentic task simulators (e.g., AI2THOR, ALFWorld, WebShop, OSWorld) which provide text-based state information for supervision. The core principle of world modeling is essential for any agent's success, making our method broadly relevant.\n\nWe thank the reviewer for this important point and acknowledge this limitation.\n1.  **Rationale for Controlled Environments**: Our use of controlled environments was a deliberate choice. These settings are essential for conducting a rigorous and systematic analysis of the core components of visual reasoning. They allow us to isolate variables and understand the fundamental training dynamics.\n2.  **Applicability to Current Agent Benchmarks**: We want to clarify that our framework is directly applicable to the majority of simulators used in current agentic research. Environments for embodied agents and GUI agents like **Maniskill, AI2THOR, ALFWorld, Minecraft, OSWorld, and WebShop** all provide access to text-based ground truth states that can be used as supervisory signals for grounding and worldmodeling. Since nearly all current agentic RL research is conducted in simulated environments, our method is highly relevant to the field's present state.\n3.  **Fundamental Importance of Grounding and WorldModeling**: While understanding current state and predicting the next state might be challenging in cluttered and dynamic environments, an agent's ability to grounding and worldmodeling is fundamentally tied to its ability to understand the task and succeed. Therefore, our method is inherently correlated with task success and is broadly applicable in principle.\n\n### **2. (Regarding Weakness 2) Discussion on Training Efficiency**\n\n**TL;DR**: We provide approximate training costs below and will add a detailed section on training efficiency to the appendix in camera ready.\n\nWe appreciate the reviewer raising this practical concern. Training efficiency is indeed an important consideration. We have compiled the training costs for our primary experiments below.\n\n| Task | H100 GPU Hours (Approx.) | LLM-as-Judge Token Cost (Approx.) |\n| :--- | :---: | :---: |\n| FrozenLake | 40 | 2.3 x 10⁷ |\n| Sokoban | 40 | 2.3 x 10⁷ |\n| Navigation | 30 | 6.0 x 10⁶ |\n| ManiSkill | 30 | 4.6 x 10⁶ |\n| SVG | 10 | \\ |\n\nWe will add a dedicated section to the appendix of our camera-ready paper with a full analysis of these costs. We also note that efficiency could be further improved by integrating advanced techniques like async RL, which we are exploring as a future optimization.\n\n### **3. (Regarding Weakness 3) Scalability and Transferability to Other VLMs**\n\n**TL;DR**: We ran new experiments on a larger model (Qwen2.5-VL 7B) and a different model family (InternVL3-2B). The results are highly positive, showing that our method can be scaled ups and generalizes across different VLM family.\n\nThis is a crucial question, which we acknowledged as a limitation of our work in our submission draft. Now we are excited to share new results that address it directly. We conducted additional experiments on the Sokoban task using both a larger model from the same family (Qwen2.5-VL 7B) and a model from a different family (InternVL3-2B).\n\n| Model | Qwen2.5-VL 7B | InternVL3-2B  | \n| :--- | :---: | :---: | \n| **Base RL** | 0.625 | 0.360 | \n| **Visual Reasoning RL** | **0.922** | **0.389** | \n\nThe results confirmed the scalibility and transferability of our method."}, {"review_id": "jtKfLUUsOu", "reviewer": "Reviewer_wZ7m", "summary": "The authors analyze VLMs in multi-turn reasoning tasks. In particular, they cast the problem as a POMDP, where agents (policies) are trained using RL in partially observed domains.\nThey test five reasoning strategies, which correspond to different elements that must be taken into consideration during the interactions with the environment, and three visual state representations. They also try to reinforce directly the quality of visual state reasoning, by proposing BI-Level GAE, a General Advantage Estimation algorithm (that propagates advantages at the episode level as well) which is better suited for this. The tests show better success than the same VLMs without training, even though some strong proprietary model are still very competitive.", "questions": "Why did you use BI-Level GAE only to train visual state representations, and not for the reasoning in the first part? It seems pretty general and an improvement over base RL, so why not testing it?\n\nWhy do you state, in the caption of Table 1, that the proprietary models are non-comparable? Because they do not apply your proposed reasoning strategy?", "limitations": "Not really discussed, only briefly", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n(Quality)\nThe evaluation is extensive, analyzing the problem along various axes, such as reasoning strategies and visual representations\n\n(Originality and significance) Casting the multi-turn problem as a POMDP is sound and interesting, especially in a field where, often, RL is conflated with 'contextual bandits'. This formalization makes clear that the action of the agent has an effect on the environment. However, it is definitely not groundbreaking, Moreover, Bi-Level GAE seems to be an improvement over base GAE, but its impact on the field or novelty is not clear to me.\n\n(Clarity) It a well-written paper.\n\nWeaknessess\n\n(Quality)\nIt would be preferrable to compare, like in Table 1, reasoning with BI-Level GAE vs base GAE and non-reasoning (see my questions)\n\n(Clarity)\nI believe after line 256 an equation is missing, an important one too.\nIn general, the introduction could be improved, especially in the end. In particular, at a first read it is not immediately clear how the last contribution differs than the first ones. Also, a clear, 'explicit' summary of the contributions is preffered at the end of the section.\n\t\n(Quality)\nI feel the SVG task is a bit worse than the other environments: it does not fit that well with the multi-turn concept\n\nIn summary, I like the paper and the contributions. However, it falls short of a stronger accept due to its potential impact and novelty which are difficult for me, at this moment, to rate too highly. For example, I would give a higher score if the obtained results were overwhelmingly better than the ones obtained by most baselines, even proprietary/advanced models (greater impact), or/and if the new reasoning strategy and training technique were to be compared to other SOTA strategies, showing improvements across the board and producing greater insights (novelty).", "was_revised": true, "final_justification": "The authors addressed my main complaints so I raised the score.", "rebuttal": "We sincerely thank Reviewer wZ7m for the detailed and constructive feedback. We are encouraged that the reviewer finds our work to be a **well-written paper** with **extensive evaluation**, and considers our POMDP formalization of the multi-turn problem as **sound and interesting**.\n\nThe reviewer’s main concerns seem to arise from the clarity of our experimental design and the significance of the performance gains achieved by our method. We believe our clarifications and the new experimental results provided below can resolve these concerns.\n\n---\n\n### **1. (Regarding Weakness 1 & Question 1) Clarification on the experiment design and new Bi-Level GAE experiments.**\n\n**TL;DR**: Bi-Level GAE was designed for fine-grained credit assignment when turn-level visual reasoning rewards are introduced. New experiments confirm that **Bi-Level GAE also improves performance with Free-Think**, but its full benefit emerges under fingrained intermediate rewards.\n\n\n*   **Why we didn't use Bi-Level GAE in the first part**:\n    In the first part, each task’s reward design was **coarse-grained**, and our focus was to study the impact of reasoning strategies rather than the RL algorithm itself. Most tasks only provide (i) a simple turn-level format reward, which is easy to learn per our experimental observations, and (ii) a trajectory-level task-success reward. Therefore, we adopted a standard RL algorithm (with modifications for multi-turn tasks) instead of introducing Bi-Level GAE at this stage.\n    \n*  **Why Bi-Level GAE was introduced later**:\nWhen we incorporated turn-level visual reasoning rewards, we observed that fine-grained credit assignment became crucial. The standard advantage estimation failed to effectively propagate supervision across turns. This motivated the development of Bi-Level GAE, which explicitly combines turn-level and trajectory-level credit assignment to better leverage intermediate reasoning signals.\n\n*   **New Experimental Results:** We appreciate the reviewer’s suggestion to test Bi-Level GAE more broadly. We have now run new experiments applying Bi-Level GAE directly to the `FreeThink` strategy.\n\n| Method | FrozenLake | Sokoban |\n| :--- | :---: | :---: |\n| FreeThink | 0.39 | 0.43 |\n| FreeThink + Bi-Level GAE | **0.62** | **0.43** |\n\nWhile the effect on Sokoban is small, FrozenLake shows a substantial gain, validating the effectiveness of Bi-Level GAE. We hypothesize that the degree of improvement varies because Bi-Level GAE is most effective when it can propagate **correct and strong intermediate reward signals**. In our full method, the visual reasoning reward provides exactly this kind of dense, turn-by-turn signal. However, in the base task setup, the intermediate rewards are often sparse, with the most significant reward only being delivered upon final task success. This sparsity can limit Bi-Level GAE from demonstrating its full advantage.\n\n### **2. (Regarding  Weakness 2) Edits for missing equation and contribution summary.**\n\n**TL;DR** We thank the reviewer for pointing out these clarity issues. We will add the missing equations and a more explicit, structured summary of contributions to the camera-ready version.\n\n*   We will add the formal definitions for the turn-level delta ($\\delta_n^{\\text{turn}}$) and advantage ($A_n^{\\text{turn}}$) after line 256:\n\n    $$\n    \\delta_n^{\\text{turn}} = r_n + \\gamma_{\\text{turn}} V_\\phi(p_{n+1}) - V_\\phi(p_{n})\n    $$\n    $$\n    A_n^{\\text{turn}} = \\delta_n^{\\text{turn}} + \\gamma_{\\text{turn}} \\lambda^{\\text{turn}} A_{n+1}^{\\text{turn}}\n    $$\n\n*   We will revise the end of the introduction as follows:\n    > Our contributions are fourfold. (1) We provide empirical evidence for the importance of explicit visual state reasoning in multi-turn RL for VLM agents. (2) We analyze how different visual state representations affect reasoning quality and task performance. (3) We introduce visual reasoning rewards to guide visual state reasoning and Bi-Level GAE for fine-grained credit assignment. (4) We present VAGEN, a scalable training framework that decouples environment setup from model training, enabling efficient experimentation and algorithmic extensibility. Together, these contributions establish a principled pathway for developing VLM agents capable of robust visual reasoning in dynamic environments.\n\n*  We will also add a brief summary of contributions at the beginning of each relevant section to improve readability.\n\n### **3. (Regarding  Weakness 3) Clarification on the inclusion of the SVG task.**\n\nWe thank the reviewer for raising this point. We included SVG:\n(1) to evaluate whether our method extends beyond tasks with well-defined success and limited action space. SVG introduces **open-ended text generation** and **non-binary evaluation metrics** (DreamSim, DINO);\n(2) we reformulated this task from single-turn to multi-turn, inspired by findings that converting single-turn problems into multi-turn dialogues with feedback can improve LLM reasoning [Liu et al., 2025; Zheng et al., 2025; Shinn et al., 2023]. Our work represents an early but important step in exploring this in the VLM domain.\n\n*Reference:*\n\n*Liu et al., 2025. \"A Simple 'Try Again' Can Elicit Multi-Turn LLM Reasoning.\"*\n\n*Zheng et al., 2025. \"What Makes Large Language Models Reason in (Multi-Turn) Code Generation?\"*\n\n*Shinn et al., 2023. \"Reflexion: Language Agents with Verbal Reinforcement Learning.\"*\n\n\n### **4. (Regarding Summary/Impact) New results demonstrating significant performance gains.**\n\n**TL;DR** We have conducted **longer training and new baseline comparisons**, which show that our method achieves **overwhelmingly better** results. Our lightweight **3B VLM** now **outperforms best evaluated proprietary model by 14% overall** and **surpasses other RL baselines by 49%**, demonstrating clear significance and impact.\n\n* **New Results vs. Advanced Models:** We trained our model for longer in FrozenLake and Sokoban (300 steps vs. 150) and added comparisons against Open AI o3 and Open AI o4‑mini. The results show our 3B parameter model now achieves a new state-of-the-art.\n\n| Model                 | FrozenLake |  Sokoban | Navigation | Primitive Skill |  SVG | **Overall** |\n| :-------------------- | :--------: | :------: | :--------: | :-------------: | :--: | :---------: |\n| Claude 3.7            |    0.69    |   0.25   |    0.56    |       0.28      | 0.88 |     0.52    |\n| GPT‑4o                |    0.54    |   0.43   |    0.72   |       0.50      | 0.81 |     0.60    |\n| Open AI o4‑mini |    **0.82**    |   0.44   |    0.75    |       0.75      | 0.56 |     0.67    |\n| Open AI o3                    |    0.78    |   0.60   |    0.81    |       0.75      | 0.66 |     0.72    |\n| **Ours (Qwen‑VL 3B)** |  0.74  | **0.79** |  **0.82**  |     **0.94**    | 0.79 |   **0.82**  |\\\n\nOur relatively small 3B model outperforms Open AI o3 by **+14% overall**.\n\n* **New Results vs. RL Baselines:** To better highlight the novelty and contribution of our method, we also ran new experiments comparing against other RL baselines.\n\n| Method            | Overall Performance |\n| :---------------- | :-----------------: |\n| Vanilla‑PPO       |         0.26        |\n| GRPO (w/mask)     |         0.54        |\n| Turn‑PPO (w/mask) |         0.55        |\n| **Ours**          |       **0.82**      |\n\nOur method leads the next‑best baseline by **+49%**.\n\n* We believe the two new experiments above confirm the effectiveness and significance of our method.\n\n\n### **5. (Regarding Question 2) Clarification on \"non-comparable\" models.**\n\nWe apologize for the confusing phrasing. \"Non-comparable\" was intended to mean \"not a controlled comparison\". Our intent was to emphasize that for a scientifically rigorous evaluation of **reasoning strategies** and **optimization algorithms**, it is crucial to hold the base model constant (i.e., same size and architecture). We will correct the phrasing in the camera-ready version to \"not a controlled comparison\" to avoid future misunderstanding."}, {"review_id": "JELskH5Yif", "reviewer": "Reviewer_ortS", "summary": "This paper explores how to improve the performance of Vision-Language Models (VLMs) on multi-turn agent tasks by augmenting reinforcement learning (RL) with explicit visual state reasoning. The authors introduce reasoning strategies such as Grounding and WorldModeling, and propose a new RL framework that integrates (1) a Visual Reasoning Reward and (2) a Bi-Level Generalized Advantage Estimation (GAE) mechanism to more effectively guide the learning process.", "questions": "- For Figure 3, would it be possible to show longer training curves? RL training is often noisy, and it is unclear whether the final decline in Base RL is due to temporary fluctuations or actual performance degradation. Also, does Table 3 report results after convergence, or at a selected checkpoint?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper is well-written and easy to follow.\n- The topic is interesting and relevant to the growing interest in multimodal agent training.\n- The experiments span a wide range of environments with different task structures and action space.\n\nWeaknesses\n- In the section “Forcing Off-the-Shelf VLMs to Reason Cannot Solve Multi-turn Agentic Tasks”, the authors conclude that most models don't perform well when being prompted to reason. However, many of the evaluated models were not explicitly trained for reasoning (except for Claude 3.7 Sonnet). Given this, poor performance may be expected because the tasks involve spatial reasoning and planning. It would strengthen the claim if the authors included stronger recent models such as Qwen3 series or GPT o3/o4-mini in the benchmark.\n- In the PrimitiveSkill task, it appears that Base RL performs better or comparably to Visual Reasoning RL in terms of sample efficiency and final performance where the plots are truncated. The paper states that “Visual Reasoning RL consistently outperforms Base RL,” but this may need to be more carefully qualified. Can the authors provide further explanation? Is there variance across seeds or task subtasks?\n- Since Base RL already includes Grounding + WorldModeling, the contrast between Base RL and Visual Reasoning RL mainly lies in reward shaping and advantage estimation. However, more baselines such as direct action-level RL would be helpful to isolate the contribution of reasoning components.\n- Since RL methods are often sensitive to random seeds, it would be better to report variance across multiple runs.", "was_revised": true, "final_justification": "The authors report additional experiments on the weaknesses, therefore I will keep my original positive score.", "rebuttal": "We sincerely thank Reviewer ortS for the detailed and constructive feedback. We are encouraged that the reviewer finds our paper to be **well-written** and **easy to follow**, with an **interesting and relevant topic** and **experiments that span a wide range of environments**.\n\nThe reviewer's main concerns are related to the strength of our baselines, the performance analysis on the PrimitiveSkill task, the need for additional RL comparisons, and the reporting of variance. We believe our clarifications and the new experimental results provided below will address these points.\n\n---\n\n### **1. (Regarding Weakness 1) Strength of Baseline Models**\n\n**TL;DR**: We added experiments with stronger proprietary models (OpenAI GPT-o3 and GPT-o4-mini). Our 3B VLM outperforms these models on most tasks, demonstrating the effectiveness of our method.\n\nWe appreciate the reviewer’s suggestion to benchmark against stronger recent models. To address this, we ran additional experiments.\n\nWe compared our method against strong proprietary models like GPT-o3 and GPT-o4-mini. It is important to note that these models were used with a 4096-token budget (much larger than the 500-token budget for our method and other baselines). We also trained the model for longer steps in Sokoban and FrozenLake using our method (300 vs. 150 in the original submission).\n\n| Model | FrozenLake | Sokoban | Navigation | Primitive Skill | SVG | **Overall** |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| GPT-o3 | 0.78 | 0.60 | 0.81 | 0.63 | 0.75 | 0.71 |\n| GPT-o4-mini | 0.82 | 0.44 | 0.75 | 0.50 | 0.78 | 0.66 |\n| **Ours**  | 0.74 | **0.79** | **0.82** | **0.94** | **0.79** | **0.82** |\n\nOur 3B VLM outperforms both GPT-o3 and GPT-o4-mini on most tasks, highlighting the effectiveness of our method.\n*(Note: Qwen3 only supports text inputs. We will test it when its vision-language version releases.)*\n\n### **2. (Regarding Weakness 2) Clarification on the PrimitiveSkill Task Performance**\n\n**TL;DR**: We acknowledge the reviewer's observation is accurate. Our method's main benefit in the PrimitiveSkill task is improved **sample efficiency in early training**, while performance converges to Base RL in later stages due to task-specific properties. We will revise our claims to reflect this nuance.\n\nWe thank the reviewer for this insightful observation. We agree that our original statement \"Visual Reasoning RL consistently outperforms Base RL\" needs to be more carefully qualified for the PrimitiveSkill task.\n\n*   We re-analyzed PrimitiveSkill by averaging results across its subtasks.\n\n| Step | Base RL| Visual Reasoning RL|\n|:--|:--|:--|\n| 0    | 0.000 | 0.000|\n| 5    | 0.000 | 0.000|\n| 10   | 0.000 | 0.007|\n| 15   | 0.000 | 0.062|\n| 20   | 0.019 | 0.311|\n| 25   | 0.061 | 0.495|\n| 30   | 0.188 | 0.717|\n| 35   | 0.283 | 0.835|\n| 40   | 0.541 | 0.887|\n| 45   | 0.842 | 0.932|\n| 50   | 0.927 | 0.993|\n| 55   | 0.925 | 0.952|\n| 60   | 0.976 | 0.969|\n\n\n\n1.  The averaged results show Visual Reasoning RL learns faster and achieves higher performance in the initial training phases, confirming its ability to improve training efficiency.\n2.  In the later stage, Visual Reasoning RL converges to Base RL. We attribute this to two reasons: (i) The PrimitiveSkill task provides object coordinates directly, reducing the need for visual grounding, which is a key strength of our reasoning component. (ii) The action space is relatively predictable, allowing the base agent to eventually learn the world model without extra supervision.\n\n*   **Edits to the paper:** We will edit our manuscript to state that Visual Reasoning RL provides significant sample efficiency improvements, especially in early training, while acknowledging that performance may converge in later stages for tasks with reduced grounding complexity. Thank you for helping us present our contributions more precisely.\n\n### **3. (Regarding Weakness 3) Isolating Contributions with More RL Baselines**\n\n\n**TL;DR:** Our paper already includes an action-level RL baseline (“No-Think RL”, Table 1), which corresponds to the model directly predicting actions without reasoning. We further added new baselines (Vanilla-PPO, GRPO w/mask, Turn-PPO w/mask). Our method outperforms the next-best baseline by **49%**, highlighting the effectiveness of Visual Reasoning RL.\n\nWe thank the reviewer for this suggestion. Our paper (Table 1) already includes an action-level RL baseline (“No-Think RL”), which removes the reasoning step entirely and lets the model output actions directly. This setting serves as a strong reference point for RL without explicit reasoning.\n\nTo further isolate the effect of our proposed components (visual reasoning reward and Bi-Level GAE), we conducted additional experiments with the following baselines:\n\n* **Vanilla-PPO:** Standard PPO without state masking.\n* **GRPO (w/mask):** Trajectory-level GRPO with state masking.\n* **Turn-PPO (w/mask):** PPO variant where all tokens in one turn share the same advantage.\n\n| Method            | Overall Performance |\n| ----------------- | :-----------------: |\n| Vanilla-PPO       |         0.26        |\n| GRPO (w/mask)     |         0.54        |\n| Turn-PPO (w/mask) |         0.55        |\n| **Ours**          |       **0.82**      |\n\nOur method outperforms the next-best baseline by **49%**, confirming that our method is effective.\n\n\n### **4. (Regarding Weakness 4) Reporting Variance Across Multiple Runs**\n\n**TL;DR**: We have conducted new experiments on FrozenLake and Sokoban with random seeds. The results show that our Visual Reasoning RL achieves a higher mean performance.\n\nWe agree that reporting variance is crucial for RL results.\n\n*   We have run 3 experiments with random seeds for both Base RL and Visual Reasoning RL on the FrozenLake and Sokoban tasks. We report the mean and variance below.\n\n| Method | FrozenLake | Sokoban |\n|--------|------------|---------|\n| Base RL | 0.7000 ± 0.0557 | 0.6233 ± 0.0808 |\n| Visual Reasoning RL | 0.7267 ± 0.0115 | 0.7800 ± 0.0854 |\n\n*(Note: Results shown as mean ± standard deviation)*\n\n\nThe results confirm that Visual Reasoning RL performs better on average.\n\n*  Due to computational constraints, we cannot feasibly run multiple seeds across all ~40 experiments (including tasks, strategies, and ablations). Each full run requires 10~40 H100 GPU hours, making multiple seeds for every setting prohibitive. \n\n### **5. (Regarding Question 1) Longer Training Curves and Table 3 Reporting**\n\n**TL;DR**: We provide longer training curves for FrozenLake and Sokoban, which show the previous decline in Base RL was a temporary fluctuation. Table 3 reports the best checkpoint for fair comparison. Our new multi-run results confirm that Visual Reasoning RL still achieves superior average performance despite this.\n\nThank you for this question. RL training can indeed be noisy.\n\n*   **Longer Training Curves:** We have extended the training for FrozenLake and Sokoban to 300 steps.\n\n| Step | FrozenLake Base RL |  FrozenLake Visual Reasoning RL | Sokoban Base RL | Sokoban Visual Reasoning RL |\n| :-- | :--- | :--- | :--- | :--- |\n| 1 | 0.125 | 0.140625 | 0.0390625 | 0.0546875 |\n| 20 | 0.21875 | 0.15625 | 0.125 | 0.125 |\n| 40 | 0.453125 | 0.4296875 | 0.25 | 0.2734375 |\n| 60 | 0.5390625 | 0.5703125 | 0.265625 | 0.390625 |\n| 80 | 0.6953125 | 0.59375 | 0.3515625 | 0.4375 |\n| 100 | 0.734375 | 0.65625 | 0.3515625 | 0.5390625 |\n| 120 | 0.7578125 | 0.71875 | 0.3828125 | 0.5 |\n| 140 | 0.78125 | 0.796875 | 0.4140625 | 0.5546875 |\n| 160 | 0.75 | 0.765625 | 0.421875 | 0.7265625 |\n| 180 | 0.78125 | 0.765625 | 0.4375 | 0.6796875 |\n| 200 | 0.7578125 | 0.7890625 | 0.5078125 | 0.75 |\n| 220 | 0.7890625 | 0.71875 | 0.5703125 | 0.71875 |\n| 240 | 0.78125 | 0.7578125 | 0.5546875 | 0.78125 |\n| 260 | 0.78125 | 0.8203125 | 0.5234375 | 0.6640625 |\n| 280 | 0.78125 | 0.84375 | 0.5546875 | 0.7265625 |\n\n\nThe longer curves show that the decline in Base RL was a temporary fluctuation. However, our Visual Reasoning RL still achieves higher final performance.\n\n*   **Clarification on Table 3:** For Table 3, we reported results from the best-performing checkpoint for all methods to ensure a fair comparison. We will clarify this reporting methodology in the paper."}]

Author rebuttal:

Meta-review: The authors explore improving the performance of vision-language models in multi-turn agentic tasks in interactive dynamic environments (modeled as POMDPs), examining the effect of methods enabling reasoning about the visual states and world models. Regarding reasoning about visual states, they compare multiple reasoning strategies (No-Think, Free-Think, Grounding, WorldModeling, Grounding-WorldModeling). Regarding visual state representations, the authors consider several variants (natural language descriptions, structured formats, symbolic representations). Regarding RL-based optimization over a sequence of turns, the authors propose Bi-Level General Advantage Estimation using turn-level visual reasoning rewards with a LLM-as-a-Judge (LLMaaJ) and the token level for action generation. This combination of methods consistently (but not universally) outperform multiple baseline foundational models (VLM-R1-3B, Qwen2.5-VL{3,7,72}B, InternVL3-2B, GPT-4o, Gemini 2.0, Claude 3.7 Sonnet) over challenging benchmark environments (Sokoban, FrozenLake, Navigation, PrimitiveSkill, SVG Reconstruction). Secondary experiments are performed to validate the individual component strategies and point towards directions for future work.

Strengths identified by the reviewers include:
- Multi-turn RL-trained VLM reasoning models haven't been studied in detail, are important (and likely more so in the future), and the authors address several issue in accruing improvements.
- Strong evidence is problem that incorporating visual state reasoning, especially WorldKnowledge and WorldKnowledge+Grounding, into the VLM reasoning trajectory improves task performance. 
- The Bi-Level GAE approach is novel and enables efficient RL learning. 
- The additional experiments regarding visual state representation is insightful and provides task-family specific guidance that also aligns with intuition. The empirical evaluation is relatively extensive overall.
- The paper is well-written and easy to understand while the formulation and considered alternatives are well-motivated.

Weaknesses identified by the reviewers include:
- Some configurations require additional interpretation (e.g., Grounding-WorldKnowledge vs. WorldKnowledge (reviewer HHiJ), off-the-shelf model performance (reviewers HHiJ, 77zp), Bi-Level GAE stability explanation (reviewer HHiJ), PrimitiveSkill task concerns (reviewer ortS)). However, these were well-rebutted during the discussion period.
- The experiments are conducted primarily on controlled environments, which partially explain why structured representations and WorldKnowledge states are helpful for improving performance. However, this is likely to transfer to a large set of practical problems also.
- The empirical results are consistently better than baseline models being augmented, but fall short of proprietary models in some cases -- meaning there is a better method out there (even if it may require a lot more data and training). This was addressed in rebuttal with longer training runs, but would still need to be integrated into the paper.
- [my own criticism] The paper reads a bit like a set of explorations instead of a coherent formulation -- many which have analogues in text-based multi-turn RL. One could imaging a 'truly exhaustive' set of ablations where you could tease apart the individual contributions and secondary experiments that emphasize interpretation of the differences in performance. The current version reads a bit like a disjoint set of findings (modulo the Bi-Level GAE formulation). Some of this can be addressed in preparing a revised version (and maybe a bit of restructuring the paper would fit better with how I interpreted the contributions), but some would require additional experiments and interpretation. 

Overall, the Bi-Level GAE is innovative for multi-turn RL with VLM agents and the findings regarding visual state type and representation are insightful. Additionally, the performance consistently shows improvement over 'vanilla' foundational models. In my reading of the paper, I would estimate it to be borderline (and leaning toward reject), but with the improved results relative to proprietary models and rebuttal overall, I lean more towards accepting as the findings are promising and I expect will be cited as a baseline in future works in this area of growing interest.
Decision: Accept (poster)