Title: VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual States

World Modeling

Multi-turn RL

VLM Agents
Abstract: A major challenge in training VLM agents, compared to LLM agents, is that states shift from simple texts to complex visual observations, which introduces partial observability and demands robust world modeling. We ask: can VLM agents build internal world models through explicit visual state reasoning? In this work, we architecturally enforce and reward VLM agent’s reasoning process via reinforcement learning (RL), formulating the problem as a Partially Observable Markov Decision Process (POMDP). We demonstrate that structuring agent’s reasoning into StateEstimation (“what is the current state?”) and TransitionModeling (“what is next?”) is critical by studying five reasoning strategies. Investigating how agents should ground visual states and represent these internal beliefs, we reveal the optimal representations are task-dependent: Natural Language excels at capturing semantic relationships for general tasks, while Structured formats are essential for high-precision manipulation. These insights motivate our approach to reward shaping and credit assignment. We leverage a WorldModeling Reward to densely rewards the agent’s turn-by-turn state predictions, while our Bi-Level General Advantage Estimation (Bi-Level GAE) enables turn-aware credit assignment. Through such world model reasoning, we enable a 3B model to achieve performance of 0.82 on a set of five diverse agent tasks, nearly 3× improvement over its untrained counterpart (0.21) and surpassing proprietary reasoning models like GPT-5 (0.75), Gemini 2.5 Pro (0.67) and Claude 4.5 (0.62). All experiments are supported by our VAGEN framework, a scalable system for training and analyzing multi-turn VLM agents across diverse visual environments


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how to improve the performance of Vision-Language Models (VLMs) on multi-turn agent tasks by augmenting reinforcement learning (RL) with explicit visual state reasoning. The authors introduce reasoning strategies such as Grounding and WorldModeling, and propose a new RL framework that integrates (1) a Visual Reasoning Reward and (2) a Bi-Level Generalized Advantage Estimation (GAE) mechanism to more effectively guide the learning process.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is well-written and easy to follow.\n- The topic is interesting and relevant to the growing interest in multimodal agent training.\n- The experiments span a wide range of environments with different task structures and action space.\n\nWeaknesses\n- In the section “Forcing Off-the-Shelf VLMs to Reason Cannot Solve Multi-turn Agentic Tasks”, the authors conclude that most models don't perform well when being prompted to reason. However, many of the evaluated models were not explicitly trained for reasoning (except for Claude 3.7 Sonnet). Given this, poor performance may be expected because the tasks involve spatial reasoning and planning. It would strengthen the claim if the authors included stronger recent models such as Qwen3 series or GPT o3/o4-mini in the benchmark.\n- In the PrimitiveSkill task, it appears that Base RL performs better or comparably to Visual Reasoning RL in terms of sample efficiency and final performance where the plots are truncated. The paper states that “Visual Reasoning RL consistently outperforms Base RL,” but this may need to be more carefully qualified. Can the authors provide further explanation? Is there variance across seeds or task subtasks?\n- Since Base RL already includes Grounding + WorldModeling, the contrast between Base RL and Visual Reasoning RL mainly lies in reward shaping and advantage estimation. However, more baselines such as direct action-level RL would be helpful to isolate the contribution of reasoning components.\n- Since RL methods are often sensitive to random seeds, it would be better to report variance across multiple runs.\n\nQUESTIONS:\n- For Figure 3, would it be possible to show longer training curves? RL training is often noisy, and it is unclear whether the final decline in Base RL is due to temporary fluctuations or actual performance degradation. Also, does Table 3 report results after convergence, or at a selected checkpoint?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze VLMs in multi-turn reasoning tasks. In particular, they cast the problem as a POMDP, where agents (policies) are trained using RL in partially observed domains.\nThey test five reasoning strategies, which correspond to different elements that must be taken into consideration during the interactions with the environment, and three visual state representations. They also try to reinforce directly the quality of visual state reasoning, by proposing BI-Level GAE, a General Advantage Estimation algorithm (that propagates advantages at the episode level as well) which is better suited for this. The tests show better success than the same VLMs without training, even though some strong proprietary model are still very competitive.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n(Quality)\nThe evaluation is extensive, analyzing the problem along various axes, such as reasoning strategies and visual representations\n\n(Originality and significance) Casting the multi-turn problem as a POMDP is sound and interesting, especially in a field where, often, RL is conflated with 'contextual bandits'. This formalization makes clear that the action of the agent has an effect on the environment. However, it is definitely not groundbreaking, Moreover, Bi-Level GAE seems to be an improvement over base GAE, but its impact on the field or novelty is not clear to me.\n\n(Clarity) It a well-written paper.\n\nWeaknessess\n\n(Quality)\nIt would be preferrable to compare, like in Table 1, reasoning with BI-Level GAE vs base GAE and non-reasoning (see my questions)\n\n(Clarity)\nI believe after line 256 an equation is missing, an important one too.\nIn general, the introduction could be improved, especially in the end. In particular, at a first read it is not immediately clear how the last contribution differs than the first ones. Also, a clear, 'explicit' summary of the contributions is preffered at the end of the section.\n\t\n(Quality)\nI feel the SVG task is a bit worse than the other environments: it does not fit that well with the multi-turn concept\n\nIn summary, I like the paper and the contributions. However, it falls short of a stronger accept due to its potential impact and novelty which are difficult for me, at this moment, to rate too highly. For example, I would give a higher score if the obtained results were overwhelmingly better than the ones obtained by most baselines, even proprietary/advanced models (greater impact), or/and if the new reasoning strategy and training technique were to be compared to other SOTA strategies, showing improvements across the board and producing greater insights (novelty).\n\nQUESTIONS:\nWhy did you use BI-Level GAE only to train visual state representations, and not for the reasoning in the first part? It seems pretty general and an improvement over base RL, so why not testing it?\n\nWhy do you state, in the caption of Table 1, that the proprietary models are non-comparable? Because they do not apply your proposed reasoning strategy?\n\nLIMITATIONS:\nNot really discussed, only briefly", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the challenge of training VLMs to act as effective agents in multi-turn tasks like Sokoban agme. The obstacle as the authors mention is the VLM's difficulty in reasoning about changing visual states over time. To address this problem, the paper proposes \"Visual Reasoning RL\". They show that making the VLM explicitly reasoning about its environment significantly boosts performance, inluding grounding and worldmodeling. The paper also explores the way to represent the visual states. It finds that natural language is effective for most tasks. There are two techniques introduced to specifically improve the quality of visual reasoning (a turn-level visual reasoning award, and Bi-level general advantage estimation). The experiments demonstrate that this approach could improve task success rates and the quality of the reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The work is complementary to existing RL-trained reasoning VLMs. It extends the current one-turn reasoning VLMs (usually tackling math problems) to support multi-turn reasoning tasks like Sokoban.\n2. The experiment is conducted on many environments/tasks, such as Sokoban, FrozenLake, Navigation, ... The ablation shows the effectiveness of VLM doing reasoning, the grounding and the worldmodeling. it also compares with a few baselines such as Qwen2.5-VL and close source models (e.g., GPT-4o), showing that the proposed framework is performant.\n3. The Bi-Level General Advantage Estimation breaks a long, single optimiation problem into a series of smaller, more manageable ones, making it possible to train the multi-turn reasoning VLMs.\n\n## Weaknesses\n1. The proposed visual reasoning framework is mainly tested on constrained, game-like environments (grid worlds, simple 3D navigation) with known coordinates. The success of Grounding and WorldModeling relies on the fact that the states in these tasks can be described with language. This may not hold in cluttered and dynamic environments with unpredictable elements.\n2. Training efficiency could also be discusses, as the trajectory in this setting will be long and LLM-as-judge is used. \n3. It is unclear that the method could be scaled up or transferred to other VLMs (other than Qwen-VL).\n\nQUESTIONS:\nThe questions are in the section \"Strengths And Weaknesses\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Visual Reasoning Reinforcement Learning (VR-RL), aiming to strengthen visual state inference, and proposes the Bi-Level GAE method, which computes advantage estimates at both the turn level and the token level. Experimental results demonstrate that, compared to baseline RL methods, this approach significantly improves task performance and the quality of reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides empirical evidence that incorporating explicit visual state reasoning into the VLM's thinking process enhances task performance.\n2. The introduction of Visual Reasoning RL, which includes a turn-level visual reasoning reward and Bi-Level GAE, is a novel contribution that effectively addresses the complexities of credit assignment and reward propagation in multi-turn visual tasks. \n3. The study systematically investigates and provides insights into the optimal choice of visual state representations, highlighting that natural language is generally effective, while structured formats are superior for manipulation-heavy tasks where precision is key and object-centric abstractions are available.\n\nWeaknesses:\n1. The paper notes a restricted model architecture as a limitation, suggesting that the findings might not generalize to all VLM families without further investigation.\n2. The experimental scenarios are relatively simple, the set of evaluation metrics is limited, and the evaluation methodologies and choice of algorithms also exhibit certain inherent limitations.\n\nQUESTIONS:\nQuestions:\n1. In Table 1, Grounding‐WorldModeling performs worse than WorldModeling on PrimitiveSkill. Could you provide some explanations and a failure analysis?\n2. Table 1 presents the results of several “off-the-shelf VLMs.” In some tasks—particularly FrozenLake and SVG—these VLMs even outperform the Grounding-WorldModeling RL, despite having received no training. Does this suggest that the chosen tasks are too simple to showcase the advantages of RL, or that the success rate of this RL algorithm is not yet high enough?\n3. The paper notes that structured representations, when inferred from \"image-only input, are noisy due to limited grounding capabilities\" in tasks like FrozenLake and Sokoban. Are there proposed architectural modifications or training paradigms that could improve the VLM's ability to autonomously generate accurate structured visual states from pixels, mitigating this performance degradation?\n4. Visual Reasoning RL, incorporating the visual reasoning reward and Bi-Level GAE, leads to \"more stable training and enhanced performance,\" with Base RL's performance sometimes declining in later stages while Visual Reasoning RL continues to increase. Could you explain what are the mechanistic reasons for Base RL's decline in performance during later training stages, and how does Bi-Level GAE specifically counteract this, leading to greater stability and sustained improvement?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}