Title: World-aware Planning Narratives Enhance Large Vision-Language Model Planner
Venue: NeurIPS 2025
Accepted: yes
Keywords: planning

embodied

LVLMs
Abstract: Large Vision-Language Models (LVLMs) show promise for embodied planning tasks but struggle with complex scenarios involving unfamiliar environments and multi-step goals. 
Current approaches rely on environment-agnostic imitation learning that disconnects instructions from environmental contexts, causing models to struggle with context-sensitive instructions and rely on supplementary cues rather than visual reasoning during long-horizon interactions.
In this work, we propose World-Aware Planning Narrative Enhancement (WAP), a framework that infuses LVLMs with comprehensive environmental understanding through four cognitive capabilities (visual appearance modeling, spatial reasoning, functional abstraction, and syntactic grounding) while developing and evaluating models using only raw visual observations through curriculum learning.
Evaluations on the EB-ALFRED benchmark demonstrate substantial improvements, with Qwen2.5-VL achieving a 60.7 absolute improvement in task success rates—particularly in commonsense reasoning (+60.0) and long-horizon planning (+70.0). Notably, our enhanced open-source models outperform proprietary systems like GPT-4o and Claude-3.5-Sonnet by a large margin.


Peer reviews:
comments: SUMMARY:
The paper presents an extension of large vision-language models by generating instructions on multiple visual and language dimensions, including reasoning chains. The enhancements show consistent improvements over generic LLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
- Well-motivated extension of large vision-language models
- Well-conducted experiments
- Includes all bells and whistles of state-of-the-art VLM work

Weaknesses:
- Many details undefined or insufficiently described

QUESTIONS:
1. If an LLM is used for instruction augmentation and reasoning generation (line 185), was the output of the LLM systematically checked in a human evaluation?
2. How is the "verification function" and the "intentions"  it is based on, defined?
3. What is the difference between "basic", "partial", and "complete" reasoning?
4. The LLM-generated reasoning chains are used as supervision signal - did you evaluate the learned reasoning process of the vision-language model separately?
5. The claim is to enhance vision-language models for difficult situations - what about moving from indoor to outdoor navigation?

LIMITATIONS:
Limitations moved to appendix
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving VLM planners that output text-based action at each step (e.g., \"find {object}\", \"pick up {object}\"). Given a dataset of instruction-trajectory pair, they propose enriching the instruction with additional visual, spatial, functional, and syntactic descriptions. Given the augmented instruction and the original action, a VLM predicts the reasoning. Then, using curriculum learning, they supervised finetune the VLM to predict reason and action given the enriched instruction. They showed that their finetune model is able to improve in EB-ALFRED benchmark when the LLM planner does not have access to environmental feedback.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\n\n## Strengths\n\n* The paper showed how training on the 81k trajectories is able to improve a weak model, approaching the performance of closed-sourced VLM (e.g., Claude-3.5-Sonnet). And they showed these on two open-sourced small models (InternVL3-8B and Qwen2.5-VL-7B-Ins)\n* The paper compares directly training on the entire augmented dataset with curriculum learning, showing that curriculum learning provides performance gains. \n\n## Weaknesses\n\n* The paper is missing a fair evaluation showing the effectiveness of their specific data augmentation approach because the number of trajectories is a confounding factor. Although Table 1 and Table 3 show that training on the original dataset alone is insufficient, the paper's approach is training on more data (which comes from the proposed augmented approach). It is unclear whether the gain is due to the augmentation approach or the increasing training data size. I suggest the authors answering:\n    * (Fix the training data size) What is the performance of training on 16k original data v.s. 16k augmentation data?\n    * (Other augmentation method) How well does your approach compare to other data augmentation approaches? e.g., asking an LLM to generate reasoning given the original language instruction (instead of the augmented instruction)\n* What is the training and testing split? It is unclear whether the training and testing sets have a strong overlap, so the SFT model can be overfitted to the environment. \n\n# Clarity\n\n## Strengths\n\n* The paper is well-written and well-organized with informative figures. \n\n## Weaknesses\n* All results are missing standard error. \n* For clarity, table 1 should only include the relevant models (GPT-4o, Claude-3.5-Sonnet, InternVL3-8B, Qwen2.5-VL-7B-Ins) for both settings. It is difficult as a reader to identify what models are relevant in the current table. The remaining models can be moved to the appendix. The paper should also include a clearer definition of the following:\n    * What are the task categories (Base, Common, Complex, Visual, Spatial, and Long)? I understand that these are from EmbodiedBench, but it will be helpful to write some explanation in the paper/appendix and point to the original paper. \n    * What are all the methods being compared? Having a baseline section in 4.1 would be helpful,. Right now, the readers have to interpret what \"+ Basic Reasoning\" v.s. \"+ Full Augmentation\" v.s. \"+ Curriculum Learning\" means.\n\n# Significance & Originality\n\n## Strengths\n* The paper shows how the LLM planner relies on environment feedback (e.g., identifying infeasible actions) to maintain good performance.\n* The paper shows that performing SFT on an augmented dataset is able to improve the performance of a weak 7B-8B model, making it approach a large-scale proprietary model. \n\n## Weaknesses\n* Although the paper observes that the LLM planner's performance drops when we remove environment feedback, how common is it to not have any environment feedback (even if it might be noisy)? For example, InnerMonologue [1] has long discussed the availability of success detectors acquired either via engineering or learning. \n* At inference time, how do you get the narration-augmented instructions that are given to you at train time (based on Figure 1 and equation 5)? If you do not get that information at test time, it seems like there is a train-test mismatch on input. \n\n[1] Inner Monologue: Embodied Reasoning through Planning with Language Models. https://arxiv.org/abs/2207.05608\n\nQUESTIONS:\n* In Table 3, when going from \"Basic Reasoning\" to \"+ Visual/Spatial Only\", why is the performance on Visual tasks and Spatial tasks worse?\n* In equation 5, is the VLM trained to predict both reason and action? I am assuming that it is doing that. \n* What exactly is the difference between \"Full Augmentation\" and \"Curriculum Learning\"? I am assuming that \"Full Augmentation\" directly train on all the data instead of following the 3-staged curriculum.\n\nSome additional questions are throughout the weaknesses sections.\n\nLIMITATIONS:\nYes, the authors have addressed both.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the embodied visual navigation problem in the EB-ALFRED benchmark where an agent is required to act based on visual observations in order to accomplish tasks. The authors observe that agents trained with data from this environment fail to learn behaviors contextualized in the environment and instead learn superficial associations. To encourage the agent to develop contextualized behaviors, the authors augment task instructions with various transformations such as adding visual and spatial modifiers to object references. Experiments with Qwen VL and Intern VL models show that this augmentation strategy leads to agents that generalize better than agents trained only on the non-augmented data.\n\nSTRENGTHS AND WEAKNESSES:\n* The motivation is sound. Tasks in benchmarks such as Alfred are generated based on templates and agents directly trained on such data fail to learn associations that are important for generalization.\n* The proposed augmentation strategies are meaningful. However, one aspect that seemed unsatisfactory is that the augmentation strategies are not sufficiently tailored for the specific environment. For example, in Figure 1, the modifiers 'cone-shaped' and 'small square' are useful when there are lamps and clocks in the environment that do not fit these specific descriptions. If there is only one lamp and only one clock in the room, how does adding these modifiers help the agent learn generalizable behaviors? Either the environment will have to be modified to add distractor objects or the task generation will need to pay attention to objects in the room to generate more tailored scenarios. \n* Performance improvements with Qwen VL and Intern VL models seem promising. Are published results included in this comparison table?\n* I am not convinced by the example in sec 5.3. This is just a regular Alfred task and it is not very surprising that the agent manages to complete it. It would be more interesting if the authors can present more non-trivial planning and reasoning behavior.\n\nQUESTIONS:\nsee strengths and weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents World-Aware Planning Narrative Enhancement (WAP), a novel framework that significantly improves Large Vision-Language Models (LVLMs) in complex embodied planning tasks. By focusing on four key cognitive capabilities, visual modeling, spatial reasoning, functional abstraction, and syntactic grounding, WAP addresses the limitations of environment-agnostic imitation learning. It uses a curriculum learning approach and operates using only raw visual inputs. Evaluated on the EB-ALFRED benchmark, WAP-powered models like Qwen2.5-VL show large gains, even outperforming proprietary models like GPT-4o and Claude-3.5-Sonnet in closed-loop settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The proposed WAP framework builds environmental understanding through four interrelated capabilities (visual appearance modeling, spatial-relational reasoning, functional abstraction learning, and syntactic grounding). This multi-dimensional approach to infusing LVLMs with environmental understanding directly addresses the core limitations of environment-agnostic imitation learning.\n2. Performs well using only vision and language inputs, showing strong results under partial observability—crucial for real-world scenarios.\n\n**Weakness:**\n1. Although computational resources are mentioned in the Appendix, the main text could better address the trade-offs between performance and the cost of using complete observation histories and generating 80k+ augmented instruction pairs.\n2. The framework's instruction augmentation and reasoning generation steps rely on a superior vision-language model (Qwen2.5VL-72B-Instruct) as the teacher. The authors could further elaborate on the sensitivity of the overall system's performance to the quality and specific characteristics of this teacher model.\n3. While the paper acknowledges that current LVLMs face challenges in unfamiliar environments, the evaluation is limited to the EB-ALFRED benchmark. To better assess the framework’s generalization capabilities, it would be valuable to examine its performance in new, unseen environments or with object configurations that differ substantially from those seen during training.\n\nQUESTIONS:\n1. The gap between explicitly guided and self-directed enhancement (Commonsense: 62 vs. 48; Long-horizon: 70 vs. 60) suggests issues in autonomous reasoning. What qualitative differences exist in the generated narratives?\n2. Could the authors provide a discussion of common failure modes or scenarios where the WAP-enhanced models still struggle?\n\nLIMITATIONS:\nnoted in weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}