World-aware Planning Narratives Enhance Large Vision-Language Model Planner
Authors: Junhao Shi
Zhaoye Fei
Siyin Wang
Qipeng Guo
Jingjing Gong
Xipeng Qiu
Venue: NeurIPS 2025 poster ()
Keywords: planning
embodied
LVLMs
Primary area: reinforcement_learning

Abstract: Large Vision-Language Models (LVLMs) show promise for embodied planning tasks but struggle with complex scenarios involving unfamiliar environments and multi-step goals. 
Current approaches rely on environment-agnostic imitation learning that disconnects instructions from environmental contexts, causing models to struggle with context-sensitive instructions and rely on supplementary cues rather than visual reasoning during long-horizon interactions.
In this work, we propose World-Aware Planning Narrative Enhancement (WAP), a framework that infuses LVLMs with comprehensive environmental understanding through four cognitive capabilities (visual appearance modeling, spatial reasoning, functional abstraction, and syntactic grounding) while developing and evaluating models using only raw visual observations through curriculum learning.
Evaluations on the EB-ALFRED benchmark demonstrate substantial improvements, with Qwen2.5-VL achieving a 60.7 absolute improvement in task success rates—particularly in commonsense reasoning (+60.0) and long-horizon planning (+70.0). Notably, our enhanced open-source models outperform proprietary systems like GPT-4o and Claude-3.5-Sonnet by a large margin.

World- Planning Narratives Enhance Large Vision-Language Model Planner
Junhao Fei'", Siyin Wang!. Qipeng Guo
Jingjing Gong"! Xipeng Qi
Fudan University SShanghai Innovation Institute SShangaai Al Laboratory jhsh 24@m. fudan. edu, en, zyfei200fudan. edu.cn ctl.ufuudn.edd..c. edu. cn
jigongj @gmail. jigmaalj@mmall
com, xpqiuεfudan.
Abstract
Large Vision-Language Models (LVLMs) show promise for embodied planning tasks but struggle with complex scenarios involving unfamiliar environments and multi-step goals. Current approaches rely aivironment-agnostic imitation learning that disconnects instructions from environmental contexts, causing models struggle with context-sensitive instructions and rely supplementary cues rather than visual reasoning during long-horizon interactions. In this work, we propose World-Aware Planning Narrative Enhancement (WAP), a framework that infuses LVLMs with comprehensive environmental understanding through four cognitive capabilities (visual appearance modeling, spatial reasoning, functional abstraction, and syntactic grounding) while developing and evaluating models using only raw visual observations through curriculum learning. Evaluations on the EB-ALFRED benchmark demonstrate substantial improvements, with Qwen2.5. VI. achieving 60.7 absolute improvement in task success rates rates- paress-aaticlllrlyy commonsense reasoning (+60.0) and long-horizon planning (+70.0). Notably, our enhanced open-source models outperform proprietary systems like GPT-4o and Claude-3.5 5-Sonnet by large margin.
1 Introduction
Recent advances in Large Vision-Language Models (LVLMs) 6 20 have expanded their applications to embodied planning tasks, where agents interpret natural language instructions into a sequence of actions which will be further executed in interactive environments (1 28 These models leverage large-scale pretraining on vision-language datasets align visual inputs with textual commands, achieving notable success in controlled scenarios with explicit object references and low environmental complexity [3] [2). However, when faced with increasingly complex real- life-like scenarios-thor those involving unfamiliar environments, varied instruction formats, and multi- step goals-current methods exhibit severe limitations in both generalization ability and reasoning consistency 26).
fundamental challenge confronting current embodied planning systems lies in their environment- agnostic imitation learning paradigm. In existing methodologies [3 12] expert demonstration trajectories are typically associated with simplified, iwvironmen-inneeeendeet instructions (e.g.. "put the apple on the table"). They force models to learn direct mappings from generic instructions to action sequences in an open-loop manner, disregarding the nuances of the changing surrounding environment. This learning approach operates disjointed fashion, treating task instructions and environmental
'Equal contribution Corresponning author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

contexts as distinct, unconnected elements. This hinders models from developing an integrated grasp of the environment's specific characteristics such as visual appearance, spatial relationships, object functionality, and linguistic
capaaiiiiissssss essential for human-like task execution [9]. While these models may perform adequately in standardized validation environments, their performance significantly deteriorates when faced with difficult situations that demand the ability to establish links between the changing surroundings and context-sensitive instructions, for example, "place the apple on the table near the television". The limitation is becomes evident during extended sequences of interaction, where models, due to their lack of detailed environmental representations, struggle to integrate previous visual observations. Hence, they resort to supplementary environmental cues, like feedback from actions taken or indicators of task progress, instead of relying exclusively on visual input for decision decision-making making.
To address these gaps, we introduce WAP, an innovative world-aware narrative enhancement approach. This method is designed to infuse LVLMSs with comprehensive information about the environment. By doing so, we aim to augment the model's understanding of the environment by incorporating relevant context from the world around it. Inspired by traditional thcories of cognitive intelligence [14, our narrative enhancement focuses on collecting data that progressively cultivates four inter- connected capabilities: (1) Visual Appearance Modeling: Detailed capture of object textures and geometries. (2) Spatial-Relanee Relational Reasoning: Understanding the spatial arrangement and room layouts. (3) Functional Abstraction Learning: Grasping tool-object relationships and symbolic representations. (4) Syntactic Grounding: Interpreting complex language to resolve ambiguity. To effectively steer the data generation process while considering various dimensions, we augment existing disjointed data by integrating complementary information from the environment or semantic spaces. Secondly, framework adheres realistic deployment scenarios: agents receive only image observations and natural language instructions in a closed-loop manner, without auxiliary privileged environmental feedback. To incrementally equip the model tackle cognitive challenges, we collect the data and train the model with a curriculum learning strategy, which enhances the model's ability to formulate sophisticated strategies in wide range of situations, markedly differentiating from traditional imitation learning techniques that do not support this level of advanced intellectual growth.
We evaluate framework through extensive experiments using Qwen2.5-VL (8 [andd Intern VL3 32 on the EB-ALFRED benchmark within EmbodiedBench (26), a challenging suite of high- level planning tasks. Our approach achieves substantial improvements over baseline methods, with Qwen2.5-VL. demonstrating a 60.7 absolute improvement in average task success rates. Particularly noteworthy are the significant gains in commonsense reasoning (+60.0) and long-horizon planning (+70.0), with similar patterns observed for InterV Remarkably, our enhanced open-source models outperform recent proprietary systems like GPT-4o and Claude-3.5-Sonne by a large margin.
This work makes three key contributions:
Our approach bridges the gap between high-level task instructions and the nuanced details of real-world environments by integrating contextual world knowledge into planning sys- tems. This multidimensional enhancement leverages narratives that are aware of various environmental factors, making the planning process more robust and adaptable to complex, real-life scenarios.
We demonstrate that closed-loop embodied agents can achieve superior planning perfor- mance using only visual observations and natural language instructions without privileged environmental beedback-challeneg
the prevailing assumption that additional auxiliary signals are necessary for robust planning in complex environments.
Our approach establishes new state-of of-the-art benchmarks on SB-ALFRED, outperforming not only existing academic baselines by 60.7 improvement but also surpassing proprietary systems like GPT-4o and Claude-3.5-Sonne by significant margins in challenging long- horizon planning scenarios.
2 Related Works
Embodied planning 4 25 represents a critical cognitive capability for embodied agents, serving as the brain that guides physical interactions within complex environments. Early approaches to embodied planning 18 27 31 relied exclusively on textual environment metadata rather than
visual perception, limiting their adaptability to real-world scenarios. Later visual-based methods
2

---

Base training set
Multi-dimension Enhancement
Visual
Wak the table with the cone-shaped lamp on it. lurn off the samp, and the small square clock noxt to it.
Initial
Walk to the table with
Unverified Enhanced Instructions i
lemp it, fum of
Spatial
Traj
lemp, pick up clock
Walr
shode beside woodenn
Turn the tamp, the clock
Self-verify (with r
belowr
Enhanced training set
Functional
#T
Walk to the inblo where the light acurce reats,
Enhanced
onlingaat the glow. Vren ralrove the timotuspar.
Reasoning
<Thinking....
Enhanced Instructions
Syntactic
As approach
Traj
Reasoning Generation
switch, the settle softer ambiance..
cosy
Model input/ output
Incons stant samples
(Instructinn
Trajectory
Ressoning
R
Figure I: Multi--iimensonal cognitive enhancement framework overview. Our approach trans- forms base instruction- instruction-rraeectory pairs through structured pipeline: (1) Four-dimensional instruction augmentation, targeting different cognitive abilities; (2) Reasoning generation with semantic verifi- cation; and (3) Curriculum learning across progressive stages. This process builds robust planning capabilities while maintaining strict visual-only observation constraints.
30 employed cascaded pipelines with external models for visual processing like semantic maps. Recent LVLM-based systems [2] 25 have begun processing visual input directly but continue depend on unrealistic forms of environmental feedback like action success signals and task progress information that would be unavailable in genuine deployment scenarios. Our work advances the field by operating solely on raw visual observations without privileged feedback and by processing complete observational history, more closely mirroring human capabilities in real-world settings.
16
Methodologically, embodied planning approaches can be categorized as either training- free or training- based. Training-free methods leverage prompt engineering [16 3 or multi-agent frameworks with specialized roles (29 5 [22 Training-based approaches include supervised fine-tuning on human
expert demonstrations 23 2, or methods that recognize the importance of trial-and- and-error learning rial-and-error
through either direct preference optimization [17] 30 21 or reinforcement learning [ 24 19].
Unlike these approaches that often treat task instructions and environmental contexts as disconnected elements, our world-aware planning narrative enhancement framework systematically develops integrated cognitive capabilities, enabling LVLMs to form sophisticated planning strategies without relying on privileged cues, more closely resembling human cognitive processes in complex real-world scenarios.
Methodology
We propose multi-dimensional cognitive enhancement framework for embodied planning that systematically develops reasoning capabilities across complementary dimensions while operating solely raw visual observations. Figure Iillustrates our approach's 's components.
3.1 Problem Formulation
In embodied planning tasks, an agent must execute a sequence of actions (a1, a2,..........................a.. .ra based on natural language instruction / and egocentric visual observations {or, o2,..., OT We operate within a closed-looy control framework, where each action decision depends critically on both current observation and historical context. This mirrors real-world robotics scenarios where agents must continuously adapt to environmental changes resulting from previous interactions. Unlike many previous approaches that focus isolated decision points, our framework explicitly models temporal dependencies by conditioning each action on the complete observation history:
a, = fe(T, (01, (01,02...,0.))
(1)
3

---

Where fe represents our rision-language model that predicts the next action based on the instruction and observation history. The key challenge to develop robust planning capabilities across diverse scenarios with only visual-context history. Our specifically addresses these challenges by systematically developing the agent's ability to extract and integrate relevant information across time.
cognitive enhancement approach
3.2 Multi-Dimensional Cognitive Enhancement
Our complete narratives enhance framework follows the pipeline illustrated in Figure ??. Our framework enhances cognitive capabilities across four complementary dimensions that systematically develop different aspects of the embodied intelligence required for robust planning.
3.2.1 Instruction Augmentation
Given an original instruction and expert trajectory = (at)) with corresponding observations {or Y we generate enhanced instructions { across four cognitive dimensions:
f = oT;))):0s) k E (Visual, Spatial, Functional, F
(2)
Where M is a superior vision-language model and 0g represents dimension-specific prompting strategies:
Visual Dimension:
Enhances object appearance modeling by explicitly describing visual attributes critical for identification (e.g., cone-shaped lamp, small square clock).
Augments positional understanding by specifying object locations rela- tive to environmental landmarks (beside the wooden chair) and precise spatial relationships (below the lamp base).
Spatial Dimension:
Functional Dimension:
Develops deeper object-interaction understanding by articulat- ing affordances and functional properties (light source, timekeeper) that capture causal relationships between objects.
Syntactie Dimension:
Introduces linguistic complexity through narrative structures, indirect references, and contextual dependencies that require resolving ambiguity beyond literal instructions.
This augmentation process creates structured hierarchy of cognitive demands, progressively chal- lenging the model's environmental reasoning capabilities.
3.2.2 Semantic Consistency Verification
To ensure augmented instructions maintain task equivalence, implement a verification mechanism:
5
c()= De (1))
Where ((; represents the verification function that checks if the generated instruction i represents the same intention as 7, I[-) is an identification function and evaluates to 1 if the input expression is true. Instructions fail meet the threshold will trigger regeneration of the instruction to maintain dataset quality.
3.2.3 Stepwise Reasoning Generation
For each action a, in trajectory T, we generate explicit reasoning r; that captures the cognitive process linking observation to action:
Tr = M(i, or, ((: :.) (1, (1, ..---,,)111))
The 1-th step of reasoning is left out for the vision-languan model M to predict. These reasoning annotations serve as intermediate supervision signals that help the model develop explicit cognitive processes that might otherwise remain implicit, including environmental state tracking, object relationship inferences, and action preconditions.

---

3.3
Curriculum Learning Framework
Our training procedure follows a three-stage curriculum that gradually increases cognitive complexity:
ee arg min Kel.)-,,
(5)
where fe represents the vision-language model being optimized and LcE the cross-entropy loss for action prediction. The curriculum stages are:
Base Stage (D1): Training on original instruction-trajectory pairs to establish foundational action mapping capabilities.
2. Environmental Understanding Stage (D2): to develop perceptual grounding and scene comprehension.
Incorporating visual and spatial augmentations
3. Conceptual Reasoning Stage (D3): Introducing functional and syntactic augmentations to develop higher-order reasoning about object relations and ambiguous references.
This progressive training scheme aligns with cognitive development theories, allowing the model first master perception-action correspondences before tackling more abstract semantic relationships. Importantly, our framework operates under strict partial observability constraints proviiding only egocentric RGB observations without privileged information to ensure real-world applicability.
Experiments
We conduct comprehensive experiments to evaluate our framework's effectiveness enhancing embodied planning capabilities. Our analysis focuses both overall performance metries and fine-grained cognitive capabilities across diverse task contexts.
4.1 Experimental Settings
Dataset We construct enhanced corpus comprising 80,875 cestructiio-trajeccte pairs derived from the original 16, 145 ALFRED trajectories through our multi-dimmnssonal augmentation approach. The dataset is structured across four cognitive dimensions: Visual (appearance attributes), Spatial (positional relationships), Functional (interaction affordances), and Syntactic (referential complexity). This structured enhancement enables systematic evaluation of specific cognitive capabilities crucial for embodied agents.
Models For instruction augmentation and reasoning generation described in Section 3, we employ Swen2.5-VLL-22B--mee as the teacher model. We evaluate our framework on two foundation model series: Qwen2.5-V1 VL (Qwen2.5- (Qwen2.5-VLL7B-- -L--.---LL----ntruue [ and Intern VL3 (Intern VL3-8B) [32].
representing state-of-the-art vision-language architectures with distinct pretraining approaches.
Evaluation We evaluate on the EB-ALFRED benchmark from EmbodiedBench 255. which pro- vides refined evaluation protocols over the original ALFRED benchmark, including streamlined action spaces and higher-quality language instructions. Beyond the standard Success Rate (SR) metric, also the standard deviation quantify model's ability to maintain consistent performance across varying task complexities:
/1 CESRR SR)2 -R)2 66
STD
(6)
Where C represents the six task categories (Base, Common, Complex, Visual, Spatial, and Long), and SR mean success rate across categories. Lower STD values indicate more balanced capabilities across different task types, while higher values suggest uneven performance that excels in some categories but struggles others, representing lower robustness. This metric provides crucial insight into model robustness beyond aggregate performance measures.
5

---

Table 1: Performance comparison on EmbodiedBench (EB-ALFRED), Results show success rates (SR) across task categories. Models marked with indicate our proposed approach and variants. Results for Intern VL3-8B and harder closed-loop settings are conducted by ourselves. Best results in bold.
Model
Avg.
STD
Base
Common
Complex
Visual
Spatial
Long
Proprietary Models (Original open-lopp setting with action feedback)
GPT-4o
7.8
64
46
52
Claude-3.5-Somee
64.0
8.6
72
76
60
58
Gemini-1.5-PP
62.3
7.8
70
64
72
58
52
58
Gemini-2.0-fflas.
52.3
6.2
62
48
54
46
46
58
Semini-1.S-fllas.
39.3
10.6
44
40
56
42
26
GPT-4o mini
24.0
13.0
34
24
22
Open- Models
Intern Mlem7WL22.5788BMMMP
40.0
4.5
48
36
42
40
40
34
Owen2.5-VLL-2BB-1
39.7
6.3
50
36
34
Owen2-VLL-72B-Im
33.7
4.8
40
30
40
30
32
30
Vision-Ins
32.0
10.1
38
34
44
28
32
16
MieemWVL2..5388BMMPPP
25.7
4.7
30
20
20
28
32
24
9.0
36
22
Llama-3.2-11B Vision-Ins
13.7
7.4
24
16
22
Vntem-112..58B-MPP
7.7
4.3
12
6
14
6
6
Dwen2.5-VI-77B-Is
4.7
3.9
10
6
2
0
Qwen2-VL-7B-Ins
1.7
2.3
6
2
InternVL3-SB
10.7
7.6
20
Proprietary Models (Under harder closed-loop settings)f
GPT-4o
26
2.8
30
26
28
22
26
24
Saude-3.5-Sonnee
$7.3
13.4
62
40
42
Our Approach with IntemVLL--BB
Intern InternVL3-88 VL3-8B
6
4.7
12
10
2
0
+ Original Reasoning
46.0
18.4
58
46
34
+ WAP Augmentation
57.0
5.5
62
52
60
58
48
62
Curriculum Learning
61.0
7.2
66
56
66
58
50
70
Our Approach with Qwen2.5-VVI-2 Inst
Qwen2.5-VL VL-7BBB--
2
2,5
6
4
0
0
Original Reasoning
47.0
14.0
64
22
48
50
44
54
WAP Augmentation
58.0
6.8
60
62
62
46
54
64
Curriculum Learning
62.7
6.3
66
62
70
56
52
4.2 Main results
As demonstrated in Table D our curriculum learning framework establishes new state-of-the-an
performance across all task categories while operating under strictly realistic observation constraints. The Qwen2.5- VL implementation achieves 13.5x improvement in average success rate (from 4.67 to 62.67) with 14% improvement compared with baseline method, surpassing even GPT-4o (56.3) and approaching Gemini-1.5-PP (62.3) under harder settings - without having access to privileged environmental information. Similar performance gains observed with InterVL3 models, which improve from 10.67 to 61.0.
Notably, our approach maintains competitive Standard Deviation (STD) of 6.3, lower than many proprietary models such Claude-3.5-Somme (8.6) and Gemini- 1.5-fil- (10.6), indicating more balanced capabilities across diverse task categories. This represents a substantial improvement over our basic reasoning approach, which exhibits a high STD 14.0, suggesting uneven performance that handles some categories well but struggles significantly with others. The progression from basic reasoning SSTD=14.0) to curriculum learning (STD=6.3) demonstrates how our multi-dimensional enhancement approach systematically builds more balanced cognitive capabilities.

---

Enhanced Environmental Cognition Our framework demonstrates substantial improvements in environmental understanding capabilities, evidenced by performance across specialized task categories:
For VL.3 the success rate improves from 46 (baseline) to 58 on tasks requiring the identification of objects based appearance (c.g.. distinguishing objects by color, shape, or pattern).
Visual Perception:
2. Spatial Reasoning:
Performance rises from 34 to 50 (IntemVL3) on tasks requiring precise positioning and relational understanding (e.g.. "place the object to the left of the sink"),
3. Semantic Grounding: For Qwen2.5-Ve VL. commonsense reasoning accuracy increases from
22 62, enabling effective handling of instructions requiring real-world knowledge and functional understanding of objects.
4. Referential Resolution:
The model successfully resolves ambiguous references (c.g., "that container mentioned before") through contextual reasoning with accuracy raising from 48 to 70 (Qwen2.5-VL), overcoming a critical limitation in conventional vision-language systems.
These improvements underscore the importance of structured environmental modeling beyond simple action prediction for robust embodied planning.
Further generalization long-horizon planning Our approach achieves remarkable 70 success rate on long-horizon tasks-those requiring 15+ sequential actions actions-reprecnn represeetrrggngg 35- 35-fold improvement over baseline models and matching Souude-3.5-Sonnee 's performance in this challenging category. Notably, while proprietary models show mixed results under the stricter closed-loop setting (without action feedback), framework maintains consistent performance across all settings.
This exceptional capability stems from two key innovations, (I) Full Temporal Context: Our training paradigm incorporates complete observation histories rather than isolated frames, enabling the model to capture causal relationships between actions and environmental changes across extended sequences. (2) Multi-dimensional Knowledge Integration: The four cognitive dimensions of our data enhancement approach collectively enable the model to maintain coherent world representations throughout extended task execution.
The dramatic performance disparity between GPT-4o (24 in closed-loop vs. 54 in open-loop settings) and Claude-3.5-Somnet (74 in closed-loop vs. 52 in open-loop) on long-horizon tasks is particularly revealing. While GPT-4o struggles without action feedback, likely falling into repetitive error patterns, Claude-3.5 3.5dde----Soonet actually improves under closed-loop constraints, suggesting superior error recognition and recovery capabilities. Our framework (70 success rate) approaches Claude's closed-loop performance without requiring proprietary model access, validating our hypothesis that proper environmental modeling serves as a critical foundation for compositional task planning. This confirms that closed- settings, despite being more challenging, better reflect the requirements for successful long-horing horizon planning in real- real-world scenarios.
Analysis
5.1
Self-Directed Enhancement Potential
To explore autonomous data augmentation capabilities, we investigate a self-directed enhancement approach where models independently select augmentation strategies based task descriptions. This implicit enhancement contrasts with our explicit curriculum learning framework.
As shown in Table self-directed enhancement achieves moderate success (56.7 average), but exhibits notable limitations compared to our explicit framework (62.7). The self-directed approach demonstrates reasonable environmental perception capabilities- agpabliites-matchinng explicit methods in Visual (52 vs. 56) and Spatial (52 vs. 52) categories. However, shows significant deficiencies in semantic reasoning: Commonsense understanding (48 62) and Long-horizon planning (60 70).
While the self-directed curriculum approach improves performance consistency (STD=7.2) compared to the basic reasoning baseline (STD=14.0) it remains less balanced than our explicit curriculum method (STD=6.3). This suggests that while the model can autonomously develop more uniform capabilities across tasks, it still benefits from structured guidance in developing complementary
7

---

Table 2: Comparison between explicit and self-directed enhancement approaches
Method
Avg
STD!
Base
Common
Complex
Visual
Spatial
Long
Original Reasoning
47.0
14.0
22
48
54
Explicit (Standard)
58.0
6.8
62
62
64
Explicit (Curriculum)
62.7
6.3
66
62
70
56
52
70
Self-Directed (Standard)
54.7
7.7
50
62
62
Self-Directed (Curriculum)
56.7
7.2
66
48
62
52
52
60
cognitive skills. The model shows particular weakness in commonsense reasoning (48), indicating tendency toward surface-level linguistic modifications rather than deeper semantic understanding.
Nevertheless, the 56.7 success rate achieved without numan-designed enhancement rules suggests promising avenues for autonomous improvement future work.
5.2 Generalization to Unseen Environments and Object Configurations
To evaluate robustness beyond the training distribution, we measured performance the unseen split of VOTA-Bende [21], which introduces novel kitchens, living rooms, and object instances absent during training. We report Success Rate (SR) and tasks, comparing our approach (WAP) with GPT-4o:
success (PL. across diverse
Table 3: Results on VOTA-Benche unseen split. SR success rate; PL path-length-weiihhed success.
Task
Examine Light
Pick & Place
Stack & Place
Heat & Place
Cool & Place
Overall
Method
SR
P1.
SR
Pl.
SR
P1.
SR
P1.
SR
PT.
SR
Pf.
GPT-4o
30.42
24.73
24.15
19.82
18.27
14.56
16.83
15.98
12.54
10.45
20.36
16.83
WAP (ours)
70.35
65.42
58.62
54.91
61.47
56.37
63.84
60.21
59.28
57.63
64.56
59.86
WAP consistently outperforms GPT-4o across all task families and both metrics, indicating improved grounding and exccution efficiency in environments and object configurations that differ substantially from those seen during training.
5.3 Sensitivity to Teacher Model Quality and Size
We assess sensitivity to the teacher model by varying size and quality using Qwen2.5-7B, Qwen2.5- 32B, and Qwen2.5-72BB [8). Larger teachers generally yield better results across most dimensions, with stronger gains in spatial and long-horizon reasoning:
Table 4: Impact of teacher model capacity performance across evaluation aspects.
Teacher
Avg
Base
Visual
Spatial
Common
Complex
Long
Qwen2.5-7B
54.67
66
54
56
48
48
56
Qwen2.5-32B
57.33
64
60
62
52
46
60
Qwen2.5-72B
62.67
62
70
56
52
70
As shown in Table average performance improves monotonically consistent gains in the Spatial and Long categories as the teacher seales up. Some dimensions (e.g.. Complex) may show non-monotonic changes at intermediate scales, suggesting that benefits from additional capacity can task- and skill-dependent.
with
5.4
Ablation Study
We conduct top-down ablation on Qwen2.5-VLL-77 by starting from the full framework and progressively disabling modules isolate their effects. We remove, sequence: (i) curriculum learning (three-stage schedule: Base Environmental Understanding - Conceptual Reasoning); (ii)

---

Table 5: Top-down ablation from the full framework by progressively disabling curriculum, step-wiid wiie reasoning, and WAP instruction enhancement
Configuration
Avg
STD
Base
Common
Complex
Visual
Spatial
Long
Curriculum (Full)
62.7
6.3
70
w/o Curriculum
58.0
6.8
60
62
62
46
54
w/o Following-steps Reasoning
54.0
9.3
62
46
64
52
40
60
w/o First-step Reasoning
46.7
17.1
60
16
56
46
42
60
w/o WAP Instruction
47.0
14.0
64
22
48
50
44
54
Base Model
4.7
3.9
10
8
6
2
0
step-wise reasoning cesponennts-frs--sttee reasoning and the reasoning chains applied at following steps; and (iii) WAP instruction enhancement linensiaimnnional prompts integrating visual, spatial, commonsense, and long-horizon cues).
According to the results in Table [] removing curriculum decreases the average by 4.7 points (62.7-58..0 00) and slightly increases variance. The largest drops occur in Visual (56-26 -10), Complex (70-62,- -8)), Long (70-64,- -6), and Base (66->60,-6) while Common remains unchanged (62) and Spatial slightly improves (52-54, +2). This indicates that curriculum primarily benefits perceptual robustness and long-horizon execution, with limited direct impact on commonsense.
Disabling reasoning chains after the first step (w/o Following-ste Reasoning) further reduces the average to 54.0 (-4.0) and increases variance (STD 6.8-9933). The strongest declines are in Common (62-46,, -16) and Spatial (54-400, -14), underscoring that sustained step-wise reasoning is crucial for coherent environment -attmnnntttttinnnno coupling beyond initial guidance. Visual and Complex rise slightly (46-52; 62 62)) suggesting that front-loaded guidance offers short-term benefits for perceptual tasks but cannot maintain commonsense spatial consistency.
Removing the initial hootstrapping chain (w/o First-step Reasoning) causes a sharper average drop to 46.7 (-7.3) and the largest variance increase (STD 9.3 97.1> Common collapses (46->16, -30), with additional declines in Complex (64-56, -8) and Visual (52-+46, -6), while Long remains unchanged (60). This pattern shows that first-step reasoning is pivotal for establishing correct task setup and commonsense grounding; without it, the model becomes highly unstable across categories.
Replacing WAP prompts with original simple instructions (w/o WAP Instruction) keeps the average similar (46.7-447.0 but changes the error profile: Common recovers modestly (16-+22, +6), whereas Complex (56-+48, -8) and Long (60-54, -6) deteriorate. This suggests WAP cues are particularly beneficial for complex and long-horizon behavior, but in the absence of step-wise reasoning they can amplify imbalance across categories.
Overall, three findings emerge: (1) Curriculum chiefly strengthens long-horizon planning and perceptual robustness while modestly reducing variance. (2) Complete step-wise reasoning is essential for stability and for commonsense and spatial coherence; removing follow-up chains disproportionately harms Common and Spatial. (3) WAP instruction is necessary to realize gains on Complex and Long tasks, but without sustained reasoning chains it yields an imbalanced skill profile. The full combination achieves the highest average with the lowest variance, improving the base model by over 13x while maintaining balanced gains across all splits.
5.5
Case Study
To provide qualitative insights into our model's capabilities, we analyze representative examples of embodied planning and reasoning. Our case studies focus (I) cognitive process transparency in complex multi-step planning and (2) the effectiveness of our multi-dimensional augmentation approach.
Figure 2 illustrates the execution of seemingly simple instruction: "Place a chilled apple section into the bin." This instruction contains implicit procedural requirements- the apple must be sliced and chilled necessitating a sequence steps beyond merely disposing of an apple.
Our Qwen2.5-Ve LLL model with curriculum learning successfully decomposes this task into 18 distinct actions across three phases. First, the model identifies necessary tools, demonstrating task decom- position by recognizing that sectioning requires knife. Second, it exhibits situational awareness,

---

User Instruction:
Place chilled apple section into the bin.
Observation
(Part 2: Find knife and slice)
Reasoning 1: escential for safely and effectivvly slicing the opple into sections.
the apple. first need find knife, A kmtfe
Action
Action 2: pick the knife
Task Decomposition: Breaking complex instructions into sub goels
Action an apple
Action 4; siice the applo
Reasoning 4: Since we have already found picked up knijee. next logicul slice the apple. This will allow create chilled appic section that can he placed into the his as per the male goal.
Obsseniation
(Part apple fridge)
Functional Understanding: Tool usage conditins and mechanisms
5: find
top
Action 6: put down the object
Reasoning 6: Since the apple been sliced, next step is to pur knife down the countertop free your hands for handling the apple.
apple
Observation on
Action 8: pick up the apple
9: fridge
Situational Awareness:
Enutronment state tracking and action planning
Action 10: open the fridge
put down the object
Reasoning 12: After puiting down the apple, the next logical close the fridge. This entures that apple properly stored the refrigerutor chilled before placed hin.
OBserpation 17
Action 12: close the fridge
-...
Object Property Reasoning: Understanding functional features and operation methods
apple bin)
open fridge
Action 14: find an apple
Reusoning Since apple needs to chilled, was placed fridge. Now, the next step (Ind sultable container bin place the chilled apple section. A garbage can common place to dispose food waste.
Obseration on
15: pick up apple
Action 16: clase the fridge
garbage
Action 18: put down the object
Commonsense Application: Intograting everyday knowledge into task execution
Figure 2: Reasoning process visualization for complex instruction execution. The figure shows our model executing the instruction "Place a chilled apple section into the bin." The model successfully decomposes this seemingly simple instruction into 18 distinct actions across three phases, demon- strating robust planning capabilities. Reasoning annotations highlight five critical cognitive abilities: task decomposition, functional understanding, situational awareness, object property reasoning, and commonsense knowledge application. This example illustrates how our model maintains coherent planning over long horizon (18 steps) while handling implicit requirements not explicitly stated in the instruction (e.g.. the apple must be chilled before disposal).
notably Reasoning where it places the knife down before handling the apple, showing safety awareness. Most importantly, the model correctly infers the need to refrigerate the apple to fulfill the "chilled" requirement- demonstrating both functional understanding of appliances and commonsense knowledge application. This example highlights improvements achieved through our curriculum learning approach, as baseline models consistently failed to maintain coherence across extended sequences, typically omitting the critical chilling step.
6 CConclusion
In this paper, address fundamental limitations in embodied agents where current LVLMMs struggle with complex real-life scenarios due to their environment-agnostie imitation learning paradigm. Previous approaches treat task instructions and environmental contexts as disconnected elements, leading to poor generalization in unfamiliar environments and inconsistent reasoning during multi- step tasks. We introduce a world-aware narrative enhancement approach that systematically develops four cognitive capabilities: visual appearance modeling, spatial-relational reasoning, functional abstraction learning, and syntactic grounding. Our experiments on EB-ALFRED demonstrate substantial improvements with Qwen2.5-VL. and Intern VL3, achieving up to 60.7 higher success rates and outperforming even proprietary models like GPT-4o in challenging scenarios. Our approach proves that high-performance embodied planning is possible using only raw visual observations without privileged feedback, establishing a new sfate-of-the-ar for embodied systems in complex, real-life-lie environments.
10

---

Acknowledgment
This work was supported by the Science and Technology Commission Shanghai Municipality (No. 24511103100)
References
[1] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online rein- forcement learning. ArXiv, abs/2302.02662, 2023. URLhttss: http:////ppp.sssttissssssieee www.nticschol..ooo
:
[2] Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang. Robogpt: intelligent agent of making embodied long-term decisions for daily instruction tasks, 2024. URL https:://axxi...tt
2361111549911 15649
[3] Jacky Liang, Wenlong Huang, Xia, Peng Xu, Karol Hausman, Brian lchter, Peter R. Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500, 2022. URL
https://api. www.mtiscctcsooooooooooooo
org/Corpus.li :D0rpus1D:2223555422 252355542.
[4] Yang Liu, Weixing Chen, Yongjie Bai, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied AI. CoRR, abs/2407.06886. 2024. doi: 10.48550/ARXIV.2407.. 066.6668.. URL htt/hhtts///oo/ org.:/....rg//... 11.p://1..8855000 48550/arXivv 2407 06886
[5] Jinjie Mai, Jun Chen, Bing chuan Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem. Ilm as robotic brain: Unifying egocentric memory and control.
ArXiv, abs/2304.09349, 2023. URL. https://api semanticcchollarr orr.alauurrrrCrruuuDDD 258212642
[6] OpenAL. Gpt-4o system card, 2024. URL htttht:::///ariii htttts://arxv..orr//a/////1/
[7] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision- and-language navigation. 202/ IEEE/CVE International Conference on Computer Vision (ICCV), pages 15922-15973 2021. URL https:://ppl semanticscholar htwpr//sccccccaaoorrrorrrrupuu
234482879
[8] Qwen, :, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://:://
org 2ttp
[9] Lawrence Shapiro and Shannon Spaulding. Embodied Cognition. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2025 edition, 2025.
[10] Suyeon Shin, Sujin Jeon, Junghyun Kim, Gi-Cheon Kang, and Byoung- -ak Zhang. Socratic planner: Inquiry-based zero-shot planning for embodied instruction following. ArXiv, abs/2404.15190. 2024. URL https://appi semanticscholar Irg/CorpusID; 269302975
[11] Keisuke Shirai, Cristian Camilo Beltran-Hernander Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, and Shinsuke Mori. Vision-language interpreter for robot task planning. 2024 IEEE International Con- ference on Robotics and Automation (ICRA), pages 2051-2058, 2023. URL, httphhttss.. ://::///ppp
semanticscholar. htg/CorppuIDD:2443355.2

---

[12] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk. Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks, 2020. URL htttpss:s//aiii.or.g.g.eeee 1992/12.. 1912. 01733
[13]
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreviie net/ forum?)d--IOXXYcGGTT
[14] R. S. Siegler. Children's 's learning. American Psychologist, 60()) 2005. doi: 10. 1003-066X.60.8.777
[15] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. 2023 IEEE International Conference Robotics and Automation (ICRA), pages 11523-11530, 2022. BRttsstp:://ap U..phttps./ppp...eeat...cchhh semanticscholar
org/ 252519594
[16] Chan Hee Song, Jiaman Wu, Clay Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2986-2997, 2022. URL https://api www.nticscooollatt....oooooomaae wwwp//rrg///oo.o 254408960
[17] Yifan Song, Da Yin, Xiang Yue, Jie Huang. Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- based trajectory optimization for LLM agents. CoRR, abs/2403.02500 2024. doi: (044850/ARXXI.2240..0222
URL httpss: org/10..00110 48550/ www....ww...........o 04440.00500
[18] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models, ArXiv, 0bss/2305.16657 2023. URL https
Dorpat/tccccccoo.ooooo/////poopo :2: 258947337
[19] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, Devon Hjelm, and Alexander Toshev. Large language models generalizable policies for embodied tasks. ArXiv, abs/2310.17722, 2023. URL https
7/api. www.sssecacicchhlla..
Ctrp//oruusTiii ID: :645555748
[20] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, fos/2403.055300 2024. URL https::/appi.semaati.coo
org/
[21] Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes a better planner: Dual preference optimization for embodied task planning. CoRR, abs/2503. 10480, 2025. doi: (0.48550/ARXV.2503.. 10480. URL
https://doi.orz: org/10...111... (err/10.88550) 48550/arXiv arxiv 2503. 10480
[22] Zidan Wang, Rui Shen, and Bradly C. Stadie. Wonderful team: Zero-shot physical task
planning with visual 1lms. 2024. URL https://api. semanticscholar wwwpcchholarroo//oooooaooo IIr
271533474
[23] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. ArXiv, abs/2307.01848, 2023. httpstts:://pp.. htttpp::/app..emmmti.csssss
org/CorpusID ID: 259342896
[24] Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang. Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Octopus: Embodied vision- language programmer from environmental feedback. In European Conference on Computer Vision, 2023. URL httpss//p//p semanticscholar, Trrg/CorpusID... (orpusIDD:633992555 263909250
[25] Rui Yang. Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiiedeench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. ArXiy, abs/2502.00560. 2025. (tIppsttss://// semanticscholar. wtwp:ooppsSDD:273312222
12

---

[26] Rui Yang. Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian. Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedhenchh Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents, 2025. URL https: //trsiv...x.vv. org/ abs/ /a///bbb//222// 2502. 09560
[27] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023. OpenReview.net, 2023. http:htttppppeereeeeeeee
[28] Fanlong Zeng. Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S. Yu. Large language models for robotics: A survey, 2023. URL htt.:/ttt///axii.oottt org/ abs//bb////// 07226
[29] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. ArXiv, abs/2307.024855, 2023. URL https: /tapss://ap1 api semanticscholar
org/Corpus ID: 259342833
[30] Qi Zhao, Haotian Fu, Chen Sun, and George Dimitri Konidaris. Epo: Hierarchical Ilm agents with environment preference optimization. ArXiv, abs/2408.16000, 2024. URL htt.s: //api
semanticscholar: wwtp/oog/////ooo :72146208
[31] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense
knowledge for large-scale task planning. ArXiv, abs/2305.14074 2023. URL https
//api. Peapi.semanticachol..o
orggCorpus ID: 258841057
[32] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao. Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang. Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: / / arxxiv org/ abs/ 2504 10479
13

---

7 Technical Appendices and Supplementary Material
A Limitation
While our framework demonstrates significant advancements in embodied planning, several limita- tions warrant further discussion. First, our world-aware narrative enhancement operates primarily the symbolic action level (e.g.. "pick up knife"), lacking explicit modeling of continuous control parameters. This abstraction necessitates future integration with low-level controllers to enable practical deployment requiring force modulation and precise trajectory optimization. Second, while our experiments demonstrate effectiveness in household environments based on the ALFRED dataset, the framework's s generalizability industrial settings or outdoor scenarios with dynamic obstacles remains unverified. These domains may present distinct challenges in spatial reasoning and adaptive navigation that our current implementation does not address. Third, in our approach we mainly focus enhancing user instruction and step-wise reasoning, limiting the fine-tuned model's capacity for mid-execution error correction. This constraint suggests the need for more dynamic enhance mechanisms in future iterations.
B
Training Details
B.1
Narrative Enhancement
Our instruction enhancement pipeline employs the -0-n225--VL-22-Innttreee model under manufacture--ecommeeddd configurations, utilizing temperature sampling (r=1.0) with nucleus filtering (top-p=0.7) for balanced diversity and coherence. This process generates both narrative- augmented instructions and associated step-wise rationales, maintaining strict alignment with the original ALFRED action trajectories through constrained decoding mechanisms.
B.2 Model Fine-tuning
implement full-parameter optimization on two vision-language architectures: Qwen2.5-V1-722 Instruct and Intern VI.3-8B. The training regime employs AdamW optimization with base learning rate 7) = 110-5, 10% linear warmup, and cosine decay scheduling over epochs. Experiments utilize contrastive context windows (16k/32k tokens) with per-device batch size 4, distributed across 8xA100- 80GB nodes via tensor parallelism. The complete training cycle requires hours per model variant, aggregating to 800 A100 GPU-hours when accounting for ablation studies and hyperparameter tuning. Memory optimization is achieved through Flash Attention v2 and BF16 mixed-precision training, maintaining numerical stability while maximizing hardware utilization.
B.3
Compute for augmentation and training.
We further report the end-to-end compute for data preparation and training on H100-class accelerators:
Table 6: Compute profile for augmentation and training.
Stage
Hardware
Compute (GPU hours)
Instruction augmentation
4x H100
20
Reasoning augmentation
4x H100
200
Training
8x H100
100
Reasoning augmentation dominates the preparation cost, followed by supervised training and instruc-
tion augmentation. In practice, practitioners can: (i) adopt complete observation history inference for stronger execution performance; and (ii) scale reasoning augmentation selectively to balance up-front cost against downstream improvements.
14

---

C Additional Results and Analyses
C.1 Performance-Cost Trade-offs: Generation
Observation History and Augmented Instruction
We study how observation history and augmentation choices affect both effectiveness and computa- tional cost. All inference-time results measured with tensor parallelism on 2 GTX 3090 GPUs. Using complete observation histories (i.e., multi-image inputs) improves performance relative to single observation, at the expense of higher inference compute. Importandy, the speed comparison here contrasts multi-image inputs against the original long textual prompts used in EmbodiedBench, isolating the impact of richer visual context versus prompt length.
Table 7: Effect observation history on performance and inference-time cost (tensor parallel on 2x GTX 3090s).
Setting
Performance
Inference compute (GPU hours)
Single observation
52.33
1.6
Complete observation history
62.67
4.2
Relative to single-observation inference, complete history yields +10.34 absolute improvement while increasing compute by roughly 2.6x (1.6-4..2 GPU hours), offering a clear latency-accuracy knob for Long-horizon tasks where temporal context is critical.
C.2 Qualitative Examples of Reasoning Effect
Example 1: Object Recognition and World Knowledge
Task:
After enjoying your morning coffee, move the beverage holder to the nightstand.
WAP Model (Success):
Action Sequence: (1) find mug (2) pick up the mug (3) find side table (4) put down object
Step Reasoning: "Since the goal is move the beverage holder to the nightstand, I need to first locate the beverage holder. Assuming it's mug, I will start by searching for in the current environment."
Action: find mug
Baseline Model (Failure):
Action Sequence: (1) find a statue (2) pick up the statue (3) find side table (4) put down object
Step Reasoning: "Since the goal is to move the beverage holder to the nightstand, I first need to locate the statue to determine its current position and plan the next steps accordingly."
Action: find statue
Figure 3: Example showing baseline's failure to understand object-level world knowledge (mug as beverage holder)
15

---

Example 2: Multi-Object Manipulation with Physical Constraints
Task:
When finished cooking, place the saucepan with a spatula inside on the kitchen counter.
WAP Model (Success):
Action Sequence: (1) find a spatula (2) pick up the spatula (3) find pan (4) put down the spatula (5) pick up the pan (6) find counter top (7) put down object
"The current visual state shows that the spatula is being held, the saucepan is still on table. To place the spatula inside the saucepan as per the user's instruction, need to put down the spatula temporarily." Action: put down the spatula
Step 4 Reasoning:
Baseline Model (Failure):
Action Sequence: (1) find spatula (2) pick up the spatula (3) find fridge (4) open the fridge (5) find a pan (6) pick up the pan (7) close the fridge (8) find a counter top (9) put down object
Step Reasoning: "After picking up the spatula, the next logical step is look for a saucepan. A saucepan is typically stored in the fridge or pantry."
Action: find fridge
Claude-sonnet-3.5 (Failure):
Action Sequence: (1) find a pan (2) pick up the pan (3) find spatula (4) pick up the spatula (5) put down the object in hand (6) find counter top (7) put down the object hand
Step 3 Reasoning: "Based on the instruction, we need to handle saucepan and spatula, placing them together on the kitchen counter. The task requires finding and manipulating multiple objects in a specific sequence to achieve the desired arrangement."
Figure 4: Example showing baseline and comparison models' incorrect spatial reasoning and failure handle one-object-att -otject-t----imee constraint
The failure cases reveal different types of reasoning deficiencies: Baseline model shows incorrect spatial reasoning by mistaking sauce and saucepan, demonstrating poor understanding of typical kitchen object locations; Claude-sonnet-3.5 fails understand the task requirement of placing the spatula inside the pan, instead treating them as separate objects to be placed on the counter; Our model correctly recognizes the one-object-at-a constraint and strategically puts down the spatula to enable proper manipulation.
C.3 Contrastive Examples of Narration-Augmentted Training
Original Training Sample
Original Instruction:
Place white coffee mug on white dresser
Action Sequence: (1) find mug (2) pick up the mug (3) find a dresser (4) put down object
Step 1 Reasoning: "Since the white coffee mug needs to be placed on the white dresser, need to locate the mug first before proceeding to the next step."
Action: find mug
Figure S: Original training sample with basic reasoning
16

---

Narration-Auumeented Training Sample
Enhanced Instruction:
Place the white vessel for enjoying hot beverages atop the surface where personal treasures rest.
Action Sequence: (1) find mug (2) pick up the mug (3) find a dresser (4) put down the mug
Step 1 Reasoning: "The user instruction requires placing white vessel (likely mug) for enjoying hot beverages on a surface where personal treasures rest. From the provided image, there is no visible mug present. Therefore, the logical next step is to find mug."
Action: find mug
Figure Narration-augmentted sample with semantic bridges ("vessel for enjoying hot beverages" mug)
The narration in the augmented data explicitly states the semantic bridges that are only implicit in the original demonstrations. Training on pairs that couple the same action with both concise and enhanced reasoning equips the policy with these paraphrase links. Consequently, at evaluation time the WAP model retrieves the correct object and surface ("beverage holder" + mug. "nightstand" + side table), whereas the baseline lachinee-lckkingg such supervision often selects objects similar to the training data ("statue") while having no semantic relationships.
D Data Generation Process
D.1 Dataset Statisties
Our enhanced dataset comprises 80,875 instruction-trajeee trrjeetort pairs derived from 16, 14,145 original ALFRED trajectories, systematically expanded through two complementary approaches. The first expansion preserves the original trajectory structure while adding four specialized narrative en- hancements (visual, spatial, functional and syntactic), yielding four distinct subsets each containing 16,145 samples. Unlike conventional datasets that provide only sparse instructions and atomic action sequences, framework enriches each trajectory with: (1) step-wise observation images capturing environmental states, and (2) step-wise reasoning annotations detailing action rationales and preconditions.
Additionally, we provide comparative dataset of 32,290 samples featuring implicit-instruction augmentation. This contrastive set employs self-supervised prompting techniques where models autonomously determine enhancement requirements through preliminary attention patterns, rather than receiving explicit annotation guidelines. This dual-structure design enables systematic evaluation of both human- human-guided and model-induced enhancement strategies, while maintaining parity in environmental complexity and task diversity with the original ALFRED distribution.
D.2 Prompt Templates for Instruction Augmentation
Here are the prompt templates used for generating the enhanced instructions across the four cognitive dimensions.
Visual Dimension Prompt
17

---

Visual Dimension Prompt:
Enhance the following instruction by adding spatial descriptions based the image. Keep natural and concise.
Examples:
Original: "Put two spray bottles in the cabinet"
Enhanced: "Place two cylindrical green cleaning spray bottles in the wooden cabinet."
2. Original: "Put knife in container"
Enhanced: "Put a 20cm silver chef knife into the blue rectangular plastic container." "
Now enhance:
Original: /human instruction
Enhanced:
Figure 7: Prompt for instruction visual enhancement
Spatial Dimension Prompt
Spatial Dimension Prompt
Enhance the following instruction by adding multi-layered spatial descriptions based on the image. Refer to objects through their positional relationships with 2-3 adjacent landmarks. Keep descriptions natural and concise.
Examples:
Original: "Put two spray bottles in the cabinet"
Enhanced: "Put two spray bottles in the white cabinet under the stainless steel sink against the wall"
2. Original: "Put knife in container" Enhanced: "Place the chef's knife holder on the granite countertop, positioned to the right of the refrigerator"
Now enhance:
Original: (human_ jintrn_isttuuction Enhanced:
Figure 8: Prompt for instruction spatial enhancement
18

---

Functional Dimension Prompt
Functional Dimension Prompt
Enhance the following instruction by adding functional descriptions based world knowledge. Replace one object or its placement with an indirect reference while keeping the sentence natural and concise.
Examples:
Original: "Put two spray bottles in the cabinet"
Enhanced: "Place items used for misting surfaces inside the cabinet."
2. Original: "Put knife in container" Enhanced: "Insert an object commonly used for cutting into a container designed for safekeeping."
Now enhance:
Original: (human instruction Enhanced:
Figure 9: Prompt for instruction functional enhancement
Syntactic Dimension Prompt
Syntactic Dimension Prompt
Enhance the following instruction into a more elaborate version by adding contextual details and symbolic substitutions. Replace objects and locations with contextual references (c.g.. pronouns or implied terms) and include irrelevant but plausible background information. Keep sentences concise and avoid adding new actions.
Examples:
Original: "Put two spray bottles the cabinet" Enhanced: "The spray bottles are on the shelf, already cleaned from yesterday's use. Move them to the cabinet for storage."
Original: "Put a knife in container"
Enhanced: "That knife the counter just finished slicing vegetables, Place it in the container to keep the edge protected."
3. Original: "Put washed lettuce in the refrigerator" Enhanced: "There's s lettuce in the sink-we'ss ve prepped enough for dinner. Wash it and store there to keep it fresh." "
Now enhance:
Original: (human instruction]
Enhanced:
Figure 10: Prompt for instruction syntactic enhancement
19

---

D.2.1 Verification Prompt for Semantic Consistency
Verification Prompt for Semantic Consistency
You are tasked with verifying two instructions describe the same task.
Original instruction: (original instruction
Enhanced instruction: (enhanced instruction] Environment images: [IMAGES]
Please verify if these instructions describe the SAME task goal, even if expressed differently. Consider only task objects and actions, not the specific methods. Respond with ONLY "Yes" if they describe the same task, or "No" if they describe different tasks.
Figure II: Verification Prompt for Semantic Consist
D.2.2 Reasoning Generation Prompt
Reasoning Generation Prompt
You are tasked with generating the reasoning process for an embodied agent executing specific action.
Instruction:
(enhanced instruction]
Current observation: [IMAGE]
Current action: {action]
Previous actions and reasoning:
{previous_actions, _nd (enddrrnnnnnn)
Please generate detailed reasoning that explains WHY the agent should take the cur- rent action. Include:
What the agent observes in the environment
2. How this relates to the instruction
3. Why this specific action is appropriate at this step
4. How this action contributes to the overall task goal
Reasoning:
Figure 12: Reasoning Generation Prompt
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately reflect the paper's contributions. We identify the limitation of environment-agnostic planning in current LVLMs and propose a world-aware enhancement framework that systematically develops four cognitive dimensions (visual, spatial, functional, and syntactic).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitation is discussed in Appendix Part with separate "Limitations" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Code and model generation configs are provided in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Our code is provided in supplementary materials and will release soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee httaa//gggiiee//eeuuabiisisssss /Cc////oddiiisssseeeeeee ssiiiiiisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Setting and Details are discussed in section 4.1 and appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the evaluation settings in Embodiedbench, which does not contain етrоr bar settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96 CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments compute resources part is discussed in Appenix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We carefully follow NeurlPS Code of Ethics in our research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This part is discussed in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use the dataset from ALFRED httpss//gtttua.oommkkkolllllllrd and evaluation benchmark from Embodied-
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
All assets are discussed in main paper and Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Details are discussed in Section 3.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hattaa/a/aaaarrsssec////oeeeeere for what should or should not be described.
27


Reviews:
[{"review_id": "U77apSyQEx", "reviewer": "Reviewer_5NkH", "summary": "The paper presents World-Aware Planning Narrative Enhancement (WAP), a novel framework that significantly improves Large Vision-Language Models (LVLMs) in complex embodied planning tasks. By focusing on four key cognitive capabilities, visual modeling, spatial reasoning, functional abstraction, and syntactic grounding, WAP addresses the limitations of environment-agnostic imitation learning. It uses a curriculum learning approach and operates using only raw visual inputs. Evaluated on the EB-ALFRED benchmark, WAP-powered models like Qwen2.5-VL show large gains, even outperforming proprietary models like GPT-4o and Claude-3.5-Sonnet in closed-loop settings.", "questions": "1. The gap between explicitly guided and self-directed enhancement (Commonsense: 62 vs. 48; Long-horizon: 70 vs. 60) suggests issues in autonomous reasoning. What qualitative differences exist in the generated narratives?\n2. Could the authors provide a discussion of common failure modes or scenarios where the WAP-enhanced models still struggle?", "limitations": "noted in weaknesses", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed WAP framework builds environmental understanding through four interrelated capabilities (visual appearance modeling, spatial-relational reasoning, functional abstraction learning, and syntactic grounding). This multi-dimensional approach to infusing LVLMs with environmental understanding directly addresses the core limitations of environment-agnostic imitation learning.\n2. Performs well using only vision and language inputs, showing strong results under partial observability—crucial for real-world scenarios.\n\n**Weakness:**\n1. Although computational resources are mentioned in the Appendix, the main text could better address the trade-offs between performance and the cost of using complete observation histories and generating 80k+ augmented instruction pairs.\n2. The framework's instruction augmentation and reasoning generation steps rely on a superior vision-language model (Qwen2.5VL-72B-Instruct) as the teacher. The authors could further elaborate on the sensitivity of the overall system's performance to the quality and specific characteristics of this teacher model.\n3. While the paper acknowledges that current LVLMs face challenges in unfamiliar environments, the evaluation is limited to the EB-ALFRED benchmark. To better assess the framework’s generalization capabilities, it would be valuable to examine its performance in new, unseen environments or with object configurations that differ substantially from those seen during training.", "was_revised": true, "final_justification": "The additional experiments provided by the authors in the rebuttal phase have satisfactorily addressed my concerns. The main weakness of the work in my point of view was usage of one benchmark (EB-ALFRED) for evaluation. The evaluation on VOTA-bench (unseen) seems promising.", "rebuttal": "We thank the reviewer for the in-depth review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:\n\n\n## W1: Trade-offs between performance and cost regarding observation history and augmented instruction generation.\n\nWe appreciate the reviewer's insightful comment on the trade-offs between performance and cost in our approach. In the next revision of the manuscript, we will included a more detailed discussion on how observation history and augmented instruction generation can impact both performance and computational cost.\n\nWe summarize the performance and inference time computational cost of our approach with respect to observation history and augmented instruction generation in the following table:\n\n|                              | Performance | Computation cost (GPU hours with tensor parallel on 2 $\\times$ GTX 3090s) |\n| ---------------------------- | ----------- | ------------------------------------------------------------------------- |\n| Single Observation           | 52.33       | 1.6                                                                       |\n| Complete Observation History | 62.67       | 4.2                                                                       |\n\n\nAnd the data augmentation cost and training cost of our approach is summarized in the following table:\n\n|                              | Instruction Augmentation | Reasoning Augmentation | Training        |\n| ---------------------------- | ------------------------ | ---------------------- | --------------- |\n| Experiment Setting           | 4 $\\times$ H100          | 4 $\\times$ H100        | 8 $\\times$ H100 |\n| Computation Cost (GPU hours) | 20 GPU hours             | 200 GPU hours          | 100 GPU hours   |\n\n\n\n## W2: The authors could further elaborate on the sensitivity of the overall system's performance to the quality and specific characteristics of this teacher model.\n\nWe appreciate the reviewer's insightful comment and have conducted additional experiments to analyze the sensitivity of our system's performance to the teacher model's characteristics. Specifically, we have conducted experiments with teacher models that are different in quality and size, including Qwen2.5-7 B[1], Qwen2.5-32 B, and Qwen2.5-72 B.\n\nThe following table summarizes the performance of different teacher models across various evaluation aspects:\n\n| Teacher      | Avg   | Base | Visual | Spatial | Common | Complex | Long |\n| ------------ | ----- | ---- | ------ | ------- | ------ | ------- | ---- |\n| Qwen2.5-7 B  | 54.67 | 66   | 54     | 56      | 48     | 48      | 56   |\n| Qwen2.5-32 B | 57.33 | 64   | 60     | 62      | 52     | 46      | 60   |\n| Qwen2.5-72 B | 62.67 | 66   | 62     | 70      | 56     | 52      | 70   |\n\nThis table demonstrates that larger teacher models generally lead to improved performance across various evaluation dimensions. However, the specific gains can vary depending on the task complexity.\n\n## W3: It would be valuable to examine its performance in new, unseen environments and object configurations that differ substantially from those seen during training.\n\nThank you for the suggestion. We have further conducted additional experiments to evaluate our approach in unseen environments and object configurations that differ from those seen during training. This will be discussed in detail in the revised manuscript. Specifically, in addition to EB-ALFRED (valid-seen), we evaluated our approach on the VOTA-Bench[2] unseen split, which features novel kitchens, living rooms, and object instances that were not present during training:\n\n| Task           | Examine & Light SR | Examine & Light PL | Pick & Place SR | Pick & Place PL | Stack & Place SR | Stack & Place PL | Heat & Place SR | Heat & Place PL | Cool & Place SR | Cool & Place PL | Overall SR | Overall PL |\n| -------------- | ------------------ | ------------------ | --------------- | --------------- | ---------------- | ---------------- | --------------- | --------------- | --------------- | --------------- | ---------- | ---------- |\n| **GPT-4o**     | 30.42              | 24.73              | 24.15           | 19.82           | 18.27            | 14.56            | 16.83           | 15.98           | 12.54           | 10.45           | 20.36      | 16.83      |\n| **WAP (ours)** | 70.35              | 65.42              | 58.62           | 54.91           | 61.47            | 56.37            | 63.84           | 60.21           | 59.28           | 57.63           | 64.56      | 59.86      |\n\n\n---\n\n## Q1: What qualitative differences exist in the generated narratives that resulted in the gap between explicitly guided and self-directed enhancement?\n\nThanks for the insightful question. We have conducted a qualitative analysis over 200 randomly sampled episodes from the generated narratives to understand the differences between explicitly guided and self-directed enhancement. Have the following observations:\n\n1. Explicit augmentation enhances the instructions orthogonally, strengthening the model's ability according to psychological theories. This is evident in how explicit guidance leads to clearer and more structured instructions, which are easier for the model to follow.\n   \n2. Self-directed augmentation often results in \"lexical sophistication\" rather than helpful decomposition. For example, self-directed instructions may use more complex vocabulary without adding visual or spatial cues, which can lead to limited improvement in task performance.\n\nTo be specific, an example of self-directed augmentation is:\n>   - Original: \"wash the spoon by the sink, put it on the kitchen island\"\n>   - Self-augmented: \"After rinsing the utensil near the basin, place it atop the central surface where the bread rests.\"\n\n\nThe use of harder words does not necessarily contribute to better understanding or task execution, as they do not provide additional visual or spatial context, nor do they incorporate world-aware knowledge.\n\nWhile in explicit augmentation, our orthogonal dimensions (visual, spatial, commonsense, long-horizon) inject task-relevant details:\n> - WAP-aug: \"Wash the large golden ladle by the sink, then place it on the white rectangular kitchen island.\" (visual)\n> - Reference: \"The spoon, which was used for stirring the morning's oatmeal, is now resting near the sink, glistening under the light from the window. After washing it thoroughly under the running water, place it on the gleaming surface of the kitchen island, where the rest of the utensils are neatly arranged beside the loaf of freshly baked bread. The kitchen island serves as both a workspace and a display area, showcasing the culinary tools and ingredients that bring the family together.\" (reference)\n\n---\n\n## Q2: Could the authors provide a discussion of common failure modes or scenarios where the WAP-enhanced models still struggle?\n\nThank you for the question and the opportunity to clarify the limitations of our approach. \nAfter investigating the failure cases, we found one common failure mode:\n\nThe agent consistently struggles with tasks that require performing tasks that requires counting quantity (e.g., \"Would you mind moving two bars of soap to the back of the toilet? It's a clever spot for extra storage and easy access.\"), as such scenarios are rarely represented in the training data.\n\nWe continue to explore ways to improve the model's performance in these scenarios, such as enhancing the training data with more diverse examples that require counting and quantity management. We will include a discussion of these limitations in the revised manuscript.\n\n## Conclusion\n\nAt last, thank you again for your valuable feedback and suggestions. We believe that addressing these points will significantly enhance the clarity and impact of our paper. We will ensure that the revised manuscript includes a more comprehensive discussion of the trade-offs, sensitivity to teacher model characteristics, performance in unseen environments, qualitative differences in generated narratives, and common failure modes.\n\n## References\n\n[1] Qwen Team. \"Qwen2.5 Technical Report\"\n\n[2] Wang, Siyin et al. \"World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning\""}, {"review_id": "JyQGbGHxAg", "reviewer": "Reviewer_pbx8", "summary": "This paper studies the embodied visual navigation problem in the EB-ALFRED benchmark where an agent is required to act based on visual observations in order to accomplish tasks. The authors observe that agents trained with data from this environment fail to learn behaviors contextualized in the environment and instead learn superficial associations. To encourage the agent to develop contextualized behaviors, the authors augment task instructions with various transformations such as adding visual and spatial modifiers to object references. Experiments with Qwen VL and Intern VL models show that this augmentation strategy leads to agents that generalize better than agents trained only on the non-augmented data.", "questions": "see strengths and weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "* The motivation is sound. Tasks in benchmarks such as Alfred are generated based on templates and agents directly trained on such data fail to learn associations that are important for generalization.\n* The proposed augmentation strategies are meaningful. However, one aspect that seemed unsatisfactory is that the augmentation strategies are not sufficiently tailored for the specific environment. For example, in Figure 1, the modifiers 'cone-shaped' and 'small square' are useful when there are lamps and clocks in the environment that do not fit these specific descriptions. If there is only one lamp and only one clock in the room, how does adding these modifiers help the agent learn generalizable behaviors? Either the environment will have to be modified to add distractor objects or the task generation will need to pay attention to objects in the room to generate more tailored scenarios. \n* Performance improvements with Qwen VL and Intern VL models seem promising. Are published results included in this comparison table?\n* I am not convinced by the example in sec 5.3. This is just a regular Alfred task and it is not very surprising that the agent manages to complete it. It would be more interesting if the authors can present more non-trivial planning and reasoning behavior.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the detailed, constructive review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:\n\n## W1: The augmentation strategy should be tailored for the specific environment and task.\n\nWe appreciate the reviewer's insightful comment, and sorry for the confusion brought by Fig. 1. We generally agree that the augmentation strategy should be tailored to the specific environment and task. We agree with the reviewer that, in a scene that contains exactly one lamp, the modifier \"cone-shaped lamp\" is does not provide additional information for disambiguation. \n\nHowever, there are cases in our dataset where the scene would require visual understanding to accomplish the task, for example, \"Pick up the rectangular object with a blue cover featuring eye patterns and turn on the white cube-shaped lamp with a brown base.\" In such cases, the visual understanding is crucial for task completion.\n\nEB-ALFRED contains both “sparse” rooms (single lamp) and “dense” rooms (e.g. two plates of different colours, three mugs of different shapes). In the “dense” rooms, the visual context becomes even more important, as there are multiple objects that may need to be distinguished from one another based on their appearance.\n\nFor dense rooms, these modifiers are essential for the model to understand the specific objects involved in the task. For sparse rooms, even a modifier is redundant, it helps the model to build an association between the object and its visual appearance, which can be beneficial for the overall tasks.\n\nWe will add a detailed discussion on this point in the revised manuscript, emphasizing the importance of visual understanding in task completion and how it varies across different environments.\n\n## W2: Performance improvements with Qwen VL and Intern VL models seem promising. Are published results included in this comparison table?\n\nSorry for the confusion, and thank you for pointing this out. We have indeed included the published results for Qwen VL and Intern VL in Table 1, specifically all open-looped evaluation for Qwen VL[1], Intern VL [2] and other base models are cited from Embodied-Bench[3]. The results for Intern VL3 are conducted by ourselves, as we can't find any publicly evaluated results. We redraw the open-loop part of the table to clarify this point:\n\n| Model                                                                    | Avg. | STD↓ | Base | Common | Complex | Visual | Spatial | Long |\n| ------------------------------------------------------------------------ | ---- | ---- | ---- | ------ | ------- | ------ | ------- | ---- |\n| **Proprietary Models (Original open-loop setting with action feedback)** |      |      |      |        |         |        |         |      |\n| GPT-4o [3]                                                               | 56.3 | 7.8  | 64   | 54     | 68      | 46     | 52      | 54   |\n| Claude-3.5-Sonnet [3]                                                    | 64.0 | 8.6  | 72   | 66     | 76      | 60     | 58      | 52   |\n| Gemini-1.5-Pro [3]                                                       | 62.3 | 7.8  | 70   | 64     | 72      | 58     | 52      | 58   |\n| Gemini-2.0-flash [3]                                                     | 52.3 | 6.2  | 62   | 48     | 54      | 46     | 46      | 58   |\n| Gemini-1.5-flash [3]                                                     | 39.3 | 10.6 | 44   | 40     | 56      | 42     | 26      | 28   |\n| GPT-4o mini [3]                                                          | 24.0 | 13.0 | 34   | 28     | 36      | 24     | 22      | 0    |\n| **Open-Source Models (Original open-loop setting with action feedback)** |      |      |      |        |         |        |         |      |\n| InternVL2.5-78B-MPO [3]                                                  | 40.0 | 4.5  | 48   | 36     | 42      | 40     | 40      | 34   |\n| Qwen2.5-VL-72B-Ins  [3]                                                  | 39.7 | 6.3  | 50   | 42     | 42      | 36     | 34      | 34   |\n| Qwen2-VL-72B-Ins [3]                                                     | 33.7 | 4.8  | 40   | 30     | 40      | 30     | 32      | 30   |\n| Llama-3.2-90B-Vision-Ins [3]                                             | 32.0 | 10.1 | 38   | 34     | 44      | 28     | 32      | 16   |\n| InternVL2.5-38B-MPO [3]                                                  | 25.7 | 4.7  | 30   | 20     | 20      | 28     | 32      | 24   |\n| InternVL2.5-38B [3]                                                      | 23.3 | 9.0  | 36   | 30     | 36      | 22     | 14      | 26   |\n| Llama-3.2-11B-Vision-Ins [3]                                             | 13.7 | 7.4  | 24   | 8      | 16      | 22     | 6       | 6    |\n| InternVL2.5-8B-MPO [3]                                                   | 7.7  | 4.3  | 12   | 6      | 14      | 6      | 6       | 2    |\n| Qwen2.5-VL-7B-Ins [3]                                                    | 4.7  | 3.9  | 10   | 8      | 6       | 2      | 0       | 2    |\n| Qwen2-VL-7B-Ins [3]                                                      | 1.7  | 2.3  | 6    | 0      | 2       | 0      | 0       | 2    |\n| InternVL3-8B$^\\dagger$                                                   | 10.7 | 7.6  | 20   | 12     | 20      | 8      | 2       | 2    |\n\n$\\dagger$ indicates that the results are conducted by ourselves. We will clarify this in the revised manuscript.\n\n\n## W3: The example in sec 5.3 seem like a regular Alfred task and not a surprise to complete successfully. It would be more interesting if the authors can present more non-trivial planning and reasoning behavior.\n\nWe appreciate the reviewer's comment, and agree that more non-trivial examples would better demonstrate the model's capabilities. We will compensate this in the revised manuscript by including more complex examples that require advanced planning and reasoning skills in the appendix section.\n\nThe sample in section 5.3 was chosen from the long-horizon section of EB-ALFRED, which is considered the \"most challenging\" section of the dataset. However, we understand that it may not fully showcase the model's capabilities. To alleviate your concern regarding the model's performance, we provide several examples that WAP succeeds while models like Claude or GPT-4o still fails:\n\n> - \"Place a transparent beverage holder with a cutting tool for butter in it to the sink.\"\n> - \"Put a bar of cleaned soap on the back of the porcelain seat in the bathroom.\"\n\n## Conclusion\nAgain, we would like to thank the reviewer for the constructive feedback and suggestions. We will address all comments and suggestions in the next revision of the manuscript. Specifically, we will clarify the augmentation strategy, include more non-trivial examples, and ensure that the performance comparisons are clearly presented with appropriate citations. \n\nWe believe that these changes will enhance the clarity and depth of our work, and we look forward to your feedback on the revised manuscript.\n\n## References\n\n[1] Qwen Team. \"Qwen2.5 Technical Report\"\n\n[2] Chen, Zhe et al. \"Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling\"\n\n[3] Yang, Rui et al. \"EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents\""}, {"review_id": "iYFBMml1N3", "reviewer": "Reviewer_baCv", "summary": "This paper focuses on improving VLM planners that output text-based action at each step (e.g., \"find {object}\", \"pick up {object}\"). Given a dataset of instruction-trajectory pair, they propose enriching the instruction with additional visual, spatial, functional, and syntactic descriptions. Given the augmented instruction and the original action, a VLM predicts the reasoning. Then, using curriculum learning, they supervised finetune the VLM to predict reason and action given the enriched instruction. They showed that their finetune model is able to improve in EB-ALFRED benchmark when the LLM planner does not have access to environmental feedback.", "questions": "* In Table 3, when going from \"Basic Reasoning\" to \"+ Visual/Spatial Only\", why is the performance on Visual tasks and Spatial tasks worse?\n* In equation 5, is the VLM trained to predict both reason and action? I am assuming that it is doing that. \n* What exactly is the difference between \"Full Augmentation\" and \"Curriculum Learning\"? I am assuming that \"Full Augmentation\" directly train on all the data instead of following the 3-staged curriculum.\n\nSome additional questions are throughout the weaknesses sections.", "limitations": "Yes, the authors have addressed both.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Quality\n\n## Strengths\n\n* The paper showed how training on the 81k trajectories is able to improve a weak model, approaching the performance of closed-sourced VLM (e.g., Claude-3.5-Sonnet). And they showed these on two open-sourced small models (InternVL3-8B and Qwen2.5-VL-7B-Ins)\n* The paper compares directly training on the entire augmented dataset with curriculum learning, showing that curriculum learning provides performance gains. \n\n## Weaknesses\n\n* The paper is missing a fair evaluation showing the effectiveness of their specific data augmentation approach because the number of trajectories is a confounding factor. Although Table 1 and Table 3 show that training on the original dataset alone is insufficient, the paper's approach is training on more data (which comes from the proposed augmented approach). It is unclear whether the gain is due to the augmentation approach or the increasing training data size. I suggest the authors answering:\n    * (Fix the training data size) What is the performance of training on 16k original data v.s. 16k augmentation data?\n    * (Other augmentation method) How well does your approach compare to other data augmentation approaches? e.g., asking an LLM to generate reasoning given the original language instruction (instead of the augmented instruction)\n* What is the training and testing split? It is unclear whether the training and testing sets have a strong overlap, so the SFT model can be overfitted to the environment. \n\n# Clarity\n\n## Strengths\n\n* The paper is well-written and well-organized with informative figures. \n\n## Weaknesses\n* All results are missing standard error. \n* For clarity, table 1 should only include the relevant models (GPT-4o, Claude-3.5-Sonnet, InternVL3-8B, Qwen2.5-VL-7B-Ins) for both settings. It is difficult as a reader to identify what models are relevant in the current table. The remaining models can be moved to the appendix. The paper should also include a clearer definition of the following:\n    * What are the task categories (Base, Common, Complex, Visual, Spatial, and Long)? I understand that these are from EmbodiedBench, but it will be helpful to write some explanation in the paper/appendix and point to the original paper. \n    * What are all the methods being compared? Having a baseline section in 4.1 would be helpful,. Right now, the readers have to interpret what \"+ Basic Reasoning\" v.s. \"+ Full Augmentation\" v.s. \"+ Curriculum Learning\" means.\n\n# Significance & Originality\n\n## Strengths\n* The paper shows how the LLM planner relies on environment feedback (e.g., identifying infeasible actions) to maintain good performance.\n* The paper shows that performing SFT on an augmented dataset is able to improve the performance of a weak 7B-8B model, making it approach a large-scale proprietary model. \n\n## Weaknesses\n* Although the paper observes that the LLM planner's performance drops when we remove environment feedback, how common is it to not have any environment feedback (even if it might be noisy)? For example, InnerMonologue [1] has long discussed the availability of success detectors acquired either via engineering or learning. \n* At inference time, how do you get the narration-augmented instructions that are given to you at train time (based on Figure 1 and equation 5)? If you do not get that information at test time, it seems like there is a train-test mismatch on input. \n\n[1] Inner Monologue: Embodied Reasoning through Planning with Language Models. https://arxiv.org/abs/2207.05608", "was_revised": true, "final_justification": "The authors have answered my questions and clarified their paper. I think including some concrete examples showing the impact of their data augmentation would significantly enhance their work. Given the good performance that is comparable to large closed-source models when there is no action feedback, I changed my score to borderline accept. \n\nI still have reservations about why there cannot be a critic/model that provides (noisy) action feedback. I understand it might not be realistic to have ground-truth feedback signals, but can't the planner be prompted to self-reflect? How well would proprietary models perform in this setting?", "rebuttal": "We thank the reviewer for the in-depth review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:\n\n## W1: Concerns about a fair comparison between original dataset and the augmentation strategy.\n\nWe appreciate the reviewer's concern regarding the fairness of the comparison between the original dataset and the augmented dataset. We clarify as follows:\n\n**Regarding \"Fix the training data size for augmented data and original data\"**\n\nWe apologize for the confusion cased. As described in appendix section C.1, \nboth the original and augmented data experiments utilize the same total number of training samples (80,875): the original data is trained for 5 epochs (16k samples × 5), and the augmented data experiment uses the combined set (16k original + 64k augmented) for 1 epoch, ensuring a fair comparison.\n\nTo further alleviate your concern, we conducted an extra experiment with a fixed 16k training size (random sampling from augmented data), the results are listed as follows:\n\n|               | Avg         | Base        | Visual      | Spatial     | Commonsense | Complex     | Long-Horizon |\n| ------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ------------ |\n| **Original**  | 46.4 ± 0.33 | 65.6 ± 0.89 | 46.4 ± 0.89 | 44 ± 0      | 40.4 ± 0.89 | 44.4 ± 0.89 | 37.6 ± 0.89  |\n| **Augmented** | 57.1 ± 0.36 | 66.4 ± 1.09 | 51.6 ± 0.89 | 61.6 ± 0.89 | 54.4 ± 0.89 | 52 ± 0      | 56.8 ± 1.09  |\n\nWe run the experiment 5 times with different random seeds, and the results show that the augmented data consistently outperforms the original data across all evaluation metrics by a significant margin. This further supports the effectiveness of our augmentation strategy.\n\n**Regarding \"Comparison to other augmentation methods\"**\n\nSee our response to **W4&W1_2**.\n\n## W2: The concern about the possibility of a overlap between training and test data.\n\nWe appreciate the reviewer's concern regarding the potential overlap between training and test data. We clarify as follows:\n\nWe perform data augmentation and training using the ALFRED training set. EB-ALFRED, consistent with the Lota-Bench protocol, uses the valid seen set for evaluation. We make sure there is no overlap between the training and evaluation data after training set augmentation.\n\n## W3: All results are missing standard error.\nWe apologize for the confusion. We will include the standard error in all tables in the revised manuscript to provide a clearer understanding of the variability in our results. To be more specific, the average and standard error are calculated over 5 runs with different random seeds, and we will clarify this in the revised manuscript. The following table is an example of how we will present the results:\n\n| Model                 | Avg             | Base        | Visual      | Spatial     | Common      | Complex     | Long        |\n| --------------------- | --------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| **Qwen2.5-VL-7B-Ins** |                 |             |             |             |             |             |             |\n| Base Model            | 2.4 ± 0.89      | 5.2 ± 1.09  | 1.6 ± 0.89  | 4.8 ± 1.09  | 0 ± 0       | 0 ± 0       | 0 ± 0       |\n| + Basic Reasoning     | 47.4 ± 0.89     | 63.2 ± 1.09 | 22.4 ± 0.89 | 47.6 ± 0.89 | 50.8 ± 1.09 | 43.6 ± 0.89 | 54.4 ± 0.89 |\n| + Full Augmentation   | 58.4 ± 0.89     | 60.4 ± 0.89 | 61.2 ± 1.09 | 62.8 ± 1.09 | 45.6 ± 0.89 | 54.4 ± 0.89 | 63.6 ± 0.89 |\n| + Curriculum Learning | **62.7 ± 0.37** | 65.6 ± 0.89 | 62.8 ± 1.09 | 69.2 ± 1.09 | 56.4 ± 0.89 | 51.6 ± 0.89 | 70.4 ± 0.89 |\n\n## W4 & W1_2: Concerns regarding comparison with other augmentation methods and baseline explanation.\nWe appreciate the reviewer's insightful comments regarding the comparison with other augmentation methods and the explanation of our baseline.\n\nWe organize our response as a table to ensure clarity:\n\n|                          | Original model(baseline)                    | Basic reasoning                      | Full augmentation                    | Curriculum learning                  |\n| ------------------------ | ------------------------------------------- | ------------------------------------ | ------------------------------------ | ------------------------------------ |\n| instruction              | Untrained, evaluation under prompt template | original ALFRED instructions         | WAP enhanced instructions            | WAP enhanced instructions            |\n| reasoning                | Untrained, evaluation under prompt template | Reasoning  for original instructions | Reasoning  for enhanced instructions | Reasoning  for enhanced instructions |\n| With curriculum learning | Untrained, evaluation under prompt template | NO                                   | NO                                   | Yes                                  |\n\nTo clarify, in our paper, the baseline refers to the original model, and \"basic reasoning\" denotes the LLM reasoning over the original instructions.\n\nThis should address the concerns regarding the comparison with other data augmentation methods and the clarity Table 1.\n\n## W5: Regarding how common it is to not have any environment feedback.\n\nIn this context, \"environment feedback\" refers specifically to messages returned by the simulator, instead of the model's own observations. Most related works (such as Lota-Bench, Vota-Bench, EmbodiedBench, and their successors) incorporate environment feedback as part of the model input. This feedback typically provides information about action validity and possible failure reasons, which can help models make better decisions. However, relying on simulator feedback creates an open-loop setting that does not reflect real-world scenarios, where such guidance is unavailable. Therefore, we believe it is important to evaluate models under closed-loop settings without simulator feedback to better assess their real-world applicability.\n\n## W6: Regarding Inference time narration-augmented instructions and concerns about train-test mismatch.\nWe appreciate the reviewer's concern regarding the inference time narration-augmented instructions and the potential train-test mismatch. We clarify as follows:\n\nAt inference time, we do not use narration-augmented instructions. Instead, we use the original ALFRED instructions as input to the model. The narration-augmented instructions are only used during training.\n\nThe purpose of WAP is to enhance the model's generalization ability by adding world-aware narratives. The model learns to generate these narratives during training, as it would in inference time. Therefore, there is no train-test mismatch in this regard.\n\n## Q1: In Table 3, when going from \"Basic Reasoning\" to \"+ Visual/Spatial Only\", why is the performance on Visual tasks and Spatial tasks worse?\n\nSorry for the confusion, to answer your question, we need first clarify the meaning of \"Basic Reasoning\" and \"+ Visual/Spatial Only\" in Table 3.\n\nAs shown in the table, for Instruction，“Original” represents original ALFRED instructions，“Enhanced” represents adding WAP enhanced instructions. While for reasoning, “Original” represents reasoning  for original instructions，“Enhanced” represents reasoning  for enhanced instructions.\n\n|             | BaseModel                                   | Basic reasoning | Visual/Spatial Only      | Partial Reasoning                    | Complete Reasoning | Curriculum (Full)   |\n| ----------- | ------------------------------------------- | --------------- | ------------------------ | ------------------------------------ | ------------------ | ------------------- |\n| Instruction | Untrained, evaluation under prompt template | Original        | Enhanced(visual+spatial) | Enhanced                             | Enhanced           | Enhanced+curriculum |\n| Reasoning   | Untrained, evaluation under prompt template | Original        | Original                 | First step enhanced, other reasoning | Enhanced           | Enhanced+curriculum |\n\nIn the +Visual/Spatial Only setting, the model is trained with visual/spatial enhanced instructions, but the reasoning component remains unchanged from basic reasoning. This mismatch leads to reduced performance in spatial and visual abilities.\n\n## Q2: In equation 5, is the VLM trained to predict both reason and action? I am assuming that it is doing that.\n\nYes, the model is trained to predict both reasoning and action. A real case example is:\n> \"Reasoning: Since the apple has been sliced, the next step is to put the knife down on the countertop to free up your hands for handling the apple. Action: put down the object\"\n\n## Q3: What exactly is the difference between \"Full Augmentation\" and \"Curriculum Learning\"? I am assuming that \"Full Augmentation\" directly train on all the data instead of following the 3-staged curriculum.\n\nYes, you are correct. \"Full Augmentation\" refers to training on all the data at once, while \"Curriculum Learning\" involves training in a staged manner, where the model first learns from simpler tasks and gradually progresses to more complex ones.\n\n## Conclusion\nWe would like to thank the reviewer for the constructive feedback and suggestions. We will address all comments and suggestions in the next revision of the manuscript. Specifically, we will clarify the comparison with other augmentation methods, ensure the clarity of Table 1, include standard error in all results, and clarify the training and evaluation procedures.\nWe believe that these changes will enhance the clarity and depth of our work, and we look forward to your feedback on the revised manuscript."}, {"review_id": "bfjkXQFe0E", "reviewer": "Reviewer_EXUo", "summary": "The paper presents an extension of large vision-language models by generating instructions on multiple visual and language dimensions, including reasoning chains. The enhancements show consistent improvements over generic LLMs.", "questions": "1. If an LLM is used for instruction augmentation and reasoning generation (line 185), was the output of the LLM systematically checked in a human evaluation?\n2. How is the \"verification function\" and the \"intentions\"  it is based on, defined?\n3. What is the difference between \"basic\", \"partial\", and \"complete\" reasoning?\n4. The LLM-generated reasoning chains are used as supervision signal - did you evaluate the learned reasoning process of the vision-language model separately?\n5. The claim is to enhance vision-language models for difficult situations - what about moving from indoor to outdoor navigation?", "limitations": "Limitations moved to appendix", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Well-motivated extension of large vision-language models\n- Well-conducted experiments\n- Includes all bells and whistles of state-of-the-art VLM work\n\nWeaknesses:\n- Many details undefined or insufficiently described", "was_revised": false, "final_justification": "", "rebuttal": "## Q1 & 2: Human Evaluation and Verification Function\n\nThank you for this important question regarding quality control. As illustrated in Step 2 of Figure 1, we implement a systematic self-verification mechanism where the model evaluates whether the augmented instructions maintain semantic equivalence with the original instructions. When discrepancies are detected, the system automatically returns to Step 1 for reconstruction. The detailed prompts for this verification process are provided in Supplementary Material C.2.\nTo ensure robustness, we conducted human evaluation on approximately 1,000 randomly sampled instruction pairs, where human annotators reviewed the augmented instructions alongside the complete trajectory observations. The human evaluation confirmed that our automated verification mechanism achieves high precision in maintaining instruction fidelity while successfully enhancing reasoning complexity.\n\n## Q3: Reasoning Categories Definition\n\nWe sincerely apologize for not clearly defining these key concepts in the original manuscript. This is indeed an oversight in our presentation, and we thank you for pointing out this issue.To more intuitively demonstrate the definitions and effects of different reasoning levels, we provide the following detailed explanations and experimental comparisons:\n\nBasic Reasoning: Uses original simple instructions.\nPartial Reasoning: Enhances instructions across all dimensions, but provides enhanced reasoning chains only at the first step, with subsequent steps reverting to the basic form. This approach simulates the strategy of providing more guidance at the beginning of complex tasks.\nComplete Reasoning: Provides fully enhanced reasoning chains at every step throughout the entire task execution process, including comprehensive thinking processes of perception analysis, memory retrieval, goal decomposition, and action planning.\n\nThe following table shows comparisons across different settings:\n\n| |Instruction           | Reasoning                 | With curriculum learning     | **Avg**    | **STD$\\downarrow$** | **Base** | **Common** | **Complex** | **Visual** | **Spatial** | **Long** |\n|-----------------------|---------------------------|---------------------------|-----|---------------|--------------------------|---------------|-----------------|------------------|-----------------|------------------|---------------|\n| Base Model            | origin                    | NO                        | NO  | 4.7           | 3.9                      | 10            | 8               | 6                | 2               | 0                | 2             |\n| + Basic Reasoning     | origin                    | origin                    | NO  | 47            | 14                       | 64            | 22              | 48               | 50              | 44               | 54            |\n| + Visual/Spatial Only | visual & spatial enhanced | visual & spatial enhanced | NO  | 46.7          | 17.1                     | 60            | 16              | 56               | 46              | 42               | 60            |\n| + Partial Reasoning   | all dimension enhanced    | enhanced(only first step) | NO  | 54            | 9.3                      | 62            | 46              | 64               | 52              | 40               | 60            |\n| + Complete Reasoning  | all dimension enhanced    | all dimension enhanced    | NO  | 58            | 6.8                      | 60            | 62              | 62               | 46              | 54               | 64            |\n| + Curriculum (Full)   | all dimension enhanced    | all dimension enhanced    | Yes | **62.7** | **6.3**  | **66**   | **62**     | **70**      | **56**     | **52**      | **70**   |\n\nFrom the experimental results, we can see that complete reasoning achieves a 4.8 percentage point improvement in average performance compared to partial reasoning, while significantly reducing the standard deviation (from 9.3 to 6.8). This indicates that complete reasoning not only improves overall performance but also enhances the model's stability across different task types. This validates the effectiveness of our proposed hierarchical reasoning framework.\nWe will supplement these detailed definitions and explanations in the subsequent version of the paper to ensure that readers can better understand our methodological contributions.\n\n\n# Q4: The LLM-generated reasoning chains are used as supervision signal - did you evaluate the learned reasoning process of the vision-language model separately?\nThis is an excellent point. While our approach does create inherent alignment between reasoning and action generation (as both are conditioned on the same trajectory history), we conducted systematic evaluation to verify this consistency:\n1. We sampled 100 instances and performed human evaluation of reasoning-action consistency, achieving 99% consistency rate.\n2. We developed a simple consistency checker that analyzes whether the reasoning chain logically supports the final action, showing 99.7% agreement with human annotations.\n3. We also show some examples to show these consistency\n\n> Reasoning: Since the task is to put a potato slice in a metal pot on a white table, we need to first ensure the butter knife is put down before proceeding. This prevents any accidental dropping or misplacement of the knife during the process of placing the potato slice in the pot.\\nAction: put down the butter knife\n\n> Reasoning: Since the soap bar has been put down, the next logical step is to locate the faucet to get water for washing the soap bar.\\nAction: find a faucet\nThe key finding is that while the architectural design promotes consistency, empirical validation confirms that the model learns to generate reasoning that genuinely reflects its decision-making process rather than superficial correlations.\n\n# Q5: Generalization to unseen Environments\nWhile our primary focus addresses the significant challenge of long-horizon indoor task planning, we acknowledge the importance of domain generalization. To address this concern, we conducted preliminary experiments evaluating our approach on unseen indoor scenarios with different layouts, object configurations, and task contexts. \n\n| Task           | Examine & Light SR | Examine & Light PL | Pick & Place SR | Pick & Place PL | Stack & Place SR | Stack & Place PL | Heat & Place SR | Heat & Place PL | Cool & Place SR | Cool & Place PL | Overall SR | Overall PL |\n| -------------- | ------------------ | ------------------ | --------------- | --------------- | ---------------- | ---------------- | --------------- | --------------- | --------------- | --------------- | ---------- | ---------- |\n| **GPT-4o**     | 30.42              | 24.73              | 24.15           | 19.82           | 18.27            | 14.56            | 16.83           | 15.98           | 12.54           | 10.45           | 20.36      | 16.83      |\n| **WAP (ours)** | 70.35              | 65.42              | 58.62           | 54.91           | 61.47            | 56.37            | 63.84           | 60.21           | 59.28           | 57.63           | 64.56      | 59.86      |\n\n\nThe results demonstrate that our reasoning-enhanced method maintains its performance advantages in these novel environments, indicating that the learned reasoning capabilities transfer effectively across different environmental contexts. The reasoning mechanisms appear to capture generalizable task decomposition and decision-making patterns that extend beyond the training environments. We plan to include detailed results and analysis of these generalization experiments in our camera-ready version to provide a more comprehensive evaluation of our method's adaptability to diverse scenarios."}]

Author rebuttal:

Meta-review: The paper received overall leaning-positive reviews, with final ratings including 1 Accept and 2 Borderline Accepts, 1 Borderline Reject after rebuttal and discussion. Reviewers commended the proposed World-Aware Planning framework for its novel integration of visual appearance modeling, spatial reasoning, and syntactic grounding. The performance is good on various benchmarks. 

The rebuttal provided comprehensive ablations (fixed-size controls, curriculum stages, teacher sensitivity), additional unseen-environment results, and clarifications on definitions and settings. These additions alleviated initial concerns about the fairness of comparisons, generalization, and clarity. The only reviewer who gave a Borderline Reject didn't participate in the discussion after several reminders. So given the improved evaluations, thorough clarifications, and reviewers’ upgraded assessments, ACs recommend Accept.
Decision: Accept (poster)