Egocentric Vision Language Planning

## Abstract

Abstract We explore leveraging large multi-modal models (LMMs) and text2image models to build a more general embodied agent. LMMs excel in planning long-horizon tasks over symbolic abstractions but struggle with grounding in the physical world, often failing to accurately identify object positions in images. A bridge is needed to connect LMMs to the physical world. The paper proposes a novel approach, egocentric vision language planning (EgoPlan), to handle long-horizon tasks from an egocentric perspective in varying household scenarios. This model leverages a diffusion model to simulate the fundamental dynamics between states and actions, integrating techniques like style transfer and optical flow to enhance generalization across different environmental dynamics. The LMM serves as a planner, breaking down instructions into sub-goals and selecting actions based on their alignment with these sub-goals, thus enabling more generalized and effective decision-making. Experiments show that EgoPlan improves long-horizon task success rates from the egocentric view compared to baselines across household scenarios.

## 1 Introduction

The advent of large language models (LLMs) [, ] and large multi-modal models (LMMs) [, , , ] has revolutionized the field of artificial intelligence. Their strong reasoning [, ] and powerful generalization capabilities allow them to be directly applied in various scenarios. In the next step toward artificial general intelligence (AGI), researchers are considering enabling large models (LMs), especially LMMs, to break through the world expressed by text and images to interact with the physical world. They aim to build a general embodied agent that intelligently interacts with the physical world.

LMMs have demonstrated an impressive capability of planning for long-horizon tasks over symbolic abstraction in the physical world []. However, there’s still a piece of the puzzle missing. They have struggled to ground the text world with the physical world. For example, GPT-4V often fails to accurately identify objects’ positions in images. LMMs seem to know what to do next but do not understand how the world works. A world model (dynamics model) is hence needed to connect the LMMs to the physical world. There are two potential solutions. One is to implicitly integrate environmental dynamics into the LMMs, that is, fine-tuning the LMMs based on a vast amount of state-action sequences, such as PaLM-E [] and RT-2 []. However, directly training large models requires extensive data and computational resources. The other is to explicitly introduce a pre-trained world model, *e.g., *text2image models [, ], which can be used by LMMs as an auxiliary tool. Our work explores the second path. We try to answer the question: how do we leverage the LMMs and text2image model to build a more general embodied agent?

Some works already train text2image/video models as world models for decision-making. However, there still exist several limitations. First, their task scenarios often involve object manipulation, a fully observable setting. This is uncommon in real-world scenarios, and their methods seem to struggle to adapt to other practical scenes. For example, SuSIE [] and VLP [] require generating images several steps ahead, yet the error introduced by long-range predictions is substantial for most partially observed scenarios, *e.g., *autonomous driving.
In contrast, we focus on a more challenging, partially observable setting. The embodied agent, like humans, tends to complete more complex tasks, *e.g., *household tasks, from the egocentric view. Second, their framework has limited generalization capability, mainly reflected in two aspects: (i) Their low-level policies are tailored to specific tasks, and the different dynamics may lead to policy failure; (ii) The dynamics can vary for the same action described by the text, *e.g., *turn left. This is because individuals from different environments, e.g., simulators or the physical world, exhibit differences. The text2image/text2video model lacks individual motion pattern information and cannot be generalized accurately to dynamics of other environments that are out of the training dataset. We hope the agent can generalize to different dynamics within the same type of scenario, *e.g., *household scenario.

In this work, we propose egocentric vision language planning (EgoPlan), a general embodied agent to perform long-horizon tasks from the egocentric view across different household environments. Under the egocentric view setting, predicting an observation even a few steps ahead is unreliable. Hence, text2image model, *i.e., *diffusion model, is adopted to realize the fundamental dynamics model under the partially observable setting, where observation and action are represented by image and text respectively. Furthermore, the main differences between the dynamics of the two different environments stem from two aspects: the style of the environment is different, and the motion pattern, *e.g., *amplitude, of the same type of action is different. To accurately generalize the text2image model to other environments, we can perform style transfer based on LoRA [] and introduce optical flow into the model to guide the motion pattern.

For decision-making, we require a generalized policy; hence, we prompt the LMM as the planner. In more detail, given the instruction, the LMM first decomposes it into many sub-goals. For sub-goal representation, we explore text-based and image-based approaches and analyze each form. After knowing the outcome of each action based on the dynamics model, the LMM can choose the proper action by judging which outcome is closer to the current sub-goal. Intuitively, if the planner and dynamics model possess a certain degree of generalization ability, the agent also inherits this ability.

We conduct a comprehensive evaluation and analysis of each module of the embodied agent. Empirically, we demonstrate the high quality of image generation by the world model and the high accuracy of optical flow prediction. Subsequently, we verify the world model’s effectiveness in aiding decision-making in more complex tasks. Lastly, we confirm the method’s generalization capabilities in a different environment.
Our major contributions are summarized as follows:

- •
We have collected a dataset on Virtualhome, which views an action of the agent as a trajectory and provides egocentric observations, visualising optical flow, depth maps and semantic segmentation maps at each time step in the trajectory, which will provide data support for navigation and manipulation tasks in the embodied environment.
- •
We propose EgoPlan, a framework for complex task planning that combines LMM and a world model that predicts an egocentric view of the scene at the next time step after an action is executed. In order to plan more complex tasks with more diverse and different viewpoints (a composite task includes navigation and production tasks), we limit the prediction step size of the world model to avoid the complexity explosion of the prediction algorithm, and introduce optical flow information into the world model to make the world model more sensitive to action position changes and adapt to scene changes during navigation. We demonstrate the effectiveness of our framework through LMM$+$world model planning experiments on comprehensive tasks.
- •
The egocentric observation considering different actions in different environments consists of the agent motion itself and the fine-grained background information of the environment. The optical flow information represents the motion information itself, which is computationally invariant to different scenes and styles, while the fine-grained background information of the environment can be fine-tuned by a small number of sample images from the perspective of the environment agent. We borrow the idea of style transfer in computer vision and adopt the Lora model to fine-tune our diffusion world model, so as to achieve the ability of our framework to achieve few-shot generalization in different embodied scenarios. Experiments on habitat show that our framework can still assist multi-modal large models for task planning in different environments.

## 2 Related Work

### 2.1 Diffusion Model

The diffusion model [, ] has been extensively studied in the field of image generation [, , ] and image editing [, , ]. Diffusion models can achieve a high degree of control during the image generation. In more detail, InstructPix2Pix (InstructP2P) [] trains a conditional diffusion model that, given an input image and text instruction for how to edit it, generates the edited image. ControlNet [] is widely used to control the style of the generated image by using various forms of prior information, *e.g., *edge information and segmentation. By adding LoRA or adapter [] modules to the network, the model trained on one data distribution can also be transferred to other data distributions (different visual styles) through a few picture examples.

The images produced by current diffusion models are of very high quality, highly realistic, and easily controllable. It prompts various fields to consider using these generated images to assist in accomplishing other tasks. Our paper adopts the diffusion model to generate task subgoals and predict the image of the next state for decision-making.

### 2.2 World Model for Decision-making

The world model is used to model the dynamics of the environment. It is crucial for building autonomous agents and enabling intelligent interactions in various scenarios. However, developing a precise world model remains a significant challenge in model-based decision-making.

The Dreamer series [, , ] model the environment dynamics in the latent space to predict future states within gaming environments. It enables game agents to learn the tasks by imagination, decreasing the interactions for effective learning. However, since the world models are learned in latent space instead of pixel space, these models often lack the generalization of unseen tasks and environments. A world model built in the pixel space may have better generalization capabilities. By carefully orchestrating rich data along different axes, UniSim [] can simulate realistic experiences in the visual form in response to actions by humans, robots, and other interactive agents. In summary, we can find out the versatile applications of world models span beyond games and robotics.

The advancements in diffusion-based world models are reshaping how we model physical motion laws in real-world settings, particularly in robotics. UniPi [] frames the decision-making problem in robotics as a text2video task. The generated video is fed into an inverse dynamics model (IDM) that extracts underlying low-level control actions, which are executed in simulation or by a real robot agent. Video Language Planning (VLP) [] introduces a novel method for task planning that integrates video generation with tree search algorithms. This methodology lets robots plan over longer horizons by visualizing future actions and outcomes. Unlike previous works, SuSIE [] leverages pre-trained image-editing models to predict the hypothetical future frame. A low-level goal-reaching policy is trained on robot data to reach this hypothetical future frame. Since one goal frame prediction does not require the model to understand the intricacies of the robot’s low-level precisely dynamics, it should facilitate transfer from other data sources, *e.g., *human videos. RoboDreamer [] advances the field by utilizing video diffusion to formulate plans combining actions and objects, solving novel tasks in unexplored robotic environments.

We find it unrealistic to apply the text2video model to partially observed scenarios. Moreover, it is still hard to predict the goal frame several steps ahead, as the shift in perspective could be significant. Therefore, we adopt the text2image model to accurately predict the short-range outcome for one-step planning.

Figure: Figure 1: An illustration sample in VH-1.5M, which includes current image observation, next image observation given the text action, semantic segmentation map, depth map, and optical flow map.
Refer to caption: x1.png

## 3 VH-1.5M Dataset

Most datasets related to embodied agents, *e.g., *RT-X [] and RH20T [], employ the third-person view to avoid the visual occlusion issue, thus lacking data regarding the egocentric view (first-person view). There are some datasets, *e.g., *Alfred [] and Procthor [], that adopt a first-person perspective, however, they simplify the state transition by assuming instantaneous completion of actions, which fails to mimic the dynamics changes in real-world environments. We propose the VH-1.5M dataset based on the VirtualHome [, ] environment to address these limitations.

We construct our dataset VH-1.5M in the VirtualHome environment, which comprises 50 distinct houses. Each house contains approximately 300 interactive objects, and the embodied agent can perform more than 10 actions. Note that the VirtualHome environment is a simulator tailored for embodied agents, offering a detailed simulation of a residential living scenario. It enables a range of household tasks, *e.g., *navigation and object manipulation.

The VH-1.5M dataset is organized in a structured manner, encapsulating the relationship between actions, houses, agents, and trajectories. Each task sequence entry follows a hierarchical structure, *e.g., *"/open/house_0/Female4/2_fridge" (female4 open the fridge2 in house0).

Dataset Details: The VH-1.5M dataset consists of:

- •
13 Actions: Various physical actions and interactions for Agents within the Houses.
- •
50 Houses: Uniquely designed houses with diverse layouts and object placements.
- •
4 Agents: Four distinct agents, each capable of performing the full range of actions.
- •
1.5M Samples: Dateset has numerous detailed sequences, each executing one action. Information from each step in the sequence is stored as one sample. One example is shown in Figure 1. We use House49 as the validation set.

More details of the dataset can be found in the appendix, and we will open-source the dataset.

## 4 Method

Our embodied agent, EgoPlan, takes input as a visual observation $x_{t}$ of the scene at the current timestep $t$ and a natural language goal $g$ and outputs an action $a_{t}$ to interact with the environment. Note that the $x_{t}$ only partially represents the current environment state. In addition, the agent uses encapsulated skills as actions, such as moving forward, turning, and grabbing objects.

EgoPlan consists of two parts, as illustrated in Figure 2. One is a dynamics model that gives the agent the concept of the current environment, and the other is the planner that endows the agent with decision-making capabilities. Intuitively, we humans first envision the outcomes of each action in our minds, and then, by comparing the results, we make the best decision.

### 4.1 Diffusion-Based Dynamics Model

#### 4.1.1 Learning Dynamics

From a first-person perspective, the view after two steps may be completely different, making it difficult to model. Therefore, we aim to model the fundamental dynamics model, $p_{\theta}(x_{t+1}|x_{t},a_{t})$, for one-step planning usage. In more detail, we want to generate a new image $x_{t+1}$, representing the next state given the current visual observation $x_{t}$ and the text of the action $a_{t}$. Then, we cast our eyes on the text2image model and resort to the diffusion model for modeling specifically. It has an irreplaceable advantage in easily incorporating other modalities as a condition.

Figure: Figure 2: Overview of EgoPlan. The left side features a one-step planner that provides the agent with decision-making capabilities, while the right side includes a world model (dynamics model) that provides the agent with an understanding of the current environment.
Refer to caption: x2.png

Although the open-sourced diffusion model [, ], $p_{\theta}(x_{\rm tar}|x_{\rm src},l)$, trained on a wealth of online videos, has demonstrated the ability to predict the future, their generated results are hard to control, and most are only semantically reasonable. Moreover, most of the text in the pre-trained dataset consists of image descriptions $l$ rather than action instructions $a$. Therefore, supervised fine-tuning is adopted based on our VH-1.5M dataset to better model the dynamics, $p_{\theta_{\rm sft}}(x_{t+1}|x_{t},a_{t})$. Formally, the training objective is given by:

$$ $\displaystyle\footnotesize\mathcal{L}_{\text{MSE}}$ $\displaystyle=\left\|\epsilon-\epsilon_{\theta}\left(q\left(x_{t+1}^{(k)}|x_{t },a_{t}\right),k\right)\right\|^{2}$ (1) $\displaystyle=\left\|\epsilon-\epsilon_{\theta}\left(\sqrt{\overline{\alpha_{t }}}x_{t}+\sqrt{1-\overline{\alpha_{t}}}\epsilon|a_{t}\right)\right\|^{2}$ (2) $$

where $\epsilon_{\theta}$ is a learnable denoising model for reverse process, $k$ is denoising steps, and $\overline{\alpha_{t}}$ are a set of $K$ different noise levels for each $k\in[1,K]$.
However, we find it difficult to generalize directly to other environments since our dataset only includes VirtualHome scenes. The difference between two environments, *e.g., *Habitat2.0 [, ] and VirtualHome, primarily lies in their different motion patterns for the same action and distinct visual styles. Especially for the former, the motion pattern, *e.g., *the amplitude of the same action, performed by agents in a different environment can be unpredictable.

#### 4.1.2 Generalization

We want to improve the model’s generalization ability from a different perspective. In other words, instead of enhancing generalization through big data and large models, we aim to explicitly address the differences between environments aforementioned at the methodological level.

Motion Regularization. Firstly, we must combine the motion information into the diffusion model to distinguish the different motion patterns. Optical flow has thus caught our attention. It refers to the pattern of apparent motion of image objects between two consecutive frames caused by objects or camera movement. In optical flow maps, colors represent the direction of motion, and the depth or intensity of the colors indicates the magnitude of the motion, which is a general feature across different environments.

However, in practice, in the absence of the next observation, we cannot obtain the current optical flow, $f_{t,t+1}$. Inspired by other motion estimation works [, ], we assume motion consistency holds over short intervals, meaning abrupt changes do not occur. Consequently, the consecutive optical flow maps are highly correlated, allowing us to predict the current optical flow map using the previous map. The previous map is calculated from the previous two frames and reflects the actual motion pattern in the current environment.

We notice that optical flow generation does not require complex texture generation, and it is expected not to cause a significant delay in the pipeline. Therefore, we adopt a less powerful but lightweight generative model, VQ-GAN [], and train it on our dataset to predict the optical flow map. Empirically, the generalization ability to predict optical flow is much better than predicting actual images. Formally, the training objective is given by:

$$ $\small\min{}\mathcal{L}_{VQ}(E,G,Z)=\|x-\hat{x}\|_{2}^{2}+\|\text{sg}[E(x)]-z_ {q}\|_{2}^{2}+\beta\|\text{sg}[z_{q}]-E(x)\|_{2}^{2},$ (3) $$

where $E$ is the encoder, $G$ is the generator, $Z$ represents the latent space, $x$ is the input image, $\hat{x}$ is the reconstructed image, $z_{q}$ is the quantized latent vector, sg denotes the stop-gradient operator, and $\beta$ is a hyperparameter that balances the commitment loss.

In summary, we use a simple model to predict motion patterns and then a more complex model to reconstruct real textures based on motion patterns. Therefore, we adopt ControlNet [] to incorporate the optical flow map, $f_{t,t+1}$, into the default diffusion model, $p_{\theta_{\rm sft}}(x_{t+1}|x_{t},a_{t},f_{t,t+1})$. Only the ControlNet part needs to be fine-tuned on VH-1.5M at this stage. Formally, the training objective is given by:

$$ $\displaystyle\footnotesize\mathcal{L}_{\text{MSE}}$ $\displaystyle=\left\|\epsilon-\epsilon_{\theta}\left(q\left(x_{t+1}^{(k)}|x_{t },a_{t},f_{t,t+1}\right),k\right)\right\|^{2}$ (4) $\displaystyle=\left\|\epsilon-\epsilon_{\theta}\left(\sqrt{\overline{\alpha_{t }}}x_{t}+\sqrt{1-\overline{\alpha_{t}}}\epsilon|a_{t},f_{t,t+1}\right)\right\| ^{2}.$ (5) $$

Style Transfer. Secondly, we use LoRA to fine-tune the diffusion model for visual style transfer. Note that LoRA requires very little data, just dozens of samples. Normally, it is convenient to collect data on such a scale in new environments. We expect the model to achieve generalization with as little effort as possible.

### 4.2 Planning with Dynamics Model

To avoid further training in new environments, we prompt the LMM, *i.e., *GPT-4V, as the planner. The LMM needs to be responsible for high-level goal decomposition as well as low-level action selection. Meanwhile, the pre-trained dynamics model can help the LMM better understand the world.

#### 4.2.1 Goal Decomposition

For long-term complex tasks, goal decomposition is an indispensable step. Subgoals can be represented in both text and image forms. For the text-based subgoal $g_{\rm tar}$, we prompt the LMM to generate a reasonable one.
In addition, we train another diffusion model, $p_{\theta_{\rm sft}}(x_{\rm tar}|x_{t},g_{\rm tar})$, to generate the image-based subgoal $x_{\rm tar}$ only based on the text-based subgoal and current observation. Note that predicting the image of the subgoal can be more challenging than predicting the next observation, which means the results are not very precise. We plan to delve into the impact of different types of subgoals on tasks. See Section 4.

#### 4.2.2 One-Step Planner

Since we can only ensure that the prediction for the next step is relatively accurate, we adopt a one-step planning method. In more detail, we utilize the pre-trained dynamics model to predict the visual outcomes of all the actions in the next state. Once the text/image-based subgoal is obtained, we send the subgoal and all the visual outcomes to the LMM. Then, we prompt it to compare all the potential outcomes with the subgoal and determine which action can bring the agent closer to the goal.

Figure: (a) Original
Refer to caption: extracted/5785729/subfig/close_the_fridge_5_normal.png

## 5 Experiment

In this section, we comprehensively evaluate and analyze each module of the embodied agent. We first evaluate the quality of image generation using the world model and the quality of optical flow prediction. Secondly, we evaluate whether our world model can assist task planners in completing more complex tasks. Finally, we assess the generalization of our method.

### 5.1 Visual Quality

We adopt two metrics, FID [] and user score, to evaluate the visual quality of the generated image of the world model. For models, InstructP2P (pre-trained) is the default model of InstructP2P. InstructP2P (fine-tuned) is the model fine-tuned on our dataset. Ours (previous flow) is the world model that conditions on the previous optical flow map, while Ours is conditioned on the predicted optical flow map. Note that the validation set of VH-1.5M has around 5k samples.

**Table 1: FID score comparison with other models on the validation set. It is calculated between the predicted observation and ground truth. The lower the number, the better the quality of the image.**
| Model | Mean | Variance |
| --- | --- | --- |
| InstructP2P (pre-trained) | 13.65 | 0.10 |
| InstructP2P (fine-tuned) | 1.06 | 0.05 |
| Ours (previous flow) | 0.83 | 0.03 |
| Ours | 0.82 | 0.03 |

FID Score. FID is a standard metric measuring the distance of two image distributions using the inception model. The smaller the FID is, the more similar the two images are. Table 1 shows the FID score of our model and baselines. We can see that using existing diffusion models as world models is ineffective because their training data often lacks state transition-related data. Meanwhile, introducing an optical flow map, which serves as motion pattern information, significantly enhances the generation results. In addition, world models based on predicted optical flow are slightly better than those based on the optical flow of the previous frame.

**Table 2: User score of the user study. The user score is the percentage of images that users consider to meet the criteria out of the total 1000 images. The higher the number, the better the quality of the image. The evaluated images are from the validation set.**
| Model | Mean | Variance |
| --- | --- | --- |
| InstructP2P (fine-tuned) | 54.10$\%$ | 1.53$\%$ |
| Ours (previous flow) | 69.35$\%$ | 1.34$\%$ |
| Ours | 74.93$\%$ | 2.57$\%$ |

User Study. We also conduct a user study on the accuracy of world models for image generation. For the criterion, users judge the correctness of the direction and amplitude of the executed action. Each user investigates a total of 1000 samples. There are 8 users participating in the survey in total. Our user study, shown in Table 2, again verifies our predicted optical flow can help generate higher-quality images.

Analysis. As illustrated in Figure 3, InstructP2P (fine-tuned) generates the scene of steering in the wrong direction. However, this flaw can be greatly improved by incorporating optical flow information. Moreover, it is observed that the dynamics of closing the refrigerator can be more accurately predicted if the prediction of the motion pattern is considered.

### 5.2 VirtualHome Tasks

Figure: (a) Ours (text subgoal) and other baselines on task 1-6
Refer to caption: x3.png

Results.
To prove that our world model can well assist the LMM in task planning, we evaluate various methods on $12$ tasks, each task described by an instruction, in the VirtualHome environment. Each task is tested $100$ times, and the maximum step in one episode is $80$. For each of the $12$ tasks, we abbreviated the task names for convenience. For example, the instruction of task 1, "take the bread from the toaster and place it on the plate on the table," consists of four subtasks: a) walk to the toaster, b) grab the bread, c) walk to the plate, and d) place the bread on the plate. We use "take and place" to refer to task 1.

These 12 instructional tasks are comprised of multiple sequential sub-tasks. For baselines, we use GPT4 combined with React [] as the task planner and policy, denoted as GPT4+React, and it takes input as the JSON format text environment description. We also directly use GPT-4V to make decisions, denoted as GPT4V, and we also combined GPT4V with React [] and Reflexion [] as the task planner and policy, denoted as GPT4V+React and GPT4V+Reflexion. For ablation baselines, we use the fine-tuned InstrctP2P as the world model, denoted as GPT4V+P2P. The world model that conditions on the previous optical flow map is denoted as GPT4V+PrevOF.

As shown in Figure 4, the world model significantly improves the GPT-4V ability on various long-horizon tasks. Moreover, the inclusion of optical flow information enhances the accuracy of image generation and further improves task planning performance. The results also demonstrate the effectiveness of the predicted optical flow map.

Image Subgoal vs. Text Subgoal. In this part, we analyze the impact of different types of subgoals on tasks. During the goal decomposition process, the text subgoal directly outputted by the LLM task planner represents a high-level, coarse-grained description. If our method can generate images of the scene at the completion time of the subgoal, a more detailed, fine-grained description can be obtained. This might enhance the action selection ability that relies on the quality of the subgoal.

Specifically, we have trained an InstructP2P model based on VH-1.5M to generate the image when the subgoal is completed, with the generation results illustrated in Figure 5. The decision-making results in Figure 4 show that fine-grained subgoal description is better than coarse-grained description, even if the generated image is not so accurate.

Figure: (a) Enclose the fridge
Refer to caption: extracted/5785729/subgoalfig/enclose_the_fridge_10_normal.png

Figure: Previous flow
Refer to caption: extracted/5785729/virtualhome_flow/switchoff/ori/0_flow.png

We also conduct a user study to evaluate the visual quality of the generated image-based subgoals. More details can be found in the appendix.

### 5.3 Motion Pattern

As mentioned before, we cannot obtain the optical flow from the current timestep to the next timestep. Therefore, we adopt the VQ-GAN model to predict the current optical flow map. As illustrated in Figure 6(a) and 6(c), the quality of prediction for details is promising. Furthermore, as demonstrated in Figure 6(d) and 6(e), the VQ-GAN trained on the VH-1.5M dataset can easily generalize to other environments. This is because the optical flow map is a universal feature and does not require the prediction of complex textures.

**Table 3: Average endpoint error (AEE) results. The lower the number, the closer the image is to the ground truth.**
|  | Previous flow | Prediction flow |
| --- | --- | --- |
| Habitat2.0 | 3.30 | 3.09 |
| AI2-THOR | 5.00 | 4.08 |
| VirtualHome | 21.22 | 15.71 |

The average endpoint error (AEE) specifically measures the average distance between two motion vectors at the pixel level. As illustrated in Table 3, the gap between the predicted optical flow map and ground truth is narrower than that between the previous flow map and ground truth (current optical flow map). In addition, the model trained on VirtualHome can still predict optical flow maps in Habitat2.0 and AI2-THOR []. This confirms the effectiveness and generalization of the VQ-GAN model.

Figure: Figure 7: Examples of the generated images of the next observation in Habitat2.0.
Refer to caption: x7.png

### 5.4 Generalization

To assess the generalization of our method, we also evaluate its performance in a new household environment. In more detail, we choose Habitat2.0 due to its high-fidelity scenes compared with other simulators, such as AI2-THOR. However, Habitat2.0 does not provide any inter-frame regarding manipulation skills, which is unrealistic. Therefore, we only carry out navigation tasks.

To enhance usability, we use the pre-trained optical flow model, RAFT [], to calculate the optical flow for the previous step since the optical flow cannot be directly obtained. The RAFT results are shown in the last 2 columns of Figure 6. Since VQ-GAN has demonstrated some degree of generalization ability to Habitat2.0 in Section 5.3, we can predict the motion pattern of the new environment. The remaining task is to transfer the visual style to a new environment, and we adopt LoRA to fine-tune the world model. As shown in Figure 8, we successfully perform style transfer with a small amount of data (tens of samples), and the results with LoRA are closer to real scene images compared to those without LoRA visually.

Figure 8 shows the success rate of all methods on navigation tasks in Habitat2.0. We can draw the same conclusion as in the VirtualHome environment: incorporating predicted optical flow into the world model enhances the agent’s decision-making capabilities. Additionally, our method achieved a high success rate, which further demonstrates its strong generalization ability.

## 6 Conclusion and Limitations

This paper introduces EgoPlan, an embodied agent, using the LMM as the one-step planner and the text2image model as the world model for long-horizon tasks. We demonstrate its high-quality image generation, precise optical flow prediction, and promising decision-making ability. More importantly, we have confirmed its generalization capabilities across different environments. It is also important to acknowledge the limitations of EgoPlan. Currently, the agent uses encapsulated skills as actions. It cannot perform low-level control, *e.g., *joint position. How to directly control low-level actions is left as future work.

## Appendix A Appendix

### A.1 Details of Virtualhome tasks

We conducted experiments to evaluate the decision-making ability of all methods in the VirtualHome environment. In total, we investigated 12 complex tasks, with detailed instructions for each task as follows:

Figure: Listing 1: Instructions and subtasks.

### A.2 Details of VH-1.5M’s text actions

The dataset includes a wide range of action sequences, each meticulously annotated with corresponding text actions. These text actions are crucial for providing contextual information that aligns visual actions with natural language descriptions. Below, we detail the process and structure used to generate the text actions for each action sequence in the dataset.

The generation of text actions for VH-1.5M involves a systematic and automated process. This process ensures consistency and variety in the text actions, which are essential for robust training and evaluation in vision-and-language tasks. The key steps in this process are as follows:

Verb Selection: A list of verbs related to various actions (e.g., "walk through," "close," "drink") is predefined. For each identified action sequence directory, a verb is randomly selected from the relevant list. This selection ensures a diverse representation of actions.

Object Name Extraction: Each directory represents the object acted upon, which signifies the object affected by the action. However, if the action does not involve an object, such as "walk through" or "turn left," no extraction is necessary.

Phrase Construction: Two types of phrases are constructed for each action sequence:

Next Timestep Phrase: Describes the immediate next action in the sequence. For example, "next timestep: redeposit the plate".

Goal State Phrase: Describes the intended final action or goal of the sequence. For example, "the goal state: redeposit plate".

Prompt File Creation: The constructed phrases are saved in a prompt json file within the respective action sequence directory. This JSON file contains two keys: "next" and "goal," corresponding to the next timestep phrase and goal state phrase, respectively.

#### A.2.1 More examples of the samples

We give some samples in the sequence of the task, which are shown in Figure 9. Note that samples in one sequence are arranged in chronological order, with the timestep increasing from top to bottom.

Figure: Figure 9: Samples in the sequence of closing the microwave.
Refer to caption: x9.png

Figure: Figure 10: Samples in the sequence of walking through the door.
Refer to caption: x10.png

Figure: Figure 11: Samples in the sequence of switching on the toaster.
Refer to caption: x11.png

### A.3 More examples of generating images

More examples of generated images from EgoPlan can be seen in Figure 12. Each line represents a task, and the task prompts are, in order: "capture the chicken", "grasp juice", "grasp the hairproduct", "open the cabinet", "open the microwave", "go left", "make a left", "make a left-hand turn", "make a right", "turn right", "turn to the right", "walk straight ahead".

Figure: (a) Original
Refer to caption: extracted/5785729/appendix_one_step/capture_the_chicken_2_normal.png

Figure: (a) grasp the cutlets
Refer to caption: extracted/5785729/goal_final/grasp_the_cutlets_0_normal.png

### A.4 User study of subgoal image generation

We also conduct a user study on the image generation of the subgoal. A total of $8$ users evaluated whether the generated image met the criteria of the subgoal described in the text. Each user evaluates $100$ generated images for each action, and the evaluation results are shown in Table 4. The results show that most of the generated subgoal images can represent the meaning of the text subgoals. More examples of generating figures can be seen in Figure 13

**Table 4: User study for the subgoal generation. The user score is the percentage of images that users consider to meet the criteria out of the total 100 images.**
|  | Close | Drink | Grab | Open | Put back | Put in |
| --- | --- | --- | --- | --- | --- | --- |
| Mean user score($\%$) | 66.5 | 71.75 | 55 | 66.375 | 62.125 | 64.625 |
|  | Sit | Stand up | Switch off | Switch on | Walk through |  |
| Mean user score($\%$) | 79.875 | 78.75 | 73.375 | 77.875 | 79 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness part
soundness: 3
strengths: 1. The framework is well-motivated and reasonable.
2. The data effort will be of good use to future works.
3. The paper is well-organized and easy to read.
4. The proposed method outperforms the baseline.
summary: This work has collected a dataset on Virtualhome viewing an action of the agent as a trajectory, with egocentric information.  The EgoPlan framework is introduced combining LMM for planning and a diffusion world model for dynamics prediction. Optical flow modality is used for advancement. The framework demonstrates improved performances on generation quality, VirtualHome and Habitat.
weaknesses: 1. Some crucial ablation studies are missing. How does the framework perform without optical flow and style transfer?
2. Some related works may share similar motivations using diffusion models for world dynamics, and dynamics for planning, you may consider to cite.

[1] 3D-VLA: A 3D Vision-Language-Action Generative World Model    
[2] Diffusion Reward: Learning Rewards via Conditional Video Diffusion

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see my weakness section.
soundness: 1
strengths: 1. The paper is well organized and easy to follow.
2. The idea of using optical flow to generalize across environments is reasonable and novel.
summary: This paper proposes EgoPlan, which casts a diffusion model as world model and an LLM as a high-level planner. Specifically, the diffusion model synthesizes future scenes to corresponding to several admissible actions, while the LLM predicts the next action based on the most probable future scene. The authors also propose a new dataset, dubbed as VH-1.5M, which annotates the segmentation map, depth map, and the optical flow for the trajectories collected from VirtualHome. They conduct experiments on various tasks in the VirtualHome environment. They also evaluate the quality of the generated images and optical flows on several different datasets.
weaknesses: 1. The idea of using generative model as world model [1,2,3,4] and LLM as task planner [5,6] have been widely studied in previous works.
2. (contd. 1.) The unique contribution of this paper appears to be the use of optical flow to generalize the world model across diverse environments. However, the experiment results are not sufficient to support this claim. Including task execution results rather than solely optical flow error across different simulators, could provide more comprehensive evidence and improve the robustness of the findings.
3. For the main experiment (Figure 4), presenting the results in a table rather than a figure could enhance clarity. It is unclear to me how the world model benefits the final task execution compared to directly employing GPT-4V for task planning. 
4. The authors are encouraged to evaluate their methods on more challenging tasks that require long-term planning capabilities, such as ALFRED or RxR-Habitat, to further validate their approach.

Overall, while the paper presents an interesting direction, my main concern is that additional foundational experiments would strengthen its claims. The authors are encouraged to consider these comments to enhance paper’s contributions.

[1] Contrastive Learning as Goal-Conditioned Reinforcement Learning. NeurIPS 2022.

[2] Mastering Atari with Discrete World Models. ICLR 2021

[3] Learning Latent Dynamics for Planning from Pixels. ICML 2019.

[4] Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020.

[5] LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models. ICCV 2023.

[6] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. CoRL 2022.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Would it be beneficial to better define the task in your dataset using a formal markov decision process?
2. The main results figure seems to not display baseline performance numbers, where are they?
soundness: 2
strengths: 1. Authors perform a number of ablation studies to demonstrate the usefulness of each model component
2. Authors compare their model to many different baselines.
summary: The authors leverage LLM/VLMs and T2I models to construct an embodied planning pipeline capable of one-step planning. The model is tested on VirtualHome and compared to various baselines. A new virtual-home based dataset is collected.
weaknesses: 1. The authors' use of "world model" to describe the paper's diffusion (image editing) component is highly exaggerated. By definition, world models should record and keep track of complete and accurate environment states. However, here the diffusion model is merely LoRA finetuned to edit the provided image in an in-distribution manner. The authors also fail to explain why their diffusion module can remotely constitute a world model in their methods section.

2. The InstructP2P model is known to be not very strong with physical understanding (when asked to generate the new scene after a significant action / significant view shift has taken place, it often fails). If the authors are able to overcome this issue, more visual examples should be demonstrated on harder cases.

3. The model is only able to perform greedy one-step planning, which means it has no way of optimizing its actions based on global goal. While world models are sometimes used to solve this problem, in this work the provided "world model" module seems far from being able to support this.

4. Although relatively easy to set-up, the VirtualHome simulator is quite old and visual simulation quality is not very good compared to newer simulators (eg. Behavior, Robosuite/Robocasa, etc.) or real world robotic datasets (eg. DROID). Experiments on VirtualHome alone is not a good-enough indicator whether or not the model can be adapted to real-world circumstances.

5. Some typos and incoherent sentences: for example "Introduce optical flow into the world model leads the world model more sensitive to action position changes and adapt to scene changes during navigation." (L92-93)

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. What Is the Duration of the Planning Process?
Could you provide more details on the time taken for the high-level planning process in Figure 2 and the specific planning instructions used?
2. Training Method for the Image Subgoal Generator:
How did you train the diffusion model used for subgoal prediction? Is it conditioned only on the final goal? When there is a significant difference between the subgoal and the initial screen, how do you ensure accurate outputs without environmental context? If the output quality is consistently high, is there a risk of overfitting?
3. Details on Dataset Partitioning and Evaluation:
How did you partition the dataset for training and evaluating the two diffusion models? How do you maintain output quality when the predicted image differs greatly from the input? Did you perform out-of-distribution (OOD) evaluations to address potential overfitting issues?
4. Assessment of Generalizability:
Have you tested your model in other scenarios, such as outdoor environments, to evaluate its generalizability and potential applicability?
5. Inference Time Efficiency and System Bottlenecks:
Could you provide more information on the system's inference time efficiency? Which components might be potential bottlenecks? Have you conducted ablation studies to assess the system’s stability?
6. Inclusion of LLM Reasoning Process in the Dataset:
Does your dataset include detailed LLM reasoning processes, or only subgoal texts? Could you provide some comparative reasoning cases with multi-modal input to offer deeper insights?
7. Clarification on Fig. 7 Results:
Could you explain why, in Fig. 7, the results without LoRA fine-tuning sometimes appear better than those with LoRA? Does this indicate potential overfitting or limitations in the model’s ability to handle significant environmental changes?
soundness: 1
strengths: 1. Innovative Integration of LMMs and Diffusion Models: The paper presents a novel approach by combining LMMs with diffusion models for planning and action prediction in egocentric embodied environments.
2. Incorporation of Computer Vision Techniques: The use of style transfer and optical flow enhances the model’s ability to generalize across different scenes and adapt to spatial changes, which is crucial for embodied agents.
3. Dataset Contribution: The authors have collected the VH-1.5M dataset on VirtualHome, providing egocentric observations, fine-grained action information, and visualizations like optical flow, depth maps, and semantic segmentation, which can benefit future research in navigation and manipulation tasks.
4. Improved Long-Horizon Task Performance: Experimental results indicate that EgoPlan outperforms baselines in long-horizon tasks from an egocentric perspective, showcasing the effectiveness of the proposed framework.
summary: The paper introduces EgoPlan, an egocentric vision-language planning framework that leverages large multi-modal models (LMMs) and diffusion models to handle long-horizon tasks in household scenarios. EgoPlan employs a diffusion model to simulate state-action dynamics and integrates computer vision techniques like style transfer and optical flow to enhance spatial modeling and generalization across different environments. The LMM serves as a planner, decomposing instructions into sub-goals and selecting actions aligned with these sub-goals. Experiments demonstrate that EgoPlan improves task success rates compared to baselines in egocentric views.
weaknesses: 1. Lack of Planning Instructions and Time Details: The paper does not provide specific planning instructions for the high-level goal decomposition shown in Fig. 2, nor does it mention the duration of the planning process. This omission makes it difficult to evaluate the efficiency and effectiveness of your planning method.
2. Insufficient Details on Diffusion Model Training: There is a lack of detailed information on how the diffusion models (particularly the World Model and the Image Subgoal Generator) were trained. Without these details, assessing the validity and reproducibility of your results is challenging.
3. Dataset Limitations and Overfitting Concerns: Relying solely on the VH-1.5M dataset may be inadequate. Additionally, there is a risk of overfitting without information on out-of-distribution (OOD) evaluations and how the training and testing data are partitioned.
4. Limited Generalizability: The applicability of the model to other scenarios, such as outdoor environments, has not been demonstrated. This raises questions about the generalization capabilities of the embodied agent in diverse environments.
5. No Discussion on Time Efficiency and System Stability: The paper lacks details on inference time efficiency and system stability, especially considering the multiple estimation components involved. Understanding potential bottlenecks is crucial for evaluating the feasibility of your method.
6. Lack of Detailed Reasoning Process: The reasoning process of the Large Language Model (LLM) is critical for evaluating and explaining outcomes, but the paper does not sufficiently discuss this aspect. It seems the dataset contains only subgoal text without detailed reasoning steps.
7. Confusion in Fig. 7 Results: In Fig. 7, the results without LoRA fine-tuning sometimes appear better than those with LoRA. This raises concerns about potential overfitting and the model’s ability to handle significant environmental changes.